Incomplete paired comparison is an important technique for color-imaging problems because it can avoid observers to compare every possible pairs since the number of paired comparisons for n stimuli is n(n-1)/2 which becomes prohibitive for large values of n. However, the experimental designer often struggles with questions such as what is the smallest limit the proportion of paired comparisons included that will still allow reliable estimations of scale values? Fortunately a Monte-Carlo computational simulation is carried out with a model of an ideal observer and the results shows that the proportion of paired comparisons that is included is more critical than the number of observers who make those observations . This work aims to test the results from computational simulation with 25 real observers and 10 stimuli from the gray scale. The work suggests when each observer estimates the same proportion of paired comparisons included the more proportion of pairs and number of observers, the more accurate scale values will be produced and the proportion of pairs is more critical than the number of observers who make those observations, which quite agrees with the findings from the computational simulation. The work also suggests when the each observer estimates a different proportion of paired comparisons the more proportion of paired comparisons will not always produce a more accurate scale values.
Yuan Li, Stephen Westland, Vien Cheung, "Evaluation of Incomplete Paired-Comparison Experiments" in Proc. IS&T 19th Color and Imaging Conf., 2011, pp 39 - 42, https://doi.org/10.2352/CIC.2011.19.1.art00009