Abstract Some experimenters have begun to carry out image preference experiments over the web, with observers completing the task in their own time and using their own display devices. This reduces the administrative overhead, and opens the possibility to huge numbers of potential observers. However, we have to surrender some control over viewing conditions. In previous work, we evaluated an existing web-based paired comparison experiment against a lab-based counterpart and found that, generally, the two variants did not correlate to a significantly high degree. In this work we extend that study with the development of our own web-based research platform with greater control over viewing conditions and much larger quantities of observers (over 1,000, with more than 26,000 individual observations). With this, we show much more positive correlation between the web- and lab-based variants. We also show the similarity or otherwise between the two variants as a function of time, which reveals how many web-based observations are required to achieve stable results.