A project to evaluate the perceived image quality of fine art reproductions was conducted in which pieces of artwork were imaged by participating institutions. As part of this project, observers were asked to rank fine art reproductions on a characterized display either with or without the presence of the originals. The goal was to see whether the availability of the original artwork influenced how people appreciated the reproductions. A low correlation was found between the ranking results with and without the originals, indicating a shift in the criterion employed by observers in evaluating perceived image quality from color accuracy to preference. A web-based experiment was designed in order to better understand the necessity of a controlled environment when evaluating image quality based on preference. A significantly high correlation was found between the results from the experiment without the original and from the one conducted online. Therefore, the preference judgments of perceived image quality were stable regardless of changes in viewing conditions. In addition, the areas that were considered most important by observers in making ranking decisions were identifiable by observers' clicks on the image in the web-based experiment. By understanding the part of the paintings to which more attention was drawn, information regarding the image saliency could be learned.