A variety of scientific disciplines are involved in pictorial research, from perception psychology and computer science to (digital) art history. Recently, the art historical study of pictures has shifted from close viewing, the study of individual pictures, towards distant viewing [1, 2], where a large body of pictures are analyzed using computational methods. This computational approach affords cultural analytics [26]: quantifying visual trends and structures in large corpora of images. The data used for these analyses are computational annotations [2], which can range from image statistics [16] and head posture [9, 43] to many other levels of visual representation.

There is thus a paradigm shift from the human eye studying a single work to a computational eye studying (large) collections of works. This paradigm shift takes place along two dimensions, from humans to computers and from close viewing to distant viewing. Although the convenience of computational scalability likely inspired distant viewing, the two dimensions are independent: it is possible to perform close viewing with computers [20] or distant viewing with humans [42]. However, there is one dimension to be added to draw a more complete picture of possible paradigms for pictorial research: the difference between subjective and objective annotations, as illustrated in Figure 1. While computational annotations are mostly deterministic and display noise that is only attributable to their performance accuracy, the variations in human annotations can be of various nature: aside from accuracy, it can also reflect individual differences.

To explore this distinction between objective and subjective annotations, we will first review existing work in these two fields and then present two case studies that exemplify an objective and a subjective annotation experiment.

One of the many elements of artistic style is perspective, i.e. how pictorial space is constructed. Radically different projection systems can be found across art history [49]. Also more subtle patterns have been found such as changes in viewpoint elevation [46]. In this experiment, we used a similar paradigm for tracing the horizon and human figures based on the principle of horizon ratio [35]. We choose to analyze the paintings of Hendrick Avercamp, a Dutch painter from the early 17th century. He is particularly famous for his winter landscapes, often depicting people ice-skating, which happen to be particularly suited for perspective reconstruction because ice is a horizontal plane.

In the second experiment, we aim to answer a question of artistic composition. Syndics of the Drapers’ Guild by Rembrandt is an intriguing painting: the unconventional viewpoint; the feeling of interrupting a meeting that just started. Moreover, x-ray studies have shown [41] that Rembrandt had doubts on where to position the servant, the person in the middle behind the others. According to [41], Rembrandt initially planned to position this person at the far right of the painting. Informal conversations with art historians led us to an interesting question: Where would a naive viewer position the servant, and how does this compare to Rembrandt’s choice and rejection?

We presented two visual experiments that demonstrate the difference between objective and subjective annotations in the context of close and distant viewing. Both studies make use of spatial annotations, and both concern artistic choices of composition. Yet the perspective study relies on objective annotations while the composition study relies on subjectivity. What makes the two results categorically different, and how do they relate to existing pictorial research?

Individual differences are the core difference between data interpretation of objective versus subjective annotations. In both experiments, the data contains substantial variance: in Fig. 2, the linear regression plots show a certain noise level and in Fig. 4, we see continuous distributions along the vertical direction and discrete, multi-modal distributions along the horizontal direction. What do these variances mean? In both cases, part of the noise emerges from human inaccuracies. This inherent inaccuracy of human judgment can be interesting in a purely behavioral context but is considered as irrelevant measurement noise in our context. However, this inherent noise does not seem responsible for all variances. In the first study, part of the variance can be attributed to the pictorial content. In Fig. 2(a), the residuals of the linear regressions denote how homogeneously the figure sizes diminished towards the horizon. Deviations from the regression line are either due to natural variations of human sizes or inaccurate perspective handling. Although we cannot dissociate between these two, we should realize that both explanations concern the depiction and not the viewer. In Fig. 2(b), the residuals refer to differences in viewpoint. We found that although the downward trend with time was significant, still 57% of the variance was unexplained by this trend. Again, this can be due to various reasons but they all concern the circumstances under which the picture was produced, e.g. artistic or practical choices for viewpoint difference. The variance in the data does not concern the observer.

On the other hand, Experiment 2 concerns the relation between the picture and the viewer. The variation in preferences shows clear clusters that arise from multiple individual judgments. The annotations are characterized by the individual differences of the observers. They reveal the different artistic choices that could have been made. Other than the viewpoint attitude of Avercamp, Syndics of the Drapers’ Guild affords immediate visualization of a compositional choice; manually varying the perspective attitude would be rather complex. The possibilities for these subjective annotations thus depend on the type of annotations and the pictures.

A painting (collection) can be visually analyzed by humans and computers. Studying a single work can be called close viewing while the comparison between works is called distant viewing [2]. While many digital art historians use computer annotations, our study was based on human annotations. The advantage of computer annotations is the scalability. The advantage of humans is that they can be relatively easily instructed and that they display individual differences, which are not to be found in computational approaches. Unfortunately, the two experiments reported here span only a relatively small space of what is possible when we combine perception experiments with the study of pictures. Nearly every existing visual paradigm can be translated towards a depiction-centered instead of a vision-centered research question. It seems an exciting new research area that appears rather timely: online experiments are mainstream, and online art collections grow every year. Furthermore, we are convinced that meaningfully pictorial research cannot do without human annotations, however advanced computer annotations may become.

The two experiments we conducted investigate pictorial characteristics of paintings that are often cataloged in databases. The (computational) analysis of image corpora can make use of both image data and metadata. The latter is created through annotations by art professionals and mostly concern objective, non-visual information such as the maker, medium, and provenance. Yet, many museum collections have been using their own archival methods, making comparison across collections challenging. This will change in the near future as initiatives like the International Image Interoperability Framework [38] start to gain traction. These new methods allow for a wider range of metadata such as user-generated annotations. In this perspective, any type of pictorial research related perception experiment can be added to these metadata. Being aware of other perception studies or annotations conducted on an image or image collection could result in unprecedentedly linked pictorial research.