Back to articles
Regular Articles
Volume: 5 | Article ID: jpi0138
Smelling Sensations: Olfactory Crossmodal Correspondences
  DOI :  10.2352/J.Percept.Imaging.2022.5.000402  Published OnlineFebruary 2022

Olfaction is ingrained into the fabric of our daily lives and constitutes an integral part of our perceptual reality. Within this reality, there are crossmodal interactions and sensory expectations; understanding how olfaction interacts with other sensory modalities is crucial for augmenting interactive experiences with more advanced multisensorial capabilities. This knowledge will eventually lead to better designs, more engaging experiences, and enhancing the perceived quality of experience. Toward this end, the authors investigated a range of crossmodal correspondences between ten olfactory stimuli and different modalities (angularity of shapes, smoothness of texture, pleasantness, pitch, colors, musical genres, and emotional dimensions) using a sample of 68 observers. Consistent crossmodal correspondences were obtained in all cases, including our novel modality (the smoothness of texture). These associations are most likely mediated by both the knowledge of an odor’s identity and the underlying hedonic ratings: the knowledge of an odor’s identity plays a role when judging the emotional and musical dimensions but not for the angularity of shapes, smoothness of texture, perceived pleasantness, or pitch. Overall, hedonics was the most dominant mediator of crossmodal correspondences.

Supplementary Material S1
Subject Areas :
Views 209
Downloads 43
 articleview.views 209
 articleview.downloads 43
  Cite this article 

Ryan J. Ward, Sophie M. Wuerger, Alan Marshall, "Smelling Sensations: Olfactory Crossmodal Correspondences in Journal of Perceptual Imaging,  2022,  pp 000402-1 - 000402-12,

 Copy citation
  Copyright statement 
Copyright © Society for Imaging Science and Technology 2022
  Article timeline 
  • received July 2020
  • accepted April 2021
  • PublishedFebruary 2022

Preprint submitted to:
Journal of Perceptual Imaging
J. Percept. Imaging
J. Percept. Imaging
Society for Imaging Science and Technology
Olfaction is ingrained into the fabric of our lives, altering the very perception of our favorite commodities. It plays a crucial role in the multisensory perception of our surrounding environment. Crossmodal correspondences are the non-arbitrary and consistent associations between stimulus features in different sensory modalities that are shared among most of the population [67]. These associations could be considered as sensory expectations; incongruency between the actual and expected attributes of an experience could lead to a “disconfirmation of expectation” [8], which results in the experience being perceived as less pleasant [12]. These expectations influence our decision process [12], the perceived quality of the experience, and its perceived value [72]. Sensory expectations may help induce bias (i.e., providing a red glass of white wine can bias the judgment of expert wine tasters [49], and they have been shown to bias the appearance of physical creations [33]). Semantic congruency can also increase speeded olfactory discrimination [15], identification [17], and perceived pleasantness [57, 64]. Olfaction is one of the oldest senses and plays a major role in social behavior, communication, and emotional evaluation; mood and emotional processes share a common neural substrate with the olfactory pathway, namely the limbic system [40]. Therefore, it is likely that olfactory information plays a major role in modulating the quality of our immersive multisensorial experiences.
Crossmodal interactions between smell and both vision and hearing have been well documented, for example, olfaction–audition [5, 10, 63], olfaction–color [25, 34, 41], olfaction–visual motion [36], and olfaction–angularity of shapes [29, 34]. The mechanisms underlying such correspondences are still not fully understood, but three main explanations have emerged: hedonics [10, 11, 29, 46, 71], semantics [14, 34, 46, 67, 69], and natural co-occurrence [35, 67, 69].
The practical implications for crossmodal interactions expand into the marketing of products and interactive and immersive experiences. Designing experiences that conform to the sensory expectations of the user can increase the perceived quality of the experience in products [68] and human–machine interfaces [30, 50]. In marketing and human–computer interaction, the stimulation of multiple senses creates a richer and more immersive experience, provided the sensory cues are consistent with each other [60, 62, 70]. In terms of multisensory perception, the most reliable sense dominates usually vision [66]. For example, changing the color of a drink influences our perception of the product, shaping the aroma, taste, or flavor [65]. With the prevalence of crossmodal correspondences and olfaction in the entertainment, analytic, and marketing domains, it is vital to uncover the crossmodal interactions underlying common aromatic compounds, which will eventually lead to better designs, more engaging experiences, and an enhanced perceived quality of experience (see [7] for an introduction to quality of experience). Lesur et al. [42] showed that congruent visual–olfactory stimulation using virtual reality could enhance illusory embodiment (bodily self-identification). Design spaces for conveying olfactory information have been proposed [58] and evaluated [4]. Maggioni et al. [45] identified four key features for an olfactory design space: (i) chemical, (ii) emotional, (iii) spatial, and (iv) temporal. We also believe that sensory expectations are an important building block for designing more refined olfactory based experiences, most useful when perceived pleasantness or conveying information is an important factor. For instance, it has been shown that augmenting a human–computer interface with crossmodal correspondences can enhance user performance [28].
The term quality of experience has been defined in [7] as “the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user’s personality and current state.” In terms of olfaction-enhanced multimedia, there is an increasing interest in modeling the users’ quality of experience (i.e., [53, 55]), where the users’ sensory expectation is a crucial factor. It has been shown that congruency with the sensory expectations of an observer can increase the perceived pleasantness (e.g., [57, 64]).
The current study was designed to characterize crossmodal interactions between olfactory stimuli and other sensory modalities and explore these associations’ origin. The olfactory stimuli used in our experiments were aromas commonly used in perfumes, olfaction-enhanced multimedia (e.g., [6, 22, 54]), and were selected due to an overlap with prior literature (e.g., [14, 29, 32, 34, 47, 64]). Based on previous research, we hypothesized that crossmodal correspondences would be obtained between olfactory and visual stimuli (shapes, colors) and between odors and haptic/auditory stimuli. To explore these crossmodal associations’ origin, we hypothesized that explicit knowledge of the odor’s identity might modulate these associations (e.g., are odor–color associations contingent on the object’s known color?). Second, we tested the hypothesis that hedonic values might mediate crossmodal associations (e.g., are crossmodal interactions linked by eliciting a pleasant or unpleasant emotional reaction, such as happiness or sadness?). We use the term “hedonics” to refer to our emotional and musical dimensions. We use the term “semantics” to refer to past experiences (i.e., learned associations explained by explicit knowledge of an odor’s identity).
The present study aims to (1) explore the crossmodal associations between olfactory stimuli and the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, colors, musical genres, and emotional dimensions, and (2) to characterize the origin of these associations. We hypothesized the following: First, consistent crossmodal correspondences would be obtained, including a novel modality (the smoothness of texture). Second, the knowledge of odors identity (semantics) would affect the associations. Third, hedonics would play a notable and dominating role in characterizing olfactory crossmodal correspondences.
Materials and Methods
Sixty-eight individuals (23 males and 45 females with a mean age of 26.75 (standard deviation: 12.75)) took part in the experiments. No participants reported any impairment that could affect their sense of smell (i.e., cold or flu). Participants were briefed about potential allergens and breaks (a minimum of a 10-minute break halfway through, or if the participant felt like they have a reduced sense of smell). They were instructed not to wear any scented deodorant/perfume on the day of the experiment. The experiment lasted about one hour. It was approved by the University of Liverpool and conducted under the declaration of Helsinki’s standards for medical research involving human subjects. Participants gave written informed consent before taking part in the experiment. Participants were recruited via mass e-mail and the University of Liverpool’s experiment participation requirement (EPR). Participants recruited through the EPR system did so in exchange for class credit.
All results were obtained through a graphical user interface programmed in MATLAB R2018b. Participants were placed in a lightproof anechoic chamber equipped with an overhead luminaire (GLE-M5/32; GTI Graphic Technology Inc., Newburgh, NY) during the experiment. The lighting in the room was kept consistent by using the daylight simulator of the overhead luminaire. The speakers were JBL Desktop speakers; the color stimuli were shown on a calibrated EIZO ColorEdge CG243W monitor. The results were analyzed using MATLAB R2018b.
Tasks and Stimuli
Participants were instructed to associate a given odor with a value along each of the following dimensions: visual shapes (angularity), textures, pleasantness (using a Likert scale), pitch, musical genres, and emotions. For the musical genres and emotions, participants were asked to select the most dominant choice. The aromas were presented in a random order (determined by a random number generator in MATLAB), and all associations were assessed in the same order for a given aroma. Participants were presented with the stimuli as long as was necessary for them to make their decision. At the end of the experiment, participants were asked to identify the odor from a precompiled list (classification task). A neutral option was available for four experimental tests—visual shapes, texture, pleasantness, and the emotion task. However, participants were strongly discouraged from using this option.
Odor Stimuli
Ten odorants were used; five from Mystic MomentsTM —caramel, cherry, coffee, freshly cut grass, and pine; five from MiaromaTM—black pepper, lavender, lemon, orange, and peppermint. These aromas were selected as they can be frequently found during everyday life and gives diversity in the aromas’ chemical makeup. For consistency and to avoid any other unwanted associations with the color of the oil, 4 mL of the respective essential oil was placed in a clear test tube, wrapped in white tape, and numbered 1 through 10 in an initial random permutation. The aromas were stored at 2.5 C to minimize oxidation; all odors were removed and placed back into the fridge at the same time to ensure approximately uniform evaporation. The odors were replaced every two weeks.
Shape Stimuli
A nine-point Likert scale was constructed with a rounded shape, “bouba” and an angular shape, “kiki” on the scale’s left and right side, respectively. Similar to an earlier experiment performed by Hanson-Vaux et al. [29]. The midpoint of the nine-point scale (5) was neutral (no opinion). The anchors on each side of the scale were the images used in Hanson-Vaux et al.
Texture Stimuli
A nine-point Likert scale was constructed with the words “smooth” and “rough” on the left and right side, respectively. Participants were supplied with physical representative textures to aid them in their decision, with silk being a representative for smooth and sandpaper being a representative for rough. Only two texture samples were provided to the participants. The midpoint of the nine-point scale (5) was neutral (no opinion). Participants felt the textures at least once during the questions’ first appearance. Optionally participants could feel the physical textures again if they felt like they needed to.
A nine-point Likert scale was constructed, ranging from very unpleasant on the left side to very pleasant on the right side. The center of the scale (5) was used as neutral (no opinion).
Pitch Stimuli
The full range of audible frequencies (20 Hz to 20 kHz) was implemented using a slider where movement from left to right corresponded to an increase in frequency. Every time the slider was adjusted, the respective frequency was played, producing a sinusoidal tone lasting 1 second in length. Due to the large volume of potential selections, participants played a sample from each end of the scale, followed by a sample at halfway between these two points. If the current pitch did not match the odor, a lower or higher pitch was selected (approximately halfway between the last two frequencies) as indicated by the participant. During the initial tones being played at either end of the scale, we determined the range of frequencies the participants could hear by selectively increasing the frequency on the lower end and decreasing the frequency on the upper until the participant could hear the tone. Eight participants did not complete the pitch question; therefore, we only used the results for the sixty who did for the pitch related analyses. This was because this question was added to the experiment at a later point. The level for the pitch stimuli is shown in the supplementary materials (Fig. S1).
Music Stimuli
Seven different music genres—classical, country, heavy metal, jazz, rap, classic rock, and soul, were presented to the participants. Six were selected from [75], with one added due to its vast popularity (soul). Each sample was 15 seconds in duration, normalized to − 3 dB (relative to the peak amplitude) and played at the same volume across participants. The stimuli were trimmed using Audacity software. Participants had to listen to each sample at least once during the questions’ first occurrence; the order was subject to the participant’s preference. The musical excerpts used in this experiment can be found in the supplementary materials (Table S1).
Color Stimuli
The CIE Lab color space was used because of its perceptual uniformity. Participants could slide through 101 linear interpolated slices from the L channel of the color space, increasing or decreasing the lightness. Only colors that fit in the sRGB color gamut were shown. This removed the limitations of earlier studies that let participants choose from a small selection of colors.
Emotion Stimuli
A subset of emotions from the Universal Emotion and Odor Scale [18] were included. These were—angry, aroused, bored, calm, disgust, excited, happy, sad, and scared. An option for neutral (no opinion) was also available.
Classification Task
A list of different aromas was compiled consisting of the ten odors used in this experiment and an additional eleven (banana, coconut, eucalyptus, fudge, honey, musk, pineapple, rose, strawberry, toffee, and vanilla). These were presented in alphabetical order. The extra eleven odors were included so that observers were less likely to base their decision on previously presented odors when identifying the current odor. The classification task was presented after the participants went through the questions for each odor.
Angularity, Smoothness, Pleasantness, and Pitch
We first standardized the ratings into z-scores so all the scales were in the same units; ratings were centered on each individual scale’s respective grand mean. Figure 1. shows the mean ratings (transformed to z-scores) for angularity, smoothness, pleasantness, and pitch for each of the ten odors. Separate one-way repeated measures ANOVAs (Greenhouse–Geisser corrected, α = 0.05) were conducted on the z-scores of the angularity, smoothness, pleasantness, and pitch ratings to test if the odors influenced the ratings. This revealed that the odors significantly affected all ratings—angularity (F (7.09, 475.52) = 16.59, p < 0.001, η2 = 0.19), smoothness (F (7.98, 534.87) = 5.53, p < 0.001, η2 = 0.07), pleasantness (F (6.97, 467.23) = 10.48, p < 0.001, η2 = 0.13), and pitch (F(8.58, 406.788) = 10.23, p < 0.001, η2 = 0.148). Post hoc, one-sample t-tests (Bonferroni corrected, α = 0.005 (0.05/10)) were conducted to determine which of the odors were significantly different from 0 (the original scale’s grand mean); each tested modality (i.e., the angularity of shapes) was tested independently from one another. The significantly “rounded” odors are caramel (t(67) = −9.88, p < 0.005) and coffee (t(67) = −3.87, p < 0.005). The significantly “angular” odors are peppermint (t(67) = 8.62, p < 0.005) and lemon (t(67) = 3.43, p < 0.005) (Fig. 1(A)). The significantly “rough” odor is black pepper (t(67) = −3.22, p < 0.005) whereas caramel (t(67) = 4.64, p < 0.005) is associated with “smooth,” as shown in Fig. 1(B). The significantly pleasant odors were lemon (t(67) = 4.27, p < 0.005) and orange (t(67) = 6.87, p < 0.005) whereas black pepper (t(67) = −5.84, p < 0.005) is “unpleasant,” as shown in Fig. 1(C). In the unstandardized data, there is a bias toward pleasant odors, with nine out of ten of the odors being considered pleasant. The significantly “higher pitch” odor is peppermint (t(59) = 3.47, p < 0.005), while “lower pitch” is linked with coffee (t(59) = −5.64, p < 0.005) and caramel (t(59) = −4.60, p < 0.005), as shown in Fig. 1(D). Our hypothesis of consistent crossmodal associations between odors and the angularity of shapes, smoothness of texture, perceived pleasantness, and pitch is therefore supported. All tests not reported in this manuscript are included in the supplementary materials.
Figure 1.
(A–C) Mean scores for the ten odors after z-score normalization using the grand mean. Asterisks mark the odors that are significantly different from the scale’s original grand mean. Errors bars show a 95% confidence interval. (D) Shows the same information as (A–C) apart from the mean value used to calculate it, z-score is log2 of the original ratings.
Genre and Emotions
To assess if the odors affected the genre and emotion selections, separate chi-squared tests of independence (α = 0.05) were conducted. This revealed that the odors impact both the choice of genre (x2(54) = 138.20, p < 0.05, Cramer’s V = 0.18) and the participant’s emotional response (x2(90) = 187.54, p < 0.05, Cramer’s V = 0.17). Chi-squared tests for goodness of fit (Bonferroni corrected, α = 0.005 (0.05/10)) were conducted to see which of the presented stimuli were significantly different from a chance selection; the emotional and genre dimensions were tested independently from one another. The odors significantly different from a chance selection in the genre association task are black pepper (x2(6) = 22.59, p < 0.005), caramel (x2(6) = 46.06, p < 0.005), freshly cut grass (x2(6) = 21.56, p < 0.005), and orange (x2(6) = 22.79, p < 0.005) (see Figure 2(A)). All odors were significantly different from chance selection in the emotion association task—black pepper (x2(10) = 43.62, p < 0.005), caramel (x2(10) = 62.71, p < 0.005), cherry (x2(10) = 56.24, p < 0.005), coffee (x2(10) = 71.76, p < 0.005), freshly cut grass (x2(10) = 57.21, p < 0.005), lavender (x2(10) = 47.82, p < 0.005), lemon (x2(10) = 94.41, p < 0.005), orange (x2(10) = 111.56, p < 0.005), peppermint (x2(10) = 50.09, p < 0.005) and pine (x2(10) = 56.88, p < 0.005) (see Fig. 2(B)). Our data therefore supports the hypothesis of consistent crossmodal associations between odors and musical genres and emotions.
Figure 2.
Asterisks mark the odors that are significantly different from chance selection. (A) Association matrix between the ten odors and the seven musical genres. (B) Association matrix between the ten odors and the 11 possible emotional selections.
For this analysis, 343 colors from the Lab color space were chosen, and the user’s color selection was then mapped to one of these 343 colors based on the lowest ΔE 2000 error. The sampling points are shown in Figure 3(A). These hue selections were then used to compile a histogram of common colors, as shown in Fig. 3(B). The median hue angles for the commonly selected colors are shown in Fig. 3(C). Each participant only reported one color for each odor. One-sample t-tests (Bonferroni corrected, α = 0.005 (0.05/10)) were conducted to determine if the selected lightness values were significantly different from the scale’s midpoint and default slice of 50 (the mean lightness, which was also used as the starting point in the color selection task). The odors that yielded an L value significantly different from the mean lightness were: caramel (t(67) = 3.93, L = 59.86, p < 0.005), cherry (t(67) = 3.41, L = 58.03, p < 0.005), freshly cut grass (t(67) = 3.43, L = 56.65, p < 0.005), lavender (t(67) = 4.53, L = 59.87, p < 0.005), lemon (t(67) = 14.63, L = 76.16, p < 0.005), orange (t(67) = 11.86, L = 69.99, p < 0.005), peppermint (t(67) = 6.61, L = 66.85, p < 0.005) and pine (t(67) = 3.67, L = 58.07, p < 0.005). Due to a large number of possible hue angles, a chi-square test for independence (α = 0.05) was conducted on the frequency distribution of the binned hue angles (N = 15). This revealed that the color selections significantly differ from chance selection (x2(126) = 588.95, p < 0.05, Cramer’s V = 0.31). To determine which odor’s produced consistent color profiles, Rayleigh’s z tests (Bonferroni corrected, α = 0.005 (0.05 / 10)) were conducted on the hue angles for the commonly selected colors (Fig. 3(B)). This revealed that the color profiles for caramel (z = 5.79, p < 0.005), cherry (z = 5.32, p < 0.005), coffee (z = 6.22, p < 0.005), lemon (z = 6.67, p < 0.005) and orange (z = 6.63, p < 0.005) are non-random; thereby supporting the hypothesis of consistent crossmodal odor–color correspondences.
Figure 3.
(A) Lab color gamut showing the interpolated points used to determine the perceptually closest color. (B) Common colors selected by the participants, where each color has been mapped more than twice. Colors at the bottom of the graph occurred more often. Asterisks denote non-random color profiles. (C) Cylindrical representation of the Lab color space showing the median hue angle of the commonly selected colors for each odor.
Classification Task
The participants’ task was to identify the given odor by selecting one of the 23 possible selections from a list. Retrospective twofold classification was considered, exact classification and categorical classification. The exact classification was achieved 45.74% of the time by correctly identifying the current odor. The top three correctly classified odors are peppermint (82.35%), lemon (80.88%), and orange (61.76%). The top three misclassified odors are black pepper (10.29%), pine (13.24%), and caramel (20.59%), see Figure 4. Retrospective category classification was determined by the participants’ ability to pick another odor in the same category following the fragrance classes outlined in [9]. An accuracy rating of 62.94% was achieved for category classification; each potential classification belonged to only one category.
Figure 4.
Classification matrix for the ten odors, along with an additional 12 misclassifications. Any options that were never selected have been removed from the graph.
Classification Dependencies
To test if crossmodal associations are mediated by knowledge of the odor, the ratings were divided into two sets—“correct” and “incorrect” (Figure 5(A–D)). Two-way repeated measures ANOVAs (Greenhouse–Geisser corrected, α = 0.05), was conducted on the z-scores for angularity, smoothness, pleasantness, and the pitch ratings using the odors and classification sets (correct versus incorrect) as within-subject factors. Each modality was tested independently from one another. This revealed that the main effect for identification was not significant for the angularity (F(1, 6) = 3.19, p = 0.124, η2 = 0.347), smoothness (F(1, 6) = 0.123, p = 0.738, η2 = 0.020), pleasantness (F(1, 6) = 0.142, p = 0.74, η2 = 0.18), or pitch (F(1, 6) = 0.540, p = 0.50, η2 = 0.119). These results indicate that the ratings for the angularity of shapes, smoothness of texture, perceived pleasantness, and pitch are not mediated by knowledge of the odors identity.
Figure 5.
Mean z-scores asterisks denote odors where there is significant variation between the correct and incorrect ratings. The green markers denote correct classification, and the red markers denote incorrect classification. The error bars show a 95% confidence interval.
To assess if explicit knowledge of the odor affected the musical genre and emotional dimensions, the relative percentage difference was first calculated using the assignment percentages from the “correct” and “incorrect” sets. This was performed because the underlying data was categorical and not numerical. One-sample t-tests (Bonferroni corrected, α = 0.005 (0.05/10)) were conducted on the relative percentage difference to determine if the incorrect and correct proportions for each odor are significantly different from 0 (no change). The musical and emotional dimensions were tested independently from one another. This revealed that only freshly cut grass (t(6) = 4.5139, p < 0.005) and lavender (t(6) = 4.8174, p < 0.005) are significantly different for the genre ratings. The significantly different odors for the emotional ratings are black pepper (t(10) = 7.9480, p < 0.005), caramel (t(10) = 4.3374, p < 0.005), cherry (t(10) = 4.3374, p < 0.005), freshly cut grass (t(10) = 3.8407, p < 0.005), lavender (t(10) = 4.8719, p < 0.005) and lemon (t(10) = 4.0871, p < 0.005). Hence, the knowledge of the odor identity affects the observer’s musical genre and emotional responses, see Figure 6.
Figure 6.
Asterisks denote the odors that are significantly different from 0 (no change). The values marked with a lighter color indicate an increase in assignment (%) when they knew what the odor was. The values marked with a darker color indicate a decrease in assignment (%) when they knew what the odor was. (A) Relative association matrix between the ten odors and the seven musical genres. (B) Relative association matrix between the ten odors and the 11 possible emotional selections.
Following the same procedure as the color analysis above, the common colors for the correctly classified odors are shown in Figure 7(A), and the common colors for the misclassified odors are shown in Fig. 7(B). To determine if the observed proportions (quantity of commonly selected colors) between the common colors for the correctly classified and misclassified odors were the same, chi-squared tests for goodness of fit (Bonferroni Corrected, α = 0.005(0.05∕10)) were conducted. This revealed that five odor–color associations are significantly different for the observed proportions between correctly classified and misclassified colors: the proportions for black pepper (x2(1) = 8, p < 0.005), cherry (x2(1) = 10, p < 0.005), lemon (x2(1) = 14, p < 0.005), peppermint (x2(1) = 10, p < 0.005) and pine (x2(1) = 8, p < 0.005) are significantly different. Comparing Fig. 7(A) with Fig. 7(B), we can see that with the knowledge of the odor’s identity, the generated color profiles are more consistent. Without knowing the identity of the odor, the generated color profiles are more diverse. Chi-squared tests for goodness of fit (Bonferroni corrected, α = 0.005 (0.05/10)) were conducted on the correctly and incorrectly classified odors’ median hues angles. This revealed that the median hue angles for all odors except for caramel are significantly different. This suggests that the explicit knowledge of an odor’s identity affects the colors associated with the odors.
Figure 7.
Common color selections where each color has been (A) correctly classified and (B) incorrectly classified. Each color had to have been mapped more than twice for it to be deemed a common color. Colors at the bottom of the graph occurred more often.
Principal Component Analysis & Relationship Testing
Principal component analysis (PCA) was conducted to determine the perceptual similarity between the odors (Figure 8(A)) and to uncover the potential underlying relationships between the principal components (Fig. 8(B)). The PCA was conducted on the mean shape ratings, texture ratings, pleasantness ratings, pitch ratings, the color dimension (lightness), classification accuracy, the musical genre dimensions, and the emotional dimensions. Due to some of the ratings being on different scales, each independent rating (i.e., lightness and emotions) was rescaled between 1 and 9, then standardized. The Kaiser–Meyer–Olkin sampling adequacy [37, 38] is 0.86, Bartlett’s test of sphericity [3] is significant (p < 0.01), meaning the data was of high enough quality to progress with the factor analysis. Based on inspection of the scree plot, we extracted the first four principal components, explaining 81.55% of the total variance and have an eigenvalue of at least 1. The principal components 1 through 4 explain 33.31%, 25.44%, 13.88%, and 7.87% of the total variance. We did not perform any rotations on the factors produced by PCA as we believe that it better expresses the underlying data. The first two principal components are shown in Fig. 8(A); this shows the perceptual similarity between the olfactory stimuli, for example, (cherry, lemon, and orange), (lavender, freshly cut grass, and pine), (coffee and caramel) obtained similar results in most, but potentially not all ratings analyzed using PCA. We only explored the first two principal components for straightforward interpretation, as the inclusion of the 3rd and 4th components did not change the conclusion. The loadings matrix is shown in Fig. 8(B); this shows how strongly each component affects the principal components, shown in 8A. The 3rd and 4th components for both the score and loadings plot are shown in Fig. S2. The PCA loadings plot (Fig. 8(B)) suggests that hedonics plays an essential role in crossmodal correspondences, for example, the loadings of soul and metal on the smoothness rating. The loadings plot also suggests that knowledge of the odor’s identity plays a vital role in modulating the hedonic ratings. Based on inspection of the PCA loadings plot Fig. 8(B), we can see that the component “Classification Rate” has a strong loading on the first component (furthest from 0 on the x axis). This suggests that the semantic involvement (knowledge of the odor’s identity) plays an essential role in explaining the first component’s variation. The negative loadings of “Bored,” “Calm,” and “Country” onto the “Classification Rate” component suggests that knowledge of the odor’s identity mainly influences the hedonic dimensions (emotional and musical). The negative loadings of the hedonic dimensions (emotional and musical) on the “Angularity,” “Smoothness,” “Pleasantness,” and “Pitch” components suggest that they are affected more by the hedonic dimensions with little involvement from the semantic (correct classification).
Figure 8.
(A) Score plot of the perceptual ratings for each odor. (B) Loadings plot based of the correlation coefficients for each of the dimensions used in the PCA.
Discussion and Conclusion
Our results further the knowledge of multisensory interactions by demonstrating how olfaction interacts with the crossmodal perception of the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, musical genres, and emotions. Our results also expand upon the knowledge of how crossmodal correspondences are mediated.
Our first hypothesis was that associations exist for common aromatic compounds, and we find evidence for such associations that are consistent with prior findings—olfactory associations between the angularity of shapes [29, 34], music [10, 43], colors [14, 32, 34], pitch [5, 10], and emotions [43, 73]. Our work supports the findings of [29, 34], where lemon is perceived to be more angular. Our results also support the findings of [43], where lemon/coffee are associated with jazz, although a larger overlap between lemon/coffee and soul is observed in our findings. Further consistency was found with [43], where both orange and lemon are associated to being pleasant odors. We also found similar color profiles compared with [14, 32, 34] between lavender, lemon, peppermint, and caramel. A novel sensory modality was explored, the smoothness of texture. The hypothesis that crossmodal associations between odors and the smoothness of texture was supported with the odor black pepper being significantly associated to being rough and caramel being associated with smooth.
Our second hypothesis was that knowledge of an odor’s identity would affect the reported associations; this hypothesis was supported in part. The knowledge of the odor’s identity did not affect our ratings for the angularity of shapes, smoothness of texture, perceived pleasantness, or pitch; it did, however, effect the color, emotional and musical dimensions. These findings corroborate previous findings [13, 19] where knowledge of the odor’s identity influenced the hedonic dimensions. We expected more semantic involvement on the perceived pleasantness of the odors [46] than what was observed; although this inconsistency could be attributed to a different approach in the statistical analysis, as we split our results into two data sets (correct versus incorrect), whereas [46] had an additional hedonic dimension (familiarity) which was used as a degree for identification.
Our third hypothesis was that hedonic values might mediate these associations (e.g., [10, 16, 29, 67]). Our PCA results suggest that there was more hedonic involvement than semantic on the “Angularity,” “Smoothness,” “Pleasantness,” and “Pitch” components. The PCA also suggested that our “Classification Rate” (knowledge of the odor’s identity) mainly influences our hedonic dimensions. These findings suggest that there is more hedonic involvement than semantics in explaining the origin of these associations and therefore supporting our third hypothesis. These results corroborate the findings by Zarzo [77], who reported that the observer’s emotional response was the most dominant underlying dimension.
The results from this study could be used as a psychophysical framework to aid in the design or development of interactive experiences involving olfaction, such as product design. It has been shown that changing the packaging of products can change the consumers’ perception of the item within [68]. A study by Piqueras-Fiszman & Spence [59] has shown that people have strong crossmodal associations with products, and incongruency with these sensory expectations “annoys” the consumer [68]. Albeit, semantic congruency can increase its perceived pleasantness [57, 61, 64]. It is well known that adding additional sensory dimensions to virtual and augmented reality applications can increase the sense of presence (see [20] for a review). However, the possibility of exploring the crossmodal interactions uncovered in this work using virtual reality remains unexplored. For example, virtually changing the color, shape, and even texture of an object to determine the impact these associations have on the perceived quality of experience, perceived pleasantness, and their effects on immersion and presence. With the recent traction of quality of experience in both olfaction-enhanced multimedia and product design, there is a need to examine the user’s perception of the multisensory components [51]. These products/applications’ success depends on the impact it has on human observers [31, 51]. Presenting olfactory information in a meaningful way can enhance reality, clarity, and enjoyment [21]. Recent work in olfaction-enhanced multimedia stems away from solely the technical challenges with more focus on enhancing the perceived quality of experience. This includes, but is not limited to, synchronization [1, 24, 52], scent type [54], and impact of information recall [2, 23]. When designing applications or packaging of a product, it is important to conform to the user’s sensory expectations to enhance the perceived quality of experience. Little work in this area considers how different sensory modalities affect each other and the impact this has on immersive and interactive experiences. Metatla et al. [47] investigated the effects of scented three-dimensional (3D) printed shapes (“bouba” and “kiki”) on children; their results did not yield any significant tendencies of associating odors to 3D shapes. Furthermore, they did find significant associations between their shapes and odors (lemon and vanilla) and an emotional dimension (arousal). Mesfin et al. [48] investigated the use of crossmodally congruent olfactory stimuli in multisensory multimedia. Their results revealed that crossmodally matched media enhances the quality of experience compared to a video only condition. Koizumi et al. [39] explored the impact audio has on augmenting food textures by crossmodally changing the auditory properties thereby, changing people’s perception of the food they are experiencing. Building on the small body of work in this area, we show an aggregate of olfactory crossmodal correspondences to aid in the design process helping to provide an enriched, consistent, and more complete set of sensory cues to eventually increase the perceived pleasantness and quality of experience in both product design and human–computer interaction.
Future work could include investigating the extent the underlying chemical properties play in explaining crossmodal correspondences (i.e., [44]). As the associations are shared across observers, further work is needed to determine these associations’ stability over time [26, 29]. Additional sensory dimensions could be added; gustatory could be explored to expand upon our framework. These interactions could be used to determine how crossmodal correspondences affect the perceived quality and pleasantness of 3D printed food. Due to the nature of olfaction, it may be the case that taste played a minor role in the associations reported in this article. For example, Ngo et al. [56] reported the bitter tastes are associated with angular shapes, whereas sweet tastes are associated with a more rounded shape. Hanson-Vaux et al. [29] reported a strong association between the angularity of shapes and the perceived sourness/bitterness of odors. It may be the case that, at least some of the associations reported in this article could be modeled without the need for psychological tests. This could be accomplished by aligning the observer’s perception to the underlying chemical features of the odor’s using electronic nose technology (i.e., [27, 74, 76]). E-noses typically consist of a fine-tuned array of gas sensors with an accompanying pattern recognition system [74]. Moreover, it has already been shown that the underlying chemical features of odors can be mapped to the perceived pleasantness [27, 76]. Therefore, it is highly likely that some aspects of crossmodal correspondences could also be predicted (i.e., between the chemical features of odors and the angularity of shapes).
In summary, our results demonstrate that there is a variety of crossmodal correspondences underlying common aromatic compounds, the odors caramel and coffee are significantly associated with being rounded, while peppermint and lemon are significantly angular. In our novel modality (smoothness of haptic texture), the odor of black pepper is significantly associated with being rough, while the caramel is perceived to be smooth. The odors lemon and orange are perceived to be pleasant, with black pepper being unpleasant. Coffee and caramel are perceived to be lower in pitch, with peppermint being associated with being higher in pitch. The participants could also consistently associate with specific musical genres, emotional values, and colors to odors. Knowledge of an odor’s identity modulates the reported associations of the color, emotional and musical dimensions, but not the angularity of shapes, smoothness of texture, perceived pleasantness, or pitch. Finally, the role of hedonics (emotional and musical) in explaining olfactory crossmodal correspondences is greater than semantics (knowledge of the odor’s identity).
1AdemoyeO. A.GhineaG.2009Synchronization of olfaction-enhanced multimediaIEEE Trans. Multimedia11561565561–510.1109/TMM.2009.2012927
2AdemoyeO. A.GhineaG.2013Information recall task impact in olfaction-enhanced multimediaACM Trans. Multimedia Comput. Commun. Appl.910.1145/2487268.2487270
3BartlettM. S.“A note on the multiplying factors for various χ2 approximations,” J. R. Stat. Soc. B 16, 296–298 (1954)
4BatchA.PatnaikB.AkazueM.ElmqvistN.Scents and sensibility: Evaluating information olfactationProc. SIGCHI Conf. on Human Factors in Computing Systems2020ACMNew York, NY1141–14
5BelkinK.MartinR.KempS. E.GilbertA. N.1997Auditory pitch as a perceptual analogue to odor qualityPsychological Sci.8340342340–210.1111/j.1467-9280.1997.tb00450.x
6BrkicB. R.ChalmersA.BoulangerK.PattanaikS.CovingtonJ.Cross-modal affects of smell on the real-time rendering of grassProc. - SCCG 2009: 25th Spring Conf. on Computer Graphics, 1, 2122009ACMNew York, NY161166161–6
7BrunnströmK.BekerS.De MoorK.DoomsA.EggerS.GarciaM.-N.HoßfeldT.Jumisko-PyykköS.KeimelC.LarabiC.LawlorB.Le CalletP.MöllerS.PereiraF.PereiraM.PerkisA.PibernikJ.PinheiroA.PibernikJ.RaakeA.ReichlP.ReiterU.SchatzR.SchelkensP.Skorin-KapovL.StrohmeierD.TimmererC.VarelaM.WechsungI.YouJ.ZgankA.“Qualinet white paper on definitions of quality of experience,” European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003) (EU COST action 1003 QUALINET, 2013), p. 26
8CardelloA. V.SawyerF. M.1992Effects of disconfirmed consumer expectations on food acceptabilityJ. Sensory Stud.7253277253–7710.1111/j.1745-459X.1992.tb00194.x
9CastroJ. B.RamanathanA.ChennubhotlaC. S.2013Categorical dimensions of human odor descriptor space revealed by non-negative matrix factorizationPLoS ONE8e7328910.1371/journal.pone.0073289
10CrisinelA.-S.SpenceC.2012A fruity note: Crossmodal associations between odors and musical notesChem. Senses37151158151–810.1093/chemse/bjr085
11CrisinelA.-S.JacquierC.DeroyO.SpenceC.2013Composing with cross-modal correspondences: Music and odors in concertChemosensory Perception6455245–5210.1007/s12078-012-9138-4
12DelizaR.1995External cues and its effect on sensory perception and hedonic ratings: A reviewJ. Sensory Stud.11103128103–2810.1111/j.1745-459X.1996.tb00036.x
13DelplanqueS.GrandjeanD.ChreaC.AymardL.CayeuxI.Le CalvéB.VelazcoM. I.SchererK. R.SanderD.2008Emotional processing of odors: Evidence for a nonlinear relation between pleasantness and familiarity evaluationsChem. Senses33469479469–7910.1093/chemse/bjn014
14DemattèM. L.SanabriaD.SpenceC.2006Cross-modal associations between odors and colorsChem. Senses31531538531–810.1093/chemse/bjj057
15DemattèM. L.SanabriaD.SpenceC.2009Olfactory discrimination: When vision matters?Chem. Senses34103109103–910.1093/chemse/bjn055
16DixonM. J.SmilekD.MerikleP. M.2004Not all synaesthetes are created equalCogn. Affective Behavioural Neurosci.4335343335–4310.3758/CABN.4.3.335
17EngenT.1972The effect of expectation on judgments of odorActa Psychologica36450458450–810.1016/0001-6918(72)90025-X
18FerdenziC.DelplanqueS.BarbosaP.CourtK.GuinardJ.-X.GuoT.RobertsS. C.SchirmerA.PorcherotC.CayeuxI.SanderD.GrandjeanD.2013Affective semantic space of scents. Towards a universal scale to measure self-reported odor-related feelingsFood Qual. Pref.30128138128–3810.1016/j.foodqual.2013.04.010
19FerdenziC.RobertsS. C.SchirmerA.DelplanqueS.CekicS.PorcherotC.CayeuxI.SanderD.GrandjeanD.2013Variability of affective responses to odors: Culture, gender, and olfactory knowledgeChem. Senses38175186175–8610.1093/chemse/bjs083
20GallaceA.NgoM. K.SulaitisJ.SpenceC.Multisensory presence in virtual reality: Possibilities & limitationsMultiple Sensorial Media Advances and Applications: New Developments in MulSeMedia2011IGI GlobalHershey, PA1381–38
21GhineaG.AdemoyeO.2010A user perspective of olfaction-enhanced mulsemediaProc. Int’l. Conf. on Management of Emergent Digital EcoSystems, MEDES’10277280277–80ACMNew York, NY
22GhineaG.AdemoyeO.“User perception of media content association in olfaction-enhanced multimedia,” ACM Trans. Multimedia Comput. Commun. Appl. 8 (2012)
23GhineaG.AdemoyeO. A.Olfaction-enhanced multimedia: Bad for information recall?2009 IEEE Int’l. Conf. on Multimedia and Expo2009IEEEPiscataway, NJ970973970–3
24GhineaG.AdemoyeO. A.2010Perceived synchronization of olfactory multimediaIEEE Trans. Syst. Man Cybern. A40657663657–6310.1109/TSMCA.2010.2041224
25GilbertA. N.MartinR.KempS. E.1996Cross-modal correspondence between vision and olfaction: The color of smellsAm. J. Psychol.109335351335–5110.2307/1423010
26GilbertA. N.MartinR.KempS. E.1996Cross-modal correspondence between vision and olfaction: The color of smellsAm. J. Psychol.109335351335–5110.2307/1423010
27HaddadR.MedhanieA.RothY.HarelD.SobelN.2010Predicting odor pleasantness with an electronic nosePLoS Comput. Biol.6410.1371/journal.pcbi.1000740
28Hamilton-FletcherG.WrightT. D.WardJ.2016Cross-modal correspondences enhance performance on a colour-to-sound sensory substitution deviceMultisensory Res.29337363337–6310.1163/22134808-00002519
29Hanson-VauxG.CrisinelA.-S.SpenceC.2013Smelling shapes: Crossmodal correspondences between odors and shapesChem. Senses38161166161–610.1093/chemse/bjs087
30HogganE.KaaresojaT.LaitinenP.BrewsterS.2008Crossmodal congruence: The look, feel and sound of touchscreen widgetsICMI’08: Proc. 10th Int’l. Conf. on Multimodal Interfaces157164157–64ACMNew York, NY
31HughesC. E.StapletonC. B.HughesD. E.SmithE. M.2005Mixed reality in education, entertainment, and trainingIEEE Comput. Graph. Appl.25243024–3010.1109/MCG.2005.139
32JacquotM.NoelF.VelascoC.SpenceC.2016On the colours of odoursChemosensory Perception9799379–9310.1007/s12078-016-9209-z
33JezlerO.GilardiM.GattiE.ObristM.2016Scented material: Changing features of physical creations based on odorsConf. on Human Factors in Computing Systems - Proceedings167716831677–83ACMNew York, NY
34KaepplerK.2018Crossmodal associations between olfaction and vision: color and shape visualizations of odorsChemosensory Perception119511195–11110.1007/s12078-018-9245-y
35KnöferleK.SpenceC.“Crossmodal correspondences between sounds and tastes”, Psychon. Bull. Rev. 19 1–15 (2012)
36KuangS.ZhangT.2014Smelling directions: Olfaction modulates ambiguous visual motion perceptionSci. Rep.4151–510.1038/srep05796
37KaiserH. F.1974An index of factorial simplicityPsychometrika39313631–610.1007/BF02291575
38KaiserH. F.1970A second generation little jiffyPsychometrika35401415401–1510.1007/BF02291817
39KoizumiN.TanakaH.UemaY.InamiM.2011Chewing Jockey: Augmented food texture by using sound based on the cross-modal effectACM Int’l. Conf. Proc. Ser.ACMNew York, NY
40LeDouxJ.2007The amygdalaCurr. Biol.17R868R874R868–7410.1016/j.cub.2007.08.005
41LevitanC. A.RenJ.WoodsA. T.BoesveldtS.ChanJ. S.McKenzieK. J.DodsonM.LevinJ. A.LeongC. X. R.van den BoschJ. J. F.2014Cross-cultural color-odor associationsPloS one9e10165110.1371/journal.pone.0101651
42LesurM. R.AicherH.DelplanqueS.LenggenhagerB.2020Being short, sweet, and sour: Congruent visuo-olfactory stimulation enhances illusory embodimentPerception49693696693–610.1177/0301006620928669
43LevitanC. A.CharneyS. A.SchlossK. B.PalmerS. E.2015The smell of jazz: Crossmodal correspondences between music, odor, and emotionCogSci.1132613311326–31
44LiconC. C.BoscG.SabriM.MantelM.FourneA.BushdidC.GolebiowskJ.RobardetC.PlantevitM.KaytoueM.BensafiM.“Chemical features mining provides new descriptive structure-odor relationships”, PLoS Comput. Biol. 15 1–21 (2019)
45MaggioniE.CobdenR.DmitrenkoD.HornbækK.ObristM.“Smell space: Mapping out the olfactory design space for novel interactions”, ACM Trans. Computer-Human Interact. 27 (2020)
46MaricY.JacquotM.“Contribution to understanding odour-colour associations”, Food Quality and Preference 27 191–195 (2013)
47MetatlaO.MaggioniE.CullenC.ObristM.‘Like popcorn’: Crossmodal correspondences between scents, 3D shapes and emotions in childrenConf. on Human Factors in Computing Systems - Proceedings2019ACMNew York, NY1131–13
48MesfinG.HussainN.Kani-ZabihiE.CovaciA.SalemeE. B.GhineaG.2020QoE of cross-modally mapped mulsemedia: An assessment using eye gaze and heart rateMultimedia Tools Appl.79798780097987–800910.1007/s11042-019-08473-5
49MorrotG.BrochetF.DubourdieuD.2001The color of odorsBrain and Lang.79309320309–2010.1006/brln.2001.2493
50MurrayN.QiaoY.LeeB.MunteanG. M.“User-profile-based perceived olfactory and visual media synchronization”, ACM Trans. Multimedia Comput. Commun. Appl. 10 (2014) SUPPL
51MurrayN.LeeB.QiaoY.MunteanG. M.“Olfaction-enhanced multimedia: A survey of application domains, displays, and research challenges”, ACM Comput. Surv. 48, 56 (2016)
52MurrayN.MunteanG.-M.QiaoY.LeeB.2018Olfaction-enhanced multimedia synchronizationMediaSync319356319–56SpringerBerlin
53MurrayN.LeeB.QiaoY.Miro-MunteanG.The influence of human factors on olfaction based mulsemedia quality of experience2016 8th Int’l. Conf. on Quality of Multimedia Experience, QoMEX 20162016
54MurrayN.LeeB.QiaoY.Miro-MunteanG.“The impact of scent type on olfaction-enhanced multimedia quality of experience,” IEEE Trans. Syst. Man Cybern. Syst. 47, 2503–2515 (2017)
55MurrayN.MunteanG. M.QiaoY.BrennanS.LeeB.2018Modeling user quality of experience of olfaction-enhanced multimediaIEEE Trans. Broadcast.64539551539–5110.1109/TBC.2018.2825297
56NgoM. K.MisraR.SpenceC.2011Assessing the shapes and speech sounds that people associate with chocolate samples varying in cocoa contentFood Qual. Preference22567572567–7210.1016/j.foodqual.2011.03.009
57ÖsterbauerR. A.MatthewsP. M.JenkinsonM.BeckmannC. F.HansenP. C.CalvertG. A.2005Color of scents: Chromatic stimuli modulate odor responses in the human brainJ. Neurophysiol.93343434413434–4110.1152/jn.00555.2004
58PatnaikB.BatchA.ElmqvistN.2019Information olfaction: Harnessing scent to convey dataIEEE Trans. Visual. Comput. Graph.25726736726–3610.1109/TVCG.2018.2865237
59Piqueras-FiszmanB.SpenceC.2011Crossmodal correspondences in product packaging. Assessing color–flavor correspondences for potato chips (crisps)Appetite57753757753–710.1016/j.appet.2011.07.012
60Piqueras-FiszmanB.SpenceC.2015Sensory expectations based on product-extrinsic food cues: An interdisciplinary review of the empirical evidence and theoretical accountsFood Qual. Preference40165179165–7910.1016/j.foodqual.2014.09.013
61SabiniewiczA.SchaeferE.ManesseC.HummelT.2021Smells influence perceived pleasantness but not memorization of a visual virtual environmenti-Perception121171–1710.1177/2041669521989731
62SchiffersteinH. N. J.SpenceC.Multisensory product experienceProduct Experience2008ElsevierNew York133161133–61
63SeoH.-S.HummelT.2011Auditory-olfactory integration: Congruent or pleasant sounds amplify odor pleasantnessChem. Senses36301309301–910.1093/chemse/bjq129
64SeoH.-S.LohseF.LuckettC. R.HummelT.2014Congruent sound can modulate odor pleasantnessChem. Senses39215228215–2810.1093/chemse/bjt070
65SpenceC.LevitanC. A.ShankarM. U.ZampiniM.2010Does food color influence taste and flavor perception in humans?Chemosensory Perception3688468–8410.1007/s12078-010-9067-z
66SpenceC.GallaceA.2011Multisensory design: Reaching out to touch the consumerPsychol. Mark.28267308267–30810.1002/mar.20392
67SpenceC.2011Crossmodal correspondences: A tutorial reviewAttention, Perception Psychophysics73971995971–9510.3758/s13414-010-0073-7
68SpenceC.Piqueras-FiszmanB.The Multisensory Packaging of Beverages2012Nova Science Publishers, Inc.
69SpenceC.DeroyO.2013How automatic are crossmodal correspondences?Consciousness Cogn.22245260245–6010.1016/j.concog.2012.12.006
70SpectorF.MaurerD.2012Making sense of scents: The colour and texture of odoursSeeing Perceiving25655677655–7710.1163/187847612X648800
71StevensonR. J.RichA.RussellA.2012The nature and origin of cross-modal associations to odoursPerception41606619606–1910.1068/p7223
72TeasR. K.AgarwalS.2000The effects of extrinsic product cues on consumers’ perceptions of quality, sacrifice, and valueJ. Acad. Mark. Sci.28278290278–9010.1177/0092070300282008
73WangQ.SpenceC.2017Assessing the role of emotional associations in mediating crossmodal correspondences between classical music and red wineBeverages3110.3390/beverages3010001
74WardR. J.JjunjuF. P. M.GriffithE. J.WuergerS. M.MarshallA.2020Artificial odour-vision syneasthesia via olfactory sensory argumentationIEEE Sensors J.21678467926784–9210.1109/JSEN.2020.3040114
75WhitefordK. L.SchlossK. B.HelwigN. E.PalmerS. E.2018Color, music, and emotion: Bach to the bluesi-Perception9610.1177/2041669518808535
76WuD.ChengY.LuoD.WongK. Y.HungK.YangZ.2019POP-CNN: Predicting odor’s pleasantness with convolutional neural networkIEEE Sensors19113371134511337–4510.1109/JSEN.2019.2933692
77ZarzoM.2008Psychologic dimensions in the perception of everyday odors: Pleasantness and edibilityJ. Sensory Stud.23354376354–7610.1111/j.1745-459X.2008.00160.x