21 participants (11 female, 10 male; age 29 ± 10 years) were recruited from the Hearing Systems Section’s volunteers’ database and from the Technical University of Denmark (DTU) student community for this experiment. All participants reported normal vision and normal hearing. This was confirmed with standard clinical tests. All participants had normal hearing thresholds at octave frequencies between 125 Hz and 8 kHz and all scored a visual acuity rating of at least 0 on a LogMAR visual acuity chart [15]. Data from participant 15 were excluded from the analysis based on extreme outliers in the unimodal visual and pointing conditions (20 datasets remained). The procedure was approved by the local ethical committee “Videnskabsetiske Komitéer for Region Hovedstaden” (H-16036391), and all participants provided written, informed consent. The participants were compensated with an hourly rate of 122 DKK.

The experiment took place in the audio visual immersion lab (AVIL) of the DTU. Auditory stimuli were presented using seven loudspeakers (KEF LS50, KEF, Maidstone, UK) that were part of a 64-loudspeaker array. The loudspeakers used were evenly positioned between ±45 degrees azimuth at a distance of 2.4 m. In the center of the loudspeaker array was a height adjustable chair. The chair was adjusted such that the height of the participants’ ears was aligned with the centers of the loudspeakers.

For the presentation of the visual stimuli, an HTC Vive HMD (Head Mounted Display; HTC Corporation) was used. This HMD was run with a separate computer, which was controlled by the computer that ran both the experiment and the loudspeaker array. A 1:1 model of the experimental room, created in UNITY3D (Unity Technologies), was used for the virtual environment. Calibration was done as in [1], ensuring spatial alignment between the real and the virtual environment. For the calibration, three HTC Vive Trackers were placed at known positions and were tracked during the experiment. A shift of more than 1 cm in the position of one of the trackers, or the HMD losing tracking, resulted in a recalibration of the virtual world.

In the virtual environment, a virtual loudspeaker array was not included until the last part of the experiment. Instead, a gray ring (10 cm in height) was used to indicate the height of the loudspeaker array. At 0 degrees azimuth, just below this ring, a white square was placed that served as a focus point during the experiment (see Figure 1). The virtual environment was continuously visible during the experiment and did not change, except in the last task where the ring was replaced by the loudspeaker array. Between trials, a small sphere was used to help participants with the visual alignment process. This sphere was positioned at about eye height at a distance of 2.4 meters straight ahead of the participants and moved synchronously with their head movements. At the start of each trial, this sphere disappeared. Only after the trial was finished did it reappear. To proceed through the experiment and record their localization judgements, the participants used a handheld HTC VIVE controller. In the virtual environment, a thin red rod was attached to simulate a laser pointer to help the participants point toward the perceived auditory stimuli. This “laser” disappeared at the start of a new trial and reappeared when a response from the participant was requested.

Three different visual stimuli and two different auditory stimuli were used in this experiment. However, not all combinations were tested. The experiment was designed such that each bimodal condition, containing a single set of stimuli, added one new factor. The baseline condition represented the commonly used laboratory conditions in audio-visual experiments, i.e., flashes and noise bursts. For these baseline stimuli, the magnitude spectrum of the realistic sound was combined with a randomized phase to obtain a noise with the same loudness as that of the original recordings of the real handball impact stimulus. The visual baseline stimulus was a 20-ms light blur that appeared synchronously with the auditory stimulus above the loudspeaker ring. The light blur was 33.56 cm in diameter, corresponding to an 8-degree visual angle, as indicated in Fig. 1. The Gaussian blur had a standard deviation of approximately 5.5 cm (standard Gaussian blur scaled to the size of the visual stimulus).

The second bimodal set consisted of the same baseline stimuli, but it introduced a distractor task where a letter was shown on the white screen in the center at the same time as the other stimuli, as indicated in the middle panel of Fig. 1 for the flash and distractor stimulus. The purpose of the distractor task was to ensure that participants were fixating straight ahead during each trial (which is particularly important for later conditions involving moving visual stimuli). As no effect of attention was expected, this condition was included as a control. The letter remained visible for only 200 ms. After the participants had finished the localization task of the auditory stimuli, they were shown a matrix of 16 letters and had to select the letter that had appeared during the collision. If the participant was incorrect, the trial was repeated at a later time chosen at random. This process was repeated if the participant continued to indicate an incorrect letter.

The third set of stimuli again used the baseline stimuli (with the distractor task) but introduced movement. The visual stimulus appeared above the ring at the start of the trial, fell for half a second, bounced once on the ring and then disappeared 20 ms after bouncing. The bouncing on the ring was the trigger for the audio stimulus.

The realistic stimuli consisted of the sound and visuals of a dropping ball. The auditory stimulus was a 20-ms recording of the impact of a handball landing on a carpeted floor, presented at a peak equivalent (pe) sound pressure level (SPL) of 65 dB. As illustrated in the right panel of Fig. 1, the visual stimulus was a blue ball, in the same size as the flash stimulus (33.56 cm, or 8-degrees visual angle in diameter). As with the moving flash stimuli, the ball appeared above the ring, fell down, bounced once on the ring, triggering the auditory stimulus, and disappeared.

Due to a miscorrected latency in the system, the audio was played, on average, 105 ms after the visual stimulus. There was a variation of ±13 ms due to the frame rate of the HMD and a variation in the communication speed between the computers running the virtual environment and the audio system. This asynchrony was the same across all conditions. Furthermore, as the visual stimuli appeared slightly above the ring, there was a slight elevation difference of 3 degrees between the auditory and the center of the visual stimulus. However, due to the low sensitivity to incongruities in elevation [17], this should not affect the integration.

The main task of the experiment consisted of a localization task, using only auditory or only visual (i.e., unimodal) stimulation, or a combination of both (i.e., bimodal stimulation). In total, the experiment consisted of ten conditions which were divided into four blocks (see Table I). The block order was fixed: unimodal audio, bimodal, unimodal visual, pointing task. However, within each block, the conditions were presented in a counterbalanced manner across participants.

The first two conditions were unimodal audio conditions. Here, the sounds were presented randomly from one of the seven loudspeakers and each position was repeated five times resulting in 35 trials each. As the HMD has a limited field of view (110 degrees), the visual stimuli were limited to a maximum eccentricity of ±45 degrees. Because of this, the two outer loudspeakers were not used in the bimodal conditions. For each of the five loudspeakers used to present sound in the bimodal conditions, visual stimuli were presented in a 30-degree range around that loudspeaker in 3-degree steps and also at the other six loudspeaker locations. As a result, the densest sampling occurred between 15 and -30 degrees audio-visual disparity, and the maximum disparity was up to ±75 degrees. The sampling is also shown in Figure 2, which shows for each auditory stimulus position, all tested visual positions. Each combination was presented three times, leading to a total of 322 trials per bimodal condition.

The next block consisted of the three unimodal visual conditions. At this point in the experiment, the task changed from localizing sound to localizing visual stimuli. These conditions included stimuli at all seven loudspeakers and each position was tested three times, resulting in 21 trials per condition.

To account for potential biases in the pointing response [1], a “pointing” condition was included where the participants had to point at a continuously present static visual stimulus. In this task, no distractor ring was used and the gray ring was replaced by a model of the loudspeaker array. Participants were then shown a number and had to point with the “laser pointer” at the center of the loudspeaker with that number. This was the final task of the experiment. As in the unimodal visual conditions, this task used all seven loudspeaker positions and three repetitions were conducted, resulting in 21 trials. The conditions and stimuli are summarized in Table I.

Data were analyzed using the statistical software R [12]. The unimodal data were analyzed using Levene’s test to evaluate differences in the variance and an ANOVA was applied to the localization data. To analyze the bimodal results, the localization error was calculated per participant, condition, auditory stimulus position, and angle by subtracting the position of the auditory stimulus from the response. This localization error was corrected by subtracting the mean localization error in the congruent trials at each loudspeaker location to account for angle-dependant localization biases. This corrected error was then divided by the spatial audio-visual disparity to calculate the visual bias. The spatial audio-visual disparity itself was calculated as the position of the visual stimulus minus the position of the auditory stimulus, with positive disparities indicating that the visual stimulus occurred more to the right compared to the auditory stimulus. An ANOVA analysis compared the visual bias with the absolute spatial audio-visual disparity, condition, absolute auditory stimulus position, and the relative stimuli positioning as potential predictors. The relative stimuli positioning refers here to if the visual stimulus occurred outwards compared to the auditory stimulus or if it occurred closer to the center. To investigate how the various factors affected the results, a Bonferroni-corrected post hoc within factor comparison analysis was used. For this analysis, the disparities larger than 30 degrees were not included (i.e., only the densely tested area was included), as the initial analysis revealed interactions which could not be explored when these data points were included. Dropping these specific points from the analysis did not affect the results.

Figure 3 shows the localization error as a function of the stimulus position when pointing at a continuously present visual target. This task was included to measure the error in pointing. As a “laser pointer” was included, the accuracy and precision of pointing is very high. As can be seen in Fig. 3, the maximum median localization error was around one degree, and the variance of the error was below one degree. There was a small dependency (i.e., bias) of the localization on the stimulus position, with slightly increased errors at higher eccentricities (±1 degree at ±45 degrees azimuth, F1,6 = 3.234, p < 0.01). The variance did not vary significantly with stimulus position [F1,6 = 3.234, p = 0.631].

The unimodal conditions were used to test if there were differences in localization between the stimuli that were used. Figure 4 shows the localization error for the auditory (left panel) and the visual stimuli (right panel) as a function of the presentation angle. For the auditory stimuli, the baseline stimulus (noise burst) is indicated in light blue, and the congruent stimulus (ball impact audio) is indicated in dark blue. Levene’s tests showed that, for the auditory stimuli, the variance varied significantly only with angle [F1,6 = 2.662, p < 0.05], but not with condition [F1,6 = 1.341, p = 0.2470]. Similarly, the localization error also varied with angle [F1,6 = 51.035, p < 0.001] and not with condition [F1,1 = 0.251, p = 0.6162]. However, an interaction between the stimulus angle and condition was found [F1,6 = 2.645, p < 0.05].

The right panel of Fig. 4 shows the localization data for the visual stimuli, with the static flash data shown in light red, moving flash data shown in red and the ball stimulus data shown in dark red. For the visual stimuli, the localization error and variance were much smaller than for the auditory stimuli. Moreover, a clear trend can be seen in the visual responses, where responses were closer to the center (positive errors at negative angles and vise versa) when the stimulus was presented more laterally. Additionally, the variance also increased with presentation angle. This was confirmed by the statistical analysis, which revealed an effect of stimulus position on both the variance [F1,6 = 4.6560, p < 0.001] and the localization error [F1,6 = 55.935, p < 0.0001], but no effect of the stimulus used on either the localization error [F1,2 = 0.305, p = 0.7375] or the variance [F1,2 = 0.721, p = 0.1520], respectively.

Figure 5 shows the localization error as a function of the spatial disparity between the auditory and the visual stimuli in the four bimodal conditions for participant 4. A bias, where responses are shifted toward the position of the visual stimulus, can be seen in all conditions. However, comparing the average (dashed line) responses, no clear effect of condition is visible for this participant.

The visual bias, averaged across participants, per condition is shown in Figure 6. The left and right panels show the relative stimuli position. In the left panels, A-V-A, the visual stimulus occurred inwards relative to the auditory stimuli, whereas in the right panels it is instead the auditory stimulus that occurs inwards relative to the visual stimuli. The upper panels show results for when the auditory stimulus was positioned at 0 degrees azimuth, the middle panels show the results for when the auditory stimulus was presented at ±15 degrees, and the bottom panels show the results for when auditory stimuli were presented at ±30 degrees. In the upper panels, the auditory stimulus is presented at 0 degrees, as such, the left panel show responses where the visual stimulus was presented in the left hemisphere and the right panel shows responses for visual stimuli presented in the right hemisphere. As no visual bias can be calculated at 0 degrees spatial disparity, where the auditory and visual position overlap, the curve is interrupted at this position. Since the visual bias is calculated by dividing the localization error by the spatial audio-visual disparity, similar errors in localization cause much larger changes in the bias at small disparities. This results in a steep increase of the standard deviation as the disparity decreases.

As visible in Fig. 6, the visual bias was found to decrease, in most cases, significantly with increasing absolute spatial audio-visual disparity [F9,6980 = 5.513, p < 0.0001] and varied depending on both the relative positioning of the stimuli [F1,6980 = 5.034, p < 0.05] and the absolute position of the auditory stimulus [F2,6980 = 10.317, p < 0.0001]. However, significant interactions between these factors were found, namely an interaction between the effect of the relative and auditory stimulus positioning [F2,6980 = 18.373, p < 0.0001], an interaction between the spatial disparity and the relative stimulus positioning [F9,6980 = 2.187, p < 0.05] and a three-way interaction [F13,6980 = 4.427, p < 0.0001]. In the A-V-A stimulus setup, at small disparities (<15 degrees), the visual bias was larger when the auditory stimulus was presented at 0 degrees compared to ±15 and ±30 degrees [3 degrees, [0–15: t6980 = 5.657, p < 0.0001; 3 degrees, 0–30: t6980 = 6.105, p < 0.0001]. On the contrary, in the V-A-V setup, the visual bias was lower when the auditory stimuli were presented at 0 degrees [0–15 : t6980 = −5.278, p < 0.0001].

The results for the various conditions (see Fig. 6, upper panels) were very similar at larger audio-visual disparities. However, when the stimuli were close together, the visual bias increased, and some differences appeared between the conditions. Although no main effect of condition was found [F3,6980 = 0.4372, p = 0.4372], there was a significant interaction between the relative stimuli positioning and the condition [F3,6980 = 10.108, p < 0.001] and a three-way interaction between the auditory stimulus position, the relative stimuli positioning and the conditions [F13,6980 = 4.427, p < 0.001]. As can be seen in Fig. 6, upper-left panel, when the visual stimulus occurred in the left hemisphere with the auditory stimulus at 0 degrees azimuth (denoted, A–V–A, but since the auditory stimulus occurred at the center, this corresponds to stimulus occurring left), the realistic stimuli produced a significantly smaller visual bias than the noise and flash (baseline condition) stimuli [t6980 = 3.354, p < 0.01]. The other combinations did not reach statistical significance.

In contrast, when the visual stimulus was presented in the V–A–V setup, these realistic stimuli evoked a much more similar visual bias, and it was instead the second set of stimuli (noise and flash with a distractor) that produced a lower visual bias. Both at 0 [t6980 = −3.372, p < 0.01] and ±15 degrees [t6980 = 3.034, p < 0.05], the difference between the second and fourth (realistic stimuli) condition was significant. Curiously, a negative visual bias can be seen for the noise and flash with distractor stimuli in the upper-right panel of Fig. 6 indicating that participants perceived the auditory stimulus to be further away from the visual stimulus. Again, the stepwise comparison between the first and second, second and third, and third and fourth conditions was not significant.

To see if introducing the additional factor (attention, movement, realism) improved the model, equality constraints were tested using a Bayes factor test [27]. The fully unconstrained model performed worse than the combined noise and flash with and without distractor model, indicating that this factor indeed did not improve the model (BF = 2.3501e−11). Similarly, the fully unconstrained model performed worse than the model with the combined noise and flash with the distractor, and moving noise and flash with the distractor stimuli (BF = 9.6465e−7), and the combined moving noise and flash with the distractor and the realistic stimuli (BF = 1.7162e−7).