Digital rear view mirrors with Augmented Reality in comparison with traditional rear-view mirrors

Recently, the traditional rear and side view mirrors have been started to be exchanged with a digital version. The aim of this study was to investigate the difference in driving performance between traditional rear-view mirrors and digital rear view mirrors which is called Camera Monitor System (CMS) in the vehicle industry. Here, two different types were investigated: CMS without or with Augmented Reality (AR) Information. The user test was conducted in a virtual environment, with four driving scenarios defined for testing. The user test results revealed that the participants’ driving performance using CMS (only cameras and 2D displays without augmented information) did not improve over traditional mirrors.


Introduction
Recently, the traditional rear and side view mirrors have been started to be exchanged with digital versions.The primary function of the rearview mirrors is to provide the driver with information about the traffic to the side and behind.The driver can use this information to anticipate the traffic situation and thus make decisions in advance.The camera monitor system (CMS) is to replace the traditional mirrors with cameras and then transmit the video to a screen in the car in real-time to provide the driver with information about the traffic behind.The development of CMS can be seen as three levels, traditional mirrors, CMS (only camera and screens), and CMS with Augmented Reality (AR) information overlay.Replacing traditional mirrors with cameras and screens can reduce air resistance, thus reducing energy consumption [1].In addition, the mirror will be affected by the weather when it rains, and snows, such as if the mirror is fogged or covered by rain and cannot work, and the screens and cameras are possible to avoid these problems.The screen placement in the car does not have to be limited to the placement of the traditional mirrors but can be placed closer to the driver so that they can pay more attention to the road.The Field of view (FOV) of the camera has the potential to be larger than the conventional mirror.And the FOV can be adjusted to cover more of the blind spot area.Adding Augmented Information to the CMS may improve safety, providing for example, highlighted pedestrians, blind-spot alerts, speed and distance information of vehicles behind, and other predictions.
The work targeted perception and decision-making in CMS involving behavioral or perceptual, as well as physical measurements.For the CMS, an important question was how depth perception and distance judgment were affected.Since the resulting image can be perceived as if located at a different distance than it would be in a standard, traditional mirror.Crucial is the impact on perception, potentially leading to erroneous decision-making.
A studies with test persons was carried out in a Virtual Reality (VR) simulator.In this article we will present part of the investigation reported in the M.Sc.thesis Zhang and Gao (2022) [2], concentrating on the time of taking a driving decision in a few different scenarios.

Indirect vision in cars
Two main methods are available to extend a driver's indirect vision in cars, namely conventional mirrors and CMS.According to Regulation No. 46 of the United Nations Economic Commission for Europe (UNECE) [3], mirrors are intended to give a clear view of the rear, side, or front of the vehicle within a pre-defined FOV using a reflective surface."Interior mirror" means, a mirror that can be installed in the passenger compartment of a vehicle, i.e., a rear-view mirror."Exterior mirror" refers a mirror that can be mounted on the exterior surface of the vehicle.i.e., side mirrors.
In terms of CMS, UNECE (2014) [3] defines it as a device that provides indirect vision using a camera-monitor combination.The camera is responsible for capturing the external image and converting it into a signal.Afterward, the monitor converts the signal into a visual image for the driver to view.
According to ISO (2019) [4] and UNECE (2014) [3] the side view displayed by CMS cannot be smaller than the FOV shown in Figure 1.The rear FOV displayed by CMS cannot be smaller than the FOV shown in Figure 2.   People can expand the FOV seen in the mirror by the body and head movements, whereas the FOV of the CMS is fixed.Regarding driving safety and the trend of automated driving, the CMS can develop an extended function that automatically adjusts the angle to the driving environment and driving conditions, increasing the FOV and reducing blind spots.Habibovic, et al. (2017) [5] proposed several advantages of CMS compared with conventional mirrors.CMS can provide a larger FOV, especially at intersections and roundabouts.In addition, with the avoidance of distractions such as blinding sunshine and the ability to adjust the parameters of the image, the CMS provides significantly better direct visibility.

Augmented Reality
Augmented reality (AR) is an emerging technology that enhances information about the real physical world through the use of digital visual elements, sound, or other sensory stimuli delivered through technology, which has a growing trend in interactive design.It combines real and virtual and can be interactive in real-time [6].
Current advances in CMS technology and AR open up an opportunity to explore the display of indirect vision in cars.AR technology increases the likelihood that the driver will be able to notice information on the road in time.The potential of AR is already being noticed in the automotive sector, with applications such as windshields to display information about road speed limits and in-car interfaces with hover buttons designed with AR [7].

Method
The experiment in this study was conducted in a virtual environment, approximating real road behavior, as suggested by Mullen, et al. (2011) [8].

Apparatus
The setup involved a physical component utilizing a VR-rig resembling a car cab (Figure 3), adjusted to mimic real vehicle conditions.Participants wore Varjo XR-3 VR headsets with precise calibration and a built-in eye-tracker.
The VR-rig used in this experiment was not a driving simulator, turning, acceleration and breaking were only simulated with binary activation of predetermined courses.This was achieved by buttons that were installed; one on the pedals and one on the steering wheel.When the participants turned the steering wheel or stepped on the pedals, the animation in the virtual environment was triggered, see Figure 4.The virtual setting, created in Unity with High Definition Render Pipeline (HDRP), simulating realistic conditions.Augmented information, emulating real-world AR, was achieved through Wizard of Oz methods, reducing technical complexity without impacting results.

Procedure
The testing process comprised four steps: consent form reading and signing, written instructions, a training session, and three randomized tests.Participants experienced four driving scenarios within three different setups: traditional mirror, CMS (camera and screen without assistance), and CMS with Augmented AR information.This testing approach encompasses a total of 12 driving tests to analyze participants' behaviors.This approach allowed a comprehensive study of driving behaviors.
After finishing each test, participants were required to take off the VR headset and complete a questionnaire for that test for about two minutes, followed by a break of about three minutes before proceeding to the next test.The questionnaire consisted of six statements for Likert scales with 7 levels between "strongly disagree" to "strongly agree".The statements were: 1.It was useful.2. It met my needs.3. Using it was effortless.4. It was easy to learn to use it.5. I was satisfied with it.6.I felt I need to have it.
There was a short interview with participants after they completed all three formal tests regarding AR information.

215--2
IS&T International Symposium on Electronic Imaging 2024 Human Vision and Electronic Imaging 2024

Scenarios
The authors first took all the scenarios related to the use of mirrors in the driving test, and then merged scenarios with similar situations (e.g., overtaking and lane changing use mirrors in almost the same way), and finally presented these four scenarios.One underlying assumption when designing the scenarios was that experienced drivers use the two-second rule when calculating the distance between two cars, i.e., the current speed times two seconds should be maintained between the two cars in order to have enough reaction time in case of sudden danger [9].Therefore, this experiment uses the time to represent both speed and distance.
The experiment considered maintaining a distance of two seconds or more as safe, while the designed AR information suggested a distance of three seconds.

Scenario A -Merging Lane
This scenario was set up with the test car merging from a single-lane road to a double-lane highway.As shown in Figure 5, the white car is the test car that the participants drove.Black cars come from behind consistently at a steady speed in the lane where the participant needs to merge.There are multiple black cars at equal distances on this road, and participants can choose their own lane change timing and they and participants were required to observe the distance between their car and the black car through the rearview mirror or digital display, see Figure 5 (right).When they thought it was a safe time to merge in, they pressed the pedal to trigger the merging lane animation.In this scenario, participants droved the test car with three different settings.So, the group variables were three different settings, namely Mirror, CMS, and AR, and the dependent variable was Td, the difference between the actual and optimal operation time.The formula was as follows.
Td = To -Ta To was the optimal operation time, and Ta was the actual operation time.The optimal operation time here refers to the point when the test vehicle merges into the gap between the two black cars at exactly the same distance as the black car in front and the black car behind.At this point, the test vehicle maintained a safe distance of 2.5 seconds multiplied by the vehicle's speed with both the front and rear.Since the user of this scenario needed to choose the appropriate gap (between two vehicles) to merge, the corresponding optimal time was different depending on the gap of the user decided to merge in:

Scenario B -Overtaking
This scenario was set up with the test car overtaking two cars in front of it.As shown in Figure 6, the white car was the test car that the participants drove.Participants were driving in a two-way lane and needed to change lanes to the left to first overtake two black cars.When participants were in the left lane, there was a silver car approaching from behind that intended to overtake the test car.Participants had to change their way back to the right lane to make way for the silver car, when they thought they had a safe distance from the black car.On top of that, participants were supposed to return to the right lane as soon as possible because the silver car behind them was showing an intention to overtake.The participants' perspectives are shown in Figure 6 (right).The silver car behind the test car gave them an added sense of urgency to change lanes back to the right.In this scenario, participants drove the test car with three different settings.So, the group variables were three different settings, namely Mirror, CMS, and AR, and the dependent variable was Td.The difference between the actual operation time and the recommended safety time.The formula was as follows: Td = To -Ta To is the optimal operating time corresponding here to the recommended safety time, and Ta was the actual operation time.The recommended safety time here refers to the point when the test car maintained a safe distance of 2 seconds multiplied by the speed of the car with the first black car, i.e., 30.714 s.So, the higher the negative value of Td, the safer the participants were driving.

Scenario C -Intersection
This scenario was set up with the test car turning right at the intersection.As shown in Figure 7, the white car was the test car that the participants drove.The initial state of the test car was to move forward at a slow speed.A cyclist was passing on the right side, and it was expected that participants could notice the cyclist and stop their car to wait for the cyclist to pass before turning.The participants' perspectives are shown in Figure 7 (right).The participant could observe the cyclist approaching from the side from the right-side mirror.However, when the cyclist was close enough to the test car, the cyclist would enter the blind spot area, and if participants did not do a shoulder check, they would not be able to see the cyclist.In this scenario, participants drove the test car with three different settings.So, the group variables were three different settings, namely Mirror, CMS, and AR, and the dependent variable was Td, the difference between the actual operation time and the critical crash time.The formula was as follows.
Td = Tc -Ta Tc was the critical crash time, and Ta was the actual operation time.The critical crash time here refers to the time when the test car crashed the cyclist, i.e., 9.849 s.So, the higher the value of Td, the safer the participant's driving.

Scenario D -Roundabout
This scenario was set up with the test car going through a roundabout.As shown in Figure 8, the white car was the test car that the participants drove.The test car was driving on a two-lane roundabout, making a 270-degree turn and exiting at 9 o'clock.Another black car behind were overtaking the test car from the outside, and the participant needed to be aware of the black car and slow down to avoid a possible accident.The participants' perspectives are shown in Figure 8   For technical reasons, the mirror could not be rendered correctly in this scenario, so the users were only driving the car with CMS and AR.So, the group variables were two different settings, namely CMS and AR, and the dependent variable is Td, the difference between the actual operation time and the critical crash time.The formula was as follows.
Td = Tc -Ta Tc was the critical crash time, and Ta was the actual operation time.The critical crash time here refers to the time when the test car crashed the cyclist, i.e. 12.888s.So, the higher the value of Td, the safer the participants were driving.

Analysis
To analyze the participants' driving performance, the test leaders measured the difference between the actual operating time (Ta) and the optimal operating time (To) or critical crash time (Tc) for the analysis.The time used for comparison with the actual operating time differs for each scenario because the driving task that the user needed to complete differs for each scenario.

Results
For the analysis, the authors tallied success rates and then chose successful cases for detailed analysis, as the data came from the same group and the variable being measured was continuous.Additionally, the data did not follow a normal distribution according to the Shapiro-Wilk test.The Friedman Test was used to identify performance variations among successful cases in scenarios A, B, and C, while the Wilcoxon Signed Ranks Test was employed for scenario D.

Participants
All participants recruited were from Volvo Car Corporation.A total of 28 participants took the test, most of them were male and three were female.All participants had at least one year of driving experience, and six of them wore glasses.Three of them helped us complete the pilot test to improve the setup and process of the experiment.4 participants participated in the test but were unable to complete it due to experiencing motion sickness in VR.

Scenarios
In each scenario, participants experienced cars with three separate settings in the VR environment, except for scenario with a Roundabout.Conventional rear-view mirrors in Roundabout scenario could not be rendered properly in the VR-environment due to technical reasons, so the data of CMS group and AR group were collected only for the Roundabout scenario.The results of the data analysis will be presented one by one according to the scenarios.

Scenario A -Merging Lane
In this scenario, the smaller the value of Td, the better the participant's driving performance.As shown in Table 1, in Scenario A, p > 0.05, i.e., the difference between the actual operation time and the ideal operation time in this scenario is not statistically significant.In Figure 9 the horizontal coordinate P of the graph represents the participants, and the vertical coordinate is Td.Each point represents a test.The car symbols on the graph represent the two adjacent black vehicles in the experiment.Line 0, the green line, represents the optimal time to operate.In the area between the blue line and the red line above, AR instruction is the green arrow, which means lane change is allowed.The AR instruction was a red cross for the area between the blue line and the red line below, which meant that lane changes were not suggested.If the point is in the area above or below the red line, it means that the test vehicle collides with the black car, and the test fails.

215--4
IS&T International Symposium on Electronic Imaging 2024 Human Vision and Electronic Imaging 2024 From Figure 9, the following points can be derived:  Most participants tended to approach the car in front of them rather than be close to the car behind them in all three groups. The CMS group failed the most out of the three groups.
According to the video recorded by the eye tracker, most of the reasons for the failure were that the participants judged the road conditions only by the digital display.When the approaching black car behind happened to disappear from the digital display, the participants immediately stepped on the pedal, resulting in a collision with the black car.


The medians of the three groups are very close.
While driving the car with conventional mirrors to perform the driving task of merging lane, the participants were not able to observe the road condition through the mirrors because of the angle between the lane where the test car was located and the target lane.An extreme case in Figure 9 is P22 in the Mirror group, who drove a car with conventional mirrors after driving a car with AR.However, he was very cautious and conservative when driving the car with conventional mirrors and waited until all the black cars in the experiment had passed before he merged into the target lane.

Scenario B -Overtaking
For the overtaking scenario, Ts is the recommended safety time, and Ta is the actual operation time.The recommended safety time here refers to the point when the test car maintained a safe distance of 2 seconds multiplied by the speed of the car with the first black car, i.e., 30.714 s.So, the higher the negative value of Td, the safer the participants were driving.
As shown in Table 2, in Scenario B, p < 0.05, i.e., the difference between the actual operation time and the safe operation time in this scenario was statistically significant.Pairwise comparison (see Table 3) of the three groups in scenario B shows a significant difference between the CMS and AR groups, and between the Mirror and AR groups at p < 0.05.No statistically significant differences existed between the Mirror and CMS groups.In Figure 10, the horizontal coordinate P of the graph represents the participants, and the vertical coordinate is Td.Each point represents a test.The car symbols on the graph represent the two adjacent black vehicles that participants need to overtake in the experiment.Line 0, the green line, represents the safe time to operate, i.e., changing the lane back to the right lane.In the area above the blue line, AR instruction was the green arrow, which means lane change was allowed.The AR instruction was a red cross for the area below the blue line, which means that lane changes were not suggested.If the point is in the area below the red line, participants changed lanes between two black cars or crashed, which were considered to have failed because the distance between the two vehicles was too close.From Figure 10, the following points can be derived:  Scenario B had a higher success rate than other scenarios, probably because the task in this scenario was relatively simple, and participants succeeded just by waiting patiently for a while before changing lanes, with a very low probability of crashing. CMS group had the worst driving performance, with multiple crashes.


In the AR group, most participants were willing to follow the AR instructions to change lanes at a greater distance.They would change lane back until the AR instruction becomes green even though the black car had become very small from IS&T International Symposium on Electronic Imaging 2024 Human Vision and Electronic Imaging 2024 215--5 the digital screen (i.e., the distance between the black car and the test car was already very long).


The difference in operation time between the Mirror and CMS groups was not significant.
In addition, there were some random cases that can be seen in Figure 10.In the AR group, P13 triggered two consecutive lane change animations due to operational errors, i.e., he changed back to the right lane immediately after changing to the left.P19's first formal test was to drive with the AR car, so he acted very cautiously and waited until he was very far away from the black car before changing back to the right lane.

Scenario C -Intersection
In Scenario C, Tc is the critical crash time, and Ta is the actual operation time.The critical crash time here refers to the time when the test car crashed the cyclist, i.e., 9.849 s.So, the higher the value of Td, the safer the participant's driving.As shown in Table 4, in Scenario C, p > 0.05, i.e., the difference between the actual operation time and the safe operation time in this scenario was not statistically significant.In Figure 11, the horizontal coordinate P of the graph represents the participants, and the vertical coordinate is Td.Each point represents a test.If the point is in the area above line 0 (the red line), participants stopped their car before crashing the cyclist.If the point is in the area below line 0, the red line, it means that participants crashed into the cyclist, i.e., their test failed.
From Figure 11, the following points can be derived:  The CMS group had the highest number of failures.The possible reason is that the CMS group only had a few people doing the shoulder check.The cyclists would enter the blind spot area of CMS before the crash, so the CMS group could not accurately predict the distance between them and the cyclist in the blind spot area.


The Mirror group kept the shortest safety distance from the cyclist.This was due to the difference between the FOV of the conventional mirror and that of CMS, which had a wider FOV.The cyclist was in the blind spot area of the conventional rear-view mirrors, and the participant could not observe the bicyclist in the mirror.So, participants could only make a decision to stop their car by shoulder check, which led to the phenomenon that the mirror group stopped before they almost crashed the cyclist.

Scenario D -Roundabout
Tc is the critical crash time, and Ta is the actual operation time.The critical crash time here refers to the time when the test car crashed the other car, i.e. 12.888s.So, the higher the value of Td, the safer the participants were driving.
As shown in Table 5, in Scenario D, p < 0.05, i.e., the difference between the actual operation time and the critical crash time in this scenario is statistically significant.In Figure 12, the horizontal coordinate P of the graph represents the participants, and the vertical coordinate is Td.If the point is in the area above line 0 (the red line), participants stopped their car before crashing the black car.If the point is in the area below line 0 (the red line), participants crashed the black car, i.e., their test failed.
From Figure 12, the following points can be derived:  There were more failures in the CMS group than in the AR group.


In the successful cases, the AR group maintained a longer safety distance overall than the CMS group.This was because, before the collision, the AR group would see the signal showing no lane change and the WARNING signal through the digital screen.


The success rate of scenario D was the lowest among the four scenarios.This may be because this scenario had the most challenging task.The black car was in the blind spot area for a long time and could only be seen through the mirror or digital screen a few seconds before the collision.
In addition, in the AR group experiment, two participants got confused between the non-changeable signal and the warning signal because both signals were red.These two participants stopped the car when they saw a red signal on display, but it was just a no-change lane signal at that time.

215--6
IS&T International Symposium on Electronic Imaging 2024 Human Vision and Electronic Imaging 2024

Shoulder check
The shoulder-check rate of each scenario is shown in Figure 13 below.The number represents the percentage of participants who performed a shoulder check in the experiment.The higher the number, the more participants performed a shoulder check in that scenario.Different color represents different driving scenarios.
From Figure 13, the following points can be derived:  The shoulder check rate in all scenarios is AR ≤ CMS < Mirror.


The shoulder check rate of the AR group in scenario B was significantly lower than that of the Mirror group, with a decrease of 18.3%.


The shoulder check rates of CMS and AR groups in scenarios A and D were the same.One of the reasons why fewer participants did the shoulder check may be that the digital screens were placed around the steering wheel in the experiment, unlike conventional mirrors that place the right-side mirror on the side of the car near the door.So, participants do not have to look out the window when checking the digital screen that simulates the right-side mirror.In addition, with the integration of AR in CMS, participants may think they do not need to do shoulder checks to understand the surrounding driving situation because AR has already collected information about the surrounding driving situation and will prompt drivers when necessary.
However, by Related-Samples Cochran's Q Test, it was found that the p-value was higher than 0.05 in all four scenarios, i.e., there was no significant difference in the shoulder-check rate of the three groups.

Conclusions
In the experiment, users were asked to experience different driving scenarios in a VR environment by driving a car with conventional rear-view mirrors, CMS, or CMS&AR, respectively.The authors identified four driving scenarios that could cover most of the driving behavior with rear-view mirrors, namely merging lane, overtaking, intersection, and roundabout, and designed four AR instructions combined with CMS.
The user study results showed that the combination of AR in CMS significantly improved driving performance and user experience compared to conventional mirrors.In contrast, CMS alone resulted in worse driving performance than conventional mirrors due to user unfamiliarity.

Figure 5 :
Figure 5: Scenario A -Merging lane First gap: 4.473 s Second gap: 10.473 s Third gap: 16.473 s Forth gap: 22.473 s

Figure 7 :
Figure 7: Scenario C -Intersection (right).Participants could observe the black car in the rear-view mirror and the right-side mirror.

Figure 9 :
Figure 9: Results of decision time (Td) for Scenario A (Merging Lane) for all test persons and for all test cases.

Figure 10 :
Figure 10: Results of decision time (Td) for Scenario B (Overtaking) for all test persons and for all test cases.

Figure 11 :
Figure 11: Results of decision time (Tc) for Scenario C (Intersection) for all test persons and for all test cases.

Figure 12 :
Figure 12: Results of decision time (Tc) for Scenario D (Roundabout) for all test persons and for all test cases.

Figure 13 :
Figure 13: The shoulder-check rate of the 4 Scenarios.
IS&T International Symposium on Electronic Imaging 2024 Human Vision and Electronic Imaging 2024 215--1