General Aviation (GA) refers to all civil aviation operations that do not transport passengers or goods for commercial or “for hire” purposes [1]. GA had the most flight hours in civil aviation and is the deadliest segment in civil aviation (regulated by Title 14 Code of Federal Regulations Part 121, 135, 91) [2]. In 2021 alone, GA had more than 21.9 million flight hours compared to 15.9 million flight hours for part 121 air carriers (regularly scheduled air carriers such as U.S.-based airlines, regional air carriers, and cargo operators). For every 100,000 GA flight hours there were 5.26 accidents with 0.95 being fatal, compared to part 121 air carriers’ accident rate of 0.49 with 0 being fatal [2]. Among those accidents, pilot decision making was a decisive factor with 55%–85% of GA accidents caused, or contributed to, by pilot error [3].

A pilot’s decision is important in dangerous weather conditions like thunderstorms. During 1996–2014, 71% of thunderstorm-related accidents resulted in fatalities, which was significantly higher than non-thunderstorm-related accidents (23%) [4]. The high percentage of thunderstorm-related accidents highlights the need to improve pilots’ decision-making skills when facing adverse weather conditions [5]. Efforts to help student pilots take correct decisions when facing severe weather conditions, include required training programs on weather theory knowledge [6]. Weather theory knowledge provides pilots with a background of weather principles, how they affect flight safety, and how to make proper decisions [7]. Without sufficient knowledge of weather theory, pilots might not be able to take safe decisions when facing adverse weather conditions [8, 9].

Student or novice pilots can learn weather theory knowledge from a Federal Aviation Administration (FAA) certificated pilot school, either in-person or at home [6]. Different training programs focus on different aspects of weather, which leads to inconsistency in pilots’ knowledge levels [10, 11]. In addition to the inconsistency in training programs, the educational materials provided to student pilots are exclusively in 2D formats such as textbooks, images, and videos [12]. These 2D materials cannot effectively communicate all the complexities of weather phenomena and require students to create their own 3D mental models leading to misunderstanding of important concepts [13, 14]. Though there are flight simulators and other expensive visualization equipment to simulate weather phenomena, most of them use cockpit views and focus on practical training instead of aeronautical knowledge instruction. Traditional materials also lack hypothetical scenarios that may be encountered in flight or clear instructions on different topics in weather aviation [9]. To effectively overcome these issues for effective weather theory training, three criteria were proposed to for educational materials: (1) allow students who do not have expensive equipment to easily visualize complex weather phenomena with hypothetical scenarios, (2) provide easily accessible equipment to teach 3D weather concepts, and (3) complement traditional classroom and 2D flight instruction materials. Due to the wide range of GA training programs, it poses a considerable challenge to ensure the availability of 3D visualization equipment to students. As a result, it is important for students to access the 3D training content through their existing devices.

Advanced flight simulators to visualize complex 3D weather phenomena are available but fail to address the proposed criteria. Most of the high-fidelity training simulators are aircraft specific and dedicated to commercial and military pilots [15–17]. In the past 10–15 years, extended reality (XR) has been used in aviation training to further reduce training costs and broaden training system availability [18, 19]. For the purposes of this work, XR will be used as an umbrella term for technologies that immerse users into different degrees of virtual environments (VE), including Virtual Reality (VR), AR, and Mixed Reality (MR) [20]. VR immerses users into a completely virtual environment [21], typically using a Head-Mounted Display (HMD) (also referred to as a Head Worn Display) or a CAVETM system. For the purposes of this paper, the term HMD will be used when referring to a display on a user’s head. AR presents 3D virtual content overlaid onto real-world environments [22]. AR allows a user to view virtual content and 2D material using either mobile devices, such as smartphones and tablets (see Figure 1) or AR HMDs such as the Microsoft HoloLens (see Figure 2) [23–25]. HMD-based VR and AR training have limitations such as requiring an external computer to power the device or a limited area of use due to constraints on tracking [19].

A critical decision in this work was to determine a suitable platform for the 3D training content. Several potential platforms were considered in this decision. First, a VR system can be in an HMD powered by a mobile device, standalone computing in the headset, or tethered to a more powerful external computer. Any VR HMD provides a fully immersive experience and can remove a user’s situational awareness of their physical environment. So, a student in a VR HMD may not be able to use a physical textbook while in a VE, requiring the simulation to include all teaching materials (i.e., no textbook). Alternatively, to use any kind of VR HMD with traditional teaching materials will require several awkward putting on and taking off of the system by a user. A standalone computing HMD or one powered by external computer can provide sufficient rendering power. This power comes at an additional cost of $300–$3,000 per user. A phone-based VR HMD reduces overall system cost but will have significantly decreased rendering capabilities. AR can present virtual content using a device that students most likely already own (e.g., a smartphone) and meets the three criteria to improve weather theory training listed above. AR on a mobile device was a favorable choice for this work due to several advantages. First, AR does not require every screen pixel to show virtual content, so rendering in real-time on a mobile device can be accomplished. Additionally, AR allows for seamless integration with traditional materials, making it easier to work with existing content. Furthermore, AR can deliver reliable tracking and spatial navigation by utilizing a simple printed image marker. Lastly, AR is widely used in many fields where training personnel on complex spatial relationships is necessary, such as learning 3D geometry [26]. After considering all these issues from the literature search, AR on a mobile device using a marker-based algorithm was chosen for this project. In this work, AR on mobile device is referred to as mobile AR.

While marker-based mobile AR technology was chosen in this work, a mobile device does not have the same graphics capability as an HMD or high-fidelity flight simulator. These hardware limitations have reduced many XR-based training materials to simple content such as videos, static 3D models, and 2D images. In addition, most of the implementations do not have complex interaction to further enhance training [27–31]. Mobile devices only have the capability to render a limited number of polygons due to their processing power and battery capacity [32] and due to the relatively small screen size, weather models might not fully fit on the screen with a corresponding user interface (UI). In addition, touch-screen interfaces limit interactions for 3D VEs. A user cannot input depth information as most touch screens only provide 2D position information. Finally, AR on mobile devices is further limited by one hand use, due to the other hand usually occupied with holding the device [33, 34].

To use AR in weather theory training, a new solution is needed to render complex 3D weather phenomena with limited processing power, an effective UI, and the ability to compensate for non-ideal tracking conditions. The research presented in this paper studied, addresses the technical challenges to use mobile AR in weather theory training for GA student pilots. The goal is to achieve high-quality rendering and real-time stable tracking of thunderstorm educational modules on a mobile device to ensure an immersive and effective user experience. This application serves as an easily accessible 3D training tool for initial student pilots, assisting their comprehension of abstract thunderstorm concepts from 2D materials.

To achieve this objective, a particle system was created to form a thunderstorm cell with a life-like cloud appearance. Specialized shaders, viewing perspectives, and an image target were developed to improve weather theory training, lower the rendering requirements for a mobile device, and maintain stable tracking of virtual content. The work also conducted a technological evaluation of the application’s rendering performance and tracking accuracy across all compatible devices.

Although this paper focuses on the technical achievements of this work, it is worth noting that a series of user studies were conducted to assess the usability and educational benefits to students of the system as well. While not the focus of this paper, they are briefly summarized here (for full details see Meister et al. [35, 36]). Evaluations determined that the students improved their factual knowledge and visual knowledge, with high levels of motivation in a statistically significant manner. A preliminary evaluation of the application was conducted with three subject matter experts in weather and aviation, and three students, to assess whether the AR thunderstorm visualization could communicate weather theory and whether the interfaces were usable for learning and task completion (for full details, see Meister et al. [35]). Students’ knowledge increased after using visualization to explore the dynamics of the thunderstorm. Subject matter experts felt the learning experiences appropriately communicated thunderstorm theory in ways that supported instruction. The AR interfaces were rated as usable for learning interactions and produced low levels of workload. In a follow-up study, 18 (17 male, 1 female) student pilots, or pilots with fewer than 250 total flight hours, were asked to complete the AR weather-related learning activities on a smartphone (for full details, see Meister, [36]). The learning outcomes were measured by pre-and post-tests concerning factual and visual knowledge and participants’ completion time. There was a statistically significant increase in student factual knowledge (from 71% to 91%), even though students already had a high level of incoming knowledge. Visual knowledge also increased in a statistically significant manner, from 55% to 90%. To evaluate user experience with the AR application, post-trial surveys were also given to evaluate the application’s usability, user motivation, and overall experience. Participants also reported positive learning experiences with high motivation, reasonable task load and completion time, and excellent usability, as rated by the system usability scale. Together these studies demonstrate that the AR thunderstorm model, and associated learning scenarios, resulted in an effective application that enhanced the learning outcomes of students and teaches aviation weather in a way that met the expectations of aviation and weather instructors.

A literature review elucidated current research efforts on weather theory training for GA pilots and mobile AR in training and education.

A thunderstorm model with learning activities and scenarios was developed using Unity and implemented in a mobile AR application for both iOS and Android devices [63]. Experienced flight instructors provided expertise throughout the development process including the thunderstorm characteristics and corresponding scenario activities [35]. This feedback was crucial to create educational materials that would be effective for GA student pilots. The following sections discuss the development and implementation process of creating the thunderstorms training content in a marker-based mobile AR application and how to overcome the technical challenges identified earlier.

Thunderstorms change dynamically with clouds forming and dissipating throughout their lifecycle. To simulate such activities in a mobile application, where computational resources are limited, a particle system was used. Unity offers a particle system Application Programming Interface (API) that allows small images to be emitted to simulate fuzzy visual effects [60]. Particle systems are a common way to create “volumetric” visual effects with reduced rendering overhead [32, 60]. Volumetric refers to visual effects that have movement on the surface and within an object such as fire, smoke, and clouds. A thunderstorm in real life transforms through its lifecycle stages (i.e., developing to dissipation) with a classic anvil shape (i.e., nonuniform in the direction of environmental winds). Controlling each particle within the model to achieve such a visual effect increases processing time and complexity, and is not ideal for mobile AR. To address this issue, the thunderstorm was formed in segments with each segment containing one particle system to form a part of the overall anvil cloud shape through its lifecycle (see Figure 3). Particles are emitted at the beginning of the simulation with random orientations and sizes. The particles’ properties (e.g., transparency and color) are controlled by a custom developed graphics shader. A graphics shader is computer code that outputs correct levels of opacity, shading, and color, and even geometry movement (i.e., vertex procedures), during the rendering of a 3D scene. The developed shader takes the height of different location on the cloud, the presented weather information, and the current simulation time. Based on these factors, the shader assigns a specific color and transparency to each location on the cloud, enabling a gradual distribution. By combining this shader with the segment approach, thunderstorm can achieve a nonuniform shape while still maintaining an overall color and transparency changes during the cycle. However, no new particle was created nor modifications to existing particles in the cell were done at run-time to keep rendering resources low. The shader code and the segment approach significantly reduced the time to create unique cloud formations (e.g., duplicating cloud segments or the entire cloud cell) and ran in real-time on a mobile device.

Based on the Pilot’s Handbook of Aeronautical Knowledge [12], wind, icing, and temperature conditions in a thunderstorm play a critical role in flight safety and are crucial for pilots to understand. Representing these various components of information required easy-to-understand visual cues that were distinguishable when viewed alone or all together.

Color is a powerful tool to display numeric information and was used in the model. The custom graphics shader developed allow gradual color changes throughout the cloud. This shader was used in presenting cloud density (see Figure 4), icing conditions (see Figure 5), and temperature information through the cloud (see Figure 6). To distinguish between cloud density, icing, and temperature, as colors will overlap with each other, icing has labels on the cloud as secondary markings, and each piece of information can be viewed separately. It is important to note that the choice of colors was arbitrary and does not have special meaning in aviation education. Vibrant colors with high contrast to one another were used with the approval of expert flight instructors to allow easy distinction for the concepts they wanted conveyed to students.

However, not all weather information could be presented with color. Unlike gradually changing data inside the thunderstorm cloud, some corresponding weather phenomena required a more realistic appearance. Particle systems were again used to produce these phenomena. For precipitation, each particle was represented as a raindrop, or hail particle, with a gravity force attached to make it descend from the cloud. Rain and hail were differentiated with unique textures and densities (see Figure 7). In addition, a microburst, an additional characteristic of a thunderstorm critical to pilot training, had to be developed. A microburst contains localized downdraft air, a dust ring, rolling clouds, and an area of intense rain called a “virga” [64]. To simulate microbursts, particles were set to a continuous cycle movement and dissipation to represent the accompanying rolling clouds, virga, and dust ring (see Figure 8). A variety of materials were implemented to attain a realistic appearance for these phenomena. Since each particle system has its own animation, it was controlled through scripting to ensure synchronization with the overall thunderstorm simulation, which was under the user’s control.

Finally, variation in wind direction and speed is one of the most crucial components for pilots to understand in real-time flight decisions. Unlike other information in and around a thunderstorm, which may be more contained to a specific region, wind constantly changes in and around the entire cloud. For example, when a microburst occurs, the wind moves toward the ground and then circles back to the thunderstorm after hitting the ground as shown in (see Fig. 8). Additionally, since the wind movement cannot be represented in a lifelike appearance, a simplified 3D arrow geometry was utilized. The model presented in this paper used Unity’s physics engine to compute the movement of these 3D arrows according to a predefined wind pattern advised by expert flight instructors [65]. However, a limitation of Unity’s physics engine was the absence of a timeline-like animation playback feature. To address this limitation, a workaround was implemented whereby the arrow movement remained constant throughout the simulation, and the activation of the arrows was adjusted accordingly to indicate the progression of the wind. By implementing this approach, the wind movement driven by the Unity physics engine can be integrated and controlled within the overall thunderstorm simulation.

To assist student pilots in understanding 2D materials, a thunderstorm simulation was developed and integrated into a mobile AR application. This allowed students to experience 3D learning content on widely available devices. In the literature review, various technical challenges were identified in relevant research, including limitations in rendering power, tracking performance, and interaction difficulties. The development and the implementation of the thunderstorm simulation has overcome these challenges by using a custom particle system and multiple developed graphics shaders to reduce rendering overhead. Additionally, different viewing perspectives, well-designed image targets, and a simple yet useful interface were developed to ensure stable tracking and improve user interaction. The result from the technical evaluations showed that the application had a stable and sufficient rendering and tracking performance on different mobile devices and operating systems. Additionally, prior study also indicated that the application had the potential to enhance students’ learning outcomes with highly usable and relevant content [35, 36].

The evaluation results proved that a low-cost device could provide a complex visual simulation for education even when it has lower processing and power requirements compared to other simulation platforms. A detailed comparison is presented in Table IV between different simulation training platforms for GA. A mobile device is relatively cheap and has similar levels of resolution to other platforms. On the other hand, the processing hardware on a mobile device is not as powerful when compared to other platforms. Compared to a standalone HMD, a mobile device’s processor is not dedicated to XR. Compared to desktop HMDs and high-fidelity flight simulators, a mobile device has a less powerful processor because of its limited power supply. A direct comparison of power consumption between a mobile device and a wall-mounted device (e.g., desktop HMD and high-fidelity flight simulation) is challenging because the source of power is different. A mobile device relies on its own battery to provide power and is measured by its battery capacity, whereas a wall-mounted device gets power from a wall outlet and is measured by the power consumed per hour. However, the battery capacity of a mobile device is significantly smaller than an hour of power consumed from a desktop HMD or high-fidelity simulation. As a result, a mobile device possesses a lower level of power consumption and processing capabilities than a desktop HMD and high-fidelity flight simulation. To highlight the graphic differences between the thunderstorm implementation and other existing mobile AR training or educational software, a comparison was made between FenAR [31] (left in Figure 19), which utilizes 3D models and animations in mobile AR for teaching physics, and the thunderstorm simulation developed in this research (right in Fig. 19). The graphic shows that the thunderstorm content exhibits better visual quality and more complex visual effects. Combined with the evaluation result, the thunderstorm simulation was able to provide a stable 60fps thunderstorm simulation with stable AR tracking on low-cost and less powerful devices.

According to reports, over 29% of GA accidents are weather-related. One of the potential causes of those accidents is a lack of fundamental weather theory knowledge by pilots [78]. Traditional GA training on weather is delivered mainly by images, text, or video, which does not provide an immersive and compelling training environment. Mobile AR is an easily accessible visualization tool that can work in conjunction with traditional material making it a better solution to assist GA weather theory training. However, mobile AR is limited in its rendering and processing power, unstable tracking, and small touch screens. The mobile AR application described in this paper provides a detailed solution that can provide an immersive environment with a lower cost than high-fidelity equipment and is more easily accessible than a VR or AR environment requiring an HMD. The model described in this paper represents thunderstorms using a volumetric and resource-efficient model. The model provides various approaches to delivering necessary weather information to overcome the limitations of mobile AR technology. A technical evaluation was performed on the thunderstorm mobile AR application. The evaluation result showed outstanding rendering performance and tracking quality across different devices. User evaluations conducted on the thunderstorm model also reiterate that this application can enhance students’ learning outcomes.

The work described in this paper was funded by the PEGASAS Center of the Federal Aviation Administration Air Transportation Center of Excellence for General Aviation Research, Cooperative Agreement 12-C-GA-ISU.

Statements and opinions expressed in this text do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.