Many extended reality systems use controllers, e.g. near-infrared motion trackers or magnetic coil-based hand-tracking devices for users to interact with virtual objects. These interfaces lack tangible sensation, especially during walking, running, crawling, and manipulating an object. Special devices such as the Tesla suit and omnidirectional treadmills can improve tangible interaction. However, they are not flexible for broader applications, builky, and expensive. In this study, we developed a configurable multi-modal sensor fusion interface for extended reality applications. The system includes wearable IMU motion sensors, gait classification, gesture tracking, and data streaming interfaces to AR/VR systems. This system has several advantages: First, it is reconfigurable for multiple dynamic tangible interactions such as walking, running, crawling, and operating with an actual physical object without any controllers. Second, it fuses multi-modal sensor data from the IMU and sensors on the AR/VR headset such as floor detection. And third, it is more affordable than many existing solutions. We have prototyped tangible extended reality in several applications, including medical helicopter preflight walking around checkups, firefighter search and rescue training, and tool tracking for airway intubation training with haptic interaction with a physical mannequin.
We have developed an assistive technology for people with vision disabilities of central field loss (CFL) and low contrast sensitivity (LCS). Our technology includes a pair of holographic AR glasses with enhanced image magnification and contrast, for example, highlighting objects, and detecting signs, and words. In contrast to prevailing AR technologies which project either mixed reality objects or virtual objects to the glasses, Our solution fuses real-time sensory information and enhances images from reality. The AR glasses technology has two advantages: it’s relatively ‘fail-safe.” If the battery dies or the processor crashes, the glasses can still function because it is transparent. The AR glasses can also be transformed into a VR or AR simulator when it overlays virtual objects such as pedestrians or vehicles onto the glasses for simulation. The real-time visual enhancement and alert information are overlaid on the transparent glasses. The visual enhancement modules include zooming, Fourier filters, contrast enhancement, and contour overlay. Our preliminary tests with low-vision patients show that the AR glass indeed improved patients' vision and mobility, for example, from 20/80 to 20/25 or 20/30.
We present a head-mounted holographic display system for thermographic image overlay, biometric sensing, and wireless telemetry. The system is lightweight and reconfigurable for multiple field applications, including object contour detection and enhancement, breathing rate detection, and telemetry over a mobile phone for peer-to-peer communication and incident commanding dashboard. Due to the constraints of the limited computing power of an embedded system, we developed a lightweight image processing algorithm for edge detection and breath rate detection, as well as an image compression codec. The system can be integrated into a helmet or personal protection equipment such as a face shield or goggles. It can be applied to firefighting, medical emergency response, and other first-response operations. Finally, we present a case study of "Cold Trailing" for forest fire prevention in the wild.
Incident Command Dashboard (ICD) plays an essential role in Emergency Support Functions (ESF). They are centralized with a massive amount of live data. In this project, we explore a decentralized mobile incident commanding dashboard (MIC-D) with an improved mobile augmented reality (AR) user interface (UI) that can access and display multimodal live IoT data streams in phones, tablets, and inexpensive HUDs on the first responder’s helmets. The new platform is designed to work in the field and to share live data streams among team members. It also enables users to view the 3D LiDAR scan data on the location, live thermal video data, and vital sign data on the 3D map. We have built a virtual medical helicopter communication center and tested the launchpad on fire and remote fire extinguishing scenarios. We have also tested the wildfire prevention scenario “Cold Trailing” in the outdoor environment.
Simulation is a recognized and much-appreciated tool in healthcare and education. Advances in simulation have led to the burgeoning of various technologies. In recent years, one such technological advancement has been Augmented Reality (AR). Augmented Reality simulations have been implemented in healthcare on various fronts with the help of a plethora of devices including cellphones, tablets, and wearable AR headsets. AR headsets offer the most immersive experience of the AR simulation as they are head-mounted and offer a stereoscopic view of the superimposed 3D models through the attached goggles overlaid on real-world surfaces. To this effect, it is important to understand the performance capabilities of the AR headsets based on workload. In this paper, our objective is to compare the performances of two prominent AR headsets of today, the Microsoft Hololens and the Magic Leap One. We use surgical AR software that allows the surgeons to show internal structures, such as the rib cage, to assist in the surgery as a reference application to obtain performance numbers for those AR devices. Based on our research, there are no performance measurements and recommendations available for these types of devices in general yet.
Color appearance of transparent objects is not adequately described by colorimetry or color appearance models. Despite the fact that the retinal projection of a transparent object is a combination of its color and the background, measurements of this physical combination fail to predict the saliency with which we perceive the object's color. When the perceive color forms in the mind, awareness of their physical relationship separates the physical combination into two unique perceptions. This is known as color scissioning. In this paper a psychophysical experiment utilizing a seethrough augmented reality display to compare virtual transparent color samples to real color samples is described and confirms the scissioning effect for lightness and chroma attributes. A previous model of color scissioning for AR viewing conditions is tested against this new data and does not satisfactorily predict the observers' perceptions. However, the model is still found to be a useful tool for analyzing the color scissioning and provides valuable insight on future research directions.
In the early phases of the pandemic lockdown, our team was eager to share our collection in new ways. Using an existing 3D asset and advancements in AR technology we were able to augment a 3D model of a collection object with the voice of a curator to add context and value. This experience leveraged the unique capabilities of the open Pixar USD format USDZ extension. This paper documents the workflow behind creating an AR experience as well as other applications of the USD/USDZ format for cultural heritage applications. This paper will also provide valuable information about developments, limitations and misconceptions between WebXR glTF and USDZ.
This paper analyses the use of Immersive Experiences (IX) within artistic research, as an interdisciplinary environment between artistic, practice based research, visual pedagogies, social and cognitive sciences. This paper examines IX in the context of social shared spaces. It presents the Immersive Lab University of Malta (ILUM) interdisciplinary research project. ILUM has a dedicated, specific room, located at the Department of Digital Arts, Faculty of Media & Knowledge Sciences, at University of Malta, appropriately set-up with life size surround projection and surround sound so as to provide a number of viewers (located within the set-up) with an IX virtual reality environment.
Augmented and Virtual Reality are proliferating throughout daily life. Everyday utilities, such as Google Maps, deploy Augmented Reality (AR) to improve the user experience and increase its effectiveness. Similarly, Augmented and Virtual Reality have been used in the educational domain, where these applications have been shown to improve the learning curve and retention. In this study, AR is proposed as an innovative method to improve the effectiveness and efficiency of the education of nursing students. We propose to introduce this innovative technology in the education of future nurses. Within this study the researchers use augmented reality (AR) to help nursing students learn about physical assessment techniques for the heart and the lungs. Researchers have demonstrated increased memory retention while using AR [14][15]. In this study, we provide a holographic overlay including the internal organs heart, ribcage, and lungs to increase the understanding of accurate placement of devices required for assessing cardiac and respiratory issues using anatomical landmarks. In addition, the visual aspects are supported with audio sound tracks to enhance learning. The ability to project images accurately placed onto real world physical objects using AR headsets could lead to increased retention and deeper understanding leading to precision performance related to techniques in physical assessment of the patient.
In this paper, we present a novel Lidar imaging system for heads-up display. The imaging system consists of the onedimensional laser distance sensor and IMU sensors, including an accelerometer and gyroscope. By fusing the sensory data when the user moves their head, it creates a three-dimensional point cloud for mapping the space around. Compared to prevailing 2D and 3D Lidar imaging systems, the proposed system has no moving parts; it’s simple, light-weight, and affordable. Our tests show that the horizontal and vertical profile accuracy of the points versus the floor plan is 3 cm on average. For the bump detection the minimal detectable step height is 2.5 cm. The system can be applied to first responses such as firefighting, and to detect bumps on pavement for lowvision pedestrians.