XR for Advanced Prototyping of Spacecraft Mechanical Systems
This article discusses how teams in the Space Exploration Sector at the Johns Hopkins University Applied Physics Laboratory (APL) are using XR as an advanced prototyping capability. Prototypes enable engineering and design teams to see or experience an object before committing resources to full-scale production. XR provides a means of digitally prototyping high-fidelity physical information in a nonmaterial form. The collaborative immersive qualities of XR appeal to human visual processing senses, enabling teams to quickly engage complex system information, make decisions, and confidently move from ideas to actions. The design intelligence gained through using XR enables teams to make faster decisions with greater confidence and less risk. APL teams introduced production-grade XR tools into mechanical design workflows in 2017, making critical contributions to Parker Solar Probe, Europa Clipper, and other programs. But XR is only a small part of a bigger picture challenging companies to rethink conventional business operations for the modern competitive global industrial ecosystem. Incorporating XR as part of a broader digital transformation (DX) strategy carves a path to greater advantages and opportunities that cannot be realized by XR alone.
HoloLens Applications for the Demonstration of an Advanced Anthropomorphic Test Device for Under-Body Blast and the Dissemination of Finite Element Analysis Results
The Warrior Injury Assessment Manikin (WIAMan) crash test dummy was developed in response to an Army need for better injury prediction capabilities in under-body blast testing. Concurrently, a finite element model (FEM, a physics-based computational model) was developed at the Johns Hopkins University Applied Physics Laboratory (APL) to accelerate the design process and provide simulated injury prediction capabilities. However, two main issues arose when presenting the work to a broad audience. First, it was difficult to convey the results and impact of the FEM. Augmented reality suited this problem well because of the technical nature of the work and the lack of an off-the-shelf software solution the layperson could use to manipulate the model. Second, it was necessary to circumvent the need for transporting the human-sized physical anthropomorphic test device (ATD). A new method for exploring the dummy was required, but one that still allowed an audience to experience the technology firsthand. Hence, two complementary Microsoft HoloLens applications were developed at APL to allow a user to explore the inner workings of the ATD and see it in a simulated blast environment. These apps connect the user with the project in their own surroundings while providing information about various ATD parts at the user’s pace. The applications have been demonstrated to diverse audiences at various venues both locally and across the country and were successful in conveying the impact of the project.
Design and Preliminary Evaluation of an Augmented Reality Interface Control System for a Robotic Arm
Despite advances in the capabilities of robotic limbs, their clinical use by patients with motor disabilities is limited because of inadequate levels of user control. Our Johns Hopkins University Applied Physics Laboratory (APL) team and collaborators designed an augmented reality (AR) control interface that accepts multiple levels of user inputs to a robotic limb using noninvasive eye tracking technology to enhance user control. Our system enables either direct control over 3-D endpoint, gripper orientation, and aperture or supervisory control over several common tasks leveraging computer vision and intelligent route-planning algorithms. This system enables automation of several high-frequency movements (e.g., grabbing an object) that are typically time consuming and require high degrees of precision. Supervisory control can increase movement accuracy and robustness while decreasing the demands on user inputs. We conducted a pilot study in which three subjects with Duchenne muscular dystrophy completed a pick-and-place motor task with the AR interface using both traditional direct and newer supervisory control strategies. The pilot study demonstrated the effectiveness of AR interfaces and the utility of supervisory control for reducing completion time and cognitive burden for certain necessary, repeatable prosthetic control tasks. Future goals include generalizing the supervisory control modes to a wider variety of objects and activities of daily living and integrating the capability into wearable headsets with mixed reality capabilities.
Novel Perception
The Novel Perception system enhances users’ awareness of the world by providing them naturalistic access to sources of information beyond what is perceivable by the basic human senses. The system, conceived of and designed by a team at the Johns Hopkins University Applied Physics Laboratory (APL), collects signals from a variety of sensors, synchronizes them in real time, registers them in real space, and then overlays them onto the real world as imagery and holograms seen through a mixed reality (MR) headset. The concept includes “virtual lenses” of hyperspectral, radio frequency, social, physiological, thermal, and radiological/nuclear overlays, so that a user can select multiple virtual lenses to create on-the-fly custom compound lenses across these modalities. Because the volume and velocity of the data streaming from these sensor modalities may be overwhelming, the system is envisioned to leverage artificial intelligence and brain–computer interfaces to sculpt the deluge per the operational tasks at hand. This article describes the approach to developing the system as well as possible applications.
Mixed Reality for Post-Disaster Situational Awareness
When disaster strikes, what once was, no longer is; and what is, is unrecognizable. The ability to understand the way things were is critical for those working in the response, rescue, and recovery phases of disaster management, from first responders to insurance claims agents. A team at the Johns Hopkins University Applied Physics Laboratory (APL) is developing a system that uses 3-D modeling data and precise positioning data from GPS to display an image of structures “in place” using a mixed reality head-mounted display for first responders in a disaster scenario. The proof of concept revealed significant challenges that need to be solved, and we have already proposed solutions and are working to test them. This technology is designed to assist first responders but will also have applications in other areas that require real-time data presentation, such as battlefield situational awareness.
Simulated X-Ray Vision Using Mixed Reality
A Johns Hopkins University Applied Physics Laboratory (APL) team created an application for the Microsoft HoloLens, a mixed reality (MR) head-mounted display (HMD), that serves as a proof of concept for a capability that would allow warfighters to observe their surroundings beyond their immediate line of sight. The approach uses a high-fidelity 3-D reconstruction of the beyond-line-of-sight (BLOS) environment in the form of a point cloud based on data from remote sensors and overlays it with the physical surfaces in the user’s surroundings as seen through the HMD. This approach allows the user to observe areas beyond their line of sight without ever physically occupying or directly observing the space.
Mixed Reality Social Prosthetic System
A Johns Hopkins University Applied Physics Laboratory (APL) team conceived of and developed a first-of-its-kind mixed reality “social prosthetic” system aimed at improving emotion recognition training and performance by displaying information about nonverbal signals in a way that is easily interpretable by a user. Called IN:URfACE (for Investigating Non-verbals: Using xReality for the Augmented Consideration of Emotion), the proof-of-concept prototype system uses infrared sensors to measure facial movements, pupil size, blink rate, and gaze direction. These signals are synchronized in real time, registered in real space, and then overlaid on the face of an interaction partner, such as an interviewee, through a mixed reality headset. The result is dramatic accentuation of subtle changes in the face, including changes that people are not usually aware of, like pupil dilation or nostril flare. The ability to discern these changes has applications in fields such as law enforcement, intelligence collection, and health care. This article describes how the system works, the technical challenges and solutions in designing it, and possible areas of application.
Minerva: Applied Software Engineering for XR
This article describes Minerva (Multiuser Intuitive Exploitation and Visualization), a proof of concept demonstrating a dynamic multiuser alternative to traditional file-oriented intelligence processing and dissemination pipelines. The main goal of the effort was to enable multiple clients to look at the same data in the ways that worked best for them depending on the type of device they were using. Although Minerva did not evolve into a fully developed software product, the team of developers at the Johns Hopkins University Applied Physics Laboratory (APL) created a functional prototype and demonstrated effective use of standards and microservices for hosting and streaming static content. APL teams are applying the lessons learned from this effort to other projects seeking to take advantage of XR to improve traditional data production and representation pipelines.
Project Minard: A Platform for War-Rooming and Geospatial Analysis in Virtual Space
The geographic movement of individuals and assets is a complicated maze of relational data, further complicated by the individuals’ relationships or allegiances to organizations and regions. Understanding this depth and complexity of information is difficult even on purpose-built systems using conventional compute architectures. A Johns Hopkins University Applied Physics Laboratory (APL) project, called Minard, upgrades the war room to a virtual reality (VR) space. This system provides analysts with a collaborative and secure virtual environment in which they can interact with and study complex and noisy data such as alliances, the transit of individuals or groups through 3-D space, and the evolution of relationships through time. APL engineers designed intelligent visualization systems to bring the best of human intuition to state-of-the-art VR, with human–machine teams interacting both through the VR headset and behind a conventional computer terminal.
Developing Project Proto-HEAD (Prototype Holographic Environment for Analysis of Data)
With Project Proto-HEAD, or Prototype Holographic Environment for Analysis of Data, a Johns Hopkins University Applied Physics Laboratory (APL) team sought to learn more about augmented reality (AR) and whether it significantly improves simulation and data analysis over using a PC workstation. Proto-HEAD had a simple concept: given a 3-D data set, visualize that data set as a hologram within the physical world. The development process afforded the team significant insight into the challenges and potential approaches for visualizing and interacting with very large data sets. These insights are useful for new applications that visualize and interact with large data sets and complex models.