Evaluation Framework for Assessing Validation Methods on Modeling and Simulation Models
Modeling and simulation (M&S) is a critical step throughout the systems engineering process for developing and fielding a combat system. Verification and, more specifically, validation are essential to determining whether a simulation is credible and reliable. Although policy and guidance increasingly emphasizes the importance of rigorous validation founded in the application of strong statistical analysis, implementation continues to be challenging. As a result, test organizations and statisticians have been interested in developing a robust approach for measuring the performance of the validation methods used to assess model accuracy. The Johns Hopkins University Applied Physics Laboratory (APL) developed a flexible and extensible framework to evaluate the performance of the validation methods. The framework provides the modularity to evaluate multiple validation methods and is sufficiently generic to support assessment of multiple simulation models. This article details the framework design and the analysis of multiple statistical validation methods, including an exemplar assessment of the methods applied for a recently accredited missile system simulation.
Toward Robust Characterization of Lung Diseases: A Sensitivity Analysis of Lung Computed Tomography Biomarkers to Registration Error
Computed tomography (CT) scans, because of their ability to differentiate tissue densities, have been widely used to evaluate lung health. Recent studies such as COPDGene have collected inhalation and exhalation CT scans from thousands of subjects, promising insight into the mechanical properties of lung tissue. These paired scans must often be spatially aligned (i.e., registered) to extract biomarkers describing the movement of lung tissue that may correlate with disease. Unfortunately, the relationship between registration and biomarker error is poorly characterized, a challenge that must be addressed before registration-based biomarkers can be used in clinical practice. In our analysis, we consider three registration-based biomarkers (Jacobian determinant, anisotropic deformation index, and slab-rod index) and demonstrate their sensitivity to modeled registration error. We provide a range of errors for a given biomarker, highlighting how both the magnitude of registration error and correlations between vectors in the registration error field can influence biomarker error. We then describe a method to measure the error field for a particular registration algorithm and compare it with modeled registration error. These estimates enable selection of an appropriate registration error model, which improves understanding of biomarker uncertainty. Quantifying the relationship between registration and biomarker error is crucial because it may inform the selection of a registration algorithm to reduce error in new research studies, and in turn, result in robust imaging biomarkers for disease characterization.
Identifying Patterns and Relationships within Noisy Acoustic Data Sets
Acoustic emissions analysis can provide key information for monitoring the structural integrity of a system, such as the behavior of bone under various loading conditions and other complex biomechanical applications. However, when analyzing acoustic emissions data from complex systems, including systems that experience high-rate (103 s–1) loading, complex bending modes, unique shape effects, and multiple failure mechanisms, it is difficult to extract meaningful information and relationships because of an abundance of confounding factors. This article presents a methodology developed at the Johns Hopkins Applied Physics Laboratory (APL) for understanding fracture and characterizing acoustic signatures with distinct failure modes, leveraging techniques such as independent component analysis, self-organizing maps, and K-means clustering algorithms.
Data Set Representation and Tagging for Automating Data Cataloging
In the last two decades, considerable increases in computing power and available data have led to an analytics and machine learning (ML) revolution. To make knowledge management less cumbersome for human operators, a team of researchers at the Johns Hopkins University Applied Physics Laboratory (APL) proposes an ML–based method to help automate knowledge management. This method discovers new data, represents it with descriptive metadata, automatically categorizes the metadata, auto-populates a data catalog with data sets, and evaluates the new data sets for data fusion options. We focus on a framework that can potentially leverage human– machine teaming to significantly reduce the human resource burden to develop and maintain an accurate accounting of existing data and capabilities within an organization. We explored numerous ML options to test our core hypothesis—that ML techniques can be employed to reliably determine the fundamental topic that an unknown data set represents, leading to increasingly granular data set recognition as more characterization and context information can be mined in the metadata extraction phase. Ultimately, we demonstrated that multiple classifier techniques exist that can predict data set topics with close to 90% accuracy, and some with 60%– 80% accuracy, across multiple topics.
AGAVE: Automated Genomics Application for Variant Exploration
The Johns Hopkins University Applied Physics Laboratory (APL) is actively developing new capabilities for genomic surveillance of viruses. APL genomicists analyze, process, and visualize viral genomic data for several sponsor organizations that require those data to inform clinical, research, and public policy decisions. Many of the final products from these processes are delivered to sponsors as static reports or slide presentations, but it can be arduous to review or extract pertinent information from these documents. APL genomicists wanted to improve their sponsors’ ability to analyze their data and rapidly identify genomic samples or sequences they find important for decision-making. With this goal in mind, a group of APL software engineers developed the Automated Genomics Application for Variant Exploration (AGAVE). AGAVE is an interactive, intuitive web-based tool where researchers can explore and analyze genomic data, draw new connections between data points, and understand the significance behind genomic variants quickly. Researchers can view their sequence data, choose a reference genome with which to compare the data, visualize the 3-D structure of proteins that would be created from particular segments of DNA, and export those visualizations as easily shared image files. AGAVE is still under development and currently supports only influenza genomes, but as it matures and its user base grows, it will expand beyond influenza to include other viruses such as SARS-CoV-2 and even bacterial genomes.
Link-Layer Identification of Device Signatures: Wi-Fi Sensing for Crowd Analytics
The Johns Hopkins University Applied Physics Laboratory (APL) Link-Layer Identification of Device Signatures (LLIDS) research effort uses machine learning techniques to identify unique wireless device signatures from patterns in link-layer data. Identifying signatures can increase situational awareness, assist in estimating crowd sizes, provide pattern of life, and protect facilities and infrastructure through activity surveillance. Link-layer Wi-Fi data are unique because they can be collected without access to a network and with devices that have low size, weight, and power (SWaP) requirements. The LLIDS multilayer system design breaks down link-layer data into unique device signatures using a combination of pattern recognition and state-of-the-art algorithms.
APL’s Discovery Program: Guest Editor’s Introduction
This issue of the Johns Hopkins APL Technical Digest focuses on the Johns Hopkins University Applied Physics Laboratory (APL) Discovery Program, a 2-year rotational opportunity for new college graduates that consists of four rotation assignments spanning multiple technical areas across the Laboratory. In addition to featuring reflections from program alumni and host supervisors and an overview of the program’s training component, the issue highlights the technical contributions of some of the staff members who have been part of the program or are currently part of it. The articles in this issue showcase the core competencies of APL but also truly highlight the core tenets of the Discovery Program: broad exposure, career foundations, and professional connections. The articles amplify how the Discovery Program accomplishes its vision of a persistent, collaborative, and innovative network of impactful staff who will lead us into the future. Some of the staff members in the Discovery Program will enable APL’s future defining innovations.
Inside Back Cover: Precision Strike Mission Area Strategic Vectors
This illustration is a notional mission-level view of the four APL Precision Strike Mission Area (PSMA) strategic vectors working together. Continuous Universal Targeting is illustrated as sensors observing Red targets and relaying targeting information to Blue tactical platforms. Control Red Perception is illustrated as Blue airborne and surface ship jamming platforms achieving nonkinetic effects against Red airborne and surface targets. Air Dominance is illustrated as crewed and uncrewed airborne platforms working together to maintain air supremacy. Resilient Time- Critical Strike is illustrated as Blue hypersonic weapons attacking Red land and sea targets.
In Memoriam: Scott T. Radcliffe (1967–2022)
Scott T. Radcliffe, a chief scientist at APL, died February 3, 2022, in Howard County General Hospital at the age of 55 due to complications from bone marrow cancer and lung disease. He was known by his coworkers as “a national treasure, and a man of honesty and integrity who could imagine great solutions to things that nobody else could see.” He held three patents and received numerous awards from government sponsors and APL, including the Lab’s esteemed Alvin R. Eaton Award for sustained performance and exceptional scientific or engineering innovations. Scott had deep expertise in computer simulation, digital signal processing, and radio communications, and he delivered several game-changing communication, radio frequency (RF) sensing, and RF geolocation innovations for the U S government. He formed deep relationships with sponsors and colleagues, some of whom contributed personal remembrances in this tribute to him.
The Boundary Layer Transition (BOLT) Flight Experiment
The Boundary Layer Transition (BOLT) flight experiment, a unique collaboration spanning academia, government, and industry, sought to obtain flight data on a critical phenomenon affecting hypersonic vehicle design. The project aimed to further understanding of the physics of boundary-layer laminar-turbulent transition on a complex geometry, a process that can significantly increase heating and can affect hypersonic vehicle drag, controllability, and engine performance. The Johns Hopkins University Applied Physics Laboratory (APL), the project’s principal investigator, led a large team of external collaborators to design a sounding rocket flight experiment over an 18-month period, while conducting an extensive campaign of wind-tunnel experiments and computational simulations to predict the flow physics on the BOLT geometry. The final flight experiment was built and instrumented at APL using Laboratory expertise in designing and prototyping hardware for extreme environments. The BOLT experiment was delivered to the US Air Force for the flight experiment, designed to gather critical validation data on BOLT’s boundary-layer transition from over 340 sensors in the hypersonic flight regime. Although the flight test ultimately did not achieve the desired experimental conditions, the BOLT research resulted in new experimental databases, new computational tool development for complicated hypersonic flows, and significant new workforce development through the inclusion of students in the program. APL’s efforts to develop BOLT led to a follow-on flight experiment focused on turbulence (BOLT2: The Holden Mission), which flew successfully in March 2022.