ISC//Research//Projects

The Active Sensing Testbed

Intelligent systems increasingly leverage deep learning to perceive and understand their operating environment. Yet, state-of-the-art computer vision algorithms can be brittle when confronted with the complexities of real-world scenes. At the Johns Hopkins Applied Physics Laboratory’s Intelligent Systems Center, researchers have created the Active Sensing Testbed – a development environment for integrated pattern recognition and reasoning – to help enable progress in machine perception beyond singular narrowly-trained algorithms and to accelerate the translation of research advancements to real-world application.

The Challenge

Machine Perception in Real-World Applications

Figure 1: The Active Sensing Testbed allows a user to build up a workflow by chaining together different data sources, image transformations, and video analytics. For example, this shows a public webcam feed from New Orleans that applies two analytics, one for human pose detection and one for object segmentation and classification.
Many real-world applications rely on the ability of human operators to perceive and understand the environment. For example, first responders arriving at a disaster site must first survey the scene before taking action, and battlefield medics must diagnose the state of a wounded soldier before deciding upon the appropriate medical intervention. As effective as human perception can be for applications like these, it is not perfect. Our ability to ingest vast, complex data is limited. Our attention spans can be short and our perception of the world is inherently biased in different ways.

Machine perception offers the ability to complement human capabilities. Recent advancements in computer vision, while impressive, largely focus on narrowly-trained pattern recognition algorithms that can be brittle in the face of real-world complexity and adversarial manipulation.

An ideal perception system could observe a sequence like the one illustrated in Figure 2, report that a Frisbee has been exchanged for a football, and represent the space behind the barrier as a likely location for the missing Frisbee. Such as a system could operate as a teammate to a human security guard, for example, charged with maintaining public safety in a large area such as a stadium or an airport.

Figure 2: A Frisbee is exchanged for a football behind an occluding barrier as part of a machine perception data collection effort in APL’s Immersion Central laboratory for mixed reality experimentation.

The Objective

Integrated Pattern Recognition and World-View Reasoning

Our research team has created the Active Sensing Testbed (AST) to help accelerate research and development in machine perception. The AST provides a modular software environment where researchers can utilize state-of-the-art computer vision algorithms as lower-level building blocks. That allows them to create more complex perception systems that combine information across multiple views, sensor modalities, and complementary analytics.

These capabilities lower the barrier to entry for researchers to explore new concepts in active sensing where an intelligent system can enhance its understanding of a scene by reasoning, forming hypotheses and acting to gather additional information.

Our Approach

Creating a Research Testbed for Real-Time Perception and Reasoning

The architecture of the AST centers around a server that receives data feeds from multiple sensors, performs selected transformations on input data and computes selected analytics, and then sends the resulting analytics and metadata to subscribers for visualization. The server is built around a containerized architecture using Docker so the system can easily scale across multiple machines and support various libraries without creating conflicts.

A visualization of this architecture is shown in Figure 3.

We have created a research testbed around the AST at the Johns Hopkins Applied Physics Laboratory’s Intelligent Systems Center. Our testbed includes four ceiling-mounted pan-tilt-zoom cameras to facilitate data collection, along with algorithm design and evaluation. Our AST implementation also includes an operator interface to enable human-machine interaction.

Through the operator interface and the associated application programming interface, a system operator or other remote user can view data feeds, overlay computed analytics and metadata, and issue commands to the system. Our aim is to leverage the AST software framework to support similar setups across a variety of different locations and applications.

Figure 3: Active Sensing Testbed server architecture.
Our baseline AST implementation includes a suite of real-time analytics and transformations. Baseline analytics include object detection and classification, semantic instance segmentation, and human pose estimation. Transformations include traditional operations such as flipping, rotating, color/brightness adjustments, and adding noise, as well as more complex operations such as face-swapping, day/night transformations, and dynamic object removal. While many of these algorithms are built upon open-source or published techniques, the novelty of the AST is that it brings these (and potentially many other) state-of-the-art techniques together as an interactive suite of machine perception capabilities. Figure 1 provides a video demonstrates the baseline AST analytics suite.

Outcomes

Project Transitions and Future Research

Now that we have established a baseline, various APL projects are beginning make use of the AST as a tool for research and demonstration. For example, we have applied the AST software to implement a data collection suite for health analytics research, which is intended to support validations of analytics from standoff biometric sensors against commercial medical-grade sensors. An example of this suite is shown in Figure 4. Additional health analytics that we have explored through this effort include face mask detection, social distancing analysis, and full-body pose estimation and emotion recognition for use during remote therapy sessions.

Figure 4. An application of the AST software toward validating standoff biometric analytics against commercial medical-grade sensors. For example, comparing a camera-based heartrate and pulse oximetry calculation against outputs from wearable sensors.
Initial explorations of the AST software’s commercial applications are also underway, including a trial deployment of the server and operator station as a framework for testing security analytics in a large commercial facility, and a technology transfer and research partnership with a West Coast startup.

The AST provides an accessible environment and toolkit for conducting research into active sensing and intelligent systems, providing tools that will allow both novice and experienced AI researchers to quickly develop novel algorithms and system architectures that enable machine perception and reasoning tasks across a broad range of applications.

Key Contributors

Neil M. Fendley

Utilized Labs

UxV Gym