Johns Hopkins APL and Microsoft’s AI Agent Orchestrates Robotic Teams

As part of the Johns Hopkins Applied Physics Laboratory’s (APL) ongoing collaboration with a variety of industries to produce cutting-edge research and development for the nation, the Laboratory and Microsoft recently demonstrated an agentic artificial intelligence planner that can coordinate heterogenous robotic teams. This new research showcases how large language models can enable robots of different types to one day operate as integrated, high-performing teams on shared, complex missions. The September demonstration, a by-product of the two organizations’ ongoing collaboration in robotics, autonomy, and artificial intelligence, took place at APL’s campus in Laurel, Maryland.

Tackling One of Autonomy’s Hardest Problems

Coordinating robotic teams in complex, dynamic environments remains one of the toughest challenges in autonomy. Many robotic systems are highly specialized, making it difficult to integrate ground, aerial, and maritime platforms into one cohesive team; this is a challenge that APL has worked on for the past several years as autonomous systems have proliferated. The new AI agent — known as MAESTRO, short for Microsoft and APL’s Ecosystem for Strategic Teaming in Robotics Operations — approaches the problem by interpreting and scaling high-level human instructions.

From Simple Commands to Complex Missions

During the demonstration, David Patrone, a leading robotics researcher at APL, issued natural language commands to MAESTRO, gradually increasing their complexity. Early instructions were simple, such as “have a ground platform take five steps forward,” while later tasks included more complex goals such as “search this area and find a cold bag.” Even a seemingly straightforward task such as finding a cold bag required MAESTRO to complete sophisticated action planning and decision-making including: identifying and tasking the robot that has a thermal imaging capability, determining its distance from the bag, moving the robot into position, and using that thermal sensor to read the temperature of the item. APL has performed similar, more complex tests outdoors, integrating both aerial and ground robotic and drone platforms to demonstrate broader variation in capability.

This approach could eventually enhance applications ranging from disaster response to logistics to defense — any scenario where operators need multiple robotic platforms to work together with minimal human intervention. Instead of tasking each robot individually, a single operator could issue a broad command such as, “search this area,” and let MAESTRO determine how each robot contributes.

Combining Strengths

The demonstration reflects the organizations’ complementary expertise. APL brought its decades of expertise in autonomy, robotics, and multi-domain operations. The Lab’s researchers designed the agentic AI architecture, built the interfaces that allow different robot types to interact, and provided the testing environment. Microsoft contributed its cloud infrastructure and AI expertise, providing access to large language models hosted on their cloud platform Azure, which enabled communication between MAESTRO and the robots.

“At APL, we specialize in creating the mission-driven software and architectures that help autonomous systems operate in challenging environments,” said Bart Paulhamus, who leads APL’s Intelligent Systems Center. “Working with a range of innovative industry partners encourages us to explore new ideas and accelerates our ability to deliver impactful systems for our sponsors, such as how those designs might scale from prototypes to systems capable of supporting hundreds or even thousands of robots.”

“MAESTRO“ shows how mission-focused research and Azure AI can turn a single intent into coordinated action across diverse robots, bringing safer, faster outcomes to complex real-world missions. By pairing APL’s autonomy expertise with Microsoft’s cloud-scale AI, operators can command outcomes, not devices, with responsible, orchestrated teamwork at scale,” said Chris Barry, corporate vice president, U.S. Public Sector Industries, Microsoft.

While the September demo was an initial proof of concept, it highlights the benefits of combining APL’s mission-driven research with the strengths of commercial AI leaders, such as Microsoft, and lays a foundation for continued collaboration with industry to develop systems that are not only intelligent individually but collaborative at scale.