From Blackboard Scribbles to Neural Network Breakthrough

Vickram Rajendran and Will LeVine were two self-described “kids” just having fun with computers and math — early-career staff members at the Johns Hopkins Applied Physics Laboratory (APL) trying to solve one of the biggest problems in machine learning, related to a concept dubbed “competence.”

On occasion, the pair would step back, look at the blackboard and think how crazy it was that the scribbled math in front of them was helping them make critical contributions to many critical challenges.

Their work led to the definition of a new competence metric for neural networks, one that the Asymmetric Operations Sector research scientists dubbed “ALICE.” The acronym is explained in their paper, “Accurate Layerwise Interpretable Competence Estimation (ALICE),” that was accepted for a poster at the annual conference on Neural Information Processing Systems (NeurIPS) this December in Vancouver, Canada.

NeurIPS, one of the premier artificial intelligence conferences worldwide, is highly selective; only 1,428 papers were picked from 6,743 submissions for this year’s poster presentations.

In their paper, Rajendran and LeVine define a new competence metric for neural networks and a framework for computing competence, and describe the first competence estimator. This innovation derives from the operational need to understand how neural networks behave “in the wild” and is the foundation for a series of newly funded tasks to better predict neural network performance, prioritize which new data should be labeled to enhance model performance, and identify which labeled data are poorly labeled or are particularly troublesome to the trained model.

A Good Idea

Rajendran joined the Lab in 2018 as a new college graduate earning a bachelor’s degree in computer science and mathematics from Swarthmore College. He came up with the idea for ALICE, earned a $10,000 seedling grant under a branch-level Independent Research and Development (IRAD) program to begin working on it, and continued his progress with midyear IRAD funds provided by the Lab’s Special Operations Mission Area.

LeVine worked on the ALICE project as an intern and had also just started full-time at APL while earning his bachelor’s degree in mathematics at Indiana University. They both worked on developing the code, running experiments, reviewing literature and constantly iterating to improve ALICE.

They decided to write a paper because they knew their work was relevant to APL and the larger artificial intelligence (AI) and machine learning (ML) communities. Colleagues’ feedback on their early results helped them to realize their ideas had potential applications they didn’t expect, so they thought it was important enough to try submitting to NeurIPS — even though they had never written a paper before.

“Personally, it feels pretty surreal,” said Rajendran, the principal investigator for ALICE. “We’ve been reading paper after paper by some of the biggest names in AI/ML at this conference, and it’s crazy to think that we’re now publishing at the same venue alongside them. Watching this project grow from a simple idea that I would excitedly talk about to my group, to a published paper at one of the most prestigious AI/ML conferences in the world, along with multiple interested sponsors, feels unreal.”

Community Validation

Rajendran said that while the feedback from APL colleagues was helpful, getting confirmation from external experts that their work deserved to be in the conference was particularly validating.

“For a while, ALICE was just our thought baby,” said LeVine. “To have the overall AI/ML community believe that we have made significant progress toward solving these problems makes me feel very lucky, both to be at APL where we are able to work on such fundamental issues and to be a member of a community that values good ideas regardless of the researchers’ backgrounds.”

“We couldn’t be more proud of Vickram’s and Will’s accomplishments with ALICE,” said APL’s Dean Fisher, their group supervisor. “Their innovation, work ethic and contributions truly demonstrate what makes APL a special place to work — making an impact on our sponsor’s hardest technical challenges. The really fantastic part is this is just the beginning, as they have a string of additional applications and ideas for extending their research to other areas that will be of interest to the AI/ML community and our sponsors.”

The duo’s next steps include extending ALICE’s capabilities, improving their methods, integrating the software into sponsor projects and continuing their research into novel applications of competence estimation.

“It’s great to confirm that we’re not just watching and implementing papers from the sidelines, but we’re actually on the cutting edge,” said Rajendran. “Publishing these kinds of papers in top-tier conferences really shows that APL is a nationally recognized leader in artificial intelligence, and being a part of this so early in our careers makes us think we have bright futures ahead. We’re not done yet, not by a long shot.”