Army researchers develop innovative framework for training AI

Army researchers have developed a pioneering framework that provides a baseline for the development of collaborative multi-agent systems.

The framework is detailed in the survey paper “Survey of recent multi-agent reinforcement learning algorithms utilizing centralized training,” which is featured in the SPIE Digital Library. Researchers said the work will support research in reinforcement learning approaches for developing collaborative multi-agent systems such as teams of robots that could work side-by-side with future soldiers.

“We propose that the underlying information sharing mechanism plays a critical role in centralized learning for multi-agent systems, but there is limited study of this phenomena within the research community,” said Army researcher and computer scientist Dr. Piyush K. Sharma of the U.S. Army Combat Capabilities Development Command, known as DEVCOM, Army Research Laboratory. “We conducted this survey of the state-of-the-art in reinforcement learning algorithms and their information sharing paradigms as a basis for asking fundamental questions on centralized learning for multi-agent systems that would improve their ability to work together.”

Sharma’s collaborators on this project include DEVCOM ARL researchers Drs. Erin Zaroukian, Rolando Fernandez, Michael Dorothy, Derrik Asher and Anjon Basak, a postdoctoral fellow under the Oak Ridge Associated Universities fellowship program.

This survey of state-of-the-art in reinforcement learning establishes a baseline for researchers seeking to develop autonomous multi-agent systems through enhanced information sharing mechanisms, for example rewarding functions or observation and state space sharing.

Training multiple agents simultaneously is more difficult due to the dynamic nature of complex environments that can suffer from the curse of dimensionality; the more agents there are, the more complicated the coordination becomes, Sharma said. This paper develops a framework to characterize the key information sharing parameters that are often confusing and not easy to understand.

The researchers predict that centralization in training may be the solution towards more rapidly developing autonomous systems that can flexibly work alongside soldiers in the future.

“Consistent, centralized training can result in multi-agent systems that work more reliably together, increasing the level of trust from the soldier of the artificial intelligence,” Sharma said. “Specifically, we focused on identifying and characterizing the underlying mathematical framework of the most recent centralized learning algorithms.”

Such a mathematical model can provide an avenue to explore alternate centralized learning techniques to gauge their effect on the learning rate and emergent collaborative behaviors, he said.

The survey exceeds prior research literature in two ways:

• Creates a consolidated view of the latest state-of-the-art in reinforcement learning algorithms
• Outlines a novel approach to describing information shared during centralized learning

The researchers focused on the algorithms published within five to six years. As these algorithms are very recent, they have not yet been explored extensively by the research community. At the time of publication, they did not find comprehensive prior work.

The researchers attempted to define and categorize the mechanisms for sharing, orienting on what is actually being shared as opposed to how it is shared. They are optimistic that they have identified gaps in the recent reinforcement learning techniques worthy of further study that could potentially enhance the agent training process.

The researchers said they are optimistic the survey will generate discussion and further exploration of the problem space of machine learning to train autonomous multi-agent systems.

“As the demand for multi-agent systems working cooperatively has become more prevalent in the commercial industry, for example, Amazon warehouse robots, Intel’s drone show at winter Olympics 2018. There is also an emerging need for these multi-agent system technologies to assist the Army in collaborative tactical operations,” Sharma said. “The research resulting from this survey document can make the goal of reliable, collaborative AI achievable.”

Moving forward, the team feels better equipped to investigate particular aspects of multi-agent reinforcement learning based approaches that train agents in centralized fashion.

Centralized techniques come with certain limitations, so they will also conduct empirical analysis of existing decentralized learning techniques, Sharma said. They plan to move towards modeling and simulation of multi-agent reinforcement learning training to validate and extend theories of agent learning, behavior and coordination.