Open up the entire process of 'self-evolution' of intelligent agents! Fudan launches AgentGym, a general-purpose intelligent body platform

AIxiv專欄是本站發布學術、技術內容的欄位。過去數年，本站AIxiv專欄接收通報了2,000多篇內容，涵蓋全球各大專院校與企業的頂尖實驗室，有效促進了學術交流與傳播。如果您有優秀的工作想要分享，歡迎投稿或聯絡報道。投稿信箱：liyazhou@jiqizhixin.com；zhaoyunfeng@jiqizhixin.com

AI通用智能體的自我進化能力，並非遙不可及。

LLM-based Agent，已經不再需要人類監督者的幫助，開始實現「自我進化」！

這個智能體在學習了專家軌跡以後，獲得了基礎的通用能力，能夠在更廣泛、更真實的未知環境與任務上進行探索和學習，在外部的回饋下不斷提升自己。

最近，復旦大學語言與視覺團隊推出的AgentGym 平台，打通了大語言模型智能體「資料取樣、訓練微調、自我進化、能力評測”全流程。基於該平台提出的 AgentEvol 演算法，首次探討了通用智能體的自我進化能力，並在多項智能體任務上表現非凡，與 GPT-4、Claude 等 SOTA 模型比肩。

• 論文連結：https://arxiv.org/abs/2406.04151
• #AgentGym程式碼倉庫：https://github.com/WooooDyy/AgentGym

研究背景

#開發一個能夠解決和適應複雜工作的多任務通用智能體，一直是人工智慧社群長久以來的重要目標。

類似於人類的學習過程，通用智能體首先透過模仿，開始學習最基礎的知識和技能。

隨著基礎能力的掌握，我們不僅期望智能體可以透過與不同環境的互動，持續學習並適應許多先前未見的任務，還能從自身經驗以及外部回饋中汲取豐富的智慧，發展出一定程度的泛化能力（圖1）。

#

^{圖1：基礎通用智能體實現「自我進化」的示意圖。這個智能體首先在人類監督下進行行為克隆，然後在不同的外在環境和任務中進行探索和學習，以實現自我進化。}

大語言模型憑藉其卓越的通用能力，被視為建構此類智能體的重要基礎之一。目前的研究領域正沿著兩個主要方向進行探索，以推動智能體技術的進一步發展。

• 依賴人類監督的行為複製（Behavior Cloning）方法，需要智能體逐步模仿專家提供的軌跡資料。這種方法雖然有效，但由於標註資源的限制，難以擴展。 對環境的探索也較為有限，容易遇到效能或泛化性的瓶頸。
• 允許智能體根據環境回饋，不斷提高能力的自我改進（Self Improving）方法，減少了對人類監督的依賴，同時豐富對環境的探索深度。然而，它們通常在特定任務的孤立環境中進行訓練，得到一群無法有效泛化的專家智能體。

面對上述挑戰，作者首次探討了一個具備基礎能力的通用智能體——在多種環境和任務中－自我進化的潛力。

為了實現這一研究目標，作者確定了推動智能體自我進化的「三大關鍵支柱」，這些支柱是研究的核心要素。

• 多樣化的環境和任務，允許智能體動態且全面地進行互動、訓練，而不是被局限於某個孤立的環境。
• 一個適當大小的軌跡資料集，幫助智能體配備基本的指令遵循能力和基礎任務知識。
• 一種有效且可擴展的演化演算法，激發智能體在不同難度環境中的泛化能力。

^{Figure 2: Schematic diagram of the AgentGym platform. The platform covers a total of 14 environments across different categories, each deployed as an HTTP service. The client provides an encapsulated unified interface for the agent to facilitate interaction with the environment. Through the AgentEvol method, the authors explore the self-evolution of agents in different environments and tasks. In addition, the platform provides the test set AgentEval to conduct a comprehensive ability evaluation of the agent.}

Around these three pillars, the author’s research work is reflected in the following aspects:

• "AgentGym", an interactive platform containing 14 specific environments and 89 specific task types (Figure 2), provides support for large language model agent training. The platform is based on HTTP services and provides a unified API interface for different environments, supporting trajectory sampling, multi-round interaction, online evaluation and real-time feedback.
• "AgentEval", a challenging agent testing benchmark. "AgentTraj" and "AgentTraj-L" are expert trajectory data sets constructed through instruction enhancement and crowdsourcing/SOTA model annotation. After format unification and data filtering, it helps the agent learn basic complex task-solving capabilities.
• "AgentEvol" is a new algorithm that stimulates the self-evolution of agents across environments. The motivation of this algorithm is to expect the agent to conduct autonomous exploration when faced with previously unseen tasks and instructions, and to learn and optimize from new experiences.

AgentGym platform is a brand new framework that supports large language model agent trajectory sampling, self-evolution, and ability evaluation, characterized by providing diverse, real-time, concurrent and unified format feedback. It aims to help the artificial intelligence community more conveniently explore LLM-based agents with general capabilities.

AgentGym——An intelligent agent platform integrating interactive training and evaluation

## AgentGym integrates multiple environments, rich trajectory data, and comprehensive benchmarking. It simplifies the environment configuration process through a

unified environment operation interface. Specifically, AgentGym has the following features:

Diverse environment:

AgentGym contains 14 environments and 89 tasks, covering categories such as web navigation, word games, embodied control, tool usage, and coding. Whether you are dedicated to building Task-specific Agents or universal Generally-capable Agents, the AgentGym framework can provide corresponding support.

Among them, each environment

is deployed independently, which avoids dependency conflicts between different environments and ensures the scalability of the platform. For example, the WebShop environment, an interactive platform for online shopping tasks, can be easily deployed with just one line of commands.

Data-driven:

AgentGym’s trajectory data adopts a unified ReAct format. This format combines reasoning steps and action sequences through "Thought-Action" pairs. An example of trajectory data is provided in the upper left of Figure 2.

The platform has built a collection of

20509 instructions through extensive collection and enhancement of instructions, and selected 1160 instructions with diversity from them. A benchmark test set AgentEval was constructed to comprehensively evaluate LLM-based agents.

At the same time, the author used GPT-4-Turbo and crowdsourcing annotation to collect trajectory data, and strictly filtered it based on rewards or correctness to build

6130 A collection of high-quality trajectories AgentTraj. In order to demonstrate the performance potential of the behavioral cloning method, the researchers further expanded and obtained AgentTraj-L containing 14485 trajectories.

^{Figure 3: Statistics of 14 environments of the agentgym platform (covering the number of tasks, the scale of the instruction set, the scale of the assessment set, the scale of the trajectory, and the average number of interaction rounds).}

Modular architecture and efficient Pipeline:

## The #AgentGym platform adopts a modular design, so developers

can easily add or change environments. The environment is deployed on different servers (EnvServers) to achieve flexible and efficient interaction through HTTP service . Clients (EnvClients) encapsulate the functions required to interact with the environment and provide corresponding operation interfaces.

The core component AgentController serves as an intermediary between the agent and the environment, providing a trainer (Trainer) that optimizes the agent strategy, and a performance evaluator that supports multiple environments ( Evaluator). The unified operating interface simplifies the interaction between the agent and the environment, allowing users to focus on algorithm optimization and agent training.

#                                               Figure 4: Overview of the AgentGym platform architecture.

##Unique advantages:

Compared with other frameworks ,The advantage of AgentGym is that it not only provides a wide range of environment collections, but also provides
real-time environmental feedback

to the agent through an interactive platform to support the training and evaluation of the agent. At the same time, AgentGym supports the "comprehensive evolution" of the agent in multiple environments, which greatly enhances the agent's generalization ability and enables it to perform well in different tasks and environments. Figure 5: Comparison of AgentGym with other agent frameworks.

##AgentEvol——General Agent Evolution Algorithm

Based on the AgentGym suite, researchers can easily sample, train and evaluate agents. In order to explore the "self-evolution" potential of general-purpose agents, the Fudan Language and Vision Team proposed the AgentEvol algorithm (Figure 6), which helps agents improve their capabilities in multiple environments and tasks. The core idea of ​​this algorithm is to allow the agent to improve its performance through exploration and learning, especially when faced with tasks and instructions that it has not seen before.

                                                                                                                                                                                                           ^{Figure 6: AgentEvol algorithm framework}

AgentEvol is first based on the collected AgentTraj trajectory data set, A "base generally-capable agent" is trained through "behavioral cloning" to equip it with basic instruction-following capabilities and necessary prior knowledge. In this process, the agent imitates the expert's trajectory step by step, including thinking process (thought) and action (action).

#Then, this basic general intelligent agent interacts with different environments to complete self-evolution. It faces more diverse instructions and queries from different environments and gradually improves its ability to complete various tasks.

This process is inspired by the RL as Inference method in machine learning, which treats interactive reinforcement learning as a probabilistic inference problem (see the original text for specific derivation and explanation). This method is different from the traditional reinforcement learning method. It does not directly find the trajectory that maximizes the expected return. Instead, it first defines an optimal policy distribution about the trajectory and then optimizes this distribution through an iterative process.

Specifically, the process includes two alternating steps:

• "Exploration Step": In this step, the agent interacts with the environment under the current strategy, generates new trajectories and evaluates their rewards, forming an estimated optimal strategy distribution. Specifically, the agent interacts with multiple environments and generates a series of behavioral trajectories. Each trajectory is the product of the interaction between the agent and the environment according to the current strategy, including the agent's thinking, the agent's behavior, and the observation of the environment. Then, the environment will give a reward signal to each trajectory based on the degree of matching between the trajectory and the task goal.
• 「Learning Step」: In this step, the agent updates the parameters according to the estimated optimal policy distribution to make it closer to optimal strategy. Specifically, the agent uses the trajectory and reward data collected during the exploration step to optimize itself through an optimization objective function based on trajectory reward weighting. Note that in the learning step, in order to reduce overfitting, the author always optimizes the "basic general agent" instead of the agent obtained in the previous round of optimization.

Through alternating exploration and learning steps, the AgentEvol algorithm gradually optimizes the agent, significantly improves its ability in multiple environments, and achieves "self-evolution" The goal.

Experiment introduction

Task overview:

This study conducted a series of cross-environment exploration and evolution experiments on the agent through the AgentGym framework. The experiment is designed to evaluate the ability of basic agents to self-explore and evolve in diverse environments. To this end, the author adopts a broader instruction set to expand the exploration space of the agent.

Main results:

Across 11 different environments, using the AgentTraj dataset The trained agent demonstrates good basic interaction capabilities.

Further, by implementing behavioral cloning on the larger AgentTraj-L data set, the agent achieved significant performance improvements.

The AgentEvol method proposed in this article, although is only based on limited expert data in the initial stage, through alternating exploration and learning steps , the agent can make correct decisions on unseen exploration sets and achieve self-evolution. On multiple agent tasks, the AgentEvol method surpasses and other SOTA models.

This discovery reveals the potential of agents to adapt to and solve more complex tasks, providing a solid foundation for the development of more advanced general-purpose agents.

                                                                                                                                                                                                                                           ^{Figure 7: Performance comparison of various models and agents in a multi-task environment}

Analysis experiments:

The team also launched a series of ablation experiments from four angles: (1) data merging strategy; (2) evolutionary iteration times; (3) exploration range; (4) sampling times.

Experiments have found that merging the trajectory currently generated by the agent with the initial set of expert trajectories can lead to more stable performance improvements. Correspondingly, using the exploration trajectory of the previous iteration may lead to overfitting and performance fluctuations.

As the number of iterations M increases during the evolution process, the performance improves, but it will eventually stabilize and converge.

#                                       Figure 8: Ablation experiment of data merging strategy and number of iterations

In the process of AgentEvol exploration, by executing sampling for each instruction,

diversified trajectories are generated, which promotes the learning of the agent.

Limit the exploration scope of the agent to

the known instruction set, that is, explore the limited space , which may limit further improvement of AgentEvol's performance.

#                                                                                                                                                                                                                                          In addition, researchers also conducted experiments on different base models. The results show that the AgentEvol method performs well on models of different sizes.

#                                                 Figure 10: Performance comparison on different base models

^{The article also explores whether} the experience trajectories of success and failure can play a role in the evolution process of general intelligent agents.

The experiment adopts the Direct Preference Optimization DPO (Direct Preference Optimization) method and conducts training based on the "success-failure" trajectory during the exploration process. The results show that the agent can learn from error experience in multi-task scenarios, but its overall performance is still inferior to the AgentEvol method.

#                                   Figure 11: DPO training based on success and failure trajectories

Fudan University Natural Language Processing Laboratory was founded by Mr. Wu Lide, chief professor of Fudan University. It is one of the earliest laboratories in my country to conduct research on natural language processing and information retrieval. With the support of the National Natural Science Foundation of China, the National 863/973/Key R&D Program, and provincial ministries and commissions funds, a large number of high-level international journals and conference papers have been published. Under the leadership of academic leader Professor Huang Xuanjing, the laboratory has carried out systematic and in-depth research on the frontiers of large models in aspects such as language large models, multimodal large models, large model alignment, and intelligent agents, resulting in MOSS, Moosi, etc. A series of work with great academic impact, and close cooperative relations with leading domestic and foreign scientific and technological enterprises. Fudan University Vision and Learning Laboratory was founded by Professor Jiang Yugang. It currently has 7 teachers, more than 80 master's and doctoral students, and more than 30 graduate students.

The laboratory is mainly engaged in research on the theory and application of computer vision and multi-modal artificial intelligence.

aims to develop accurate, fast, scalable and trustworthy AI algorithms so that machines can have human-like capabilities. The ability to learn, perceive and reason. The laboratory has undertaken important national and local scientific research projects such as the Science and Technology Innovation 2030-"New Generation Artificial Intelligence" major project, the National Natural Science Foundation of China Key Fund, the National Key R&D Plan Project, the Shanghai Science and Technology Innovation Action Plan, etc., as well as Huawei, Tencent, The technical research needs of enterprises such as Baidu.

The above is the detailed content of Open up the entire process of 'self-evolution' of intelligent agents! Fudan launches AgentGym, a general-purpose intelligent body platform. For more information, please follow other related articles on the PHP Chinese website!