Skip to content
CoMLRL

Welcome to CoMLRL's documentation

Cooperative Multi-LLM Reinforcement Learning (CoMLRL) is an open-source library for training multiple LLMs to collaborate using Multi-Agent Reinforcement Learning (MARL). It provides implementations of various MARL algorithms for LLM collaboration and supports different environments and benchmarks.

Animated overview of CoMLRL: pretrained language models, cooperative MARL trainers, and multi-agent environment interactions.

About#

“What is LLM collaboration?"

LLM collaboration involves agents working together to solve tasks in multi-agent systems. Each agent receives a task prompt and generates a response synchronously according to its instructions. The agents’ responses jointly form a solution. Users and systems may validate the solution and provide additional requirements or suggestions. These interactions form part of the environment, whose state may change based on the agents’ outputs. State updates are included in prompts for subsequent turns until the task is complete or a turn limit is reached.

“Why should we fine-tune multi-LLM systems with MARL?"

Many studies use LLM-based multi-agent systems to solve tasks through agent interaction. Most models are pretrained separately without optimization for coordination, which may limit performance. Effective prompt design also remains challenging. Cooperative MARL methods optimize teams of agents toward a shared objective. These established methods provide a natural foundation for training LLM-based multi-agent systems.

“What are the benefits of decentralized reasoning?"

During decentralized execution, agents use their own prompts and policies. Independent generations can run concurrently when the device configuration permits, supporting role-specific models and modular task decomposition. Latency, memory use, and task quality depend on model sizes, prompt lengths, communication, and devices. Decentralization alone does not guarantee a speedup or preserve all information available to a centralized model.

Features#

  • Multi-agent learning methods for LLM collaboration:
    • Multi-Agent REINFORCE: Critic-free policy gradient methods, including MAREINFORCE, MAGRPO, MARLOO, and MAReMax.
      • Index-aligned joint responses with joint_mode='aligned'.
      • Cartesian-product joint responses with joint_mode='cross'.
      • Decentralized actors or one centralized actor with collaboration_mode='centralized'.
    • Multi-Agent Actor-Critic: Actor-Critic methods, including IAC and MAAC.
      • Independent actor-critic (separate critic or value head over the LLM backbone).
      • Centralized critic over joint prompts with separate actors.
      • Decentralized actors or one centralized actor with collaboration_mode='centralized'.
    • Multi-Agent PbRL: MADPO, MARLHF, and their iterative variants.
      • Direct joint preference optimization or a learned joint reward followed by RL.
      • Decentralized actors or one centralized actor with role-aware joint responses.
  • Environments that simulate real-world tasks for training and evaluating LLM collaboration:
    • Writing: Multiple LLM agents collaborate to process articles.
      • TLDR - Summarizing Reddit posts.
      • arXiv - Expanding abstracts into introductions.
    • Coding: Collaborative code generation for programming problems.
      • MBPP - Mostly basic Python problems.
      • HumanEval - Handwritten evaluation problems.
      • CoopHumanEval - A cooperative variant of HumanEval.
      • ClassEval - Completing class-level code based on attributes and docstrings.
    • Travel: Collaborative travel planning under user constraints.
    • Tool-Use: Collaborative function calling on BFCL v4.