What are the differences between CoMLRL and other multi-LLM training frameworks?
Many frameworks support training role-specific LLM-based multi-agent systems (PettingLLMs, Dr. MAS, MARTI). Most of them are formulated as partially observable stochastic games (POSGs), in which each agent has its own reward. Hand-crafted reward rules for each agent or role make training signals denser, but they are onerous to design.
Cooperative MARL has been studied for a long time and is normally formulated as a Dec-POMDP. In a Dec-POMDP, all agents share one reward, so the optimal joint policy is, in theory, globally optimal for the collaboration. This formulation also allows agents to find more specific tasks from loosely divided roles (see the CoopHumanEval coding domain). In addition, unlike parameter-sharing approaches, CoMLRL mainly supports optimizing systems of truly distinct LLM agents, which better represent heterogeneous LLMs with different capabilities and directly reflect the multi-agent setting.
Does CoMLRL support single-agent RL post-training?
Yes. Users can set num_agents=1 for single-agent RL post-training by following the Training Initialization guide. However, CoMLRL prioritizes simple multi-agent training and omits some optimizations specific to the single-agent case. For single-agent training, LLaMA-Factory, trl, OpenRLHF, and verl are better choices.
Does CoMLRL support self-play or RSI with RL post-training?
Yes. CoMLRL supports multi-turn RL post-training, so environment interactions can be customized for these pipelines.
Self-play (Spiral, SPADE) and self-improvement (MAFT) can be implemented with a single agent in one trainer (num_agents=1), whose external transition adjusts a self-play-style prompt at each turn, for example by switching the agent’s role or asking it to revise its previous answer.
Multi-agent recursive self-improvement (RSI) is more involved: each agent runs in its own trainer, and the system coordinates these trainers. At each turn, the system sends different prompts to one or more trainers to collect their agents’ responses, and it triggers each trainer’s training at its scheduled steps, for example by alternating between agents.
Does CoMLRL support distributed training?
CoMLRL can place whole models, such as actors, critics, and reward models, on different GPUs of a single machine with parallel_training="mp". It does not provide multi-process data-parallel training or shard a single model across devices. Training Parallelization describes device configuration.
Can I use parameter sharing to save VRAM?
Yes. As described in Training Entities, agents accepts loaded model objects, so one loaded model can be passed for every agent. The agents then share its weights, and formatters can give them different roles. Each agent still keeps its own optimizer state, so sharing saves memory for weights and gradients but not for optimizer states.
Can I use LoRA or lower-precision training to save VRAM?
Not recommended. In our extensive experiments, neither PEFT methods such as LoRA nor lower-precision training worked well. LoRA could not fully tune the models, and effective changes may even require larger LoRA adapters, which can end up increasing memory use. Instead, parameter sharing can be used when agents start from models with similar initial performance, and gradient checkpointing can reduce activation memory, although these approaches have not been widely tested. Users are welcome to test them and share their findings in the CoMLRL issues.