Multi-Turn Interaction

Many complex problems require more than one turn to solve. LLM agents interact with the environment to obtain useful feedback from other models or tools in the system.

Multi-Turn MAGRPO#

Multi-turn MAGRPO builds a rollout tree whose branches represent different joint responses (TreeRPO).

Joint rollout tree Joint observations and actions branch into independent rollouts.

Each episode begins with a task sampled from the dataset, which provides the initial observations $\mathbf{o}_0=\{o_{1, 0}, \cdots, o_{n, 0}\}$ and histories $\mathbf{h}_0=\{h_{1, 0}, \cdots, h_{n, 0}\}$ for all agents. At each turn, agents generate a group of joint responses $\mathbf{a}^{\mathcal{G}}_t\gets\boldsymbol{\pi}^{\mathcal{G}}(\cdot|\mathbf{h}_t)$ from their current observation-action history $\mathbf{h}_t$ . Each response starts a distinct rollout. Agents receive joint rewards $r^{(g)}_{t}$ for each response, based on the accumulated history and current joint action $\mathbf{a}^{(g)}_{t} \in \mathbf{a}^{\mathcal{G}}_{t}$ . Each rollout then evolves independently, producing new joint observations $\mathbf{o}^{\mathcal{G}}_{t+1}$ as the environment changes and branching into further rollouts at the next turn $$ t+1 $$ . This process continues until the terminal turn $$ H $$ .

Joint Mode#

MAGRPO supports two modes for forming joint responses at each turn:

Aligned (joint_mode="aligned"): Responses with the same generation index are combined across agents. Each agent samples $$ G $$ responses (the fixed num_generations value) at every node. With $$ T $$ turns and no early termination, the tree has $$ G^T $$ leaves.

Aligned joint responses Responses with matching generation indices are combined across agents.

Cross (joint_mode="cross"): The Cartesian product of the agents’ sampled responses yields $$ G^N $$ joint actions per node. With no early termination, the tree has $G^{N \cdot T}$ leaves. Responses are reused across combinations, so the joint samples are not independent. This increases the number of reward evaluations and the rollout size without guaranteeing more accurate value estimates.

Cross joint responses The Cartesian product forms joint responses within each rollout.

Only responses from the same rollout can be combined because rollouts evolve independently. The current MAGRPO implementation accumulates undiscounted returns: discount does not change its tree recursion. MAREINFORCE describes return aggregation and advantage normalization.

Multi-Turn MAAC/IAC#

The MAGRPO rollout tree generates numerous samples per episode. In the multi-turn setting, MAAC/IAC generate only one sample per agent per turn, and the episode continues across turns until termination. MAAC/IAC use one-step temporal-difference targets, with discount applied to the next rollout value and no bootstrapping after the final or early-terminated turn. The current implementation collects the full episode before buffering its samples. Policy updates occur when rollout_buffer_size is reached, and any remaining buffer is flushed at the end of the epoch. Policies are not updated between turns of the same episode.

Actor-critic trajectory One joint response per turn advances the episode from left to right.

IAC/MAAC multi-turn training requires num_generations to be set to 1.

Environment Transition#

External feedback mechanisms control how environment observations are incorporated into prompts for subsequent turns. These mechanisms are usually implemented in CoMLRL’s downstream environments.

Environment transition At turn t, the agents' responses enter the external transition, which returns each agent's own observation for turn t+1.

Custom External Feedback#

Users can implement custom external feedback by defining a function with the following interface:

External Feedback Parameters

prompt: Original task prompt or problem description (required)
agent_completions: List or tuple of completions from the previous turn, one per agent (required)
num_agents: Number of agents in the system (required)
prompt_history_per_agent: List containing one list of previous prompts per agent (optional)
response_history_per_agent: List containing one list of previous responses per agent (optional)

The function must return a list or tuple of prompts for the next turn, one for each agent. The trainer passes only the arguments above, with no additional keyword arguments. Mode-specific parameters can be configured through a closure or functools.partial.

By default, each returned prompt becomes the new prompt field and is passed through the corresponding agent’s formatter. If external_prompt_passthrough=True, the returned prompts are used directly without reformatting. In MAGRPO, the external transition is called for each rollout branch with that branch’s histories.

The following example constructs prompts for the next turn:

from typing import List, Optional

def custom_external(
    prompt: str,
    agent_completions: List[str],
    num_agents: int,
    prompt_history_per_agent: Optional[List[List[str]]] = None,
    response_history_per_agent: Optional[List[List[str]]] = None,
) -> List[str]:
    # Custom logic to format next-turn prompts
    # Access environment feedback, tool outputs, etc.
    next_turn_prompts = []
    for i in range(num_agents):
        # Format prompt for agent i based on history and feedback
        next_prompt = f"{prompt}\nPrevious attempt: {agent_completions[i]}\nPlease revise."
        next_turn_prompts.append(next_prompt)
    return next_turn_prompts

Example Modes#

An environment for code generation includes three example external transition modes:

external.mode=expert_edits: An external LLM (default: DeepSeek-Coder) proposes code edits. Follow-up prompts include edit suggestions and context from previous turns. With the required API keys, users can select a different expert (e.g., Claude, GPT) through expert_model.
external.mode=level_feedback: Static AST checks and dynamic code execution provide a diagnosis. By default, the sandbox includes only the first test. The sandbox_slice parameter selects all tests (0, None, or ‘all’) or a specific number of tests (negative values are supported).
external.mode=plain: This self-improvement mode includes only the prompts and responses from previous turns and a revision instruction.