How to Evaluate Multi-Turn AI Conversations

Most LLM evaluations follow a single turn eval. That works for classification, extraction, and single-turn question answering. It is not enough for conversational agents.

A multi-turn agent must retain context, update state, ask for missing information, call tools, handle corrections, and complete a real-world task. Every individual response can appear reasonable while the conversation still fails.

A scheduling agent might sound helpful while it:

• Forgets the requested duration

• Books the wrong date

• Invites the wrong person

• Acts without confirmation

• Creates the same event twice

• Claims success after an API failure

Multi-turn evals must therefore measure three different things:

Response quality: Was each message appropriate?

Trajectory quality: Did the agent take the right sequence of actions?

Outcome correctness: Did the intended result actually occur?

A polished response cannot compensate for an incorrect transaction.

1. Define What Success Means

Before building an eval, define the expected final state.

For a scheduling agent, success might mean:

• An event was created

• The correct person was invited

• The duration was 30 minutes

• The event was scheduled during the requested window

• The user confirmed the time

• No duplicate event was created

This is more useful than defining one ideal final response.

A multi-turn test case should describe the required state, actions, constraints, and outcome.

#python
from dataclasses import dataclass, field
from typing import Any


@dataclass
class Scenario:
    name: str
    initial_message: str
    expected_outcome: dict[str, Any]

    required_tools: set[str] = field(default_factory=set)
    forbidden_tools: set[str] = field(default_factory=set)
    required_state: dict[str, Any] = field(default_factory=dict)

    max_turns: int = 10
    requires_confirmation: bool = False


scenario = Scenario(
    name="schedule_meeting",
    initial_message=(
        "Schedule a 30-minute meeting with Sarah "
        "next Tuesday afternoon."
    ),
    expected_outcome={
        "event_created": True,
        "attendee": "[email protected]",
        "duration_minutes": 30,
        "day": "Tuesday",
    },
    required_tools={
        "search_contacts",
        "check_calendar",
        "create_event",
    },
    forbidden_tools={"delete_event"},
    required_state={
        "duration_minutes": 30,
        "confirmation_received": True,
    },
    max_turns=8,
    requires_confirmation=True,
)

This gives the eval system a concrete specification:

• What the agent must remember

• Which tools it should use

• Which actions are prohibited

• What must change in the environment

• How efficiently the task should be completed

2. Capture the Complete Trace

You cannot evaluate what you do not record.

For every conversation, capture:

• User and assistant messages

• Tool calls and arguments

• Tool results and errors

• Agent state after each turn

• Retries

• Latency and token usage

• Final environment state

State snapshots are especially important.

Suppose the user initially requests Tuesday, then says:

The agent should update the day while preserving the other information:

Before: day=Tuesday, duration=30
After:  day=Thursday, duration=30

If the meeting is eventually created on Tuesday, the trace should reveal whether:

• The correction was misunderstood

• State was never updated

• Updated state was later overwritten

• The tool received stale arguments

• The event was correct, but the final response was wrong

Without intermediate state, all of these failures look identical.

Production observability and offline evals should use the same trace format. That makes it possible to turn a production failure directly into a regression test.

3. Evaluate at Three Levels

A useful eval system should grade the turn, trajectory, and outcome separately.

Turn-level evaluation

At each turn, ask:

• Did the agent understand the latest message?

• Did it preserve previous constraints?

• Did it ask for information already provided?

• Did it make an unsupported assumption?

• Did it choose the correct next action?

Turn-level grading helps identify exactly where the conversation started to drift.

Trajectory-level evaluation

Inspect the complete sequence of actions:

• Were the correct tools called?

• Were they called at the right time?

• Were unnecessary calls made?

• Did the agent repeat an action?

• Did it recover from a tool failure?

• Did it wait for confirmation before acting?

Do not require one exact path unless the workflow demands it. Two agents may take different but equally valid routes to the same outcome.

Outcome-level evaluation

Finally, inspect the external environment:

• Does the calendar event exist?

• Was the correct database row updated?

• Was the refund actually issued?

• Was the ticket routed correctly?

• Were duplicate actions avoided?

The environment is the source of truth.

The statement “Your meeting has been booked” is not evidence that the meeting exists.

4. Prefer Deterministic Graders

When a requirement can be checked with code, check it with code.

Deterministic graders are faster, cheaper, more stable, and easier to debug than model-based judges.

def get_called_tools(trace):
    return [
        call["name"]
        for turn in trace["turns"]
        for call in turn.get("tool_calls", [])
    ]


def grade_scenario(scenario, trace):
    failures = []

    for key, expected in scenario.expected_outcome.items():
        actual = trace["environment_outcome"].get(key)

        if actual != expected:
            failures.append(
                f"{key}: expected {expected!r}, got {actual!r}"
            )

    tools = set(get_called_tools(trace))

    missing = scenario.required_tools - tools
    forbidden = scenario.forbidden_tools & tools

    if missing:
        failures.append(f"Missing tools: {sorted(missing)}")

    if forbidden:
        failures.append(
            f"Forbidden tools called: {sorted(forbidden)}"
        )

    if len(trace["turns"]) > scenario.max_turns:
        failures.append("Conversation exceeded turn limit")

    return failures

Deterministic checks work well for:

• Tool names and arguments

• Required state fields

• Valid state transitions

• Database changes

• Duplicate operations

• Confirmation requirements

• Maximum retries

• Authorization rules

• Latency and cost limits

• Forbidden actions

Avoid making the checks unnecessarily rigid. Requiring an exact tool sequence can incorrectly reject valid alternative trajectories.

Test what matters to the product, not incidental implementation details.

5. Use LLM Judges for Semantic Quality

Some criteria cannot be reduced to exact comparisons:

• Did the agent understand the user’s real goal?

• Was a clarification question necessary?

• Did it handle a correction naturally?

• Was the conversation repetitive?

• Did it make an unsupported assumption?

• Was the final explanation clear?

These are good uses for an LLM judge.

However, avoid vague prompts such as:

Give the judge a narrow rubric:

Evaluate whether the agent:

1. Preserved all user constraints
2. Avoided asking repeated questions
3. Avoided unsupported assumptions
4. Waited for confirmation before acting
5. Correctly interpreted tool results
6. Reached the requested outcome

Identify the first turn where a meaningful failure occurred.

The judge should receive the complete trace, including tool results and final environment state. Otherwise, it may trust the agent’s unsupported claim that an action succeeded.

Model-based grading works best when you:

• Score individual criteria separately

• Use binary decisions for critical behavior

• Require structured output

• Define each score clearly

• Validate the judge against human-reviewed conversations

• Test whether its verdict remains stable across repeated runs

LLM judges should complement deterministic graders, not replace them.

6. Test More Than the Happy Path

Scripted conversations are the foundation of a regression suite.

They are useful for testing:

• Core workflows

• Known production failures

• Tool-use rules

• State transitions

• Confirmation requirements

• Safety policies

But scripted tests cover only the paths you anticipate.

User simulators can explore more varied behavior:

• The user changes the date midway

• The user contradicts an earlier statement

• The user interrupts the workflow

• The user provides incomplete information

• A tool fails

• The user refuses a proposed option

• The user changes the goal entirely

A simulator should have a hidden goal, known facts, and behavioral constraints. It should not volunteer every detail or automatically accept weak responses.

The strongest eval suite combines:

Scripted regression tests
+ Constrained user simulations
+ Real production conversations

Each serves a different purpose.

Scripted tests protect known behavior. Simulations discover unexpected behavior. Production traces reveal how users actually interact with the system.

7. Measure Reliability, Not One Successful Run

LLM behavior is nondeterministic.

A conversation that succeeds once may fail on the next attempt. Important scenarios should therefore be run multiple times.

Track metrics such as:

• Task success rate

• Critical failure rate

• Context-loss rate

• Recovery rate

• Incorrect tool-call rate

• Duplicate-action rate

• Average turns to completion

• Cost per successful conversation

• P50 and P95 latency

Do not rely only on an average score.

Consider two agents:

Agent A
Average quality: 91%
Critical failure rate: 8%

Agent B
Average quality: 88%
Critical failure rate: 1%

For payments, healthcare, account changes, or destructive actions, Agent B is likely the safer system.

Failures should be classified by severity.

Critical failures

• Unauthorized action

• Wrong customer record modified

• Private information exposed

• Irreversible action taken without confirmation

Major failures

• User correction forgotten

• Wrong tool called

• Required workflow step skipped

Minor failures

• Repetitive wording

• One unnecessary question

• An overly verbose response

A critical failure should never disappear inside a high aggregate score.

Find the First Incorrect Turn

The final error is often only the symptom.

Suppose the agent creates an event on the wrong day. The root cause may have happened four turns earlier when it failed to process the user’s correction.

Useful failure categories include:

• Intent recognition

• Memory

• State updates

• Planning

• Tool selection

• Tool arguments

• Tool execution

• Tool-result interpretation

• Response generation

A useful failure analysis might look like this:

Wrong final date
    ↓
create_event received Tuesday
    ↓
planner read Tuesday from state
    ↓
state was not updated after the correction
    ↓
root cause: correction-handling failure

This is far more actionable than simply marking the conversation as failed.

Every meaningful production failure should become:

1. A saved trace

2. A minimized reproduction

3. A named failure category

4. A regression test

5. A release-blocking test when appropriate

Over time, the eval suite becomes the behavioral specification for the agent.

The Core Principle

A multi-turn eval is not testing whether a model can produce a good sentence.

It is testing whether the entire system can move from the user’s initial goal to the correct final state while preserving context, following policy, using tools correctly, and recovering when the conversation changes.

For production agents: