Free guides

How can I baseline AI agent performance metrics before detecting skill drift?

If you run a conversational AI agent in production, you will eventually notice that it feels worse than it used to. Answers get a little less relevant, tasks fail more often, or the bot asks for information it already has. The hard part is proving that this is happening, and proving it by how much. Without a baseline, every complaint is anecdotal and every dashboard spike is ambiguous.

The fix is to capture a reference snapshot of how the agent performs right now, while you still trust it, and then compare future measurements against that snapshot. This article walks through a technique you can finish in under an hour for a small agent, and it stands on its own whether or not you ever automate anything further.

Step 1: Pick three to five core skills

Resist the urge to measure everything. A baseline with twelve metrics is noisy, and when one of them wiggles you will not know whether it matters. Choose a small set that maps to what users actually need from the agent.

For most agents, intent accuracy, slot precision and recall, task success rate, and latency are enough. Add conversation length if you care about efficiency.

Step 2: Build a reference dataset with human labels

Export a recent window of production logs, for example the last 30 days. You do not need to label all of it. A sample of one to two thousand turns, drawn at random, is usually enough to estimate the metrics with reasonable confidence.

Store annotations in a flat CSV so that any script can read them:

turn_id,session_id,user_input,agent_intent,intent,slot_filled,success,turn_latency_ms
1,s01,"Book a flight",BookFlight,BookFlight,yes,no,350
2,s01,"Paris to Tokyo",BookFlight,BookFlight,yes,yes,420

Here agent_intent is what the agent predicted and intent is the human-verified label. Keeping both columns lets you measure accuracy directly.

A common mistake is relying only on automated signals such as whether the session ended without an error. That misses the cases that matter most: a confident wrong answer, or a session that technically completed but left the user with the wrong booking. Human validation of intent and slots is what makes the baseline trustworthy.

Step 3: Compute the metrics with a small script

The following script reads the annotated CSV and writes a JSON report. It is deliberately short so you can read and adapt it.

import csv, json, argparse, statistics
from collections import Counter

def load_rows(path):
    with open(path, newline='') as f:
        return list(csv.DictReader(f))

def compute_metrics(rows):
    n = len(rows)
    intent_correct = sum(1 for r in rows if r['intent'] == r['agent_intent'])

    # Slot precision: of the turns where the agent filled a slot,
    # how many were filled correctly?
    filled = [r for r in rows if r['slot_filled'] == 'yes']
    slot_precision = (sum(1 for r in filled if r['success'] == 'yes') / len(filled)) if filled else 0.0

    success = sum(1 for r in rows if r['success'] == 'yes')
    latency = statistics.mean(int(r['turn_latency_ms']) for r in rows)

    sessions = Counter(r['session_id'] for r in rows)
    avg_turns = statistics.mean(sessions.values())

    return {
        'intent_accuracy': round(intent_correct / n, 4),
        'slot_precision': round(slot_precision, 4),
        'success_rate': round(success / n, 4),
        'avg_latency_ms': round(latency, 1),
        'avg_turns_per_session': round(avg_turns, 2),
    }

if __name__ == '__main__':
    p = argparse.ArgumentParser()
    p.add_argument('--data', required=True)
    p.add_argument('--output', required=True)
    args = p.parse_args()

    metrics = compute_metrics(load_rows(args.data))
    with open(args.output, 'w') as f:
        json.dump(metrics, f, indent=2)
    print(json.dumps(metrics, indent=2))

Run it with:

python evaluate_metrics.py --data annotations.csv --output baseline_report.json

Notice that slot precision is computed only over turns where a slot was filled. A high precision with low recall can hide missed entities, so if you have the data, also compute recall as the share of required slots that were filled. Measuring both prevents a tidy-looking number from concealing a gap.

Step 4: Store the baseline under version control

A baseline is only useful if you can find it later and trust that it has not changed. Commit baseline_report.json to a metrics repository and tag it with the date, for example baseline-2026-11-01.json. Keep the annotated CSV in a secure archive so you can recompute or audit the numbers later. Record the agent version, model version, and prompt version alongside the report, because those are the usual suspects when a metric moves.

Step 5: Set tolerance thresholds before you need them

Decide now how much deviation is acceptable, so that a single bad afternoon does not trigger a panic. A reasonable starting point:

These numbers are starting points, not universal rules. Tune them to your agent's normal variance. Measure the spread over two or three repeated evaluations on the same data before you fix a threshold, so you know what noise looks like.

Step 6: Re-run the same evaluation on a schedule

The whole point of a baseline is comparison. Re-run the identical evaluation on a fresh labeled sample at a regular cadence, such as weekly, and compare each metric against the stored baseline. Keep the sampling method and annotation guidelines constant. If you change how turns are sampled or how labels are defined, you have created a new baseline, not a comparison.

Common pitfalls

• Single-metric thinking. Accuracy alone can look healthy while task success quietly falls. Watch several signals together.
• Unstable reference data. If the annotated sample changes between runs, differences may reflect the sample, not the agent.
• Labels that drift too. Annotators apply guidelines differently over time. Periodically re-label a small overlap set to check consistency.
• No record of versions. Without agent, model, and prompt versions attached to the baseline, you cannot attribute a change to its cause.

Checklist

• Choose three to five skills and their metrics
• Export roughly 30 days of production logs
• Human-label one to two thousand sampled turns
• Run the evaluation script to produce baseline_report.json
• Commit the report with a date tag and record the model and prompt versions
• Set tolerance thresholds based on repeated runs
• Schedule the same evaluation on a fresh sample

Once you have a baseline you trust, detecting drift becomes a comparison problem rather than a debate about how the bot feels. You know what good looked like, by how much things have moved, and when the movement is large enough to act on.

If you want a more complete treatment of this workflow, including longer checklists and worked examples, the full "AI Agent Skill Drift Prevention Guide" is available at https://heracles-store.pages.dev/p/ai-agent-skill-drift-prevention-guide/.

Written with AI assistance and reviewed for accuracy before publishing.

This guide was researched and written with the help of AI and reviewed by an automated quality check. It is general educational information, not professional advice.