How to Evaluate AI Coding Agents: A Practical Scoring Framework

Why Evaluation Matters

Installing an AI agent is easy. Knowing whether it’s actually good is hard. Teams deploy agents and hope for the best, but without structured evaluation, you can’t answer fundamental questions:

• Is the agent getting better or worse over time?
• Which agent is best for which task type?
• Are there systematic failure patterns we should fix?
• Is the output safe and compliant with our standards?

The Four Dimensions

We recommend scoring every agent session across four dimensions:

1. Accuracy (40% weight)

Did the agent produce correct output? This includes:

• Code correctness (does it compile, pass tests?)
• Logical correctness (does it solve the actual problem?)
• Completeness (did it handle edge cases?)

2. Efficiency (25% weight)

Did the agent use resources wisely?

• Token consumption relative to task complexity
• Number of tool calls (fewer is better for equivalent outcomes)
• Session duration
• Retry count (retries indicate wasted work)

3. Safety (20% weight)

Did the agent avoid harmful actions?

• No destructive file operations without confirmation
• No exposure of secrets or credentials
• No execution of dangerous commands
• Proper handling of sensitive data

4. Style (15% weight)

Does the output match team standards?

• Code style consistency with existing codebase
• Appropriate documentation and comments
• Following project conventions (naming, structure)
• Clean diffs (no unnecessary changes)

Building Scorecards

A scorecard is a per-session evaluation record. It captures:

• Overall composite score (0-100)
• Per-dimension scores
• Specific penalty events (what went wrong and where)
• Trend data (is this agent improving?)

Automated vs Manual Scoring

The best approach combines both:

• Automated: Token efficiency, error rates, retry counts, test pass/fail, lint violations
• Manual: Code review quality, architecture decisions, creative problem-solving

Start with automated scoring (it scales). Add manual review for a random sample (10-20% of sessions) to calibrate.

Using Evaluations to Improve

Evaluations are only useful if they drive action. Here’s the feedback loop:

1. Score sessions continuously across all agents
2. Identify patterns in low-scoring sessions (what triggers failures?)
3. Implement fixes (better prompts, pre-checks, model upgrades)
4. Measure improvement (did scores go up after the fix?)

Example: Detecting Drift

Agent performance can degrade over time as codebases change, dependencies update, or provider models shift. Weekly score trends reveal drift early. If an agent’s accuracy drops from 94% to 87% over two weeks, you investigate before it becomes a team-wide problem.

Where Agentic Resource Registry (ARR) Fits

A scorecard is only as good as the sessions behind it. Agentic Resource Registry (ARR, formerly Observal) captures those sessions for you:

• Every session from every coding agent your team runs is traced through hooks: prompts, tool calls, MCP round trips, tokens and errors
• Each agent gets an Insights report that names what keeps going wrong and the numbers that prove it, such as how many tool errors came from edits against stale files
• The report suggests prompt changes, and applying them creates a new version of the agent that goes through review before anyone pulls it

Use the framework above to decide what good looks like, and ARR’s traces and Insights to see where each agent falls short.

---

Related reading:

• How to Reduce AI Agent Costs by 40%
• Setting Up Self-Hosted Observability in 10 Minutes
• Book a demo of ARR