---
title: "Evaluation and Observability for Claude Agents"
url: "https://vanducng.dev/2026/08/15/evaluation-and-observability-for-claude-agents/"
date: "2026-08-15"
series: "agentic-coding-with-claude-code"
categories: ["AI"]
tags: ["evaluation","observability","agentic-coding","claude-code","agent-sdk"]
---

# Evaluation and Observability for Claude Agents

[Part 4](https://vanducng.dev/2026/08/15/mcp-and-tool-governance/) ended with a debt: whether the loop is actually doing the job. When a skip was correct, when a tool lied, and when the audit trail is the only honest record.

This is Part 5 of 6 in my **Agentic Coding with Claude Code** series. The working model I am keeping is simple: evaluation is how I score a run. Observability is how I see the run. Structured output is the seam between them.

A harness that cannot tell success from a polite hallucination is a demo. Later I will talk about bounded autonomy. This one stays on scoring the loop and seeing it.

## Silent Failures Hide in Free-Form Text

[Part 3](https://vanducng.dev/2026/08/15/the-harness-models-agent-sdk-and-stop-reason-loops/) already said free-form text is a chat. A harness needs a value it can branch on. I now treat that as an eval problem, not only a parsing problem.

The failure I keep seeing is not a crash. It is a run that looks finished.

| What I see | What it actually was |
| --- | --- |
| A paragraph that says "extracted" | Two required fields were invented |
| Schema-valid JSON | The numbers do not add up, or a field is a guess |
| `action: "skip"` | The case needed a human, not a quiet pass |
| `ResultMessage.subtype == "success"` | The last text was polite and empty of evidence |
| A tool result that looks like `ok` | The handler returned a plausible row the source never had |

[Part 4](https://vanducng.dev/2026/08/15/mcp-and-tool-governance/) already wrote result classes (`ok`, `not_found`, `denied`, `error`) into an append-only audit. That file is not a score. It is the raw material. Evaluation is the next sentence: given this fixture, this trace, and this structured decision, did the agent do the job?

The takeaway I kept: schema-valid is not truth-valid. A skip can be the right answer. A skip can also be the agent ducking a case it should have escalated. I cannot tell those two apart from prose.

## Structured Output Is the Contract

On the Agent SDK, I pass [`output_format`](https://code.claude.com/docs/en/agent-sdk/structured-outputs) as `{"type": "json_schema", "schema": ...}`. TypeScript uses `outputFormat`. The agent may still use tools. When the loop ends, `ResultMessage.structured_output` is validated JSON. The SDK validates against JSON Schema draft-07. Zod schemas need `target: "draft-7"` or the run fails at startup.

On the Messages API, the equivalent is [`output_config.format`](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) with `type: "json_schema"`. Same idea: the last text block is parseable JSON matching the schema.

I treat that object as the contract the harness branches on. I do not treat it as a grade.

Official result contract I actually check, from the [agent loop](https://code.claude.com/docs/en/agent-sdk/agent-loop#handle-the-result) and the [Python `ResultMessage`](https://code.claude.com/docs/en/agent-sdk/python#resultmessage):

| `ResultMessage.subtype` | Official meaning | What I do before I score |
| --- | --- | --- |
| `success` | The loop finished | Use `structured_output` only if it is present. `success` with no structured payload is still a failure. |
| `error_max_structured_output_retries` | Schema validation never succeeded, or a model fallback retracted the output and no retry replaced it | Score as a harness failure. Read `errors` before I blame the schema. |
| `error_max_turns` | Hit `max_turns` (tool-use turns only) | Incomplete. Do not invent the missing fields. |
| `error_max_budget_usd` | Hit `max_budget_usd` | Same. The spend cap fired. |
| `error_during_execution` | The loop broke (API failure, cancel) | Incomplete. Check `errors`. |

A single-shot `query()` then raises a plain `Exception` after yielding an error result. Wrap the loop. A result can also be `success` with `is_error=True` when the last API call failed. I check `subtype`, then `structured_output`, then `is_error`. I do not score `result` text as if it were the contract.

```python
# Sketch. Official fields only. Not a course file.

from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage

EXTRACT_SCHEMA = {
    "type": "object",
    "properties": {
        "status": {
            "type": "string",
            "enum": ["extracted", "incomplete", "escalate"],
        },
        "loan_amount_cents": {"type": ["integer", "null"]},
        "rate_bps": {"type": ["integer", "null"]},
        "missing_fields": {"type": "array", "items": {"type": "string"}},
        "reason": {"type": "string"},
    },
    "required": ["status", "missing_fields", "reason"],
    "additionalProperties": False,
}

async def run_extract(prompt):
    async for message in query(
        prompt=prompt,
        options=ClaudeAgentOptions(
            output_format={"type": "json_schema", "schema": EXTRACT_SCHEMA},
            max_turns=12,
        ),
    ):
        if not isinstance(message, ResultMessage):
            continue
        if message.subtype == "success" and message.structured_output:
            return message
        return None
```

The schema is allowed to say `null`. That is the point. A field the document does not contain stays `null` and lands in `missing_fields`. Fabricating a rate so the object "looks complete" is the silent failure I am trying to make expensive.

I use structured output for decisions the harness must act on (`extracted` / `incomplete` / `escalate`). I do not use it for the essay a reviewer reads. The reviewer reads the reason string and the audit.

## What I Measure

Course 3's public syllabus names task completion, quality, and reliability. Those are not SDK field names. They are scores I attach to a run. Official fields I *can* attach, from [`ResultMessage`](https://code.claude.com/docs/en/agent-sdk/python#resultmessage) and [cost tracking](https://code.claude.com/docs/en/agent-sdk/cost-tracking):

| Working-model score | Question I ask | Official fields I attach |
| --- | --- | --- |
| **Task completion** | Did the definition of done happen? | `subtype`, `structured_output`, `stop_reason` |
| **Quality** | Was the answer right, not just shaped? | Fixture gold vs extracted fields. Not an SDK field. |
| **Tool-use correctness** | Right tool, right args, right result class? | Audit line from [Part 4](https://vanducng.dev/2026/08/15/mcp-and-tool-governance/). OTEL `claude_code.tool` spans if I export them. |
| **System metrics** | How expensive was the path? | `num_turns`, `duration_ms`, `duration_api_ms`, `total_cost_usd`, `usage`, `model_usage` |

`total_cost_usd` is a client-side estimate. Official wording: do not bill from it, and do not trigger financial decisions from it. I use it as a budget signal next to `max_budget_usd`. Authoritative spend lives on Anthropic's Usage and Cost API, which I am not wrapping here.

`usage` covers the top-level loop only. Nested work (subagents, compaction) is missing from that dict. For whole-tree tokens I read `model_usage`. I am not standing up subagents in this post. I still refuse to treat `usage` as the whole bill.

`stop_reason` on the result is the same field [Part 3](https://vanducng.dev/2026/08/15/the-harness-models-agent-sdk-and-stop-reason-loops/) already branched on (`end_turn`, `tool_use`, `refusal`, and the rest). A `refusal` is not a quality miss. It is a stop policy. I score it as `halted`, not as `wrong_extract`.

The takeaway I kept: completion without quality is a silent failure. Quality without a trace is a story I cannot replay.

## Response, Step, Trajectory

I score at three units. The public syllabus names them. The labels below are my working model, not product APIs.

| Unit | What I score | When it is the right unit | When it lies |
| --- | --- | --- | --- |
| **Response** | The final `structured_output` against a gold fixture | Extraction, a classify, any task whose done-state is one object | A lucky last object after a bad tool path. Schema-valid and still wrong. |
| **Step** | One hop: this tool, these args, this result class | Tool-use correctness, a permission deny, a validator that should have fired | A perfect hop inside a trajectory that never should have started |
| **Trajectory** | The whole path: tools, order, skips, retries, final object | Routing, multi-source fusion, anything where *how* it got there matters | Expensive to label. Overfit to one happy path. |

Response eval is cheap. I use it first. If an invented Harbor Credit extractor (below) returns `loan_amount_cents: 45000000` and the fixture says `45000000`, the response score passes. It does not tell me the agent called `lookup_rate` on the wrong product, then "corrected" the math.

Step eval is the Part 4 audit, scored. Did `fetch_doc` return `ok` with the expected fields? Did a deny stay a deny? I write this as assertions on one line, not as a second model.

Trajectory eval is the unit I reach for when the product is a path. An insurance router that retries a schema miss and then escalates a coverage conflict can have a perfect final object and still be the wrong story if it never opened the human queue. I score the sequence: which tools, which result classes, whether a retry happened, whether the last `status` matches the policy.

```mermaid
flowchart TB
    subgraph Path["Trajectory score"]
        direction LR
        T1[Tools] --> T2[Result classes] --> T3[Retries] --> T4[Final object]
    end

    subgraph Last["Response score"]
        direction LR
        R1[Final object] --> R2[Gold fixture]
    end

    classDef greenClass fill:#27AE60,stroke:#333,stroke-width:2px,color:#fff
    classDef orangeClass fill:#F39C12,stroke:#333,stroke-width:2px,color:#fff
    classDef blueClass fill:#4A90E2,stroke:#333,stroke-width:2px,color:#fff
    classDef purpleClass fill:#9B59B6,stroke:#333,stroke-width:2px,color:#fff
    classDef tealClass fill:#16A085,stroke:#333,stroke-width:2px,color:#fff

    class T1 blueClass
    class T2 tealClass
    class T3 purpleClass
    class T4 greenClass
    class R1,R2 orangeClass
```

Response on the left is the baseline. Trajectory on the right is the score I add when the path is the product. I still keep a response assertion. A beautiful path that fabricates the rate is a fail.

## The Eval Loop

I do not have an official Anthropic "eval product" to turn on. What I do have is a working loop I can run in CI, plus official telemetry I can export if I want the same run in a collector.

```mermaid
flowchart LR
    Fix[Fixture] --> Run[Agent run]
    Run --> Trace[Trace + ResultMessage]
    Trace --> Score[Score]
    Score --> Store[Store]
    Store --> Next[Next fixture]

    classDef blueClass fill:#4A90E2,stroke:#333,stroke-width:2px,color:#fff
    classDef orangeClass fill:#F39C12,stroke:#333,stroke-width:2px,color:#fff
    classDef greenClass fill:#27AE60,stroke:#333,stroke-width:2px,color:#fff
    classDef purpleClass fill:#9B59B6,stroke:#333,stroke-width:2px,color:#fff
    classDef tealClass fill:#16A085,stroke:#333,stroke-width:2px,color:#fff

    class Fix blueClass
    class Run orangeClass
    class Trace purpleClass
    class Score greenClass
    class Store tealClass
    class Next orangeClass
```

1. **Fixture in.** A document (or a stubbed tool result), a gold object, and a policy: extract, skip, or escalate. Invented files. Not production PII.
2. **Agent run.** `query()` with `output_format`, a tight `allowed_tools` list, `permission_mode="dontAsk"`, and `max_turns`.
3. **Score.** Response assertions on `structured_output`. Step assertions on the audit. Trajectory assertions when the path matters. System metrics as budgets, not as grades.
4. **Store.** The fixture id, `session_id`, `subtype`, scores, and the audit lines. Append-only, same rule as Part 4.

The trace is the raw material. I keep two layers, and I do not mix their names.

**Working-model trace** (what I log in my process, every run):

| Field | Why I keep it |
| --- | --- |
| Tool name and args | The public API the model actually called |
| Result class (`ok` / `not_found` / `denied` / `error`) | Part 4's honest record |
| `stop_reason` | Why that turn ended |
| `ResultMessage.subtype` | Why the loop ended |
| `structured_output` | The contract I scored |
| `num_turns`, `duration_ms`, `total_cost_usd` | The budget |

**Official export** (what the CLI can send if I turn telemetry on):

The Agent SDK does not emit telemetry itself. It runs the Claude Code CLI as a child process. That CLI can export OpenTelemetry metrics, log events, and (in beta) traces to any OTLP collector. Official page: [Agent SDK observability](https://code.claude.com/docs/en/agent-sdk/observability). The catalog of names lives on [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage).

| Signal | Official enable | What I use it for |
| --- | --- | --- |
| Metrics | `CLAUDE_CODE_ENABLE_TELEMETRY=1` and `OTEL_METRICS_EXPORTER` | Token and cost counters (`claude_code.token.usage`, `claude_code.cost.usage`) |
| Log events | `CLAUDE_CODE_ENABLE_TELEMETRY=1` and `OTEL_LOGS_EXPORTER` | `claude_code.user_prompt`, `claude_code.api_error`, `claude_code.tool_result`, `claude_code.tool_decision` |
| Traces (beta) | `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `OTEL_TRACES_EXPORTER`, and `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` | Spans: `claude_code.interaction`, `claude_code.llm_request`, `claude_code.tool`, `claude_code.tool.execution` |

Telemetry is off until I set those variables. I can put them in the process environment or in `ClaudeAgentOptions.env`. Prompt text and tool bodies stay out of the export unless I opt in (`OTEL_LOG_USER_PROMPTS`, `OTEL_LOG_TOOL_DETAILS`, `OTEL_LOG_TOOL_CONTENT`). I leave those unset on anything that touches documents.

I do not treat OTEL as the score. A span says a tool ran. It does not say the extract was right. The working-model loop still owns the grade. Official tracing is also labeled beta: span names may change. If the collector is down, the CLI fails silent on export by default. The agent still runs. That is why the append-only audit stays in my process.

`session_id` on the `ResultMessage` is how I join a CI fixture to a later replay. Official traces attach a `session.id` span attribute by default. Claude Code omits it if `OTEL_METRICS_INCLUDE_SESSION_ID` is falsy.

## Retry vs Escalate

The SDK already retries structured-output validation. If the object never matches, I get `error_max_structured_output_retries`, not a best-effort blob. That is a harness retry. I am not inventing a second retry API on top of it.

The product question is narrower: which mistakes may the model fix, and which ones must a human see?

| Failure class | Example | Retry? | Escalate? |
| --- | --- | --- | --- |
| **Shape miss** | JSON failed the schema | Yes. The SDK already does this. | After the retry cap. |
| **Fixable arithmetic** | Line items do not sum to the stated total | Yes, once, with the validator error in the next user turn | If the second pass still disagrees. Do not "correct" the source. |
| **Transient tool error** | Result class `error` (timeout, 5xx) | Yes, once, same args | If it fails again. Fail closed. |
| **Honest empty** | Result class `not_found`, or a required page is missing | No | `incomplete` or `escalate`. Never fill the field. |
| **Denied call** | Result class `denied` | No | Halt the write path. Same as Part 4. |
| **Conflict** | Two sources disagree on a date or a rate | No blend | Annotate both, then `escalate`. |
| **Policy / harm** | Fraud language, legal threat, a coverage fight | No | Human queue. Do not retry the classify. |
| **Refusal** | `stop_reason == "refusal"` | Official docs allow a fallback model. I still fail closed on extract. | Yes. Do not invent a workaround. |

```mermaid
flowchart TB
    Out[structured_output or error subtype] --> Kind{Failure class}
    Kind -->|shape / arithmetic / transient| Retry[Retry once]
    Retry --> Ok{Valid and consistent?}
    Ok -->|yes| Accept[Accept]
    Ok -->|no| Human[Escalate]
    Kind -->|empty / denied / conflict / policy / refusal| Human
    Kind -->|extracted and gold matches| Accept

    classDef blueClass fill:#4A90E2,stroke:#333,stroke-width:2px,color:#fff
    classDef orangeClass fill:#F39C12,stroke:#333,stroke-width:2px,color:#fff
    classDef greenClass fill:#27AE60,stroke:#333,stroke-width:2px,color:#fff
    classDef redClass fill:#E74C3C,stroke:#333,stroke-width:2px,color:#fff
    classDef purpleClass fill:#9B59B6,stroke:#333,stroke-width:2px,color:#fff

    class Out blueClass
    class Kind orangeClass
    class Retry purpleClass
    class Ok orangeClass
    class Accept greenClass
    class Human redClass
```

Retry is for mistakes the model can see in the next window: a schema error string, a validator diff, a transient `error` class. Escalate is for missing evidence, a deny, a conflict, or a case I do not want a cheaper model to close. I do not retry a skip into an extract. That is how a silent failure gets a second chance to look like success.

## Three Small Analogs

Course 3 trains this on insurance extraction, mortgage documents, and multi-source supply-chain risk. I am not reprinting those pipelines. Invented analogs only. Same fail-closed rule I used on shelfwatch: a skip is better than a blended guess.

### Harbor Credit: extract, refuse to fabricate

Fictional lender. One messy application packet. The agent may read a stubbed `fetch_packet` tool and return the schema above.

```text
Goal: fill the fields the packet actually states.
Definition of done: status extracted with cents and basis points, or incomplete / escalate.
Fail closed when: a required page is missing, a total does not match the line items,
  or fetch_packet returns not_found / error / denied.
Never: invent a rate, average two disagreeing figures, or coerce a skip into extracted.
```

Fixtures I will actually score (invented):

| Fixture | Gold `status` | What a silent failure looks like |
| --- | --- | --- |
| Complete packet, totals match | `extracted` | Extra fields the page never had |
| Rate page missing | `incomplete`, `rate_bps: null` | A made-up `rate_bps` that still schema-validates |
| Line items sum to 44900000, header says 45000000 | retry once, then `escalate` | The model "corrects" the header so the object is clean |
| `fetch_packet` denied | `escalate` | A guessed packet assembled from the prompt |

Response eval catches the invented rate. Step eval catches the deny that was ignored. Trajectory eval catches the "corrected" total: the path had a validator error and still ended `extracted`.

### Northline Mutual: retry the fixable, escalate the rest

Fictional carrier. One inbound policy PDF stub. The harness routes `renew` / `refer` / `escalate`.

```text
Goal: a structured route, plus a reason a human can audit.
Retry when: schema miss, one arithmetic mismatch, one transient tool error.
Escalate when: coverage language conflicts, a missing declarations page,
  fraud or legal wording, or the retry cap fires.
Do not retry a refer into a renew.
```

The structured object is the route. The score is whether the route matches the fixture policy, and whether a retry happened only on the allowed classes. A trajectory that retries a coverage conflict is a fail even if the last object says `escalate`.

### Keel Freight: fuse evidence, keep provenance

Fictional briefing. Four disagreeing stubs: carrier ETA, warehouse scan, weather bulletin, customs hold.

```text
Goal: one briefing object with one row per source, plus a conflicts list.
Definition of done: every source is present or marked failed, conflicts are named,
  escalate_on is filled when two timestamps or two risk flags disagree.
Never: average ETAs, drop a failed source, or emit a single "most likely" date.
```

Provenance is a field, not a paragraph. Each row keeps `source`, `observed_at`, and `value`. The conflicts array points at the two rows. If customs is down, that row is `error` and the briefing still ships. The silent failure here is a blended Wednesday that no source stated.

Quality on this analog is not "did we pick the right date." It is "did we refuse to pick." Trajectory eval asks whether a failed source stayed visible.

## Putting the Concepts into Practice

Pick one workflow that already emits a decision (extract, route, a briefing) and write this brief before you add a score:

```text
Definition of done (the structured object):
Gold fixtures (happy, missing field, conflict, deny, transient error):
Response assertions (fields that must match, fields that must stay null):
Step assertions (tool, args shape, result class):
Trajectory assertions (allowed retries, required escalate):
Budgets (max_turns, max_budget_usd, duration_ms you will page on):
What you will log (audit line + ResultMessage fields):
What you will export (OTEL off, metrics only, or traces beta):
What fail-closed means (do not guess X):
```

Then inspect a real run:

1. Is `structured_output` present on `subtype == "success"`, or did you score `result` text?
2. Did any fixture pass response eval and fail step eval (wrong tool, ignored deny)?
3. Did a schema-valid object invent a field the source never had?
4. If you force `not_found` on the only evidence tool, does the status become `incomplete` or `escalate` instead of a guess?
5. If traces (beta) are on, does a `claude_code.tool` span exist for every audit line? Metrics or log events alone will not emit that span. If a span is missing, trust the audit.

If you cannot answer those, you have a typed demo, not an evaluated agent.

## Key Terms

- **Evaluation**: How I score a run against a fixture and a policy
- **Observability**: How I see a run: the working-model audit, `ResultMessage` fields, and optional OTEL export
- **Silent failure**: A finished-looking run that skipped, guessed, or blended when it should have stopped
- **`output_format` / `outputFormat`**: Agent SDK knob. `{"type": "json_schema", "schema": ...}`. Draft-07.
- **`output_config.format`**: Messages API equivalent
- **`structured_output`**: Validated JSON on a successful `ResultMessage`. Absent means do not branch.
- **`ResultMessage.subtype`**: `success`, `error_max_turns`, `error_max_budget_usd`, `error_during_execution`, `error_max_structured_output_retries`
- **`num_turns` / `duration_ms` / `total_cost_usd`**: Official system metrics on the result. `total_cost_usd` is a client-side estimate.
- **Response / step / trajectory**: Working-model eval units. Final object, one hop, or the whole path.
- **Working-model trace**: Tool name, args, result class, `stop_reason`, `structured_output`
- **OTEL export**: Official CLI telemetry (`CLAUDE_CODE_ENABLE_TELEMETRY`). Traces are beta (`CLAUDE_CODE_ENHANCED_TELEMETRY_BETA`).
- **`claude_code.interaction` / `claude_code.llm_request` / `claude_code.tool`**: Official span names when traces are on
- **Retry vs escalate**: Fixable mistakes go back to the model once. Missing evidence, denies, conflicts, and policy cases go to a human.
- **Fail closed**: Skip, mark incomplete, or escalate when evidence is missing. Do not guess.
- **Provenance**: Keep the source next to the value. Do not blend disagreements into one number.

## Final Thoughts

The model is still the reasoning engine from [Part 1](https://vanducng.dev/2026/08/09/core-concepts-of-agentic-coding-with-claude-code/). The harness is still the runtime from [Part 3](https://vanducng.dev/2026/08/15/the-harness-models-agent-sdk-and-stop-reason-loops/). MCP is still the governed surface from [Part 4](https://vanducng.dev/2026/08/15/mcp-and-tool-governance/). None of that tells me the loop did the job.

Structured output is the seam: the object I branch on, and the object I score. Observability is the audit plus, if I opt in, official OTEL spans. Evaluation is the fixture, the three units, and the retry-versus-escalate rule. Schema-valid JSON that is still wrong is the failure mode I now look for first.

The next post moves from scoring the loop to bounding it: autonomy I am willing to grant, and guardrails that hold even when the model would rather keep going.

## References

- AI Engineering with Claude
- [Agent SDK structured outputs](https://code.claude.com/docs/en/agent-sdk/structured-outputs)
- [Agent loop](https://code.claude.com/docs/en/agent-sdk/agent-loop)
- [Python ResultMessage](https://code.claude.com/docs/en/agent-sdk/python#resultmessage)
- [Cost tracking](https://code.claude.com/docs/en/agent-sdk/cost-tracking)
- [Agent SDK observability](https://code.claude.com/docs/en/agent-sdk/observability)
- [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage)
- [Platform structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)
- [Stop reasons](https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons)
- [Tool use](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview)

---

*This is Part 5 of 6 in the Agentic Coding with Claude Code series. Earlier: [Part 1](https://vanducng.dev/2026/08/09/core-concepts-of-agentic-coding-with-claude-code/), [Part 2](https://vanducng.dev/2026/08/15/context-engineering-for-claude-code/), [Part 3](https://vanducng.dev/2026/08/15/the-harness-models-agent-sdk-and-stop-reason-loops/), [Part 4](https://vanducng.dev/2026/08/15/mcp-and-tool-governance/).*