Evaluation and Observability for Claude Agents

Source: AI Engineering with Claude (Udacity ND7426), plus current Claude docs I fetched: Agent SDK structured outputs, agent loop, Python ResultMessage, cost tracking, Agent SDK observability, Claude Code monitoring, platform structured outputs, stop reasons, and tool use

Part 4 ended with a debt: whether the loop is actually doing the job. When a skip was correct, when a tool lied, and when the audit trail is the only honest record.

This is Part 5 of 6 in my Agentic Coding with Claude Code series. The working model I am keeping is simple: evaluation is how I score a run. Observability is how I see the run. Structured output is the seam between them.

A harness that cannot tell success from a polite hallucination is a demo. Later I will talk about bounded autonomy. This one stays on scoring the loop and seeing it.

Silent Failures Hide in Free-Form Text

Part 3 already said free-form text is a chat. A harness needs a value it can branch on. I now treat that as an eval problem, not only a parsing problem.

The failure I keep seeing is not a crash. It is a run that looks finished.

What I see	What it actually was
A paragraph that says “extracted”	Two required fields were invented
Schema-valid JSON	The numbers do not add up, or a field is a guess
action: "skip"	The case needed a human, not a quiet pass
ResultMessage.subtype == "success"	The last text was polite and empty of evidence
A tool result that looks like ok	The handler returned a plausible row the source never had

Part 4 already wrote result classes (ok, not_found, denied, error) into an append-only audit. That file is not a score. It is the raw material. Evaluation is the next sentence: given this fixture, this trace, and this structured decision, did the agent do the job?

The takeaway I kept: schema-valid is not truth-valid. A skip can be the right answer. A skip can also be the agent ducking a case it should have escalated. I cannot tell those two apart from prose.

Structured Output Is the Contract

On the Agent SDK, I pass output_format as {"type": "json_schema", "schema": ...}. TypeScript uses outputFormat. The agent may still use tools. When the loop ends, ResultMessage.structured_output is validated JSON. The SDK validates against JSON Schema draft-07. Zod schemas need target: "draft-7" or the run fails at startup.

On the Messages API, the equivalent is output_config.format with type: "json_schema". Same idea: the last text block is parseable JSON matching the schema.

I treat that object as the contract the harness branches on. I do not treat it as a grade.

Official result contract I actually check, from the agent loop and the Python ResultMessage:

ResultMessage.subtype	Official meaning	What I do before I score
success	The loop finished	Use structured_output only if it is present. success with no structured payload is still a failure.
error_max_structured_output_retries	Schema validation never succeeded, or a model fallback retracted the output and no retry replaced it	Score as a harness failure. Read errors before I blame the schema.
error_max_turns	Hit max_turns (tool-use turns only)	Incomplete. Do not invent the missing fields.
error_max_budget_usd	Hit max_budget_usd	Same. The spend cap fired.
error_during_execution	The loop broke (API failure, cancel)	Incomplete. Check errors.

A single-shot query() then raises a plain Exception after yielding an error result. Wrap the loop. A result can also be success with is_error=True when the last API call failed. I check subtype, then structured_output, then is_error. I do not score result text as if it were the contract.

# Sketch. Official fields only. Not a course file.

from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage

EXTRACT_SCHEMA = {
    "type": "object",
    "properties": {
        "status": {
            "type": "string",
            "enum": ["extracted", "incomplete", "escalate"],
        },
        "loan_amount_cents": {"type": ["integer", "null"]},
        "rate_bps": {"type": ["integer", "null"]},
        "missing_fields": {"type": "array", "items": {"type": "string"}},
        "reason": {"type": "string"},
    },
    "required": ["status", "missing_fields", "reason"],
    "additionalProperties": False,
}

async def run_extract(prompt):
    async for message in query(
        prompt=prompt,
        options=ClaudeAgentOptions(
            output_format={"type": "json_schema", "schema": EXTRACT_SCHEMA},
            max_turns=12,
        ),
    ):
        if not isinstance(message, ResultMessage):
            continue
        if message.subtype == "success" and message.structured_output:
            return message
        return None

The schema is allowed to say null. That is the point. A field the document does not contain stays null and lands in missing_fields. Fabricating a rate so the object “looks complete” is the silent failure I am trying to make expensive.

I use structured output for decisions the harness must act on (extracted / incomplete / escalate). I do not use it for the essay a reviewer reads. The reviewer reads the reason string and the audit.

What I Measure

Course 3’s public syllabus names task completion, quality, and reliability. Those are not SDK field names. They are scores I attach to a run. Official fields I can attach, from ResultMessage and cost tracking:

Working-model score	Question I ask	Official fields I attach
Task completion	Did the definition of done happen?	subtype, structured_output, stop_reason
Quality	Was the answer right, not just shaped?	Fixture gold vs extracted fields. Not an SDK field.
Tool-use correctness	Right tool, right args, right result class?	Audit line from Part 4. OTEL claude_code.tool spans if I export them.
System metrics	How expensive was the path?	num_turns, duration_ms, duration_api_ms, total_cost_usd, usage, model_usage

total_cost_usd is a client-side estimate. Official wording: do not bill from it, and do not trigger financial decisions from it. I use it as a budget signal next to max_budget_usd. Authoritative spend lives on Anthropic’s Usage and Cost API, which I am not wrapping here.

usage covers the top-level loop only. Nested work (subagents, compaction) is missing from that dict. For whole-tree tokens I read model_usage. I am not standing up subagents in this post. I still refuse to treat usage as the whole bill.

stop_reason on the result is the same field Part 3 already branched on (end_turn, tool_use, refusal, and the rest). A refusal is not a quality miss. It is a stop policy. I score it as halted, not as wrong_extract.

The takeaway I kept: completion without quality is a silent failure. Quality without a trace is a story I cannot replay.

Response, Step, Trajectory

I score at three units. The public syllabus names them. The labels below are my working model, not product APIs.

Unit	What I score	When it is the right unit	When it lies
Response	The final structured_output against a gold fixture	Extraction, a classify, any task whose done-state is one object	A lucky last object after a bad tool path. Schema-valid and still wrong.
Step	One hop: this tool, these args, this result class	Tool-use correctness, a permission deny, a validator that should have fired	A perfect hop inside a trajectory that never should have started
Trajectory	The whole path: tools, order, skips, retries, final object	Routing, multi-source fusion, anything where how it got there matters	Expensive to label. Overfit to one happy path.

Response eval is cheap. I use it first. If an invented Harbor Credit extractor (below) returns loan_amount_cents: 45000000 and the fixture says 45000000, the response score passes. It does not tell me the agent called lookup_rate on the wrong product, then “corrected” the math.

Step eval is the Part 4 audit, scored. Did fetch_doc return ok with the expected fields? Did a deny stay a deny? I write this as assertions on one line, not as a second model.

Trajectory eval is the unit I reach for when the product is a path. An insurance router that retries a schema miss and then escalates a coverage conflict can have a perfect final object and still be the wrong story if it never opened the human queue. I score the sequence: which tools, which result classes, whether a retry happened, whether the last status matches the policy.

flowchart TB
    subgraph Path["Trajectory score"]
        direction LR
        T1[Tools] --> T2[Result classes] --> T3[Retries] --> T4[Final object]
    end

    subgraph Last["Response score"]
        direction LR
        R1[Final object] --> R2[Gold fixture]
    end

    classDef greenClass fill:#27AE60,stroke:#333,stroke-width:2px,color:#fff
    classDef orangeClass fill:#F39C12,stroke:#333,stroke-width:2px,color:#fff
    classDef blueClass fill:#4A90E2,stroke:#333,stroke-width:2px,color:#fff
    classDef purpleClass fill:#9B59B6,stroke:#333,stroke-width:2px,color:#fff
    classDef tealClass fill:#16A085,stroke:#333,stroke-width:2px,color:#fff

    class T1 blueClass
    class T2 tealClass
    class T3 purpleClass
    class T4 greenClass
    class R1,R2 orangeClass

Response on the left is the baseline. Trajectory on the right is the score I add when the path is the product. I still keep a response assertion. A beautiful path that fabricates the rate is a fail.

The Eval Loop

I do not have an official Anthropic “eval product” to turn on. What I do have is a working loop I can run in CI, plus official telemetry I can export if I want the same run in a collector.

flowchart LR
    Fix[Fixture] --> Run[Agent run]
    Run --> Trace[Trace + ResultMessage]
    Trace --> Score[Score]
    Score --> Store[Store]
    Store --> Next[Next fixture]

    classDef blueClass fill:#4A90E2,stroke:#333,stroke-width:2px,color:#fff
    classDef orangeClass fill:#F39C12,stroke:#333,stroke-width:2px,color:#fff
    classDef greenClass fill:#27AE60,stroke:#333,stroke-width:2px,color:#fff
    classDef purpleClass fill:#9B59B6,stroke:#333,stroke-width:2px,color:#fff
    classDef tealClass fill:#16A085,stroke:#333,stroke-width:2px,color:#fff

    class Fix blueClass
    class Run orangeClass
    class Trace purpleClass
    class Score greenClass
    class Store tealClass
    class Next orangeClass

1. Fixture in. A document (or a stubbed tool result), a gold object, and a policy: extract, skip, or escalate. Invented files. Not production PII.
2. Agent run. query() with output_format, a tight allowed_tools list, permission_mode="dontAsk", and max_turns.
3. Score. Response assertions on structured_output. Step assertions on the audit. Trajectory assertions when the path matters. System metrics as budgets, not as grades.
4. Store. The fixture id, session_id, subtype, scores, and the audit lines. Append-only, same rule as Part 4.

The trace is the raw material. I keep two layers, and I do not mix their names.

Working-model trace (what I log in my process, every run):

Field	Why I keep it
Tool name and args	The public API the model actually called
Result class (ok / not_found / denied / error)	Part 4’s honest record
stop_reason	Why that turn ended
ResultMessage.subtype	Why the loop ended
structured_output	The contract I scored
num_turns, duration_ms, total_cost_usd	The budget

Official export (what the CLI can send if I turn telemetry on):

The Agent SDK does not emit telemetry itself. It runs the Claude Code CLI as a child process. That CLI can export OpenTelemetry metrics, log events, and (in beta) traces to any OTLP collector. Official page: Agent SDK observability. The catalog of names lives on Claude Code monitoring.

Signal	Official enable	What I use it for
Metrics	CLAUDE_CODE_ENABLE_TELEMETRY=1 and OTEL_METRICS_EXPORTER	Token and cost counters (claude_code.token.usage, claude_code.cost.usage)
Log events	CLAUDE_CODE_ENABLE_TELEMETRY=1 and OTEL_LOGS_EXPORTER	claude_code.user_prompt, claude_code.api_error, claude_code.tool_result, claude_code.tool_decision
Traces (beta)	CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_TRACES_EXPORTER, and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1	Spans: claude_code.interaction, claude_code.llm_request, claude_code.tool, claude_code.tool.execution

Telemetry is off until I set those variables. I can put them in the process environment or in ClaudeAgentOptions.env. Prompt text and tool bodies stay out of the export unless I opt in (OTEL_LOG_USER_PROMPTS, OTEL_LOG_TOOL_DETAILS, OTEL_LOG_TOOL_CONTENT). I leave those unset on anything that touches documents.

I do not treat OTEL as the score. A span says a tool ran. It does not say the extract was right. The working-model loop still owns the grade. Official tracing is also labeled beta: span names may change. If the collector is down, the CLI fails silent on export by default. The agent still runs. That is why the append-only audit stays in my process.

session_id on the ResultMessage is how I join a CI fixture to a later replay. Official traces attach a session.id span attribute by default. Claude Code omits it if OTEL_METRICS_INCLUDE_SESSION_ID is falsy.

Retry vs Escalate

The SDK already retries structured-output validation. If the object never matches, I get error_max_structured_output_retries, not a best-effort blob. That is a harness retry. I am not inventing a second retry API on top of it.

The product question is narrower: which mistakes may the model fix, and which ones must a human see?

Failure class	Example	Retry?	Escalate?
Shape miss	JSON failed the schema	Yes. The SDK already does this.	After the retry cap.
Fixable arithmetic	Line items do not sum to the stated total	Yes, once, with the validator error in the next user turn	If the second pass still disagrees. Do not “correct” the source.
Transient tool error	Result class error (timeout, 5xx)	Yes, once, same args	If it fails again. Fail closed.
Honest empty	Result class not_found, or a required page is missing	No	incomplete or escalate. Never fill the field.
Denied call	Result class denied	No	Halt the write path. Same as Part 4.
Conflict	Two sources disagree on a date or a rate	No blend	Annotate both, then escalate.
Policy / harm	Fraud language, legal threat, a coverage fight	No	Human queue. Do not retry the classify.
Refusal	stop_reason == "refusal"	Official docs allow a fallback model. I still fail closed on extract.	Yes. Do not invent a workaround.

flowchart TB
    Out[structured_output or error subtype] --> Kind{Failure class}
    Kind -->|shape / arithmetic / transient| Retry[Retry once]
    Retry --> Ok{Valid and consistent?}
    Ok -->|yes| Accept[Accept]
    Ok -->|no| Human[Escalate]
    Kind -->|empty / denied / conflict / policy / refusal| Human
    Kind -->|extracted and gold matches| Accept

    classDef blueClass fill:#4A90E2,stroke:#333,stroke-width:2px,color:#fff
    classDef orangeClass fill:#F39C12,stroke:#333,stroke-width:2px,color:#fff
    classDef greenClass fill:#27AE60,stroke:#333,stroke-width:2px,color:#fff
    classDef redClass fill:#E74C3C,stroke:#333,stroke-width:2px,color:#fff
    classDef purpleClass fill:#9B59B6,stroke:#333,stroke-width:2px,color:#fff

    class Out blueClass
    class Kind orangeClass
    class Retry purpleClass
    class Ok orangeClass
    class Accept greenClass
    class Human redClass

Retry is for mistakes the model can see in the next window: a schema error string, a validator diff, a transient error class. Escalate is for missing evidence, a deny, a conflict, or a case I do not want a cheaper model to close. I do not retry a skip into an extract. That is how a silent failure gets a second chance to look like success.

Three Small Analogs

Course 3 trains this on insurance extraction, mortgage documents, and multi-source supply-chain risk. I am not reprinting those pipelines. Invented analogs only. Same fail-closed rule I used on shelfwatch: a skip is better than a blended guess.

Harbor Credit: extract, refuse to fabricate

Fictional lender. One messy application packet. The agent may read a stubbed fetch_packet tool and return the schema above.

Goal: fill the fields the packet actually states.
Definition of done: status extracted with cents and basis points, or incomplete / escalate.
Fail closed when: a required page is missing, a total does not match the line items,
  or fetch_packet returns not_found / error / denied.
Never: invent a rate, average two disagreeing figures, or coerce a skip into extracted.

Fixtures I will actually score (invented):

Fixture	Gold status	What a silent failure looks like
Complete packet, totals match	extracted	Extra fields the page never had
Rate page missing	incomplete, rate_bps: null	A made-up rate_bps that still schema-validates
Line items sum to 44900000, header says 45000000	retry once, then escalate	The model “corrects” the header so the object is clean
fetch_packet denied	escalate	A guessed packet assembled from the prompt

Response eval catches the invented rate. Step eval catches the deny that was ignored. Trajectory eval catches the “corrected” total: the path had a validator error and still ended extracted.

Northline Mutual: retry the fixable, escalate the rest

Fictional carrier. One inbound policy PDF stub. The harness routes renew / refer / escalate.

Goal: a structured route, plus a reason a human can audit.
Retry when: schema miss, one arithmetic mismatch, one transient tool error.
Escalate when: coverage language conflicts, a missing declarations page,
  fraud or legal wording, or the retry cap fires.
Do not retry a refer into a renew.

The structured object is the route. The score is whether the route matches the fixture policy, and whether a retry happened only on the allowed classes. A trajectory that retries a coverage conflict is a fail even if the last object says escalate.

Keel Freight: fuse evidence, keep provenance

Fictional briefing. Four disagreeing stubs: carrier ETA, warehouse scan, weather bulletin, customs hold.

Goal: one briefing object with one row per source, plus a conflicts list.
Definition of done: every source is present or marked failed, conflicts are named,
  escalate_on is filled when two timestamps or two risk flags disagree.
Never: average ETAs, drop a failed source, or emit a single "most likely" date.

Provenance is a field, not a paragraph. Each row keeps source, observed_at, and value. The conflicts array points at the two rows. If customs is down, that row is error and the briefing still ships. The silent failure here is a blended Wednesday that no source stated.

Quality on this analog is not “did we pick the right date.” It is “did we refuse to pick.” Trajectory eval asks whether a failed source stayed visible.

Putting the Concepts into Practice

Pick one workflow that already emits a decision (extract, route, a briefing) and write this brief before you add a score:

Definition of done (the structured object):
Gold fixtures (happy, missing field, conflict, deny, transient error):
Response assertions (fields that must match, fields that must stay null):
Step assertions (tool, args shape, result class):
Trajectory assertions (allowed retries, required escalate):
Budgets (max_turns, max_budget_usd, duration_ms you will page on):
What you will log (audit line + ResultMessage fields):
What you will export (OTEL off, metrics only, or traces beta):
What fail-closed means (do not guess X):

Then inspect a real run:

1. Is structured_output present on subtype == "success", or did you score result text?
2. Did any fixture pass response eval and fail step eval (wrong tool, ignored deny)?
3. Did a schema-valid object invent a field the source never had?
4. If you force not_found on the only evidence tool, does the status become incomplete or escalate instead of a guess?
5. If traces (beta) are on, does a claude_code.tool span exist for every audit line? Metrics or log events alone will not emit that span. If a span is missing, trust the audit.

If you cannot answer those, you have a typed demo, not an evaluated agent.

Key Terms

• Evaluation: How I score a run against a fixture and a policy
• Observability: How I see a run: the working-model audit, ResultMessage fields, and optional OTEL export
• Silent failure: A finished-looking run that skipped, guessed, or blended when it should have stopped
• output_format / outputFormat: Agent SDK knob. {"type": "json_schema", "schema": ...}. Draft-07.
• output_config.format: Messages API equivalent
• structured_output: Validated JSON on a successful ResultMessage. Absent means do not branch.
• ResultMessage.subtype: success, error_max_turns, error_max_budget_usd, error_during_execution, error_max_structured_output_retries
• num_turns / duration_ms / total_cost_usd: Official system metrics on the result. total_cost_usd is a client-side estimate.
• Response / step / trajectory: Working-model eval units. Final object, one hop, or the whole path.
• Working-model trace: Tool name, args, result class, stop_reason, structured_output
• OTEL export: Official CLI telemetry (CLAUDE_CODE_ENABLE_TELEMETRY). Traces are beta (CLAUDE_CODE_ENHANCED_TELEMETRY_BETA).
• claude_code.interaction / claude_code.llm_request / claude_code.tool: Official span names when traces are on
• Retry vs escalate: Fixable mistakes go back to the model once. Missing evidence, denies, conflicts, and policy cases go to a human.
• Fail closed: Skip, mark incomplete, or escalate when evidence is missing. Do not guess.
• Provenance: Keep the source next to the value. Do not blend disagreements into one number.

Final Thoughts

The model is still the reasoning engine from Part 1. The harness is still the runtime from Part 3. MCP is still the governed surface from Part 4. None of that tells me the loop did the job.

Structured output is the seam: the object I branch on, and the object I score. Observability is the audit plus, if I opt in, official OTEL spans. Evaluation is the fixture, the three units, and the retry-versus-escalate rule. Schema-valid JSON that is still wrong is the failure mode I now look for first.

The next post moves from scoring the loop to bounding it: autonomy I am willing to grant, and guardrails that hold even when the model would rather keep going.

---

This is Part 5 of 6 in the Agentic Coding with Claude Code series. Earlier: Part 1, Part 2, Part 3, Part 4.