Skip to content
SearchSubscribe
Expert · Lesson 11 — Production debugging playbook
E11Expert
Expert · Lesson 11● live

Production debugging playbook

When an agent misbehaves in prod: the diagnostic ladder. Five rungs from “claim says done but isn’t” through context drift, tool failures, schema mismatches, upstream model regressions.

20 min read · 60 min applyprereq: Expert 04 (accountability)

The broken agent problem

Something is wrong. An agent that was working last week is producing bad outputs today. Your first instinct is to ask why — and the instinct immediately generates hypotheses: the model API changed, the data is different, the prompt is drifting. Each hypothesis feels plausible. Each one points to a different investigation. Without a systematic approach, you’re picking among them by intuition, and intuition in production debugging is expensive.

The core insight: production failures in agent systems form a natural ordering by diagnosis cost. The cheapest failure to detect (claim/artifact mismatch) is also the most common. The most expensive (upstream model regression) is the rarest. Operators who work from the top of the cost ordering to the bottom spend hours on unlikely causes before reaching the probable one. The diagnostic ladder inverts this: start cheap, climb only when the cheaper rungs are genuinely clear.

The ladder has five rungs. Each rung takes roughly five minutes to clear. A complete five-rung climb costs approximately 25 minutes. In most incidents the ladder stops before rung 5 — often before rung 3. The 25-minute ceiling is the worst case, not the median.

RungFailure classTime to clearPrevalence
1Claim verification — agent says done but artifact doesn’t match~2 minMost common
2Context drift — long session has decayed constraints~5 minCommon in sessions >2 hrs
3Tool failures — a dependency is silently returning wrong data~5 minCommon when tools touch external APIs
4Schema mismatches — agent’s data model inconsistent with prod~5 minCommon after data model changes
5Upstream model regression — model itself has changed~10 minRare; never the first hypothesis

The ladder is not a guarantee of a fast resolution. Some failures are genuinely hard. Rung 5 can consume significant investigation time when it is the actual cause. What the ladder prevents is spending rung-5 time on rung-1 problems — which, in practice, is the most common way production debugging goes wrong.

The diagnostic ladder

Rung 1: Claim verification

The agent says it finished. Verify the artifact directly. Do not read the agent’s summary of what it produced — open the file, query the record, load the URL, run the test. Does the artifact match the claim?

This is the most common failure mode in production agent systems. An agent can believe with complete internal consistency that a write succeeded when the write never landed. It can summarize a document it hallucinated rather than fetched. It can report a test passed without running the test. None of these failures involve the agent lying — it genuinely believes the claim. The only check that catches them is direct artifact inspection.

Operating L01 covers the full claim-verification diagnostic. The habit it builds (spot-checking one agent claim per session against the actual artifact) catches this failure class before it reaches production. If that habit isn’t yet in place, most rung-1 failures become production incidents by default.

Time to clear: 2 minutes. If the artifact matches the claim, rung 1 is clear. Climb to rung 2.

Rung 2: Context drift

Long sessions degrade. A session running for several hours on a complex task accumulates context that can crowd out or override the original constraints. The agent doesn’t forget constraints deliberately — they simply become less salient as the session depth increases. Behavioral drift is the symptom: outputs that would have been correct at session start are subtly wrong by hour four.

The anchor check is the rung-2 diagnostic: ask the agent to restate its current constraints, then compare the restatement against the original brief or CLAUDE.md. Any divergence (a narrowed constraint, a missing rule, a reworded requirement that subtly changes behavior) is context drift. Expert L08 covers the detection patterns in depth. TPL-2026-007 provides the quantitative analysis: anchor-check detection rates by drift category, with constraint-forget (0.87 detection rate) and scope-drift (0.71) as the two categories it catches most reliably.

Fix: restart hygiene. If drift is detected, restart the session with the original brief and the artifact from where the session left off. Do not try to patch a drifted session in place — the drift is in the context window, not in any single output. Only a new session with a clean context baseline resolves it.

Time to clear: 5 minutes. If the anchor check shows no divergence, rung 2 is clear. Climb to rung 3.

Rung 3: Tool failures

A tool the agent depends on is silently returning wrong data. The failure is invisible from the agent’s perspective: the tool call succeeds (returns a response, no exception), but the response is incorrect — an HTTP 4xx absorbed as success, a stale cache returned instead of live data, an empty payload treated as a valid result. The agent reports success because its tool reported success. The tool reported success because the wrapper has no status-code validation.

The diagnostic is to check the tool’s own logs, not the agent’s description of the tool call. Every tool that calls an external endpoint should produce its own execution record — HTTP status codes, response sizes, latency. If the tool is a vendor API, check the vendor’s request log. If it’s an internal endpoint, check the endpoint’s access log. Compare what actually happened to what the agent reported.

Common rung-3 patterns: a rotated API key producing 401s that the wrapper swallows; a rate-limited endpoint returning 429s that the wrapper treats as empty success; a database endpoint returning a partial result set due to a timeout that the wrapper does not surface. In all cases, the agent is not at fault — the fault is in the reliability contract between the tool layer and the agent.

Time to clear: 5 minutes for tool logs inspection. Fixing a silent-failure tool is a separate task — but identifying it as the cause takes minutes, not hours.

Rung 4: Schema mismatches

The agent’s model of the data is internally consistent but inconsistent with the actual production data shape. This failure is invisible in development because dev data matches the schema the agent was designed around. It surfaces in production when the data has evolved: a field was renamed, a previously required field became nullable, a new field was added that the agent doesn’t know to look for.

The pattern is distinctive: the agent works correctly on development or sample data, fails on production data, and fails in a structured way — not randomly. The outputs are wrong in a consistent direction that traces to a specific field or type assumption.

The diagnostic is a direct comparison: dump a sample of actual production data and compare it field by field against the agent’s expected schema (documented in the agent’s brief, CLAUDE.md, or the tool definitions). TPL-2026-001 covers eval/train overlap as one specific form of schema mismatch — where the evaluation set shares characteristics with training data that don’t hold in production. The same diagnostic logic applies: the mismatch between what the agent was calibrated against and what it now receives.

Time to clear: 5 minutesfor a field-level comparison. Fixing the mismatch (updating the agent’s schema assumptions or the data pipeline) is a separate task.

Rung 5: Upstream model regression

The model itself has changed. A vendor shipped a silent update; a model version that was stable has been deprecated and replaced. The same prompts produce structurally different outputs. This is the rarest failure class and the hardest to diagnose because it requires a historical baseline to detect — you need to know what the outputs looked like before to see that they’ve changed now.

The diagnostic has two steps. First, pin the model version: if you are not specifying the exact model version in your API call, you may be consuming whatever the vendor’s “latest” alias points to, which changes without notice. Pinning to a specific version ID stops the bleeding. Second, compare outputs: take a representative prompt from the pre-failure period and run it against the current model. Compare the response structure, length, and key output fields against a stored historical example.

Rung 5 is the correct explanation for a small number of incidents. It is the hypothesis most commonly invoked at the start of debugging and the least often confirmed. The asymmetry is worth internalizing: when a production agent misbehaves, model regression is the last thing to check, not the first.

Time to clear: 10 minutes with a stored baseline. Without a stored baseline, rung 5 clearance requires rebuilding one from available context — which is a reason to maintain baselines proactively.

Three failure patterns

Where the debugging process itself goes wrong. These are not agent failures — they are operator failures in applying the ladder. Hover any card to see the diagnosis.

№ 01

The rung-skipper

claim looks like“An operator's CV pipeline starts producing incorrect outputs overnight. The operator immediately suspects the upstream model API has been updated — the vendor has shipped model changes without notice before. They spend four hours testing prompt variations, comparing outputs against historical baselines, and filing a support ticket. The actual cause: the pipeline's claim-logging code has a bug that marks every job as successful regardless of output. The artifact was never inspected. Rung 1 would have caught this in ten minutes.”
what’s missingThe operator skipped the bottom of the ladder because it felt too simple. Rung-skipping is ego-driven debugging: the operator assumes the failure is sophisticated because sophisticated failures are more interesting to investigate. Most production failures are not sophisticated. The majority trace to rung 1 or rung 2. Skipping the bottom costs hours on the top.
the moveMake rung 1 non-negotiable. Before any other investigation, verify the artifact the agent claims to have produced. Inspect it directly: open the file, query the record, load the URL. This takes two minutes. It costs nothing. Any failure that would have been caught here cannot justify the time spent on higher rungs.
№ 02

The same-rung loop

claim looks like“An operator notices their scheduling agent is producing incorrect time-zone conversions. They spend three hours on rung 1: checking the output artifact, verifying the agent's claim, comparing the reported and actual values. The claim verification keeps confirming the agent believes it is correct. The operator keeps re-running the same checks. The actual cause: the time-zone lookup library the agent calls through a tool is returning stale cached data. This is a rung 3 failure (silent tool malfunction). The operator never checked the tool's own logs.”
what’s missingPersistent failure at one rung is a signal to climb, not to repeat. Each rung has a resolution: either the rung explains the failure (stop here, fix here) or it doesn't (climb). If a rung produces no explanation after one complete pass, it is cleared. Staying on it compounds time cost without advancing toward a resolution. The ladder is a progression, not a loop.
the moveSet a time budget per rung: approximately five minutes for a complete pass. If the rung is clear after that pass, climb. Document what you checked and what you found — 'rung 1 clear: artifact inspected, contents verified against agent claim, no discrepancy' — and move on. The documentation prevents re-checking the same rung on the same incident.
№ 03

Escalation by frustration

claim looks like“An operator's data processing agent has been misbehaving intermittently for two days. Initial checks cleared rungs 1 and 2 quickly. Rung 3 (tool failures) produced one suspect: a vendor API that occasionally returns 200 with an empty payload. The operator verified the API issue once, filed a bug report with the vendor, and moved on. The intermittent failures continued. Rather than holding rung 3 until the API issue is confirmed resolved, the operator escalated to rung 4 and rung 5 — schema mismatches, model regression — spending eight hours on investigations that found nothing. The API was the issue throughout.”
what’s missingEscalation by frustration is what happens when the operator cannot confirm a rung 3 cause is fully resolved before climbing. The suspected cause was real but intermittent — hard to reproduce on demand. The operator treated 'I checked it once and it wasn't failing that time' as 'cleared.' Intermittent tool failures require sustained observation, not a single check. Frustration with the unresolved state produced premature escalation.
the moveA rung is not cleared until the failure mode it covers is demonstrably absent — not just absent at the moment you checked. For intermittent failures, rung clearance requires either: (a) a confirmed fix with no recurrence over a meaningful observation window, or (b) a controlled bypass that reproduces the same failure without the suspected tool. If neither is achievable quickly, hold the rung and instrument it — add logging, add health checks — before escalating.

The common thread: all three patterns involve operators applying the wrong diagnostic intensity to the wrong rung. The rung-skipper applies maximum intensity too high. The same-rung-looper applies repeated intensity to a cleared rung. The escalation-by-frustration operator leaves a rung before it is genuinely resolved. The ladder only works if each rung is climbed once, completely, and in order.

The rung-test

The decision gate for each climb. Before ascending from rung N to rung N+1, demonstrate that rung N does not explain the failure. The demonstration takes roughly five minutes per rung. A complete five-rung pass costs 25 minutes in the worst case.

The test is explicit: for each rung, run the specific check that would confirm or clear it. Document what you checked and what you found. A rung is cleared only when the check produces no evidence that the failure lives there. If the check produces ambiguous results, the rung is not cleared — stay and refine the check rather than climbing on uncertainty.

The full diagnostic ladder
Run the diagnostic ladder on this agent failure.

Failure description: [describe what the agent is doing wrong]
Session duration: [how long has this session been running?]
Last known good state: [when did it last work correctly?]
Any recent changes: [code changes, config updates, model updates, tool changes]

Work each rung in order. For each rung:
1. What would this rung look like if it were the cause?
2. What is the specific check that would confirm or clear it?
3. Run the check. What did you find?

RUNG 1 — Claim verification (~2 min)
Check: Inspect the actual artifact the agent claims to have produced.
Do NOT read the agent's summary — open the file, query the record,
load the URL. Does it match what the agent said?
Result: [ ] Matches claim (rung 1 clear) / [ ] Does not match (stop here)

RUNG 2 — Context drift (~5 min)
Check: How long has this session been running?
If > 2 hours on a complex task, run the anchor check from TPL-2026-007:
ask the agent to restate its current constraints and compare to the
original brief. Does anything differ?
Result: [ ] No drift detected (rung 2 clear) / [ ] Drift found (stop here)

RUNG 3 — Tool failures (~5 min)
Check: Look at the tool's own execution logs, not the agent's
description of tool results. Are any tools returning errors, empty
payloads, or wrong data that the agent is treating as success?
Common signals: HTTP non-2xx absorbed silently, empty response body
treated as valid data, stale cache returned instead of live data.
Result: [ ] All tools healthy (rung 3 clear) / [ ] Tool issue found (stop here)

RUNG 4 — Schema mismatches (~5 min)
Check: Does the agent's mental model of the data match the actual
data shape in production? Dump a sample of actual prod data and
compare field names, types, and value ranges to what the agent expects.
Common signals: works on dev data, fails on prod; field names changed;
nullable fields the agent treats as required.
Result: [ ] Schema consistent (rung 4 clear) / [ ] Mismatch found (stop here)

RUNG 5 — Upstream model regression (~10 min)
Check: Has the model version changed without your action?
Pin the model version explicitly if you haven't already. Compare
a representative prompt/response pair against a historical baseline.
Do outputs for identical inputs differ structurally?
Result: [ ] No regression detected / [ ] Regression confirmed (stop here)

OUTPUT: The first rung where you found a cause. That rung is
where the fix lives. Do not continue climbing once a cause is found.

Works each rung in order, documents the check and result, stops at the first rung where a cause is found. Run at the start of any production incident. Takes ~25 minutes for a complete five-rung pass; most incidents resolve before rung 4.

The ladder also works as a post-incident review tool. After any production failure is resolved, reconstruct which rung it would have resolved on and how much time was spent above that rung. The pattern across incidents reveals which rung catches the most failures in your system, and where the highest-payback preventive investment is.

Annotated transcript

Operator reports a broken agent and immediately hypothesizes model regression (rung 5). Agent walks down the ladder. Rung 1 confirms a claim/artifact mismatch in two minutes. Rung 3 identifies the cause (silent tool failure on a write endpoint). Rungs 4 and 5 are never reached. Click any pinned annotation to expand the “why this turn matters” commentary.

Diagnosing a production write failure

Three operator turns, three agent turns. Result: incident resolved at rung 3, cause identified (absorbed 401 errors on write tool), structural fix specified — before any model regression investigation begins.

  1. operator

    The agent is broken. It was working fine last week. Now it's producing wrong outputs and I can't figure out why. I'm starting to think the model API changed on us.

  2. agent

    Before we go to the API, let's work the ladder from the bottom. Most production failures resolve on the first two rungs — and those take under ten minutes to clear.

    Rung 1: claim verification. What exactly does the agent say it produced, and what does the actual artifact contain? Have you opened the output file, queried the record, or loaded the URL directly — not through the agent's summary of it?

  3. operator

    I've been looking at the agent's output summary. The agent says the records were updated. Let me actually query the database directly... okay. The records were not updated. The agent is claiming success on writes that didn't happen.

  4. agent

    Rung 1 caught it. The agent's claim diverges from the artifact — this is claim/artifact mismatch, the most common production failure mode.

    Two directions from here:

    1. The agent is failing silently on the write operation. The tool call appears to succeed from the agent's perspective but the write isn't landing. Check the tool's own logs — not the agent's summary of the tool call, but the actual execution log on the write tool or the database transaction log. Look for error codes that the tool is swallowing.

    2. The session has been running long and the write-confirmation logic has drifted. If this session is multi-hour, the agent may have drifted to a version of its confirmation logic that marks writes as successful without verifying them. See Expert L08 and TPL-2026-007 for the anchor-check pattern.

    Which is more likely given the context? How long has this session been running, and is the write tool one you control or a third-party integration?

  5. operator

    The session has been running for about six hours. It's a write tool we control — an internal REST endpoint. Let me check the endpoint logs... I'm seeing 401 Unauthorized errors on the write calls. But the agent is getting back a 200.

  6. agent

    That's rung 3: silent tool failure. The write endpoint is returning 401 but the agent's tool wrapper is either not checking the status code or is treating the response as success regardless of status. The agent sees a response and marks the write done.

    Fix at the tool layer, not the agent layer: the tool wrapper needs to check the HTTP status code and return a structured error to the agent when the status is not 2xx. Right now it's absorbing the 401 silently. Once the tool returns an error correctly, the agent will surface it rather than claiming success.

    The 401 itself needs a separate investigation — that's an auth credential issue, probably a rotated token or an expired API key. But the deeper fix is the tool's error handling. A tool that can receive a 401 and report success is a reliability debt that will surface again whenever auth rotates.

    You don't need rungs 4 or 5 here. Schema and model regression are not in play — the issue is upstream of the model entirely.

  7. operator

    That makes sense. The API key was rotated last week — I didn't update the tool config. And the tool wrapper does just check for a response body, not the status code. I'll fix both. Is there anything else I should do to prevent this class of failure?

  8. agent

    Two structural fixes worth adding while you're in the code:

    1. Status-code validation in every tool wrapper. Any tool that calls an external endpoint should treat non-2xx as an error, return a structured error object to the agent, and never silently absorb failure codes. This makes the agent a reliable reporter of what actually happened rather than a reporter of what it hoped happened.

    2. A rung-1 verification step in your post-session review. Operating L01 covers the claim-verification diagnostic in detail — the habit of spot-checking one agent claim per session against the actual artifact. That habit catches this class of failure before it runs six hours. Most claim/artifact mismatches are visible within the first 30 minutes if you're checking.

    The ladder took about 15 minutes this time. The model regression investigation you were heading toward would have taken most of the day and found nothing.

Production incident checklist

A structured walk for any production agent incident. Each rung has a specific check. Clear each rung before climbing. If a rung reveals the cause, stop — the fix lives at that rung, not above it.

PRODUCTION INCIDENT — DIAGNOSTIC LADDER CHECKLIST

BEFORE STARTING
[ ] Write down the failure description in one sentence
    (what the agent claims vs. what you observe)
[ ] Note: session duration, last known good state,
    any recent changes (code, config, model version,
    tool dependencies)
[ ] Time-box: 25 minutes max for a full ladder pass.
    If resolution takes longer, escalate to async
    investigation — do not debug indefinitely live.

RUNG 1 — CLAIM VERIFICATION (~2 min)
[ ] Identify the specific artifact the agent claims
    to have produced (file, record, URL, test result)
[ ] Inspect the artifact directly — do NOT use the
    agent's summary
[ ] Rung 1 clear if: artifact matches agent's claim
[ ] Rung 1 stop if: artifact diverges from claim

If stopped at rung 1:
[ ] Check tool execution logs for the write/create
    operation (not the agent's description of the call)
[ ] Look for silently absorbed errors in the tool layer
    (see rung 3 checks below — often the root cause
    of rung 1 claim/artifact mismatch)

RUNG 2 — CONTEXT DRIFT (~5 min)
[ ] Check session duration. If < 1 hour on simple
    task: rung 2 unlikely, but still run anchor check.
[ ] Run anchor check: ask agent to restate its current
    constraints on the failing behavior
[ ] Compare restatement to original brief / CLAUDE.md
    verbatim
[ ] Rung 2 clear if: restatement matches original
[ ] Rung 2 stop if: any constraint narrowed, broadened,
    missing, or reworded in behavior-changing way

If stopped at rung 2:
[ ] Restart session with original brief + current
    artifact state
[ ] Do NOT patch a drifted session in place
[ ] See Expert L08 and TPL-2026-007 for restart hygiene

RUNG 3 — TOOL FAILURES (~5 min)
[ ] List tools the agent called in the failing session
[ ] For each tool: open the tool's own execution log
    (not the agent's summary of the call)
[ ] Check for: non-2xx status codes, empty payloads,
    stale cache responses, timeout-truncated results
[ ] Bypass test: call the tool directly with the same
    inputs and compare raw response to what agent got
[ ] Rung 3 clear if: all tools healthy on direct check
[ ] Rung 3 stop if: any tool showing absorbed errors
    or incorrect data

If stopped at rung 3:
[ ] Fix is at the tool wrapper layer — not the agent
[ ] Add status-code validation to the failing tool
[ ] Identify the credential/API issue separately
[ ] Do not climb to rung 4 until tool is repaired
    and verified

RUNG 4 — SCHEMA MISMATCHES (~5 min)
[ ] Dump a sample of actual production data
[ ] Compare field by field to agent's expected schema
    (documented in brief, CLAUDE.md, or tool defs)
[ ] Look for: renamed fields, changed types, newly
    nullable fields, added fields agent doesn't handle
[ ] Rung 4 clear if: prod data matches agent's schema
[ ] Rung 4 stop if: any structural mismatch found

If stopped at rung 4:
[ ] Fix is at the schema/brief layer — update agent's
    data expectations to match prod reality
[ ] Or fix the data pipeline if the shape change
    was unintentional

RUNG 5 — UPSTREAM MODEL REGRESSION (~10 min)
[ ] Confirm you are pinning an explicit model version
    (not "latest" or an alias that floats)
[ ] If not pinned: pin now and re-run. Observe if
    behavior changes.
[ ] Compare a representative prompt/response pair
    to a stored historical baseline
[ ] If no baseline exists: reconstruct from available
    session logs, then store for future incidents
[ ] Rung 5 stop if: outputs for identical inputs
    differ structurally from baseline

If stopped at rung 5:
[ ] Lock to the last known good model version
[ ] Document the regression: prompt, baseline output,
    current output, model versions compared
[ ] Check vendor release notes / model changelog
[ ] Do not update model version in production until
    you have tested the new version against your
    representative prompt set

POST-INCIDENT
[ ] Document: which rung resolved it, time spent
    per rung, time spent above the resolution rung
[ ] If time above resolution rung > 15 min: flag
    for postmortem — that's a ladder discipline failure
[ ] Update CLAUDE.md with any structural fix applied
    (tool wrapper change, schema update, restart rule)
[ ] If rung 1 resolved it: add artifact spot-check
    to post-session review habit (see Operating L01)

The post-incident log is the compounding mechanism. Tracking which rung resolves each incident builds a failure distribution for your specific system. Most teams find 60-70% of incidents resolve at rungs 1-2 after the first month of tracking. That number drives where preventive investment pays most: rung-1 habit formation and rung-2 restart discipline return more per hour than rung-5 model-pinning infrastructure.

Prompt kit

Three prompts: the full five-rung ladder, the rung-2 anchor check, and the rung-3 tool health audit. Save in your CLAUDE.md or a personal incident response snippets file.

The full diagnostic ladder
Run the diagnostic ladder on this agent failure.

Failure description: [describe what the agent is doing wrong]
Session duration: [how long has this session been running?]
Last known good state: [when did it last work correctly?]
Any recent changes: [code changes, config updates, model updates, tool changes]

Work each rung in order. For each rung:
1. What would this rung look like if it were the cause?
2. What is the specific check that would confirm or clear it?
3. Run the check. What did you find?

RUNG 1 — Claim verification (~2 min)
Check: Inspect the actual artifact the agent claims to have produced.
Do NOT read the agent's summary — open the file, query the record,
load the URL. Does it match what the agent said?
Result: [ ] Matches claim (rung 1 clear) / [ ] Does not match (stop here)

RUNG 2 — Context drift (~5 min)
Check: How long has this session been running?
If > 2 hours on a complex task, run the anchor check from TPL-2026-007:
ask the agent to restate its current constraints and compare to the
original brief. Does anything differ?
Result: [ ] No drift detected (rung 2 clear) / [ ] Drift found (stop here)

RUNG 3 — Tool failures (~5 min)
Check: Look at the tool's own execution logs, not the agent's
description of tool results. Are any tools returning errors, empty
payloads, or wrong data that the agent is treating as success?
Common signals: HTTP non-2xx absorbed silently, empty response body
treated as valid data, stale cache returned instead of live data.
Result: [ ] All tools healthy (rung 3 clear) / [ ] Tool issue found (stop here)

RUNG 4 — Schema mismatches (~5 min)
Check: Does the agent's mental model of the data match the actual
data shape in production? Dump a sample of actual prod data and
compare field names, types, and value ranges to what the agent expects.
Common signals: works on dev data, fails on prod; field names changed;
nullable fields the agent treats as required.
Result: [ ] Schema consistent (rung 4 clear) / [ ] Mismatch found (stop here)

RUNG 5 — Upstream model regression (~10 min)
Check: Has the model version changed without your action?
Pin the model version explicitly if you haven't already. Compare
a representative prompt/response pair against a historical baseline.
Do outputs for identical inputs differ structurally?
Result: [ ] No regression detected / [ ] Regression confirmed (stop here)

OUTPUT: The first rung where you found a cause. That rung is
where the fix lives. Do not continue climbing once a cause is found.
Rung 2 anchor check (context drift)
Run an anchor check on this session before climbing to rung 2.

SESSION CONTEXT
Original brief or CLAUDE.md constraints (paste relevant section):
[paste]

Current session duration: [estimate]
Task at the point of failure: [describe]

Ask the agent the following questions in sequence:
1. "What are your current constraints on [the failing behavior]?"
2. "What does the original brief say about [the failing behavior]?"
3. "Do your answers to 1 and 2 differ in any way?"

Compare the agent's answers to the original brief text directly.
Look for:
- Constraints that appear in the original but are absent from the agent's restatement
- Constraints the agent has narrowed, broadened, or reworded in a way that changes behavior
- New constraints the agent has introduced that weren't in the original

If any divergence is found: restart the session with the original brief.
If no divergence: rung 2 is clear. Proceed to rung 3.

Output: divergence found (describe) or rung 2 clear.
Rung 3 tool health audit
Audit the tools this agent depends on for rung 3 verification.

List the tools the agent called in the session leading up to the failure:
[list tool names or describe what they do]

For each tool, check:
1. Does the tool have its own execution log separate from the agent's session?
   If yes: open it. Look for errors, warnings, unexpected status codes.
2. Can you call the tool directly (bypassing the agent) with the same
   inputs and verify the output manually?
   If yes: do it. Compare the raw tool response to what the agent reported.
3. Are there any HTTP status codes the tool wrapper might be absorbing
   (non-2xx treated as success, empty payload treated as valid data)?

Flag any tool where:
- Its own log shows errors the agent didn't report
- A direct call produces different output than what the agent described
- The wrapper has no status-code validation

Output: for each tool — healthy / suspect (describe) / not inspectable (reason).
If any tool is suspect: rung 3 is not clear. Fix the tool before climbing.

Apply this — run the ladder on a real incident

~60-minute exercise.One real incident, one complete ladder pass, one post-incident log entry. Don’t investigate above the rung where the cause lives.

Your first ladder-disciplined incident response

Each step takes 10-15 minutes. Progress saves automatically.

0/5
  1. 01The next time an agent produces unexpected output, run rung 1 before anything else. Inspect the actual artifact — do not read the agent's summary of it.This takes two minutes. Set a personal rule: no investigation above rung 1 until rung 1 is cleared. Track how many incidents resolve at rung 1 over the next month. Most operators find the number is higher than expected.
  2. 02Find one recent incident where you spent significant time debugging. Reconstruct where it would have resolved on the diagnostic ladder. Calculate the time you spent above that rung.Use the full ladder prompt from the kit as a retrospective template. The goal is calibration: knowing which rung catches your most common failure class makes future incidents faster to resolve. This exercise usually surfaces one or two recurring failure classes.
  3. 03Audit the error-handling in every tool wrapper your agents depend on. For each tool that calls an external endpoint: does the wrapper check HTTP status codes? Does it return a structured error to the agent on non-2xx? If not, fix it.Silent tool failures (rung 3) are the most common cause of claim/artifact mismatch (rung 1). Fixing the tool layer prevents the surface symptom. A tool that reports success on a 401 or 503 is a reliability debt that will surface repeatedly.
  4. 04Add the rung-1 verification habit to your post-session review: once per session, pick one agent claim at random and verify it directly against the artifact. Log the result.Operating L01 covers the full claim-verification diagnostic. This exercise converts that lesson into a habit. After 10 sessions, the habit is established and requires no deliberate effort. The compounding effect is that most claim/artifact mismatches are caught within the session rather than discovered in production.
  5. 05Write a one-paragraph rule for your CLAUDE.md that governs how your agents report tool results. Include: status-code validation requirements, what counts as a successful write, and the structured error format for failed calls.This converts the rung 3 discipline into structural enforcement. Future agents read the rule; they don't rely on the wrapper catching errors. The rule and the wrapper together create a two-layer defense against silent tool failure.
Expert tier · what's next

After this lesson