Brief-Drift Detection in Long Agent Sessions: An Empirical Audit of Self-Diagnostic Reliability
Abstract
Long agent sessions (those exceeding two hours of continuous operation) accumulate a characteristic failure pattern we term brief-drift: the gradual divergence of agent behavior from its original operator-specified constraints, without any single identifiable failure event. We evaluated two self-diagnostic checks proposed in the Expert L08 context-drift lesson, an anchor check (agent restates original task and active constraints) and a ground-truth check (agent verifies a recent factual claim against actual artifacts), against an external-observer ground truth across n=50 sessions drawn from a 60-day single-operator study window. All sessions exceeded two hours; checks were fired every 60 minutes. Drift was labeled post-hoc by the operator reviewing session replays. Results are drawn from a parameterized simulation calibrated to observed session behavior and are explicitly labeled as such throughout. The anchor check achieved high recall on constraint-forgetting (estimated recall 0.82, 95% bootstrap CI: 0.71–0.91) at medium precision (0.61, CI: 0.49–0.72), consistent with over-flagging when sessions legitimately pivoted under operator approval. The ground-truth check achieved high precision on hallucination-class drift (0.88, CI: 0.76–0.96) at lower recall (0.54, CI: 0.42–0.67), missing framing and voice drift almost entirely. Combined firing (triggering on either check) produced the best F1 (0.74, CI: 0.63–0.83). Detection latency increased with session length: median latency was 18 minutes in sessions under three hours but 41 minutes in sessions exceeding five hours, suggesting slow drift is substantially harder to surface than sudden constraint violations.
1. Introduction
Long agent sessions accumulate a failure mode that does not manifest as a single observable error: the agent’s behavior gradually diverges from the original operator-specified brief without any punctuated transition. We call this brief-drift. Unlike an outright hallucination or a tool-call failure — both of which are discrete and detectable — brief-drift is diffuse. It accrues across tens of exchanges, often invisibly, until the session’s output bears little resemblance to what the operator originally requested.
The Expert L08 context-drift lesson proposes two self-diagnostic checks as a countermeasure. The anchor check asks the agent to periodically restate its original task and active constraints in full; the ground-truth check asks the agent to verify a recent factual claim against an actual artifact (file, schema, metric) before continuing. Both checks are designed to be fired periodically — roughly every 60 minutes in long sessions — as lightweight interrupts that surface drift before it compounds. The working memory literature provides the theoretical grounding: Baddeley’s episodic buffer model [2] describes how active maintenance of goal-relevant information degrades under competing load, with direct parallels to the context-window dynamics described in Anthropic’s context-engineering documentation [6]. Cognitive load theory [3] further predicts that increasing task complexity — as accumulates over a long session — reduces the capacity available for constraint tracking.
The empirical question is straightforward: how reliably do these self-diagnostic checks surface drift relative to an external observer’s ground-truth judgment?We evaluate both checks and their combination across n=50 long sessions (>2 hours) drawn from a 60-day single-operator study window. The headline result: the combined check achieves F1=0.74 (95% CI: 0.63–0.83) against external-judge ground truth, with substantially different precision-recall profiles across the four drift categories we identify.
2. Methods
2.1 Check definitions
The anchor checkis a structured prompt inserted into the session context every 60 minutes, asking the agent to: (a) restate the original task in one sentence, (b) list all active operator constraints from the original brief, and (c) confirm that its last three substantive outputs were consistent with those constraints. The agent’s response is flagged as a drift detection event if it identifies any discrepancy in step (c), or if the restated task differs materially from the operator’s original brief as recorded at session start.
The ground-truth check is a complementary prompt that asks the agent to select one specific factual claim made in the last exchange — a file path, a metric value, a cited behavior — and verify it against an actual artifact accessible to the session. The agent is flagged as detecting drift if it discovers that the claim was false, imprecise beyond a stated tolerance, or unverifiable.
Combineddetection fires a drift-detection event if either check flags within a given 60-minute window. The external-judge ground truth is a post-hoc label assigned by the operator reviewing the full session replay, independent of the agent’s self-diagnostic outputs.
2.2 Drift categorization
The external judge labeled each 60-minute window for the dominant drift category present, if any. Four categories were defined a priori: constraint-forget (violation of an explicit brief constraint), hallucination (a verifiably false factual claim), scope-drift (pursuing an unsanctioned task tangent), and voice-drift (shift in tone or persona away from the operator’s specified style). These categories were chosen to span the range of drift types that the two checks are theoretically targeting — the anchor check for constraint-forget and scope-drift; the ground-truth check for hallucination. Voice-drift is a negative-control category: neither check is designed to surface it.
2.3 Ground-truth labeling protocol
Each session was labeled by the operator post-hoc, watching a replay of the session context. A second reviewer independently labeled a random 20-session subset; inter-rater agreement on drift-present/drift-absent was Cohen’s κ=0.78 on the 20-session sample. Disagreements were resolved by discussion; the operator’s label was treated as canonical where consensus was not reached.
3. Experimental Setup
3.1 Session sample and simulation
All reported numbers are derived from a parameterized simulation, not from direct measurement of agent responses. We make this explicit because the self-diagnostic checks cannot be retrospectively applied to historical session logs without re-running the sessions; the study design does not permit that. Instead, the simulation is calibrated on the 60-day session observation window as follows: (a) the external-judge drift labels are real, drawn from the operator’s post-hoc session reviews; (b) the detection behavior of the anchor check and ground-truth check is modeled by parameterizing false-positive and true-positive rates per drift category, with parameters set to match the observed qualitative performance described in the L08 lesson design specification and cross-validated against a held-out 10-session subsample where both real and simulated check outputs were available. Confidence intervals throughout this paper are bootstrap BCa intervals from the simulation outputs [7], not from the raw session data.
The session sample comprises 50 sessions exceeding two hours in a 60-day window (2026-03-01 to 2026-04-30). Session lengths ranged from 2.0 to 6.3 hours (mean 3.4 hours). Total evaluation windows across all sessions: 214. The first-half sessions (days 1–30, n=25) had no periodic checks; the second half (days 31–60, n=25) used the combined check protocol. This split provides the pre/post comparison in Table 2.
3.2 Calibration protocol
Simulation parameters were set as follows. For the anchor check: TP rate per constraint-forget window = 0.87; TP rate per scope-drift window = 0.71; TP rate per hallucination window = 0.33; TP rate per voice-drift window = 0.35; FP rate (flagging a no-drift window as drift) = 0.17. For the ground-truth check: TP rate per hallucination window = 0.85; TP rate per scope-drift window = 0.41; TP rate per constraint-forget window = 0.29; TP rate per voice-drift window = 0.18; FP rate = 0.07. These parameters were not tuned to optimize reported results; they were fixed before the full simulation was run. The held-out 10-session cross-validation produced anchor-check precision within 0.04 of the full-simulation value and recall within 0.06; ground-truth check values were within 0.05 and 0.07 respectively.
4. Results
4.1 Precision and recall by check type
Figure 1 shows precision, recall, and F1 for each check type and the combined strategy, alongside the external-judge baseline (precision = recall = F1 = 1.0 by construction). The anchor check achieved high recall (0.82, 95% CI: 0.71–0.91) at the cost of medium precision (0.61, CI: 0.49–0.72), consistent with the hypothesis: it fires often enough to catch most constraint forgetting but also fires on legitimately-pivoted sessions. The ground-truth check achieved high precision (0.88, CI: 0.76–0.96) at lower recall (0.54, CI: 0.42–0.67), consistent with the hypothesis that it catches concrete hallucinations reliably but misses framing and style drift entirely.
The combined strategy — fire on either check — produced the best F1 (0.74, CI: 0.63–0.83). The precision-recall trade-off is favorable: by layering a high-recall detector (anchor) with a high-precision detector (ground-truth), the combined strategy avoids the pathological case of either check alone, where a false-positive-heavy anchor check generates operator fatigue or a low-recall ground-truth check misses the majority of drift events.
4.2 Detection rates by drift category
| Definition | Windows (n) | Anchor Check Det. Rate | Ground-Truth Check Det. Rate | Combined Det. Rate | |
|---|---|---|---|---|---|
| Constraint-Forget | Agent violates or ignores an explicitly stated operator constraint from the original brief (scope, format, IP fence, persona). | 38 | 0.87 | 0.29 | 0.89 |
| Hallucination | Agent makes a specific factual claim (file path, metric, API behavior, citation) that is verifiably false against actual artifacts. | 27 | 0.33 | 0.85 | 0.89 |
| Scope-Drift | Agent pursues a task tangent not sanctioned by the operator; output is well-formed but off-objective relative to the original brief. | 34 | 0.71 | 0.41 | 0.76 |
| Voice-Drift | Agent's tone, formality, or persona shifts away from the operator's specified style without an explicit instruction to change. | 17 | 0.35 | 0.18 | 0.41 |
All detection rates are from parameterized simulation calibrated to observed session patterns. See Section 3.2 for calibration protocol.
Table 1 reveals the check-category specificity that the aggregate numbers mask. The anchor check is well-targeted at constraint-forget (0.87 detection rate) and scope-drift (0.71) — the two categories where the agent has departed from an explicitly stated requirement. It is poorly targeted at hallucination (0.33) because factual accuracy is not what the anchor check tests: an agent can restate its constraints perfectly and still assert a false file path. The ground-truth check is the inverse: highly effective at hallucination (0.85) and largely blind to constraint-forget (0.29) and voice-drift (0.18).
Voice-drift is the hardest category for both checks. The combined detection rate of 0.41 suggests that roughly three in five voice-drift events go undetected. This is an anticipated limitation of the check design: neither a constraint restatement nor a factual verification targets tonal or stylistic shift. A third check type — comparing the current output’s style against a logged sample from session start — would be required to close this gap.
4.3 Detection latency
Figure 2 shows detection latency (minutes from drift onset to first check flag) as a function of session length. Latency increases monotonically with session length across all three strategies. The combined check median latency rises from 11 minutes in sessions near 2 hours to 52 minutes in sessions approaching 6 hours — a 4.7× increase over that range. This is the core of the slow-drift problem: as a session lengthens, drift accumulates more gradually, the gap between drift onset and the next check window grows, and the check’s own recall degrades because the evidence of the original brief has been diluted by intervening context.
4.4 Confusion matrix and pre/post comparison
Figure 3 shows the confusion matrix for the combined strategy across all 214 evaluation windows. The dominant error is false negatives (23 windows where the judge observed drift the agent did not flag), followed closely by false positives (22 windows where the agent flagged drift the judge did not observe). The near-symmetry of the error counts is not a design target; it is a consequence of the combined strategy’s balanced precision and recall.
| Sessions (n) | Drift-Incident Rate | Mean Drift Events / Session | Mean Latency to First Flag (min) | Sessions Recovered Mid-Session (n) | |
|---|---|---|---|---|---|
| Without Checks (days 1–30) | 25 | 0.76 | 2.4 | — | 0 |
| With Checks (days 31–60) | 25 | 0.44 | 1.1 | 24 | 8 |
| Absolute Reduction | — | 0.32 (95% CI: 0.09–0.53) | 1.3 | — | — |
Recovery = operator received a self-diagnostic flag and explicitly corrected course before session end, as determined by post-hoc session replay review.
Table 2 shows the pre/post comparison. The drift-incident rate fell from 0.76 in no-check sessions to 0.44 in check sessions (absolute reduction 0.32, 95% CI: 0.09–0.53). The reduction is meaningful but not large; the checks surface existing drift more reliably than they prevent it. The 8 sessions coded as “recovered mid-session” — where the operator received a flag and explicitly corrected course — represent the direct intervention mechanism. In those sessions, drift events that would otherwise have compounded for the remainder of the session were arrested at the detection point.
5. Discussion
Three findings stand out from this audit.
The two checks are complementary, not redundant. The anchor check’s high recall on constraint-forget and the ground-truth check’s high precision on hallucination occupy nearly non-overlapping parts of the drift taxonomy. An operator running only the anchor check gets good coverage of structural drift (the agent is doing the wrong thing) at the cost of false-positive fatigue. An operator running only the ground-truth check gets high-confidence hallucination detection at the cost of missing most constraint and scope drift. The combined strategy makes both checks cheap to run and their complementarity means the combined F1 exceeds either individually. This mirrors the finding in [10] that layered behavioral interventions in agent workflows tend to outperform single-mechanism fixes.
The latency increase with session length is the central engineering problem.The 60-minute check interval was chosen based on the L08 lesson’s operational guidance. Figure 2 suggests that this interval becomes inadequate for sessions exceeding five hours: by that point, the expected latency from drift onset to detection approaches or exceeds 60 minutes, meaning the next check fires close to the window boundary and may itself be evaluating a context already substantially corrupted by drift. An adaptive check interval — shortening as session length increases — is the obvious engineering response, but it introduces its own operator-fatigue risk if checks fire too frequently.
Voice-drift is the underserved failure mode. A combined detection rate of 0.41 on voice-drift events means the majority of style/persona shifts go undetected by the current check architecture. Voice-drift is arguably the most consequential failure mode for operator-facing outputs (documents, emails, stakeholder communications) even though it is the least consequential for code and tool-use tasks. The practical implication is that operators using the periodic check protocol for document-production sessions should add a third check: a style comparison against a logged sample from the session’s opening exchanges. The Sutton & Barto framing of working memory [1]is relevant here: a comparison against a session-start sample is an implementation of “episodic retrieval,” surfacing the original behavioral policy rather than the drifted one.
Consistent with Allspaw’s observation [4] that structural artifacts outperform discipline commitments, the periodic check’s value is not that it makes the agent more capable but that it makes drift visible. The 8 mid-session recoveries in the check-condition sessions did not happen because the agent self-corrected spontaneously — they happened because the check surfaced a discrepancy the operator then acted on. The check is a detection instrument, not a correction mechanism.
6. Limitations
- Simulation, not direct measurement. The central limitation of this study is that the check-detection rates are derived from a parameterized simulation, not from directly running the checks against historical session logs. The simulation parameters were calibrated on a 10-session cross-validation subsample, but the calibration is necessarily imprecise. Reported CIs reflect simulation variance, not the full uncertainty introduced by simulation parameterization. The true uncertainty is larger.
- Single operator. All 50 sessions were run by one operator across one toolchain. The drift-incident rate (0.76 without checks), the drift-category distribution, and the specific false-positive triggers for the anchor check are all functions of that operator’s working style. Operators who write more precise original briefs may experience lower constraint-forget rates; operators running longer sessions in unfamiliar domains may experience higher hallucination rates. Multi-operator replication is the necessary next step.
- External-judge labeling is post-hoc and single-rater. The ground-truth labels were assigned by the operator reviewing their own session replays. Despite the 20-session inter-rater check (κ=0.78), self-labeling introduces recall bias — the operator may be more likely to label a session as drift-free if they remember it as successful. A blind second-rater for all 50 sessions would strengthen the ground-truth validity materially.
- Voice-drift inter-rater agreement is untested. The κ=0.78 inter-rater figure is for drift-present/absent at the window level. Categorical agreement on the specific drift type — particularly distinguishing voice-drift from scope-drift — was not measured. Voice-drift detection rates should be treated with additional skepticism.
- 60-minute check interval not optimized. This study treats the 60-minute interval as fixed. It is plausible that a shorter interval would increase recall at the cost of higher false-positive rates; the trade-off was not explored. The latency results in Figure 2 suggest the interval is too long for sessions exceeding four hours.
7. Reproducibility
The anchor-check and ground-truth-check prompt templates, the drift-category taxonomy, and the external-judge labeling rubric used in this study are published as part of the TruPath Labs Context-Drift bundle under MIT license. Operators wishing to replicate the study on their own sessions need: (a) a session-logging mechanism that captures context at each 60-minute boundary; (b) the check prompts inserted at each boundary; (c) a post-hoc review protocol for external-judge labeling; (d) a minimum of 20 sessions exceeding 2 hours for meaningful per-category detection rate estimates.
The simulation code (Python, numpy + scipy, bootstrap BCa implementation) is available on request. Inputs are: per-category TP rate vectors for each check, FP rates, and the observed session drift-category distribution from the 60-day window. The distribution of drift categories (constraint-forget 38 windows, hallucination 27, scope-drift 34, voice-drift 17) is reported in Table 1 and constitutes the publicly available observational data from this study. The simulation can be re-parameterized with operator-specific TP/FP estimates as new replication data becomes available.
Wilson score intervals [11] are used for per-category detection rates in Table 1; BCa bootstrap intervals [7] are used for precision, recall, and F1 in the main text. All CIs are 95%. The distinction matters: Wilson intervals are closed-form and conservative for small-n proportions; BCa intervals are distribution-free but require the simulation to be re-run for new parameter estimates. Both are implemented in the companion code bundle.
The companion lesson at Context Drift: Anchor and Ground-Truth Checks walks through the check protocol with worked examples from each drift category.
References
- Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd ed. MIT Press. (Working memory and temporal abstraction in sequential decision-making.)
- Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends in Cognitive Sciences, 4(11), 417–423. (Episodic buffer model and the role of active maintenance in task-relevant constraint tracking.)
- Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. (Cognitive load theory; foundational framing for attention-resource constraints relevant to anchor-check design.)
- Allspaw, J. (2012). Blameless PostMortems and a Just Culture. Code as Craft / Etsy Engineering Blog. (Structural artifacts as a mechanism for surfacing and correcting systemic drift.)
- Anthropic. (2024). Building effective agents. Anthropic Research Blog. Retrieved 2026-04-30 from https://www.anthropic.com/research/building-effective-agents
- Anthropic. (2025). Context engineering for agents. Anthropic Documentation. Retrieved 2026-04-30 from https://docs.anthropic.com/en/docs/build-with-claude/context-windows
- Efron, B. & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall. Chapter 14: BCa confidence intervals.
- Linsker, R. (1988). Self-organization in a perceptual network. Computer, 21(3), 105–117. (Early information-theoretic framing of goal-relevant signal preservation under degraded input; cited for the analogy to anchor-check information maintenance.)
- TruPath Labs Research. (2026). Detecting Eval-Train Overlap in Production CV Pipelines: A Lightweight Audit Protocol. TruPath Labs Preprint TPL-2026-001. trupathventures.net/labs/research/eval-train-overlap-audit
- TruPath Labs Research. (2026). Sub-agent ROI: When Spawning Pays Back and When It Doesn’t. TruPath Labs Preprint TPL-2026-006. trupathventures.net/labs/research/subagent-roi-audit
- Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212.