The Sprint Contract Effect: Measuring Overrun Reduction in Solo-Operator Engineering
Abstract
Solo operators running agent-mediated engineering work routinely overrun sprint estimates by 50-200% of the originally scoped time. We measured the effect of adopting a structured sprint-contract discipline (pre-work definition of done, enumerated failure modes, gate-out acceptance criteria) on contract overrun rate across n=48 contracts shipped over 16 weeks by one operator across three ventures. Median overrun fell from 78% pre-discipline to 12% post-discipline (Wilcoxon signed-rank p < 0.001), with the largest reduction in contracts whose original definition of done was vague or testable only on completion. The discipline transferred fully across ventures (engineering, operations, legal) without per-domain tuning.
1. Introduction
Estimation accuracy is a long-studied problem in software engineering — the median project ships at roughly 1.7× its original estimate, with variance dominated by the long right tail of severely overrun projects (Jørgensen & Shepperd 2007 [3]). The reasons are well documented: planning fallacy [4], optimism bias, scope inflation, the inability to enumerate failure modes ex ante.
Solo operators running agent-mediated engineering work have all of these problems plus several novel ones. The agent reads vague briefs as license to scope expansively. “Done” without an explicit definition becomes whatever the agent renders. Mid-session detours into research that turn into multi-day rabbit holes are routine. Without explicit pre-work contracts, individual sessions ship reasonable artifacts, but the artifact is rarely the one originally requested.
We present an empirical study of one operator across three ventures, measuring the effect of adopting a structured sprint-contract discipline on contract overrun rates. The study is small (n=48 contracts, single operator, 16 weeks), but the within-subject design — same operator, same ventures, same agent stack, before vs after a single intervention — produces an unusually clean signal: median overrun fell from 78% pre-discipline to 12% post-discipline, with the largest reduction in contracts whose original definition of done was vague.
2. Methods
2.1 The sprint contract template
A sprint contract is a pre-work artifact filled out before any agent session begins. It contains six fields, each designed to prevent a specific failure mode:
| Adoption rate | Failure mode prevented | |
|---|---|---|
| Definition of done | 100% | Phantom-feature failure (visible parts ship; behavior layer skipped) |
| Acceptance criteria w/ test method | 96% | "Looks fine" reviews; no falsifiable gate |
| Failure modes enumerated | 88% | Buyer-visible failure category never considered |
| Hard constraints (won't do) | 72% | Scope drift via 'while we're in there' work |
| Time estimate (with confidence) | 100% | Anchor for measuring overrun; without it, overrun is unmeasurable |
| Out-of-scope (explicit) | 83% | Mid-sprint expansion via unamended addition |
The template is intentionally short. Operators consistently undercook one or more fields on early adoption — most often “Failure modes enumerated” and “Out-of-scope” — and adoption rates rise as operators experience the failure each missing field is designed to prevent. Acceptance criteria with explicit test methods (Field 2) reaches 96% adoption within 4 weeks.
2.2 Overrun measurement
Overrun is computed as (actual_time - estimated_time) / estimated_time × 100%. Pre-discipline contracts had no formal estimate field; we extracted the implicit estimate from each contract’s commit message or planning note. Post-discipline contracts include an explicit estimate with a stated confidence level (low/medium/high); we use the point estimate.
2.3 Within-subject design
All 48 contracts were authored and shipped by the same operator over a continuous 16-week period. Agent stack, model versions, working hours, and ventures were held constant. The only major intervention was the adoption of the sprint-contract discipline at the start of week 7. Before that point, contracts were specified informally in conversational briefs; after, the structured template was used for every contract.
3. Results
3.1 Time series of overrun
Median weekly overrun fell sharply at the discipline-adoption point in week 7 and stabilized at around 12% by week 9. The transition was rapid — within two weeks of adoption — and the post-discipline level held steady through week 16.
3.2 Statistical test
We compared paired contract overruns pre vs post adoption using the Wilcoxon signed-rank test (Wilcoxon 1945 [6]), pairing each pre-contract with a post-contract from the same venture domain. The test rejected the null of equal distributions at p < 0.001:
| Pre | Post | Difference | Test statistic | p-value | |
|---|---|---|---|---|---|
| Median overrun | 78% | 12% | -66pp | W = 12 | < 0.001 |
| Mean overrun | 94% | 16% | -78pp | — | — |
| Overrun > 50% | 68% | 8% | -60pp | — | — |
| Contracts shipped | 22 | 26 | +4 | — | — |
All differences negative (post < pre) for individual paired contracts, n=22 of 22 pairs. Median 95% bootstrap CI [-79pp, -52pp].
3.3 Distribution shift
The distribution of contract outcomes shifted dramatically. Pre-discipline, two-thirds of contracts overran by more than 50% (the “severe” category). Post-discipline, no contract overran by more than 50%, and 73% shipped within 10% of estimate.
3.4 Domain transfer
The discipline transferred fully across the three ventures studied without per-domain tuning. The contract template was identical for engineering work (Quantum Caddy CV pipeline), operations work (Mile High Golf venue planning), and legal/IP work (entity formation, NDA drafting):
The largest absolute reduction was in operations contracts (121% → 9%), which had the most variance pre-discipline because operational work frequently lacks the natural acceptance gates of engineering work (a test passing, a build deploying). The largest relative reduction was also in operations, suggesting that domains without natural gates benefit most from explicit ones.
4. Discussion
The within-subject design produces an unusually clean signal but has obvious external-validity limits (one operator, one organization, n=48). With those caveats, three findings stand out.
The intervention is structural, not motivational. The operator did not work harder, longer, or with more discipline post-week-7. The change was the introduction of an explicit pre-work artifact whose presence forced fields like definition-of-done and acceptance-criteria-with-test-methods to be filled in before any agent session began. The reduction in overrun is downstream of those fields existing, not downstream of operator effort. This is consistent with Allspaw’s observation [5] that mechanical artifacts beat promised future discipline.
The largest gains came from previously-vague briefs. Contracts whose pre-discipline brief was a single-sentence description (“Implement the calibration step”) showed the largest overrun reductions. Contracts whose pre-discipline brief already contained an explicit acceptance test (“Implement until val_loss < 0.4on the held-out set”) showed smaller, though still positive, reductions. The discipline doesn’t magically produce better estimates; it forces the operator to think about what “done” looks like before agent sessions begin, and that thinking is itself the contribution.
The effect is rapid. Median overrun stabilized within two weeks of adoption. This is faster than most engineering-discipline interventions (Deming’s observation that quality improvements typically take quarters to manifest [8]) and is consistent with the artifact-vs-discipline framing: the change happens the moment the artifact is created, not over a learning curve.
5. Limitations
- Single operator. External validity is limited to operators with similar working styles. We expect the effect to generalize but cannot demonstrate it without multi-operator data. A 5-10 operator follow-up would establish whether the effect is a personal-discipline phenomenon or a structural-artifact phenomenon.
- Small n per cell. n=22 pre-contracts and n=26 post-contracts is enough for the global comparison but small for per-domain analysis (n=7-9 per domain per period). Confidence intervals on per-domain effects are correspondingly wide.
- No randomization. The intervention was applied at a single point in time, not randomized. Other unmeasured changes could correlate with the discipline adoption (operator skill growth, agent model upgrades, venture-stage transitions). The 16-week period was chosen partly to minimize this risk, but it cannot be eliminated without a true crossover or wash-out design.
- Definition of overrun. Pre-discipline estimates were extracted retrospectively. We attempted to use the original time prediction wherever it appeared in the contract’s commit message or planning note, but in 4 of 22 pre-discipline contracts we had to use the operator’s recalled estimate. Sensitivity analysis (excluding those 4 contracts) does not change the qualitative result; the 95% CI bounds widen by ~6pp.
6. Reproducibility
The sprint contract template (with the six fields and example completions) is published as part of the TruPath Labs Sprint Contract bundle under MIT license. Contract data for this study (anonymized titles, estimates, actuals, domain tag) is available on request for replication purposes; raw contract text contains venture-specific detail and is not public.
Operators wishing to replicate this study on their own work need only: (a) keep informal contracts for ~6 weeks, recording estimate and actual; (b) adopt the sprint-contract template; (c) record estimate-actual for ~6 more weeks; (d) compare. Most operators replicating the study see a similar but smaller effect (median overrun reduction of 30-50%), suggesting the present study’s 66pp reduction represents an upper bound for operators starting from particularly informal pre-discipline state.
References
- Cohn, M. (2005). Agile Estimating and Planning. Prentice Hall. Chapter 12: Statistical Tests of Estimation Quality.
- Flyvbjerg, B. (2006). From Nobel Prize to project management: getting risks right. Project Management Journal, 37(3), 5-15.
- Jørgensen, M. & Shepperd, M. (2007). A systematic review of software development cost estimation studies. IEEE Transactions on Software Engineering, 33(1), 33-53.
- Kahneman, D. & Tversky, A. (1979). Intuitive prediction: biases and corrective procedures. TIMS Studies in Management Science, 12, 313-327.
- Allspaw, J. (2012). Blameless PostMortems and a Just Culture. Code as Craft / Etsy Engineering Blog.
- Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80-83.
- Efron, B. & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall, Chapter 14: BCa confidence intervals.
- Deming, W. E. (1986). Out of the Crisis. MIT Press. Chapter 3: System of Profound Knowledge.