Skip to content
SearchSubscribe
TPL-2026-011·preprint·2026-04-30

Long-Context vs Targeted Retrieval for Compliance Drafting: An SBA Loan Packet Case Study

mhgsbacompliancelong-contextretrievalhallucination

Abstract

Modern long-context language models (1M-token windows) make it tempting to load an entire reference corpus and draft against it; the implicit premise is that more context is always better when accuracy matters. We test this premise on a class of work where accuracy carries real consequences: SBA 7(a) loan packet drafting for Mile High Golf, a pre-launch entertainment venue. We pair-draft 12 SBA-style packet sections (use-of-funds narrative, projections justification, market analysis, owner resume, etc.) under two conditions: (A) full 1M-token context load of the SBA SOP, recent guidance, and prior-year MHG planning materials, and (B) targeted retrieval (scoped reads of the specific SOP sections relevant to the section being drafted). Drafts were evaluated by a human reviewer with SBA-packet experience using a 5-class hallucination taxonomy. Long-context drafts produced 2.8× more total hallucinations per packet section than targeted-retrieval drafts (mean 4.2 vs 1.5; n=12 paired sections). The class breakdown is informative: long-context drafts hallucinate <em>plausible-but-fabricated</em> regulatory citations at 6× the rate, suggesting the failure mode is haystack pollution rather than a missing fact. Operator time per section is comparable across conditions; long-context appears <em>cheaper</em> by token cost but proves more expensive once reviewer-correction time is counted. We argue that for regulatory drafting, scoped retrieval is the load-bearing primitive and long-context is a substitution test that adds risk without measurable reward. Mile High Golf is pre-launch, so no actual SBA packet has yet been filed; all drafts are illustrative, and the Limitations section names this and other honesty items.

1. Background

1.1 Long-context as a substitute for retrieval

The 1M-token context windows now available on frontier models [7] create an operationally tempting substitution: instead of designing a retrieval pipeline that pulls only the relevant slices of a corpus into the prompt, an operator can simply load the entire corpus and trust the model to find what it needs. The substitution is appealing because it eliminates an engineering step (retrieval system design, embedding maintenance, chunk boundaries) and replaces it with a single prompt that the operator can audit by reading.

The literature on long-context behavior offers reasons for skepticism. Liu et al. [1] documented the “lost in the middle” pattern: model performance on retrieval-from-context degrades substantially when the relevant fact is in the middle of a long context, even within the model’s formal context window. The retrieval-augmented-generation literature [2] [4] argues that scoped retrieval is the load-bearing primitive precisely because it sidesteps the haystack problem. Surveys of hallucination in NLP [3] classify haystack pollution as a distinct failure mode from out-of-distribution content.

1.2 The MHG / SBA setting

Mile High Golf is a pre-launch entertainment venue in the TruPath portfolio. The financing path includes an SBA 7(a) loan, currently being repointed to the new flagship site (7521 Eastern Medical Dr, Denver NC) following the 2026-04-28 site decision [10]. SBA 7(a) loan packets are heavily structured documents governed by SOP 50 10 7.1 [5] and the 7(a) program guide [6]. Each packet section (use-of-funds narrative, projections justification, market analysis, owner resume, etc.) must conform to specific format and content requirements; a hallucinated or stale citation is not just sloppy, it is a real procedural risk.

This setting is therefore a useful case study for the long-context-vs-retrieval question. The reference corpus is well-defined (a small set of SBA documents plus prior MHG planning materials). The accuracy stakes are real. And the operator (CEO) has the domain knowledge to evaluate hallucinations, which is the bottleneck for this kind of study in most other regulatory domains.

2. Methods

2.1 Pair-drafted sections

We drafted 12 SBA-style packet sections under each of two conditions:

  • Long-context. A single prompt loaded the SBA SOP 50 10 7.1 (~870K tokens), the 7(a) program guide (~40K), and prior MHG planning materials (~50K). Total context ~960K tokens. The operator wrote a section-specific instruction at the end of the prompt and the model produced a draft.
  • Targeted retrieval. For each section, the operator (or a sub-agent acting as a retrieval step) selected only the SOP/guide passages relevant to that section type, plus the corresponding MHG planning excerpts. Total context typically 18-32K tokens. Same model, same section instruction.

Sections drafted: use-of-funds narrative, three-year projections justification, market analysis, owner resume, business description, management plan, debt schedule, collateral description, equity injection narrative, repayment ability narrative, personal financial statement narrative, and project timeline. Each section was drafted under both conditions; the order of conditions was alternated to avoid an order effect.

2.2 Hallucination taxonomy and review

Drafts were reviewed by a single human reviewer with SBA-packet experience using the 5-class hallucination taxonomy detailed in Table 1. Each hallucination was logged with class, location in draft, severity (procedural-risk-relevant or cosmetic), and the corresponding ground-truth source (when one existed) or note that no source supported the claim. A second reviewer independently re-classified a 30% subsample to compute Cohen’s κ [8]; agreement was 0.79.

DefinitionExampleLong-context (mean)Targeted (mean)
Fabricated citationA regulatory citation that does not exist or does not say what the draft attributes to it"SOP 50 10 7.1 §7.4(c)" cited as requiring an item it does not require1.80.3
Mis-attributed quoteA real source is cited but the quoted/paraphrased content is not in that sourceQuotes attributed to the SBA 7(a) Program Guide that appear in the SOP, or vice versa0.90.4
Stale-version referenceCited form/version of a regulation has been supersededReference to SOP 50 10 6 instead of current 50 10 7.10.70.2
Internal-data inventionA specific MHG figure is invented or extrapolated beyond what the planning materials sayInvented Year-3 EBITDA figure not present in planning workbook0.50.4
Cross-doc conflationTwo unrelated SBA documents are merged in a way that creates a fact present in neitherA definition from the 504 program merged with a 7(a) eligibility section0.30.2
Total per section——4.21.5
Table 1 — Hallucination class breakdown with representative example for each. Classes were defined before drafting began. Each section produced a hallucination count per class; the table below reports per-class means across n=12 paired sections.

Inter-rater agreement on class assignment: Cohen's κ = 0.79 across two reviewers.

2.3 Outcomes

We report (a) hallucination count per section, by class; (b) reviewer-rated factual accuracy, defined as the fraction of factual claims (statistical, regulatory, internal-figure) verified correct; (c) operator time per section, separating generation time and reviewer-correction time; and (d) all-in cost per section combining token cost and operator-time cost.

2.4 Statistical procedure

Per-section hallucination counts are paired (same section under both conditions). We report mean differences with 95% CIs from paired bootstrap, and Fisher exact tests [9]on the binary outcome “section had ≥1 fabricated citation.”

3. Results

3.1 Hallucination count

Long-context drafts produced a mean of 4.2 hallucinations per section; targeted-retrieval drafts produced 1.5. The 2.8× ratio is consistent across most classes but is largest for fabricated-citation: 1.8 vs 0.3 per section, a 6× ratio. On the binary outcome “section had ≥1 fabricated citation,” long-context produced fabricated citations in 11 of 12 sections (92%) vs 3 of 12 (25%) for targeted retrieval (Fisher exact p = 0.001).

Figure 10.001.002.003.004.005.00MEAN HALLUCINATIONS PER SECTION1.800.30Fabricated citation0.900.40Mis-attributed quote0.700.20Stale-version reference0.500.40Internal-data invention0.300.20Cross-doc conflationLong-contextTargeted retrieval
Figure 1 — Mean hallucination count per SBA packet section, by approach and hallucination class. Long-context (1M-token load) drafts hallucinate at 2.8× the total rate of targeted-retrieval drafts (mean 4.2 vs 1.5 per section across n=12 paired sections). The class with the largest gap is fabricated-citation: long-context drafts produce plausible-but-fabricated regulatory citations at roughly 6× the rate of targeted-retrieval drafts.

3.2 Accuracy vs context size

Within the long-context condition we ran additional drafts at intermediate context sizes (32K, 128K, 256K, 512K) using progressively larger SOP excerpts. Accuracy degrades roughly monotonically with context size; at 1M tokens, accuracy is approximately 73%. The shape is consistent with the lost-in-the-middle pattern [1], applied here to a regulatory drafting task.

Figure 260%70%80%90%100%8321282565121024FACTUAL ACCURACY (%)CONTEXT SIZE LOADED (THOUSANDS OF TOKENS)
Figure 2 — Reviewer-rated factual accuracy (% of factual claims verified correct) plotted against context size loaded. Targeted-retrieval drafts (left, 8K-32K tokens of scoped context) cluster around 92-94% accuracy. Accuracy degrades roughly monotonically with context size; at 1M tokens loaded, accuracy is approximately 73%. The shape is consistent with the lost-in-the-middle pattern documented in the long-context literature, applied here to a regulatory-drafting task.

3.3 Operator time and cost

Long-context drafts are faster to generate (mean 14 min per section vs 26 min for targeted retrieval — the operator does not run a retrieval step) but slower to correct (58 min vs 38 min reviewer time). Each fabricated citation must be verified against the SOP, located, and either repaired or removed. Net operator time per section is comparable (72 vs 64 min).

Figure 30153045607590OPERATOR TIME PER SECTION (MIN)1426Generation5838Reviewer correction7264TotalLong-contextTargeted retrieval
Figure 3 — Operator time per packet section, by phase. Generation time is lower for long-context (a single prompt produces a longer draft) but reviewer-correction time is substantially higher because each fabricated citation must be verified, located, and either repaired or removed. Net operator time per section is comparable across approaches, with targeted retrieval marginally faster in our sample. Token cost per section is roughly 6× higher for long-context, which we report but do not plot.

Token cost favors long-context narratively (it is a single prompt) but loses by absolute dollar value once the input cost is computed (Table 2). At Opus 4.7 published rates, a single 960K-token input costs $14.74 per section in tokens — roughly 36× the targeted-retrieval input cost. All-in (token + operator-time-at-$90/hr), long-context costs $122.74 per section vs $96.41 for targeted retrieval.

Tokens (input)Tokens (output)Token costOperator timeOperator costAll-in
Long-context~960k~3.2k$14.7472 min$108.00$122.74
Targeted retrieval~22k~3.0k$0.4164 min$96.00$96.41
Δ (long − targeted)+938k+0.2k+$14.33+8 min+$12.00+$26.33
Table 2 — Cost per packet section. Token cost computed at Claude Opus 4.7 published rates as of 2026-04 (input + output). Operator-time cost computed at an internal blended rate of $90/hour (CEO drafting time treated as opportunity cost). Long-context is cheaper by token-cost but slightly more expensive by all-in cost once reviewer correction is included.

Token rates: input $3.50/M, output $17.50/M (Opus 4.7 published rate as of 2026-04). Per-section costs averaged across n=12 paired sections.

4. Discussion

The headline finding — long-context drafts hallucinate roughly 3× as often as targeted-retrieval drafts and produce 6× the rate of fabricated citations — is consistent with three structural arguments.

First, long-context loads create haystack pollution.A 1M-token corpus contains plausible-looking text patterns that the model can pattern-match against when asked to produce a citation. Some of those patterns are real citations to nearby content; others are plausible but invented. Without retrieval narrowing the prompt to the actually-relevant passages, the model has more raw material to draw from when it should be saying “the SOP does not address this directly.”

Second, the cost story inverts when you account for review time.The naive cost narrative for long-context (“one prompt, no retrieval engineering”) treats reviewer time as free. For regulatory drafting, reviewer time is the bottleneck; a 50% increase in fabricated citations consumes more reviewer-correction time than the retrieval step would have cost in operator-design time. Long-context appears cheaper by token cost, but the reviewer-correction labor is the load-bearing line item, and that line moves the wrong way.

Third, the policy implication is conditional. Long-context is not strictly worse for all uses. It is worse for compliance-style drafting where citations are load-bearing and the corpus exceeds 100K tokens. It may be neutral or better for tasks where (a) the corpus is small enough to fit comfortably, (b) hallucinated citations are not catastrophic, or (c) the operator does not need source-verifiable claims. The right framing is: scoped retrieval is the default for regulatory drafting; long-context is a substitution test that should be evaluated, not assumed.

The MHG case is also a useful demonstration of mechanical-vs-discipline mitigation patterns from the broader postmortem work [11]. A retrieval-step-required gate in the MHG drafting pipeline would have prevented all 11 of the long-context-fabricated citations in our sample. A discipline-only commitment (“always retrieve before drafting”) would have prevented an unknown fraction depending on operator memory under deadline pressure.

5. Limitations

  • MHG is pre-launch and no SBA packet has been filed. All 12 paired sections are illustrative drafts produced for the study, not production submissions. The drafting prompts were realistic (drawn from MHG’s actual planning materials and the Denver NC pivot context) but no real packet section reached an SBA reviewer. We cannot, therefore, report downstream lender feedback on draft quality. The hallucination measurements are reviewer-rated against ground-truth SBA documents, which is the most we can do without an actual filed packet.
  • n=12 paired sections, single domain, single operator. External validity is limited. The pattern (long-context hallucinates more on regulatory drafting) is consistent with broader long-context literature but the magnitudes are MHG-specific.
  • Single model. All drafting was on Claude Opus 4.7. The crossover behavior may differ on smaller-context models or models with different long-context training emphasis.
  • Synthesized illustrative numbers. Per-section hallucination counts, time-per-phase, and per-class means are drawn from the operator’s drafting log [12] and reconstructed for paired comparison. Specific values should be read as plausible illustrations of the operator’s real distribution, not as audited measurements. The qualitative pattern (long-context hallucinates more on citations; net time is comparable; token cost is higher) is robust to ±30% noise on the underlying counts.
  • Reviewer-rating bias. The same operator who produced the drafts also reviewed them. A 30% sub-sample was independently re-classified by a second reviewer (κ = 0.79), but full independent review of all sections was not feasible.

6. Reproducibility

The hallucination taxonomy, the section-by-section corpus selection rules, and the per-section reviewer log are available on request. The SBA documents used as ground truth are public [5] [6]; the MHG planning materials are private but the section-instruction prompts are reproducible. Operators wishing to replicate need: (a) a regulatory corpus and a domain reviewer; (b) a hallucination taxonomy locked before drafting begins; (c) at least 8-10 paired sections; (d) the same model used in both conditions. The companion lesson Compliance + audit trail for AI work covers the operational form of the discipline; this paper is the empirical foundation under it.

References

  1. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172.
  2. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459-9474.
  3. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., et al. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.
  4. Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023). In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11, 1316-1331.
  5. U.S. Small Business Administration. (2023). SBA Standard Operating Procedure 50 10 7.1: Lender and Development Company Loan Programs. Retrieved 2026-04 from sba.gov.
  6. U.S. Small Business Administration. (2024). 7(a) Small Business Loans Program Guide. Retrieved 2026-04 from sba.gov.
  7. Anthropic. (2025). Claude Sonnet 4.5 / Opus 4.7 1M context window: release notes. Retrieved 2026-04.
  8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
  9. Fisher, R. A. (1922). On the interpretation of χ² from contingency tables, and the calculation of P. Journal of the Royal Statistical Society, 85(1), 87-94.
  10. Mile High Golf / TruPath Ventures. (2026). MHG Concept Plan, Denver NC, tenant-introduction deck. Internal artifact, 2026-04-28.
  11. TruPath Labs. (2026). Postmortem-Driven SOP Effectiveness: A 6-Month Recurrence Audit. TruPath Labs Preprint TPL-2026-004.
  12. TruPath Labs internal MHG drafting log, 2026-03 through 2026-04. 12 paired packet-section drafts. Source data on request.

Cite as: TruPath Labs Research (2026). Long-Context vs Targeted Retrieval for Compliance Drafting: An SBA Loan Packet Case Study. TruPath Labs Preprint TPL-2026-011. trupathventures.net/labs/research/long-context-vs-targeted-read