You can falsify a simulator without a single real measurement.

There's a trap that catches teams sitting on a simulator and no real data: "we can't validate anything until we collect measurements, so until then the sim is what we have." The first half is true. The conclusion doesn't follow, and believing it nearly sent us downstream on outputs that were quietly wrong.

We were in exactly that position with Quantum Caddy's throw simulator. Tens of millions of simulated throws, zero measured ones. The July 5th audit session, run under the experimentalist contract I wrote about last issue, never touched a real measurement and still produced findings strong enough to change our plans the same night. The method is worth writing down, because nothing about it is specific to physics.

Two models, no oracle

The move is to build a second, independent model of the same system and study the disagreement. Neither model is ground truth. That feels like it should be a dead end, and it isn't, because the two models earn trust differently. The as-built simulator encodes what your team assumed under deadline. The reference model encodes first principles and published priors, built slowly, with every choice documented. Where they agree you've learned little. Where they diverge, at least one of them is wrong, and the structure of the divergence usually tells you which one and why.

Five moves, in the order they have to happen.

Reproduce the as-built model from the outside

Before you correct anything, prove you can regenerate the thing you're auditing. Our simulator's actual generator script wasn't even in the repo, so the session fit the shipped data from the outside and recovered a closed-form model that matched flight outputs to about 1.5 percent. It also surfaced internal inconsistencies along the way, including an output field that was inflated a third in one regime.

This step is not optional. If you skip it, every later disagreement is ambiguous: is the reference model exposing a real defect, or did you just mis-guess what the sim does? Reproduction converts "these disagree" into "they disagree because of this specific mechanism."

Build the reference from first principles

The corrected model came from physics and literature priors, with an uncertainty band on every coefficient we couldn't measure. The discipline that matters: the reference is a *reference*, not truth. Its job is to be independently derived and honestly uncertain, so that disagreement with the as-built model is informative rather than a coin flip.

Map the divergence, not the average

Running both models over the same balanced sample of inputs is where the picture snapped into focus. On trajectories, the two models were nearly identical, median landing difference of a hundredth of a foot. On scored outcomes, 48.3 percent flipped. Agreement in one observable had told us nothing about the observable we actually cared about, because the models diverged in a regime the trajectory numbers never exercise.

That's the general lesson, and I'd bet on it holding for demand models and risk models as much as physics models: validation doesn't transfer across regime boundaries. Check the output you'll act on, in the regime you'll act on it in, or you've checked nothing.

The divergence map also told us the disagreement was broad, spanning the middle of every parameter range rather than hiding in a corner. That killed the comfortable theory that a single knob was mis-set and could be patched.

Budget the uncertainty, then rank it

The last analytical move was propagating every coefficient's uncertainty through the reference model and asking where prediction error concentrates. In our case the answer was lopsided: the in-flight prediction was tight to a couple of inches, while the after-contact prediction carried fifteen inches of uncertainty, and one coefficient drove three times more of it than any other.

A tornado chart like that is a budget in the literal sense. It tells you where distrust should live and where the next dollar of measurement effort pays most. Ours said: measure surface friction first, everything else is secondary.

Ship a measurement list, not a verdict

The audit's final artifact wasn't "the simulator is wrong." It was a ranked list of experiments, each tied to the uncertainty it retires and the observable it unlocks. That reframing matters for morale as much as method. The simulator team didn't get told their work was garbage. They got a map of exactly which stage was sound, which stage wasn't, and the shortest path to knowing for sure.

What this method cannot do

A model-vs-model audit establishes internal consistency and locates risk. It cannot validate either model against reality, and pretending otherwise recreates the original sin with more steps. Our corrected model's absolute predictions still carry unmeasured coefficients with honest error bars, and the write-up says so in bold at the top. The audit's authority comes precisely from refusing to claim more than the data supports.

So the honest framing for the whole exercise: with zero measurements you can find defects, rank uncertainties, and buy your future measurements at the highest possible exchange rate. You cannot buy truth. But you can make sure that when the real data finally arrives, you already know exactly which questions to ask it.