The number that vanished

A few weeks ago I went looking for a number I was sure I had, and it wasn’t there anymore.

The context: I’m building a small piece of software that takes ODMR spectra — the resonance fingerprints of nitrogen-vacancy centers in diamond — and works backward to the magnetic field that produced them. To know whether the model is any good, you ask how much of the measured signal it actually explains. Earlier in the project I had a figure written down: the fit explained about 44 percent of the variance on real data. Not spectacular, but a real foothold, and I’d recorded it like you record any measurement — as a fact.

When I re-ran the same analysis with the current code, the number came back around 12 percent.

There was no error message. Nothing crashed. The tests were green. The code did exactly what it was told — it just told a different story than the note I’d written to myself. Somewhere between “then” and “now” the dataset, the configuration, or some quiet assumption had shifted, and the 0.44 had evaporated without leaving a trace.

I find this kind of thing more instructive than any clean success. We talk about reproducibility as a property of other people’s work — can a second lab confirm your result. But the first lab that needs to reproduce your number is your own, three months later, and it’s startlingly easy to fail that test. A figure in a notebook is only as trustworthy as the exact pipeline that produced it, and “the exact pipeline” is usually the part nobody writes down.

What made it worse — and more interesting — is that the disappearance wasn’t a bug to be squashed. The newer code was, if anything, more correct. The earlier 0.44 may have been flattered by a looser assumption I later tightened on purpose. So the honest conclusion isn’t “something broke.” It’s “I never knew which version of the analysis produced the number I trusted, because I hadn’t pinned it.” The forensics are still open. I can narrow down the suspects, but I can’t yet point at the one config that gives 0.44 and say there, that was the truth.

The fix is boring and it’s the whole job: every reported number gets chained to the exact code and data that made it, so that future-me can re-run it byte for byte. In a field where a single measurement is built from millions of repeated cycles just to claw a signal out of the noise, the discipline of also nailing down the software that interprets those cycles is not optional. It’s part of the measurement.

I’ll write the proper version of how spin-to-field reconstruction works — and why it’s genuinely hard — as a Fundamentals piece. This is just the smaller, truer story underneath it: even in physics, a documented number can quietly walk out of the room, and the only defense is making your past self reproducible.