Deciding When a System Is Allowed to Ship

There's a moment on every project where someone decides the thing is ready. Often that decision is a vibe: it ran a few times, the demo went fine, ship it. That works until the one input you didn't try shows up in production. The alternative is to decide in advance what "ready" means — to write the bar down before you're emotionally invested in clearing it — and then to hold the system to it. This post is about that discipline, where I learned to take it seriously, and how the same shape applies whether you're gating a trading strategy or an AI feature.

I built quantlab, a quantitative-trading research system, specifically to force this rigor on myself. Markets are the most honest teacher of the difference between "it looked good" and "it works," because a strategy that overfits its backtest will happily separate you from real money. But nothing here is specific to trading. The pattern — pre-register the bar, gate the rollout, degrade safely — is how I think about putting any system into production.

Pre-registration: write the bar before you run the test

Pre-registration is a research idea: you commit to your hypothesis and your success criteria before you see the results, so you can't quietly move the goalposts once the data comes in. It's the antidote to the most natural bias there is — deciding something worked because you wanted it to.

In quantlab, that means the risk limits and the definition of an acceptable strategy are written down first, as hard rules, before a strategy is ever evaluated. Not "let's see how it does and then decide if that's good enough" — the threshold exists before the number does. The value isn't the specific limits; it's the ordering. Once you've seen a result, you are no longer a neutral judge of it. Deciding the bar in advance is how you keep a promising-looking outcome from talking you into a decision you'd never have made cold.

Gating: earn each promotion, from backtest to paper to live

The second discipline is refusing to let a system jump straight from "works on my machine" to "runs against the real world." quantlab moves a strategy through explicit stages, and each transition is a gate it has to earn.

A backtest is the cheapest, weakest evidence — it proves the idea isn't broken on historical data, nothing more. Passing it earns a strategy the right to paper-trade: to run forward against live conditions with no money at risk, which is the first honest test of whether it survives contact with the present rather than the curated past. Only after it holds up there does the conversation about real capital even open. The point of the ladder is that each stage catches a different class of failure the previous one couldn't — and a strategy that dies at the paper stage cost you nothing but time, exactly as intended.

This maps directly onto shipping software. A test suite is the backtest — necessary, cheap, and nowhere near sufficient. A staging or limited rollout is the paper trade. Full production is live capital. The engineers who get burned are the ones who treat a green test suite as permission to ship to everyone, skipping the stage whose entire job is to fail cheaply.

Fail-safe by default: what happens when a limit is hit

Pre-registered limits are worthless if hitting one is a suggestion. The third discipline is that the safe response is automatic and built in, not a decision made in the moment by someone watching a dashboard — because the moment you actually need the limit is the moment you're least inclined to respect it.

So the halt conditions are wired into the system: cross a pre-set loss threshold and the system stops itself, rather than trusting a human to intervene calmly during exactly the drawdown that makes calm hardest. The design assumes the worst case will happen and makes the default response to it survivable. This is the same instinct behind the sandbox in my other writeup — assume the bad case, and make its blast radius something you've decided in advance rather than something you discover under load.

The same shape, outside markets

Strip away the trading vocabulary and this is just a reliability discipline. Pre-register the bar so you can't rationalize past it. Gate the rollout so nothing reaches the real world without earning it one stage at a time. Make the failure mode safe by default so the worst case is bounded before it arrives.

I've applied the same thinking to AI systems, where "it seems to work" is even more seductive because a language model is fluent enough to look right while being wrong. The answer is the same: decide what acceptable output actually means before you're staring at a demo that charmed you, gate the rollout instead of trusting a good first impression, and design the system to degrade into something harmless when it inevitably meets an input you didn't foresee. The bilingual voice system I wrote about elsewhere was built to always hand back a usable result even when a call went sideways — that's this same principle, wearing a different hat.

The through-line across everything I build is a distrust of the moment where confidence substitutes for evidence. Deciding when a system is allowed to ship — deliberately, in advance, with the bar written down — is, to me, most of what separates something you demoed from something you can actually stand behind in production.