Skip to content
Engineering9 Sept 20267 min read

Latency budgets: making performance a build failure

Performance regressions are found in production because nothing fails when they are introduced. We fixed that with per-subsystem budgets enforced in CI.

DR

Davide Rinaldi

Head of Systems Engineering

A performance requirement written in a specification document has no enforcement mechanism. Six months later, nobody remembers it and the p99 has doubled.

Budgets as code

On the Kestrel trading terminal, every subsystem carries a latency budget expressed as a number in the repository:

market_data.decode:      12us p99
order.validate:          38us p99
order.serialize:         21us p99
ui.depth_render:        900us p99

The benchmark suite runs on dedicated hardware after every merge to main. If a subsystem exceeds its budget by more than 5%, the build fails and the merge is reverted.

The hardware matters

We learned this expensively: benchmarks on a developer laptop are noise. Thermal throttling, background processes and different cache hierarchies produce variance larger than the regressions you are looking for. Our benchmark runner is a pinned, isolated machine with the same CPU family as the trading desk.

What a budget change looks like

Budgets are not immovable. Changing one requires a pull request that includes the new number, the measurement that justifies it, and an approval from the practice lead. That friction is deliberate — it means every regression is a conscious decision rather than an accident.

Results

Across fourteen months on Kestrel, four budget violations were caught in CI. All four were merged code that looked correct in review. None of them reached production.

Tell us what you are building

Send a brief and a solution architect replies within one business day — with questions, not with a template.