How We Found a Value Leak in a Live AI System

The Problem

A live approval layer in a trading system was making decisions about which recommendations to pass through to execution. Nobody could vouch for whether those decisions were adding value. The system worked. The metrics looked fine. But the approval layer itself was unexamined.

The Methodology

We ran a counterfactual replay against frozen inference caches — 906 events, 898 cached model inferences, zero parse failures. The approach:

1. Replay approved decisions: What would have happened if every approved decision had been executed?
2. Replay rejected decisions: What would have happened if every rejected decision had been executed?
3. Ablation controls: Isolate the cause — was it the model's selection or the gate's approval logic?

The Finding

Approved decisions earned −0.175% per trade. Rejected decisions would have earned +0.151% per trade. The approval filter was inverted — it was approving losers and rejecting winners.

Approved vs rejected expectancy	−0.175% vs +0.151% per trade
Deterministic gate-flip control	+11.1pp profit factor (0.86→1.14)
Simulation scale	906 events · 898 cached inferences · 0 parse failures
Finding→production patch	<24 hours, zero downtime

The Fix

A reversible, provenance-marked guardrail shipped to production inside 24 hours. The deterministic gate-flip control alone improved profit factor by 11.1 percentage points (0.86→1.14) — and this was from frozen caches, meaning zero new model compute was required.

Key Takeaway

Most AI audits measure model accuracy. We measure business outcomes. An approval layer can be 100% accurate at its own logic while destroying value at the portfolio level. Counterfactual replay catches what accuracy metrics miss.