The Things I Learnt Building Agencio Predict's Signal Fabric — and Where They Travel

Eighteen months of building an autonomous trading platform, honestly accounted for — and why almost none of the important lessons turned out to be about trading.

In The Prediction Decade I made a promise: that I wouldn’t cover financial trading systems there, because I’d built one and it deserved its own article. This is that article. But I should warn you up front that it’s a bait-and-switch, and a deliberate one — because after eighteen months of hardening the Signal Fabric inside Agencio Predict, an autonomous trading platform that turns market evidence into governed, self-improving trading decisions, the single most valuable thing I learnt is this:

The trading was never the point. The engine underneath it was.

Strip the trading vocabulary out of the platform — the prices, the indicators, the brokers, the Greeks — and what remains is something far more interesting than a trading system. It’s a governed decision engine: a system that turns messy evidence into an explainable probability, lets a human or an LLM express a policy over it, proves that policy on history before it is allowed to touch reality, rolls it out through graduated exposure, wraps it in guardrails with an instant kill switch, and records every decision in an audit trail a regulator can actually read.

Nothing in that sentence mentions a market. And that’s the thesis of this piece: the engine — not the strategies — is the asset. A bid/no-bid decision, an underwriting decision, a litigation-funding decision, a campaign-spend decision and an “is this security alert real?” decision all share the same skeleton as a trade: uncertain evidence, a policy, a need to be right before you commit, and a need to explain yourself afterwards.

Here’s what the loop looks like, and then the honest accounting of what generalises, what doesn’t, what it cost to learn, and where I’d point it next.

The pipeline that survived eighteen months of contact with production: evidence → signal primitives → policy → simulated evaluation → gates → graduated rollout → guardrails and kill switch → audit — with autonomous improvement looping back under a jury and cross-validation, failing closed.

What is actually generic — and what is not

This is the section to argue with, because the honest boundary between “generic” and “trading-shaped” is the whole point.

The genuinely domain-agnostic core — the parts that would port to another industry essentially unchanged — turns out to be the governance machinery, not the market maths. The sandboxed rule DSL and its evaluator: a whitelisted, eval-free, no-look-ahead policy engine that walks a registry of primitives and genuinely does not care whether a primitive returns a momentum indicator or a supplier’s default rate. The anti-hallucination stack: schema-validated LLM output, a whitelist enforced at parse time, cite-or-refuse, a proposer/critic/judge jury, confidence thresholds, and a persisted log of every rejection. The guardrail layers and the kill switch: hard limits, consecutive-failure breakers, event blackouts, distribution-drift detection, and a platform-wide stop measured in milliseconds — rename “max drawdown” to “max exposure” and it is an insurance or marketing-spend guardrail, unchanged. The graduated rollout: mock, then paper, then live, with statistical gates in between. The self-modification controller: an LLM proposes a policy change, the change is materialised and backtested against the incumbent on identical data, and it is rejected if it degrades — failing closed. And the audit trail: not logging, but a decision provenance graph — which inputs, which model, which critique, which cross-validation result, which human approved.

Then there’s the layer that’s trading-shaped but structurally reusable. The backtest engine’s state machine is positions and P&L — but the discipline underneath it, replay history step by step and evaluate the policy at each step using only the information available at that step, is the single most transferable idea in the whole platform. The walk-forward and Monte Carlo machinery generalises the moment you swap Sharpe ratios for win rates or loss ratios. The executor — a leased, idempotent, guardrailed loop that evaluates policy against fresh evidence and acts through an adapter — only needs a different adapter: a broker today, a CRM, a policy-admin system or a SOAR platform tomorrow.

And finally, the honest residue: the genuinely trading-only parts. Market data integrations, broker adapters, derivatives maths, order-book microstructure. Of the nearly six hundred signal primitives in the registry, perhaps sixty to ninety are domain-neutral — statistics, composers, LLM primitives, significance testing. The rest are a market vocabulary. And that is fine, because the registry was designed to be swapped. The vocabulary is the cheap part. The grammar is the asset.

When we costed it honestly: roughly seventy per cent of the machinery is reusable across domains; thirty per cent — the primitive registry, the outcome model, the execution adapter — is new per vertical. The thirty per cent is where the domain expertise goes.

The five assets, in order of value

If I had to rank what eighteen months actually produced, it wouldn’t be strategies. It would be these five things.

First: “prove it on history before it touches reality.” The graduation gate. Most AI-decision products in every vertical I’ve examined ship a model straight into production, because they have no equivalent of a backtest. This platform’s entire architecture is organised around not doing that — and that discipline, not any model, is the number-one asset.

Second: the anti-hallucination jury. Proposer, critic, judge; schema validation; cite-or-refuse; confidence escalation; persisted rejections. This is the working answer to the question every serious buyer eventually asks: how do you let an LLM change a rule that decides who gets insured, funded, or bid on? The LLM gets a pen. It never gets the last word.

Third: guardrails and a kill switch that actually work. Hard limits evaluated on every tick, breakers, blackouts, drift detection, and a near-instant platform-wide stop. Autonomy without an off-switch isn’t a product; it’s a liability with a dashboard.

Fourth: the audit trail. Built for a regulator asking “why did the machine do that?” — a question trading gets asked occasionally and insurance, legal and security get asked constantly.

Fifth: the signal fabric itself. The platform computes evidence-quality signals that sound financial but aren’t: cross-channel disagreement (“do my sources agree?”), connectivity and percolation (“has my population quietly become correlated?”), discontinuity density (“is this process becoming unstable?”). Those three questions are asked, in different words, in every industry below.

The four lessons I paid for — so you don’t have to

These came out of real audits of a real system, and each one cost more than I’d like to admit.

One: the validation chain is the product. In one audit we found a headline performance metric computed roughly a hundred times wrong — in two engines, in opposite directions, so the errors cancelled and every dashboard looked plausible. Every individual safety mechanism was well designed; the chain feeding them numbers was broken. A gate that computes on a wrong number is worse than no gate at all, because it manufactures confidence.

A gate that computes on a wrong number is worse than no gate at all — because it manufactures confidence.

Two: mocked tests hide the bugs that matter. At one point more than a thousand unit tests were green while a core run path threw on every single call — because every test mocked the database, and the bugs were the kind only a real database can see. Any new vertical gets an integration test against real infrastructure in CI from day one, or it isn’t real.

Three: write the units contract down. A percent-versus-fraction mix-up survived months and two audit passes because nothing anywhere stated the convention. Put it in the code, next to the functions, and pin it with a regression test. Boring. Non-negotiable.

Four: fail closed, always. The dangerous defaults are the permissive ones — the sizing fallback that returned full size when the model meant to shrink it was down; the crisis check that read zero during an actual crisis; the validation gate that passed vacuously when it had nothing to compare. When the system doesn’t know, it must do less, not more.

Above — the fabric: hundreds of raw evidence threads woven into one cable, driven through the gates, and fanned back out to whichever world needs the decision. Below — the same engine in cutaway. Swap the registry, the outcome model and the adapter; keep the governance.

Where the engine travels

Every vertical below follows one template: swap the primitive registry, the outcome model and the execution adapter. Keep the governance.

Tendering and bids. The decision is bid/no-bid, and at what price. The evidence is tender documents, incumbent history, competitor win-rates, your own capacity and margin history — even an LLM-scored ambiguity read of the RFP text. The “backtest” is replaying your last hundred tenders using only what was knowable at bid time — which is exactly the no-look-ahead problem the engine already solves. Bidding has the same brutal feedback structure as trading: a discrete decision, a measurable outcome, a long history to replay, and a powerful temptation to fool yourself with hindsight. The walk-forward machinery answers a question most bid teams cannot answer at all: is our sixty per cent win-rate skill, or a lucky year? And the signal fabric transfers directly — cross-channel disagreement, where the commercial team is bullish while the capacity signal and the pricing model quietly disagree, is precisely the tender that becomes a loss-making win.

Insurance. Accept, decline, refer; price; reserve. The regulatory fit is the headline: insurance regulators ask precisely the questions the audit trail was built to answer — what did the model see, why did it decline, can you prove it hasn’t drifted, can you turn it off. Cite-or-refuse and the rejection log are close to purpose-built for that conversation. Aggregate exposure caps per peril and region are the same hard guardrail as a position limit, renamed. And percolation is accumulation risk: the book that looked diversified until a correlation crossed a threshold and one storm hit all of it — the same mathematics, different disaster.

Legal. Fund or decline a case; settle or proceed. Judge and counsel histories, procedural posture, an LLM reading the document corpus under cite-or-refuse. The honest caveat: legal outcomes are low-frequency and heavy-tailed, which makes the sample-size gate more important here, not less. The platform’s ingrained instinct — “your sample doesn’t support that claim” — is the single most valuable thing it brings to a field where that discipline is usually absent entirely.

Marketing. The shortest hop, since campaign prediction and ad-platform adapters already exist in the platform — arguably it’s already a second instantiation of the engine, just an under-governed one. The trap is severe, though: attribution is retrospective by nature, which makes marketing the look-ahead-bias capital of the business world. The engine’s frozen-as-of evaluation context exists for exactly this. And discontinuity density across spend, conversion and sentiment channels flags “the funnel has become unstable” before the acquisition cost blows out, not after.

Cyber security. Escalate or suppress an alert; prioritise a vulnerability. This is, quietly, the best structural fit of the five: the outcome model is the cleanest (precision and recall against a labelled incident history, with a real cost function in analyst hours), the telemetry is abundant and replayable, and shadow mode beside a SOC is the natural rollout. The adversarial mathematics the platform already uses to detect market manipulation — clustering, coordination scoring, anomaly detection — transfers to entity behaviour analytics almost verbatim. And cross-channel disagreement is the highest-yield triage signal in security: the endpoint says benign, the identity signal says anomalous, the network says exfiltration. Almost nobody computes that disagreement explicitly. The engine already does.

What it would actually take

Be sober about it, because the reusable core is real but so is the work. A new vertical means, in order: define the outcome model — the hard intellectual work, not the code, because until you can say what “P&L” means in your domain (won-bid margin? loss ratio? analyst hours saved?) every downstream gate is theatre; build the primitive registry, forty to eighty primitives before it’s useful, mostly plumbing to your data sources; replace the backtest’s state machine while keeping its no-look-ahead discipline; write one execution adapter against your system of record; and reuse the rest — evaluator, jury, guardrails, kill switch, gates, audit — unchanged.

If I were choosing the first step beyond trading: cyber-security alert triage is the best first technical vertical — cleanest outcome model, abundant replayable data, a painful status quo, and the lowest regulatory blast radius while the engine is re-proved in a new domain. Bidding is the best first commercial vertical — the highest willingness to pay per decision, and a question (“skill or luck?”) no incumbent tool answers. Insurance is the biggest prize and the slowest path: the governance is a genuine moat there, but the regulatory bar means the validation chain must be beyond reproach — which, per the lessons above, is exactly the thing that took two audits to get right.

The closing thought

A fair reader should discount all of this appropriately: the seventy-per-cent claim is a thesis from one platform, not a track record across five. Paid for once, portable in principle is the honest status of every generalisation here. But that’s also precisely why I trust the lessons more than the architecture — the lessons were extracted under adversarial conditions, by production systems that punished every assumption I got wrong.

Eighteen months ago I thought I was building a trading platform. It turns out I was building a machine for governing decisions under uncertainty, and trading was merely its first — and loudest — customer. The prediction was never the hard part. It still isn’t. Governing it was, and is, and that’s exactly why the engine, not the strategy, is the thing worth owning.

---

The companion paper

This piece is the engine-room account. The wider landscape it operates in — the four engines, the economics of foresight, and why the real product is calibrated confidence — is the companion essay, The Prediction Decade, also available as a designed PDF.

---

These are just my own words — one founder’s account of his own build, not advice of any kind. More at justinjames.agencie.io · The Possible Labs — thepossible.agencie.io