Every figure below is measured in a live production system or on a public benchmark using its own unmodified scoring harness. Nothing here is a projection. Built, live and proven are kept distinct. We never price a grade-one claim as a grade-three result.
Troml was built inside a live property operation. The public benches are how we test whether the substrate travels. They are not the production corpus.
The first RAG benchmark built on company-internal data, not Wikipedia. 500 questions. 511,963 documents. Nine source types. Judge model GPT-5.4. Official leaderboard published by Onyx, the benchmark’s authors — the live board.
| Rank | System | Overall | Correctness | Completeness | Recall |
|---|---|---|---|---|---|
| 1 | Troml | 76.79 | 83.8 | 81.84 | 86.55 |
| 2 | Skyller | 71.93 | 77.0 | 79.14 | 81.6 |
| 3 | OpenClaw | 68.22 | 81.6 | 72.86 | 79.02 |
| 4 | fgroo | 63.27 | 71.0 | 71.03 | 72.5 |
| 5 | OpenAI File Search | 61.03 | 69.8 | 67.87 | 71.65 |
Then RAGFlow, Amazon Q (Kendra), Azure AI Search, Vertex AI Search, NVIDIA AI Blueprints. LlamaIndex and LangChain, on default configs, sit at the bottom. The board’s own header: Total systems: 19. We quote the board, not a count we invented.
The board as captured, 17 August 2026. If the remote board moves, the numbers on this page still have a date.
Leaderboard last updated 12 August 2026. Verified on the public board 17 August 2026. Same harness, same judge for every system. Closed-book control: the score comes from the system, not the model’s prior knowledge of the corpus.
One figure we will not hide. Invalid extra documents: Troml 12.65, OpenClaw 0.47. We retrieve more. We also drag more junk. Lower is better on that column.
The ranking is Onyx’s to state. The result files sit in their repository. We have not claimed a separate independent re-grade, because we cannot evidence one beyond the board they publish.
SEC-filings QA. 9–10 July 2026. 84 filings, 150 open questions, frozen dev-30 / test-120 split. The second foreign-domain replication, and the strongest portability evidence on file.
| Run | Score | Status |
|---|---|---|
| Modern naive baseline, same model | 56.7 (dev) | Anchor |
| Clean one-shot, full stack | 77.5 (93/120) | The quotable number |
| v2, post-diagnosis | 82.5 | Disclosed iteration |
| v3, second iteration | 90.0 (banked; recount 90.8) | Disclosed iteration |
| Paper’s best, 2023, handed gold documents | 89.3 | Reference |
77.5 is the clean one-shot number and the one we lead with. 90.0 is labelled as post-diagnosis. v3 exceeds the paper’s best configuration, and it did so with realistic retrieval rather than gold documents.
The rule was written the moment test-120 was diagnosed, and before v2 ran:
any v2 test score must be disclosed as post-diagnosis; 77.5 stays the clean one-shot number
Independently re-verified 17 August 2026 from the raw run files, not from this page’s ancestors. v1 recounts 94/120 = 78.3 against the banked 93/120 = 77.5 — one-row judge variance. We kept the lower number. v2 recounts exactly 82.5. v3 recounts 90.8 against a claimed 90.0. The claims are the cautious pick.
Judge calibration: n=300, 91.7% human agreement, confusion errs harsh 15:3 against the system. Calibration was run against human-labelled ground truth, not against system outputs. The scores are a floor, not a ceiling. Gold document present in retrieval 120/120; every remaining error is extraction or reasoning, not recall.
This run was not for the score. It was for the transfer. The decisive lever was the platform’s own entity resolution and document scoping — +20 points — in a domain it was never built for.
Mechanical levers convert. Prompt levers churn. The v2 prompt pack was net-neutral: fixed 2, broke 3.
The founding vertical is a property-management operation. Every figure below is measured in that production system.
Production figures measured June 2026. Document-identity precision audited July 2026.
Same type size as the wins.
An early blinded comparison ranked our output last. It is on file. It is also why the numbers that survived later measurement are worth something.
We upgraded a core component to a materially better one and the whole system got worse — 74.4 against 76.9. It fixed 16 questions and broke 31. A better part does not add to a system that selects a fixed number of things. It displaces. We do not accept “we upgraded the model” as evidence of anything, from a supplier or from ourselves.
Our original token-reduction study was invalidated by our own audit. It measured about 5% of true cost. The honest instrumentation now exists. The re-run has not been published, so the claim is absent rather than softened.
On a third public benchmark, generation quality placed top-tier (RACE 52.14) while citation accuracy reached 56.8%. The fix is scoped. We do not quote the good half alone.
Portability is demonstrated, not proven at scale. Roughly 3,000 lines of substrate primitives were carried into a second vertical and ran unchanged in four of five cases. That vertical is a pilot, same jurisdiction, adjacent workflow. A paying second vertical in a structurally different workflow is what converts this to proof. We have not done it yet.
One pilot shipped the structured layer and deferred the intelligence layer. Extraction and storage are live. The obligation extractor and the learning loop exist as importable code with no scheduled execution. We name that rather than counting the pilot as a full deployment.
How the machine works is on How. If you have a real corpus: Evaluate. The full technical document is available on request: [email protected].