Three failure modes matter most in legal AI: inventing authority, drifting from the source, and treating like cases differently. This page describes how Constat measures each one — at the level of enforced outcomes, honestly, including the limits of what is measured.
The render gate. Every reference must verify against its official public source before it can appear on screen or in an export. An unverified citation is not flagged for review — it is technically impossible to display.
Safety floors in the evaluation gate. The gold-set evaluation enforces a hallucination floor of zero and abstention accuracy of 1.0. These floors are identical for every model tier — no model is ever let off the hook on safety. A change that scores below a floor cannot ship.
Abstention by design. When the verified basis is thin, the platform abstains and routes the matter to the lawyer. Abstention is a designed, measured outcome — not a failure mode.
Verbatim, hash-pinned sources. Operative legal texts enter the corpus byte-exact, pinned with cryptographic hashes and counsel sign-off. Output cites the pinned text; a paraphrase is never presented as the law.
Client facts stay client facts. Facts extracted from client documents are always labelled as the client's assertions and can never be presented as law or merged into it.
Deadlines are computed, never generated. Statutory deadlines come from counsel-signed deterministic rules with their derivation shown — never from text generation.
The question the suite asks: does the outcome change when a characteristic that must not matter changes?
Counterfactual scenario families. The same legal matter is re-posed across systematically varied client characteristics — names of different origins, gender, role and register — currently 58 scenario families and 310 fixtures, growing with the corpus.
A merge gate, not a report. The floor is zero strict violations, wired into the development gate: a change that breaks it cannot be merged.
The honest boundary. The suite covers the probes we have written; it cannot certify the absence of all conceivable bias. The underlying model inherits the biases of its training — the verification and abstention layers mitigate the consequences, they do not eliminate the cause.
These are internal evaluation results, not third-party audits; no external benchmark certification exists yet and none is implied. Mechanisms are described here at the level of enforced outcomes, not implementation. Family law remains deliberately out of scope: the platform abstains entirely rather than serve a domain where errors are hardest to bound.
