Merlise achieves breakthrough results on SciFact-Open
Merlise
Legal · Failure mode

A source that reads like a real one and is not

March 20264 min read

A phantom citation carries the surface marks of a real one. Plausible party names. A court abbreviation that looks familiar. A volume and page that seem plausible enough not to question. Nothing announces the problem until someone tries to find the actual case and cannot.

The source looks real. It is not. That is the entire pattern, and it keeps recurring.

A pattern, not an incident

One nonexistent citation could be dismissed as an unlucky mistake. A running count changes the story. A database maintained outside any single firm or court has tracked over a thousand documented instances of fabricated legal citations, rising from roughly two hundred a year earlier to several new ones logged most days. Courts have imposed six figures in sanctions in a single quarter for this exact failure.

That scale is the point. A tool built to catch this cannot treat a single instance as a novelty. It has to check for the pattern by default, on any filing, the same way a proofreader checks spelling without being asked.

A missing case and a mistyped one are not the same failure

Failures here do not all look the same underneath. A citation nobody can find after a reasonable search is one problem. A docket number typed wrong, or a case cited under its short name, is another, and it happens to be recoverable. Treating a typo like a fabrication is a failure of its own, one that erodes trust just as fast as missing a real one. So the ledger shows its work instead of handing over a bare verdict. What got tried. What came back. Whether something close turned up under a slightly different name.

When a court has already ruled that a citation does not exist, that ruling becomes the strongest possible anchor for the verdict. When only an automated search comes back empty, Merlise reports a lower confidence and asks for a person to look.

The evidence ledger

The same claim by claim view the product shows on a live document, built from this case.

VERIFICATION LEDGERPattern check · sample citations
Reliance placed on Am. Trucking Ass'ns v. Whitfield, cited as controlling.Citation
6%
No case under this name found in any federal reporter.Refuted
Case law index
"Roberts v. Hale" cited with a docket number that does not match any filed case.Citation
9%
Docket number resolves to an unrelated matter.Refuted
PACER / RECAP
"Coleman v. Briggs" appears to be a short form of a properly cited earlier case.Citation
71%
Possible match under full case name; recommend human confirmation.Uncertain
Case law index

How it resolves

The resolving record

A continuously updated hallucination tracking database maintained outside any law firm or court provides the running count. Individual entries link to the underlying sanctions orders.

AI hallucination case database, HEC Paris Smart Law Hub