Merlise achieves breakthrough results on SciFact-Open
Merlise
Editorial · Failure mode

Quotation marks are a promise. This one had nothing behind it.

20264 min read

A paraphrase leaves room for interpretation. Quotation marks do not. They tell a reader that a named person said precisely these words, and that promise is either backed by something real or it is not. When a publication cannot point to a transcript, a recording, an email, or a public statement, the quote is standing on nothing, whether or not anyone has noticed yet.

A summary can be debated. Quotation marks say the person used these exact words.

A ledger built around one question

The same set of questions gets asked of a direct quote Merlise identifies. Who is this attributed to. Where does the source record live. Does the quoted text actually appear there. Has the subject disputed it, or has an outlet already corrected or retracted it.

A sentence like "the meeting happened on Tuesday" can be checked against a calendar or a public record. A quoted sentence needs something more specific: not just that the event is plausible, but that these exact words came from that exact person. Merlise keeps that distinction rather than folding a claim into one generic truth score.

This is not only a journalism problem

Marketing pages lean on customer testimonials. Legal filings quote deposition transcripts. Research papers quote interview subjects. Sales materials quote analysts. The mechanism is identical everywhere quotation marks appear: who said this, where is the record, and how confident can a reader be that it holds.

An unsupported quote is not always malicious. Notes get lost, a paraphrase gets cleaned up until it reads like a direct quote, a translation shifts a few words. Merlise cares about the evidence, not the motive behind it, and asks that question before publication instead of after a dispute.

The evidence ledger

The same claim by claim view the product shows on a live document, built from this case.

VERIFICATION LEDGERQuote ledger, sample entries
"We've never seen results like this in twenty years of practice" (attributed to a named physician).Attribution
14%
No interview record, transcript, or public statement locates this quotation.Unsupported
No primary record located
"The merger will close in the second quarter" (attributed to a named executive, sourced to a press release).Attribution
92%
Quotation appears verbatim in the cited press release.Supported
Company press release

How it resolves

The resolving record

The Ars Technica and WIRED incidents both illustrate this pattern from different angles, alongside the wider run of newsroom incidents a continuously updated industry tracker has documented through 2026.

Press Gazette, ongoing tracker of AI related editorial incidents