Having AI check AI does not work. A script does.

What I built

Raadzoeker writes summaries of what the Rotterdam city council discusses. An AI writes them, based on the transcribed debates. That is exactly the kind of work AI seems good at: reading a lot of text and turning it into something readable. It is also work where mistakes are serious. A summary that has a council member say something that was not said does harm.

What went wrong

My first version let a lighter model write summaries without having to prove anything. They contained names carried over from example files and party clichés that came from nowhere in the debate. For another topic, the model attributed statements by an alderman to a party group.

My second idea seemed logical: have a second model check the work. That did not work. The checker approved almost everything, including summaries with partly invented quotes. A model that finds the text plausible also finds the check plausible.

What worked

Every claim in a summary now has to come with a verbatim quote from the source. A script then checks word for word whether that quote really appears in the debate. If it does not, the claim is removed from the summary. For statements attributed to a party group, the script also checks that the quote comes from a fragment by that group.

The script has no opinion and cannot be talked round. That makes it the one part of the chain I can really trust.

What it does not catch

The check catches invented quotes. It does not catch a summary that stretches a real quote too far. Automatic subtitles can also name the wrong speaker, and the script sees nothing of that. That is why every summary is labelled as AI on the site and the source is always one click away.

What I take from this

• Do not ask an AI for a judgement, ask it for evidence in a form a machine can test, such as a verbatim quote or a source reference.
• Let a simple script check that evidence instead of a second model.
• Say honestly what has not been checked. That builds trust rather than breaking it.

This applies far beyond council debates. Wherever an organisation uses AI to summarise, classify or answer, the question is not whether the model is good enough. The question is how you show that the result is right.