Skip to Content
Meubles Morinière

Engineering choicesTechnical case studies6 min read

An assistant that does not bring the facts

A language model writes plausible figures. In an ERP, a figure commits money, the law or a promise made to a customer. So Fisher drafts, reads and answers without the model supplying a single fact.

The problem

Fisher is the ERP’s assistant. It does three things. It reads a paper document (a delivery note, a supplier’s order acknowledgement, a quote request) to fill in a form. It drafts a message to the customer or the carrier. And it answers the team’s questions about the stock, an order or how to do something (Fisher).

In all three cases, the model can write a wrong number as confidently as a right one: a quantity received, a price, a delivery date, a discount nobody decided. A message that promises “a 10% discount” or “delivery on Monday” commits the store as soon as it’s sent. And a team of a few people isn’t going to check every figure against the record open beside it.

The constraints

  • Every figure comes from the ERP. That’s my decision of October 26, 2026: a figure, a date or a status that Fisher gives comes from a read, from a calculation in the code or from an excerpt of the manual, never from the model. The code checks it and flags what has no source (AGENTS.md).
  • Documents come from outside. A supplier’s acknowledgement or a customer’s email was written by someone else. A link or an address in it may be an instruction slipped to the model.
  • Customers’ contact details stay masked, and the data isn’t used to train the model.
  • Spending is capped. Above the monthly cap, Fisher stops and the forms go back to manual entry (isBudgetExhausted).

The alternatives

  • Ask the model not to make anything up. The instructions say so, but there’s no way to verify that it obeys. And a sentence of instruction doesn’t stand up to a document that contains another one.
  • Ask it to cite its sources. A citation is generated text too. The model can cite the right record with the wrong figure.
  • Have a second model review the answer. That doubles the cost, and you replace one probabilistic opinion with another.
  • Take away the model’s ability to carry facts, then verify with an exact computation the little it still writes. That’s the option that was kept. The model isn’t asked to be reliable on facts. It’s set up so that it doesn’t supply any.

The decision

The model writes the form and makes the connections. The facts come from the ERP. For each verb, what the model could make up is removed from what it can express. In a chat answer, it still has to write numbers. There, an exact check compares each number with what the ERP gave it. What leaves the store is refused in full at the slightest gap. What stays within the team is flagged to the person reading it.

The implementation

Drafting: placeholders the ERP fills in

The model writes {destinataire}, {adresse}, {suivi}, and the ERP replaces each placeholder with its value after the response (fillPlaceholders). So names, addresses and tracking numbers are never sent to the model. In the free text the team types, each known value becomes its placeholder again before sending (maskPlaceholderValues). Then the draft is refused in full if it contains (apps/erp/shared/server/ai/draft.ts, shared/lib/draft-placeholders.ts):

  • A required placeholder that’s missing, an unknown placeholder or a stray brace.
  • A number absent from the facts, in digits or in words (“within eight days”, “ten percent”).
  • A commitment that isn’t in the facts: a discount, a goodwill gesture, a refund, something free, a warranty, a percentage.
  • A deadline the facts don’t set: tomorrow, Monday, next week.
  • A link, an email or a phone number absent from the facts: “an instruction obeyed, or an invention”.

Reading: a schema with no room for facts

The schema for reading a quote request has no price field and no VAT field, because prices come from the catalogue. A read can only attach a line to a piece in the ERP if a read tool returned that identifier during the call. Identifiers that no tool supplied are removed from the read that’s returned (Citations, seenIds, apps/erp/shared/server/ai/tools.ts). Each tool declares which kind of identifier it’s allowed to cite (FISHER_TOOLS). A dictated stock count can only name the pieces of the batch on screen, with a closed enumeration.

Reading again: only when the ERP has a doubt

On the actions the evaluation set singled out, the small model reads first. The large one only reads the whole document again if the ERP’s exact cross-check finds a gap: a quantity or a price that doesn’t match the purchase, a line outside the order (doubtsOf, shared/server/ai/second-reading.ts), or if the first read returned nothing. The code sums it up in one sentence: “Never based on a confidence the model gives itself”. For now, the evaluation set hasn’t singled out any action: the two candidates, reading a delivery note and reading a supplier’s order acknowledgement, are still read by the large model alone (firstReading, FISHER_GESTURES).

Answering: every number cross-checked at the end of the turn

A chat answer can’t do without numbers. At the end of each turn, figureCheck (shared/server/ai/stream.ts) gathers what the model received from the ERP (the question, the excerpts of the manual, the record that’s open) and everything the tools returned during the turn. Then it looks for each number of the answer in that set. The ones it doesn’t find are underlined “to be checked” where they’re written. Step numbers (“1.”, “2)”) don’t count.

figureCheckapps/erp/shared/server/ai/stream.ts · lines 151–181
/**
 * Chaque chiffre d'une réponse vient de l'ERP : à la fin du tour, ceux que ni
 * la question, ni les extraits, ni la fiche, ni une lecture ne portent sont signalés à la personne,
 * sous la réponse. Un instantané par tour : le dernier remplace les précédents.
 */
export function figureCheck(runId: string, facts: string): ChatMiddleware {
	const known: string[] = [facts];
	let text = "";
	return {
		name: "fisher-figure-check",
		onAfterToolCall: (_ctx, info) => {
			if (info.ok) known.push(JSON.stringify(info.result));
		},
		onChunk: (_ctx, chunk) => {
			if (chunk.type === "TEXT_MESSAGE_CONTENT") {
				text += chunk.delta;
				return;
			}
			if (chunk.type !== "RUN_FINISHED" || text.trim() === "") return;
			const unsupported = inventedNumbers(text.replace(LIST_NUMBER, ""), known.join("\n"));
			return [
				activitySnapshot({
					messageId: `chiffres-${runId}`,
					activityType: FISHER_FIGURES_ACTIVITY,
					content: { chiffres: unsupported },
				}),
				chunk,
			];
		},
	};
}

Writing: the person confirms

Fisher proposes. Every write to the ERP waits for someone to confirm it with a button.

The results

  • The model can’t make up what it can’t express. In a draft, it isn’t the model that writes a customer’s name or address. A quote that was read has no price from the model. And a line doesn’t point to a piece the ERP didn’t return.
  • A gap shows, so you don’t need to trust anything. A draft that makes something up is refused before it reaches the team, and the log counts refusals by reason, without the text. An answer that contains a number with no source shows it right where it is.
  • Spending can follow the real difficulty: on a singled-out action, the large model only rereads the documents in which the ERP finds a gap, or that the small one couldn’t read. No action is singled out yet.
  • The evaluation set comes from real mistakes. In the ERP, “Report a bad read” files the case in the private store. pnpm ai:eval:pull brings it back into apps/erp/ai-eval/cases/, which git ignores because it holds real documents, and pnpm ai:eval replays it against the real model. I don’t publish any success rate here, because the set isn’t in the repository and is run by hand.

The limits

  • The check is lexical. It compares sequences of digits, not quantities. A “15” passes if there’s a “15” somewhere in the facts, even in an identifier or another quantity. Conversely, “1 500” is flagged if the facts write “1500”. The check catches a crude invention, but not a subtle error.
  • A correct calculation made by the model is flagged, because a sum computed by the model isn’t in any read. That’s intended. Calculations belong to the code, and that’s how a missing tool gets noticed.
  • In the chat, the check flags but doesn’t block anything. The person has to read what’s underlined. That’s the trade-off recorded in ADR 0012.
  • Numbers written in words are only looked for in French and in English. “un”, “une” and “neuf” are excluded, because they’re too common to be reliable (“un meuble neuf”, a new piece of furniture).
  • Masking only covers what the ERP sends. A document that’s dropped in goes to Anthropic as it is, contact details included, and Anthropic keeps it for up to thirty days.

On this page