Engineering choicesTechnical case studies10 min read
A morning check that says what it could not read
There’s no queue and nobody on call, so nothing guarantees that a paid order ships or that a refund goes out. Every morning, the ERP searches its own data for what shouldn’t last, and it tells “nothing to report” apart from “I couldn’t look”.
The problem
The software has no durable queue and nobody on call (Debt, risks and evolutions). When a step throws an error, the alert goes out right away. But a good part of what goes wrong in a store doesn’t throw any error. For example: a paid order nobody ships, a withdrawal received whose refund never went out, money that reached Stripe for an order the ERP rejected, or a gap in invoice numbering. To the code, each of these situations is a valid state.
The backend audit of August 31, 2026 made it its first item. The one of September 6 put the question like this: “if the shop took payments tomorrow, what would break silently, and who would find out?” (c71be614ed).
The constraints
- The team has no IT person and doesn’t watch any monitoring screen. It reads its e-mail and its phone. So the alert has to arrive there, in the team’s own words, with the action that fixes the problem.
- Some delays have a price set by law. A withdrawal must be refunded within fourteen days of the request (French Consumer Code, article L221-24), and each step of delay increases the amount owed (article L242-4).
- There’s only one scheduled task. It runs once a day (
cronsinapps/erp/vercel.json), in a function limited to five minutes (maxDuration,apps/erp/app/api/veille/route.ts). - A check that’s broken must not look like a check that found nothing.
- The alert goes out by e-mail, through a provider that may be down on the same morning.
The alternatives
- A durable queue with retries. It guarantees that a message reaches its recipient. But in most of the situations above, no message is lost, since nobody sent anything. This is a question a reader would ask, and no record shows that this option was studied for this problem.
- Error monitoring. The collector already receives server and browser errors (
packages/infra/src/collector.ts). It sees what breaks, but a forgotten order doesn’t break anything. The commit that added most of the readings puts it like this: “Twenty readings that look for the gap rather than the outage” (dc334345d7). - Say nothing when all is well, and let silence be the signal. That was the first version, on August 31: the task “sends an e-mail ONLY if one of them returns rows. Silence is the dead-man’s switch” (
63ae5133dc). Six days later, the audit noted that this silence “had no witness” (c71be614ed). The environment schema keeps the reason: “a check that stops beating goes quiet exactly like a healthy one” (apps/erp/shared/schemas/env.schema.ts).
The decision
Every morning, the ERP rereads its own data and looks for situations that shouldn’t last. Each situation is a reading: a query, a title and the action that fixes it. When a reading fails, it’s reported as “blind”. An outside service receives a signal only when the check has read everything and its report has gone out. And that service is the one that warns when the signal is missing.
The implementation
A reading looks for a state that should not last
- anomalies the ERP’s morning watch looks for in production
- 28
- apps/erp/modules/monitoring/constants/anomaly-readings.ts#ANOMALY_READINGS
Each reading is a file in apps/erp/modules/monitoring/data/ that queries the database and returns rows. This one looks for money taken with no sale. It’s a storefront order that has a Stripe payment, that the ERP hasn’t collected, that nothing has touched for an hour and that no refund in progress covers.
apps/erp/modules/monitoring/data/get-collected-without-sale.ts · lines 19–42export async function getCollectedWithoutSale(): Promise<CollectedWithoutSale[]> {
return prisma.order.findMany({
where: {
channel: SaleChannel.WEB,
stripePaymentIntentId: { not: null },
paidAt: null,
paymentStatus: {
in: [PaymentStatus.PENDING, PaymentStatus.CANCELED, PaymentStatus.FAILED],
},
updatedAt: { lt: new Date(wallClockMs() - COLLECTED_WITHOUT_SALE_MIN_AGE_MS) },
refunds: { none: { status: { notIn: FALLEN_REFUND_STATUSES } } },
},
orderBy: [{ updatedAt: "asc" }, { id: "asc" }],
take: ANOMALY_MAX_ROWS,
select: {
id: true,
orderNumber: true,
amountTotalCents: true,
paymentStatus: true,
stripePaymentIntentId: true,
createdAt: true,
},
});
}The thresholds are grouped in apps/erp/shared/constants/anomalies.ts, and they’re computed from the mechanism they watch. A payment session is considered stuck one hour after the lifetime of a session (STALE_PENDING_SESSION_MAX_AGE_MS). So if you change that lifetime, the threshold moves with it. A cancelled order that hasn’t been refunded yet enters the report at half the legal refund period (CANCELED_ORDER_REFUND_MAX_AGE_MS, derived from WITHDRAWAL_REFUND_DEADLINE_MS), while there’s still time to act. A test compares these constants and rejects a threshold that wouldn’t fall before the deadline it watches (apps/erp/test/monitoring-anomalies.regression.test.ts).
A reading returns a limited number of rows (ANOMALY_MAX_ROWS). When it reaches that cap, the report writes “at least” before the count and says there may be more.
Each section comes with its action
The ANOMALY_READINGS registry (apps/erp/modules/monitoring/constants/anomaly-readings.ts) gives each reading a title, the action that fixes it and a stake. Stakes follow a single ranking, the one in INCIDENT_STAKES: what we lose if we wait, a customer who is waiting, what blocks an order, what is owed with no urgency. The dashboard sorts its incidents into these groups, the morning e-mail sorts its sections into them, and the phone notification names the first one.
In the e-mail, each section also has an instruction written for the team and the passage of the manual that explains it (MORNING_GUIDES). This guide is typed on the keys of the registry. So a reading added without its instruction doesn’t compile.
The lines of the report contain neither a customer’s e-mail address nor a tracking number, because they land in a mailbox. They name the order, and the link leads to its page, behind the login. The regression test checks this on every line built.
The dashboard rereads part of these readings every time it’s opened, with the same “blind” state (getAdminWatchReadings). So someone who opens the ERP doesn’t wait until the next morning to see them.
“I could not look” is not “there is nothing”
The readings start together, and each one succeeds or fails on its own (Promise.allSettled). A reading that fails becomes a “blind” section of the report, with its cause. The report still goes out with the others.
This choice comes from a defect found on September 1. Back then, the readings ran in Promise.all, and a single rejection took the whole report down. That included the section on cancelled orders to refund, the only one with a legal deadline. A single reading left the database, to reread documents in file storage. If that storage had been unavailable for three days, we’d have lost three reports while the deadline was running (3bd0e66aa5).
In the log, a blind reading is written as null. A zero would read as a healthy section. When every reading is blind, the database is probably unreachable. In that case, the check sends an alert, answers with an error and stops there, without running the purges.
apps/erp/modules/monitoring/services/monitoring.service.ts · lines 60–119async function runMonitoring(
request: Request,
secret: string | undefined,
options: MonitoringOptions,
): Promise<MonitoringOutcome> {
if (secret === undefined || !isAuthorized(request, secret)) {
return { status: 401, body: "Battement de veille\u00A0: jeton absent ou invalide.\n" };
}
const withdrawalAcknowledgements =
options.acknowledgeWithdrawal === undefined
? undefined
: await resendWithdrawalAcknowledgements(options.acknowledgeWithdrawal);
const outcomes = await Promise.allSettled(READINGS.map((entry) => entry.run()));
const sections: MorningSection[] = READINGS.map(({ key, title, geste }, index) => {
const outcome = outcomes[index]!;
return outcome.status === "fulfilled"
? { key, title, geste, status: "read", entries: outcome.value }
: { key, title, geste, status: "blind", cause: describeError(outcome.reason).message };
});
const blind = sections.filter(
(section): section is Extract<MorningSection, { status: "blind" }> =>
section.status === "blind",
);
if (blind.length === READINGS.length) {
await sendAlert({
source: "veille",
context: "veille",
message: "la vérification du matin n’a pu lire aucune rubrique",
jour: day(wallClock()),
cause: blind[0]?.cause ?? "cause inconnue",
digest: VEILLE_DOWN_DIGEST,
});
return { status: 500, body: "Battement de veille\u00A0: lecture des anomalies en échec.\n" };
}
const total = anomalyCount(sections);
const outcome = await reportReadings(sections, blind, total, options);
const retention = await runRetention(options.retentionSteps);
logInfo({
source: "veille",
context: "veille",
message: "battement de veille",
anomalies: total,
sections: Object.fromEntries(
sections.map((section) => [
section.key,
section.status === "read" ? section.entries.length : null,
]),
),
retention,
...(withdrawalAcknowledgements === undefined ? {} : { withdrawalAcknowledgements }),
});
return outcome;
}An outside witness, told when everything has been read
After a successful run, the check calls an outside heartbeat service (VEILLE_HEARTBEAT_URL). When the call is missing, that service warns through its own channel.
The environment schema requires this address on a preview. In prod, it doesn’t require it at startup, because the code still treats that environment as a site under construction (isConstructionEnvironment). A reading of the check takes over. Once the site is open, it reports every morning the opening variables that are missing, this one included (getOpenSiteMissingVariables, OPENING_REQUIRED_VARIABLES).
The run only counts as successful if the token is right, if every reading could read, and if the report went out when there was something to say. A single blind reading is enough to answer with an error, even if the report arrived. A report the e-mail provider didn’t take also answers with an error. In both cases, the witness receives nothing and raises the alert on its side. So it covers the scheduled task that doesn’t fire, the unreachable database and the e-mail outage.
When it’s the witness that doesn’t answer, the check tells the team by e-mail (signalHeartbeat).
On a morning with no anomaly, the team receives nothing. On Mondays, it still receives an “all is calm” report (isWeeklyReportDue). That shows the team the check is running, without anyone having to go and look at the witness.
apps/erp/modules/monitoring/services/monitoring.service.ts · lines 216–225export async function processMonitoringRequest(
request: Request,
secret: string | undefined,
heartbeatUrl: string | undefined,
options: MonitoringOptions,
): Promise<MonitoringOutcome> {
const outcome = await runMonitoring(request, secret, options);
if (outcome.status === 200) await signalHeartbeat(heartbeatUrl);
return outcome;
}We repair before reading, and purge after
The check starts by resending the withdrawal acknowledgements that are still pending. It reads afterwards, so the report describes the state after the repair.
Once the report has gone out, it runs its housekeeping steps. The route passes them to it as a list (apps/erp/app/api/veille/route.ts): the verification of the payment chain, the till closures, the retention periods, the erasure register. Each step is isolated. If one fails, it alerts and is written as null in the log, and the following ones still run (runRetention). Confirming deliveries through La Poste tracking comes last, so a slow API doesn’t take the others’ time (apps/erp/test/veille-step-order.regression.test.ts).
If you replay the check on the same day, the same report doesn’t go out again. The idempotency key of the e-mail is a hash of its content (contentKey, apps/erp/shared/server/alert.ts), and the provider refuses a second send with the same key.
A reading that’s written is always wired
A test lists the reading files and the table of the service, and pairs them in both directions. A reading written without being wired would be “an observer that observes nothing” (63ae5133dc), and a reading imported without being called wouldn’t run. Another test pairs each scheduled task with its route (apps/erp/test/vercel-crons.regression.test.ts).
apps/erp/test/monitoring-anomalies.regression.test.ts · lines 55–63it.each(readings)("%s est APPELÉE, pas seulement importée", (file) => {
const functionName = readingFunctionName(file);
expect(
service.includes(`read: ${functionName},`),
`${file} est importée par ${SERVICE} sans être branchée sur une entrée ` +
`\`read:\` du tableau \`READINGS\` — elle ne sera jamais exécutée.`,
).toBe(true);
});What a reading finds is verified on a real database. There’s one suite per family, which creates the situation and expects its row (apps/erp/test/db/monitoring-sequence-gap.db.test.ts, monitoring-payments-without-sale.db.test.ts).
The results
On August 31, 2026, the check had four readings. Today, it has the count shown above, read from the repository at every build of this page. The commits that added the first ones each cite the audit that had found the gap by hand (3bd0e66aa5, c71be614ed).
For each failure of the check itself, there’s someone who finds out.
| What fails | What the check does | Who finds out |
|---|---|---|
| One reading | the section turns “blind”, the report goes out with the others | the team by e-mail, then the witness |
| The database | an alert, an error response, no purge | the team by e-mail, then the witness |
| The e-mail provider | the report does not go out, the response is an error | the witness, through its own channel |
| The scheduled task, or the whole deployment | nothing runs | the witness, through its own channel |
| The witness | the check ran, and an alert reports it | the team by e-mail |
| A housekeeping step | an alert, null in the log, the following steps run | the team by e-mail |
The limits
- The check runs once a day, at 6 a.m. UTC. An anomaly that throws no error can go almost a day without an alert. The dashboard shows it earlier, but someone has to open it.
- A reading only finds the gap someone thought to describe. A new kind of gap stays invisible until the audit or the incident that brings it out.
- Each reading added requires a query, a title, an action, an instruction and a passage of the manual. The service that assembles them is already over the 500-line threshold (Debt, risks and evolutions).
- The witness is one more provider, and nobody watches it. If it and the e-mail both fail on the same morning, the team learns nothing.
- In prod, nothing prevents the ERP from starting without the witness’s address. In that case, the check runs without it, and only its own report says so, once the site is open.
- The signal to the witness says that the readings read and that the report went out. A failed housekeeping step doesn’t block it. It alerts by e-mail, and if e-mail is down at that moment, only the log is left.
- The route is protected by a bearer token, with no timestamped signature (Webhooks and exchanges).