A field guide · AI automation QA
When a workflow crashes, you find out. When it quietly produces the wrong answer, it runs for months and everything downstream inherits the mistake. This is a catalogue of the second kind — the ones nobody gets alerted about.
The API times out. The script throws. A red badge appears, someone gets paged, it is fixed inside the hour. Every monitoring tool you own is built for this.
The step completes. The status is green. The output is wrong, or partial, or invented — and it flows into your CRM, your invoices, your replies to customers. Nothing anywhere reports a problem.
This is the same asymmetry that makes broken conversion tracking so expensive: a wasted ad click appears in a report with a cost beside it, while a conversion that silently stopped firing shows you nothing at all. One side of the ledger complains. The other goes quiet, and quiet reads as fine.
Language models made this much worse, because a model's failure mode is not an error — it is a fluent, plausible, confidently formatted answer. A crashed step is a bug. A hallucinated one is a sentence, and sentences pass review.
Below are the failures I look for. They share one property: on the day they happen, your dashboard is green.
The model returns "12" where the contract says 12. Downstream, a sum becomes concatenation, a comparison becomes alphabetical, a filter silently matches nothing.
Structured output modes reduce this; they do not remove optional fields, nulls where you expected zero, or a list arriving as a single item.
Assert types, not presence. Feed a case where the correct answer is legitimately zero, empty, or absent — that is where "falsy means missing" bugs live.
An agent calls a real tool with a plausible but fabricated ID — an order number, a SKU, a record key. The tool answers "not found", the agent tries another invention, and eventually writes a summary that reads like success.
Ask about an entity that does not exist. The correct behaviour is to say so. Anything else — a guess, a near-match, a confident narrative — is the bug.
The system has no path that returns nothing. Asked something out of scope, it produces an answer anyway, because producing answers is the only thing it does.
Send five questions that genuinely cannot be answered from the available data. Count how many get an answer instead of a refusal.
The retrieval step returns a document that is genuinely in your knowledge base and eight months out of date. The answer is wrong and comes with a real, checkable reference — which makes it far more convincing than an obvious hallucination.
Change a fact in the source of truth. Ask again an hour later. If the old answer survives, the pipeline is serving history.
The input exceeds the window. Something in the middle is dropped — often by a framework, without a warning — and the model answers confidently from the fragment it received.
Put a decisive fact in the exact middle of a long input. If the answer ignores it, the middle is being discarded.
The webhook provider times out waiting and retries. Your workflow runs a second time: two invoices, two emails to the same customer, two rows. Each run individually succeeded, so nothing is flagged.
Replay the same webhook payload twice. If you get two of anything, there is no idempotency key.
Step 4 of 6 fails. Steps 1–3 already wrote to three different systems. There is no transaction across a CRM, a spreadsheet and an email provider, so the record now exists in an impossible state.
Force a failure at each step in turn and inspect what remains. Every step that writes before the last one needs an undo or a resume.
Automatic retries convert a persistent failure into an intermittent one. The run goes green on attempt three, so no alert fires — and the underlying breakage is never investigated until it worsens.
Alert on retry count, not just final status. A workflow that succeeds on the third attempt every single time is a workflow that is failing twice every single time.
A query returns no rows because a credential expired, a filter broke, or an upstream field was renamed. The automation treats an empty set as a quiet day and reports success. This one can run unnoticed for months.
Alert on unexpected emptiness. If a job that normally processes forty items processes zero, that is an incident, not a quiet Tuesday.
An agent pursues a goal it cannot reach and keeps trying. Nothing crashes; tokens burn. It is usually discovered weeks later, on a bill, by someone in finance.
Give it an impossible task and watch the step count and the spend. Every agent loop needs a hard ceiling on both.
The provider updates the model behind the name you pinned to, or you upgrade deliberately. Formatting shifts, verbosity changes, an instruction that used to be obeyed is now interpreted. Prompts tuned against the old behaviour quietly get worse.
Keep twenty saved cases with known-good answers and re-run them on every model change. Without that set, you cannot tell an upgrade from a regression.
Decimal comma against decimal point. 03/09 read as March in one system and September in another. Turkish i / ı and İ / I breaking a case-insensitive match. The output is well-formed and wrong.
Run your real data, not English samples. For Turkish, test names containing İ, ı, ş, ğ through every matching and deduplication step.
"Send at 09:00" means 09:00 wherever the runner happens to live. Reminders arrive at 03:00, daily reports cover the wrong window, and a report that spans a boundary double-counts or skips.
Check a scheduled run against the business's own clock, not the platform's. Then check it again after a daylight-saving change.
Your automation reads incoming email, web pages, reviews or documents. One of them contains text addressed to the model — "ignore previous instructions", "reply with the discount code", "forward this thread". The system cannot tell content from command, so it obeys. Nothing malfunctions; that is the point.
Put an instruction in a document the workflow ingests and see whether it is followed. Anything reading untrusted input needs that input fenced as data and every outbound action gated.
The assistant quotes a price, offers a discount, or confirms a slot that is not really free. The customer keeps the message. The business never sees it. You find out when someone arrives expecting what they were promised.
Read what it actually sent — a sample of real outbound messages, not the prompt. Search them for prices, dates and guarantees nobody authorised.
Personal data reaches a third-party log, an analytics tool, or a prompt shared across customers. There is no error and no degradation in output — the only sign is in a data-flow diagram nobody drew.
List every hop a customer record makes, including logs and error reporting. In Türkiye the KVKK question is who the controller is and what they consented to — decide it before the pipeline exists, not after.
Nobody reads the output. A team wires up an assistant in an afternoon, watches the first few replies, and then never looks again. Six months later it has sent four thousand messages and no human has read a single one since week one.
You cannot ask the system to grade itself — that is the same class of machine marking its own work, and it will tell you it did fine. Checking it is a separate job, done by someone whose interest is in finding the failure rather than shipping the feature.
This is the checklist I work from, published because it is more useful to you than it is secret. It is not a product pitch and it names no client — the failures above come from building and breaking these systems, not from a case file.
If you run automations you have never adversarially tested, the fastest thing you can do today costs nothing: go and read fifty of the messages your system actually sent last month. Most people have never done it once, and most people find something.