Skip to content
EKB Labs

Patterns

What reliable automation actually looks like

Most automation fails in a small number of ways: a credential expires, a value arrives in a shape the next step cannot read, a run finishes clean and produces nothing, or a vendor goes down and takes the line with it. These are the six patterns we install against those failures, drawn from the workflows that run our own business.

Every plate is read from our live system rather than illustrated. No endpoint, credential, client or vendor appears on any of them; suppliers are labelled by role. If you want the number these patterns are measured against, including the parts of our own estate that do not yet pass, the self-audit publishes it.

§ 01 · Global error handler

A failure nobody hears about.

Every active workflow reports into one place. The alert path is ordered so that it survives the failure class it most often reports: the email escalation goes first over its own authentication, the chat alert is set to stop the workflow if it cannot deliver, and the log write is deliberately last and best effort, because it writes with the same credential class whose expiry it is usually reporting. A repeat inside the suppression window posts a quiet counted line instead of waking anyone twice.

Global error handler: the alert path, ordered to survive the failure it reports A hairline wiring diagram of one live n8n workflow. On the left, an arc of small circles represents any active workflow in the estate; dashed lines fan from them into an error trigger. The trigger feeds a format and classify step, which de-duplicates repeated failures inside a four hour window and marks credential failures as a distinct class. A gate then asks whether this is the first occurrence. If it is, the signal goes first to an email sent over an API key, which retries three times, and only then to a chat alert. If it is a repeat, it skips the email and goes straight to the chat channel as a quiet counted line. The chat step is configured to stop the workflow on failure, so an undelivered alarm is never recorded as a success. Last in the chain, and deliberately last, an append to an error log which retries twice and is best effort, because it writes with the same credential class whose expiry it is most often reporting. EKB LABS / WORKFLOW TOPOLOGY / PLATE I Global error handler The alert path is ordered so that it survives the failure class it most often reports. ANY ACTIVE WORKFLOW a failure anywhere in the estate arrives on this one trigger retry x3 / 2 s retry x2 / 2 s error trigger format + classify first occurrence? email / api key chat alert append error log 4 h suppression window credential failures are classified separately no. repeat posts a counted line and sends no email api-key auth. survives the oauth expiry it reports. runs before any oauth node. onError: stop. a delivery failure is not a false success. best effort, and last. same credential class it is usually reporting. trigger step gate retry error path EKB Labs / read from the live n8n instance, 2026-08-21 / topology only, no data 6 functional nodes + 1 documentation note / 5 connections
Every active workflow reports here. The alert path is ordered so it survives the failure class it most often reports.

§ 02 · Pipeline heartbeat

A pipeline that succeeds every day and produces nothing.

An error alarm cannot catch a workflow that runs clean and returns empty. This one watches the output instead. Both reads continue on error, so a dead credential arrives as ordinary data rather than raising, which lets the classifier separate three states rather than two: cannot read, read fine but nothing there, and healthy. The reads are chained rather than fanned out, because a parallel fan ran the workflow twice per cycle and doubled every alert.

Pipeline heartbeat: detecting the failure that never throws A hairline wiring diagram of one live n8n workflow that runs once a day. A schedule trigger feeds two reads, chained one after the other rather than fanned in parallel, because a parallel fan caused the workflow to execute twice per run and send every alert twice. Both reads are set to continue on error, so a dead credential returns an error object as ordinary data rather than raising an exception. A classify step then separates three outcomes: a read that could not complete, a read that completed and produced nothing, and a healthy pipeline. A single gate asks whether a silent failure has occurred. If it has, a chat alert fires, and that step is configured to stop the workflow on failure so an undelivered alarm is not recorded as a success. If it has not, the run ends at a no-operation step. The thresholds are stated on the plate: intelligence older than two days, or fewer than one staged draft. EKB LABS / WORKFLOW TOPOLOGY / PLATE II Pipeline heartbeat An error alarm cannot catch a workflow that runs clean and produces nothing. This one checks output. daily 09:00 read intelligence read content queue classify output silent failure? chat alert healthy / no-op onError: continue . chained, not fanned a parallel fan ran this twice per run and doubled every alert onError: stop. an alarm that fails to deliver is not a success. DETAIL / WHAT THE CLASSIFY STEP SEPARATES cannot read a dead credential returns an error object as data, not an exception no output every read succeeded and the pipeline still produced nothing healthy intelligence newer than 2 days, and at least 1 draft staged A monitor that misdiagnoses is worse than one that stays quiet, so the three are never collapsed into one message. trigger step gate alarm path EKB Labs / read from the live n8n instance, 2026-08-21 / topology only, no data 7 nodes / 5 connections / error workflow attached
An error alarm cannot catch a workflow that runs clean and produces nothing. This one checks the output instead, and separates a failed read from an empty week.

§ 03 · Credential canary

The single most common cause of automation failing after day 90.

Three credentials are asked a question every morning, with a manual entry point alongside. Each probe continues on error and always returns output, because a canary has to keep working when the thing it measures is dead. Escalation fires on any probe failure, or when a token comes inside fourteen days of a known expiry, which turns an outage into a diary entry. A green run posts one quiet line and mentions nobody.

Credential canary: testing the credentials before the work needs them A hairline wiring diagram of one live n8n workflow. Two entry points, a daily schedule and an on-demand call, converge on a config step holding the known expiry date of the publishing token. Three credentials are then probed in series: a mail credential, a spreadsheet credential and a social publishing credential. Every probe is set to continue on error and to always emit data, so a dead credential produces a reading rather than an outage; the canary must survive the thing it measures. A compose step turns the three readings and the remaining token life into one line. A gate then asks whether to escalate. It escalates on any probe failure, or when the token is inside fourteen days of expiry, in which case an email goes out over an API key with three retries before the chat line is posted. Otherwise the line is posted quietly with no mention attached, because a daily green ping is a ping that gets muted, and the red ones die with it. EKB LABS / WORKFLOW TOPOLOGY / PLATE III Credential canary Expired credentials are what actually kills automation. This asks them a question every morning. daily 07:00 on demand retry x3 / 2 s config mail probe ledger probe publish probe compose reading escalate? escalation email canary line known token expiry date onError: continue . alwaysOutputData a dead credential must produce a reading, not an outage. the canary has to survive the thing it measures. api-key auth, so it reaches a human even when every oauth token is dead. ESCALATION RULE Escalates on any probe failure, or when the publishing token is inside 14 days of its known expiry. A green run posts one quiet line with no mention attached. A daily ping you mute takes the red ones with it. trigger step gate retry escalation path EKB Labs / read from the live n8n instance, 2026-08-21 / topology only, no data 10 nodes / 9 connections / error workflow attached
Expired credentials are what actually kills automation. This asks three of them a question every morning, and it is built to keep working when the answer is no.

§ 04 · Three gates

An automated system sending something it should not have sent.

Three gates carry the reliability story. The content gate throws if any source read failed, so an outage can never be reported as a quiet week. The compliance gate fails closed: without a working unsubscribe link and a valid postal address nothing is built and nothing is previewed, and the refusal is a loud stop rather than a silent skip. The token gate sits on the human leg and refuses outright if the signing secret is unset. The machine can research, write, illustrate and typeset the entire issue. It cannot send it.

Three gates: a publishing pipeline that can build an issue but cannot send it A hairline wiring diagram of one live n8n workflow, drawn as three bands. The upper band is the scheduled leg. Four sources fan into a merge tree and then into a combine step, which throws if any source read failed, so an outage is never reported as an empty week. A first gate asks whether there is content; if not, a notice records a genuinely empty week. If there is, a language model writes the issue, the output is parsed, and an image sub-chain fans out one item per illustration, generating and hosting each with two retries and soft failure, so one bad image never blocks the issue. The chain then turns at the right edge and the middle band runs right to left. A compose step builds the final document and hands it to a compliance gate. If the issue lacks a working unsubscribe link or a valid postal address, the gate routes to a deliberate stop, which throws so the estate error handler names the missing requirement. If it passes, a soft image failure is logged, a broadcast is created but not sent, a preview is emailed, and an approval request is posted. A dashed rule then breaks the diagram: a person clicks, and no wire crosses that line. The lower band is a separate entry point that only a human action reaches. It opens an interstitial page stating there is no undo, posts a confirmation, and passes a token gate that refuses outright if the signing secret is unset, rejects automated fetches, and requires a broadcast id. Only past that gate does the send happen, after which the source rows are marked processed and a notification is posted. EKB LABS / WORKFLOW TOPOLOGY / PLATE IV Three gates The machine can research, write, illustrate and typeset the whole issue. It cannot send it. SCHEDULED LEG / BUILDS THE ISSUE 4 sources / merge tree a source read that fails throws here. an outage is never an empty week. combine + gate has content? llm writer parse output fan out per image image gen host collect urls empty-week notice the reads succeeded. there is genuinely nothing. retry x2 each . soft-fail one bad image never blocks the issue COMPLIANCE AND APPROVAL / READS RIGHT TO LEFT compose html COMPLIANCE GATE image soft-failed? log the soft-fail create broadcast preview email approval request BLOCK . stop and error to the error handler, plate I created. not sent. the machine stops here FAIL CLOSED No working unsubscribe link, or no valid postal address, and nothing is created and nothing is previewed. It throws on purpose, so the error handler names the exact requirement that is missing. A silent skip would be worse than a loud stop. A PERSON CLICKS . NO WIRE CROSSES THIS LINE HUMAN LEG / SEPARATE ENTRY POINT approval link interstitial page confirm post TOKEN GATE send broadcast mark rows notify respond there is no undo, and the page says so. refuses outright if the signing secret is unset. rejects automated fetches. requires a broadcast id. trigger step gate deliberate stop retry error path human break, nothing crosses it EKB Labs / read from the live n8n instance, 2026-08-21 / topology only, no content and no data 36 nodes / 31 connections / 3 entry points / error workflow attached
The machine can research, write, illustrate and typeset the whole issue. It cannot send it. Sending lives past a fail closed compliance gate, a human click, and a token gate.

§ 05 · Provider cascade

One vendor outage stopping the whole line.

Three independent suppliers in series. None is retried against itself, because the retry is a different supplier. Every gate that comes back empty steps down to the next, and when all three are exhausted the run reaches an explicit failure that writes a named reason to the queue row rather than going quiet. The row is claimed before the first call, so two concurrent runs cannot take the same work. Before that explicit failure step existed, an exhausted run produced no output, no reason and no alert.

Provider cascade: degrade to the next supplier, and record it when they all fail A hairline wiring diagram of the media generation section of one live n8n workflow. A queue read pulls the next staged row and immediately marks it as processing, before any model is called, so a concurrent run cannot claim the same row twice. A router then enters a descending cascade of three independent image providers. Each provider is followed by a gate asking whether an image came back. A yes at any stage exits right onto a shared bus that leads to normalise and post. A no steps down and to the right, to the next provider. No provider is retried against itself: the retry is a different supplier. If the third gate also returns no, the run reaches an explicit recorded failure, drawn as a circle with a bar, which writes a named reason to the queue row. It does not throw. Both the published path and the failure path converge on the same queue row update, so every row ends in a stated outcome. Before that explicit failure step existed, the exhausted branch was wired straight to the queue update, producing no post, no reason and no alert. EKB LABS / WORKFLOW TOPOLOGY / PLATE V Provider cascade Three suppliers in series. Every step down is a demotion, and the bottom of the stair is not silence. read next queued row mark row: processing route by provider provider 1 got an image? provider 2 got an image? provider 3 got an image? yes no normalise and post explicit failure recorded queue row updated it records the reason. it does not throw. NOTES The row is marked processing before the first model call, so a concurrent run cannot claim it twice. No provider is retried against itself. The retry is a different supplier. Before the explicit-failure step existed the exhausted branch wired straight to the queue update: no post, no stated reason, no alert. A silent drop is the most expensive kind of failure. trigger step gate explicit recorded failure demotion path EKB Labs / read from the live n8n instance, 2026-08-21 / topology only, no prompts and no data detail of a 51-node workflow / 47 connections
Three suppliers in series. No supplier is retried against itself, because the retry is a different supplier, and the bottom of the stair writes down why rather than going quiet.

§ 06 · Drift reconciler

Two systems of record diverging until somebody notices months later.

Two systems are compared every morning. The planning step emits one action per discrepancy and performs none of them itself, then routes each action to send, log or hold. Sends carry an idempotency key unique to the subscriber and the stage, so a re-run cannot double send. The hold branch reports whatever the fail-closed rule has blocked: if the postal address required by the sending jurisdiction is unset, every due message is held and reported rather than sent.

Drift reconciler: two systems of record, compared every morning A hairline wiring diagram of one live n8n workflow that runs daily. A schedule trigger reads the contact list held by the sending platform, then reads the separate subscriber ledger. Both reads retry three times. A planning step compares the two and emits an action per discrepancy rather than acting directly. A distribution bar then routes each action to one of three branches. The send branch dispatches the next email in the sequence, carrying an idempotency key unique to that subscriber and stage so that a re-run cannot double send, then advances the stage on the ledger row. The log branch appends any subscriber found in the sending platform with no matching intake record, so drift is written down rather than ignored. The hold branch reports anything the fail-closed rule has blocked. That rule is stated on the plate: if the postal address required by the sending jurisdiction is unset, every due email is held and reported instead of being sent. EKB LABS / WORKFLOW TOPOLOGY / PLATE VI Drift reconciler Two systems of record will disagree. This one finds the disagreement before a subscriber does. x3 x3 x2 x3 x3 daily 08:00 read platform contacts read subscriber ledger diff and plan route send next email advance stage log the drift notify hold and report emits an action per discrepancy, and acts on none of them itself idempotency key per stage. a re-run cannot double send. a subscriber found in the platform with no intake record is written down due emails held, not sent, and the hold is reported FAIL-CLOSED RULE If the postal address required by the sending jurisdiction is unset, every due email is held and reported. The sequence also stops for anyone whose ledger row is marked replied. A human wins over the schedule. trigger step route by action retry, with count held or reported EKB Labs / read from the live n8n instance, 2026-08-21 / topology only, no subscriber data 10 nodes / 7 connections / error workflow attached
Two systems of record will disagree. This one finds the disagreement every morning, writes down what drifted, and holds anything it is not allowed to send.

§ 07 · Does any of this fire in practice

Yes, and the evidence is the two plates together. On the morning these were drawn, the provider cascade on plate 05 failed twice inside three minutes. Both times the global error handler on plate 01 fired within about twenty seconds and completed. That is one pattern catching another, in production, on an ordinary Friday.

It is worth being precise about what these plates prove and what they do not. They show design, not performance. They are evidence that the failure paths were thought about and built, not a claim that nothing ever breaks. Our own audit says plenty still does.

§ 08 · What it runs on

None of the six patterns above are hypothetical. They run on six services, named here because the claim only means something if the names are real.

The EKB operating stackSix services, GitHub, n8n, Supabase, Vercel, Resend and Cloudflare, connected to a central operator hub by hairline wires.EKBGITHUBN8NSUPABASEVERCELRESENDCLOUDFLAREFIG. 07 · THE OPERATING STACKEKB LABS · 2026
Six services carry every workflow on this site: GitHub for source and the blog pipeline, n8n for orchestration, Supabase for data, Vercel for hosting, Resend for mail, Cloudflare for the edge. Nothing on the self-audit runs on a platform we do not also operate ourselves.
Book the DiagnosticRead the self-audit