# deadman — a dead-man's switch for n8n

You built an automation. It ran every night for four months. Then the scheduler
stopped queueing it, or the credential behind it lapsed, and it produced nothing
for 30 hours and **nothing on your instance told you so**, because an error
workflow only fires when something *ran and failed*. Nothing ran. There was no
error to catch.

deadman is a small outside observer that looks at the absence. It reads your
n8n instance's own execution history over its public REST API and answers one
question per workflow: *did anything start when it should have?*

- One program file, Python 3.11+ standard library only. No packages, no daemon,
  no database, no agent inside n8n. It runs from cron and exits.
- Read-only: it issues `GET` requests to your instance and never starts, stops,
  retries or edits an execution.
- Alerting by webhook POST and SMTP, **both optional** — with neither
  configured it still works, because stdout plus the exit code is a complete
  answer.

## What it catches

| class | what happened | why n8n's own error workflow cannot see it |
|---|---|---|
| **SILENCE** | a workflow due on a schedule produced **no execution** inside `every + grace` | nothing ran, so there is no failed execution to alert on. This is the 30-hours-nobody-noticed case. |
| **ERROR** | a **new** failed execution appeared (n8n statuses `error`, `crashed`) | it can see this one; deadman reports it without you wiring an error workflow into every board, and it dedupes so one failure is reported once. |
| **PROBE-FAULT** | the n8n API did not answer, or answered 401/403, or answered garbage | this is not about your workflows at all: it is how a rotated API key or a lapsed Cloud trial looks. |

The order matters, and it is deliberate: **PROBE-FAULT is decided first and it
suspends every workflow verdict.** If deadman cannot read your instance, it says
"could not read n8n" and exits 2. It does not report 47 workflows as dead
because *its own* key expired. Reporting someone else's outage as your
workflows' death is the most expensive lie a monitoring tool can tell.

## Five minutes on a VPS

```bash
mkdir -p ~/deadman && cd ~/deadman
# copy deadman.py, config.example.toml here, then:
cp config.example.toml config.toml
chmod 600 config.toml
```

1. In the n8n UI: **Settings → n8n API → Create API key**.
2. Keep the key out of the file:

```bash
printf 'N8N_API_KEY=<paste-the-key>\n' > /etc/deadman.env && chmod 600 /etc/deadman.env
```

   with in `config.toml`:

```toml
[instance]
base_url    = "https://n8n.yourcompany.com"
api_key_env = "N8N_API_KEY"
```

3. Declare the workflows that must keep running. The id is in the n8n URL when
   you open a workflow (`/workflow/<id>`), or `n8n list:workflow`:

```toml
[[workflow]]
id    = "AbC123xyZ"
name  = "Nightly customer report"
every = "1d"
grace = "30m"
```

4. Run it once by hand so you see it work:

```bash
set -a; . /etc/deadman.env; set +a
python3 deadman.py --config config.toml; echo "exit $?"
```

   **Do this before you trust it**: stop a workflow on purpose, wait past
   `every + grace`, and confirm the alert reaches your phone. A monitor nobody
   proved is a rumour.

5. Put it in cron. Every 10 minutes is a sensible poll rate; the alert
   cadence is governed by `cooldown_s`, not by how often cron runs it:

```cron
*/10 * * * * . /etc/deadman.env && cd /home/deadman && /usr/bin/python3 deadman.py --config config.toml >> /var/log/deadman.log 2>&1
```

cron mails stderr to the operator, which is how a failing alert channel reaches
you (see *Alert channels*).

## See it work with nothing of your own

You do not need an n8n instance, an API key or an alert channel to watch the
logic. `--simulate` answers the queries from a file on disk and runs **the same
decision path** — only the rows come from somewhere else.

```bash
python3 deadman.py --config config.example.toml --simulate demo_fixture.json \
        --now 2026-11-13T12:00:00Z --state demo_state.json; echo "exit $?"
```

```
2026-11-13T12:00:00Z ALERT error:PLACEHOLDER_WORKFLOW_ID 1 new failed execution(s): 4412 ...
2026-11-13T12:00:00Z ALERT silence:PLACEHOLDER_SECOND_WORKFLOW_ID no execution in 1200s ...
exit 1
```

Run the same command again (any `--now` later) and it sends **nothing new** —
and it prints `unchanged: 2 alert(s) still open (...)`, never "healthy",
because a poll with nothing new to send is not a healthy poll. Then set
`"fault": {"kind": "auth", "detail": "HTTP 401"}` in the fixture and run it
with a fresh `--state` path: exit 2, `PROBE-FAULT`, and **no workflow is judged**.
`demo_fixture.json` explains all three acts at its top. The timestamps are fixed
and the command pins `--now`, so this behaves identically forever.

## Exit codes

`--help` prints these too.

| code | means |
|---|---|
| **0** | healthy: n8n was reached and no alert is open |
| **1** | one or more alerts open (silence or error) |
| **2** | the probe could not do its job: bad config, unreachable instance, auth refused, malformed API reply |

`2` never means "your workflows are fine", and it never means "your workflows
are dead". It means deadman could not look. Treat it as *your monitoring is
down*, which is a different urgent thing. With cron, `MAILTO` or a
healthchecks-style "am I still running" ping on the exit code covers this.

## State, and why re-running is safe

State is one small JSON file (`[probe] state_file`): last-seen execution id and
time per workflow, the last successful check time, which alerts are open, and
the failed execution ids already reported. Written atomically (temp file +
rename).

- **An alert fires once** when it opens.
- **It re-fires only after `cooldown_s`** (default 6h; `0` = once per episode,
  never nag).
- **When it clears you get exactly one `RECOVERED` notice**, then silence.

That dedupe-and-recover behaviour is the product. A monitor that re-alerts every
10 minutes gets muted by the second week; a monitor that never re-alerts is
forgot. This is what the suite spends most of its assertions on.

Delete the state file and deadman is *not* broken — it just forgets what it
already told you, so whatever is still wrong alerts once again. Keep the file on
persistent storage.

## Alert channels

Both optional; configure either or both, and several of each.

```toml
[[alert.webhook]]
url = "https://hooks.slack.com/services/T000/B000/XXXX"
```

The POST body carries `{"text": "...", "subject": "...", "action": "...",
"key": "...", "at": "...", "detail": {...}}` — `text` first, so a Slack,
Mattermost or Discord incoming webhook renders it with no glue code, and the
other fields are there for anything that parses JSON.

```toml
[[alert.email]]
host = "smtp.gmail.com"
port = 587
from = "you@example.com"
to   = ["you@example.com"]
security    = "starttls"
username    = "you@example.com"
password_env = "DEADMAN_SMTP_PASSWORD"   # keep the password out of the file
```

`security` is `starttls` (587), `ssl` (465) or `none` (a local MTA). `to` may be
a list.

**If a channel refuses a notice**, the alert is still open and the exit code is
still 1, and every following poll says on stderr that the channel keeps
refusing it (cron mails stderr, so you find out). The cooldown clock keeps
running regardless — one dead webhook cannot turn one alert into an infinite
loop, and it cannot make an alert look delivered when it was not.

## Time

Everything is UTC. Every window is computed on absolute instants, so a
daylight-saving change cannot shift an alert, and a host set to the wrong zone
cannot either. A `--now` or state timestamp without an explicit offset is
**refused with an explanation** rather than guessed at, because guessing the
host's zone is precisely how a monitoring window ends up an hour off twice a
year. Two tests prove the point by running the check with `TZ=America/Toronto`
across both 2026 transitions: fall-back, where naive wall-clock subtraction
sees zero elapsed and would hide a real silence, and spring-forward, where it
sees two extra hours and would invent a false one.

## The tests are the deliverable

```bash
./selfcheck.sh          # or: python3 -m pytest test_deadman.py -q
```

`test_deadman.py` is plain asserts and needs no pytest: **51 rules, 270
assertions**, all driving the real `main()` through a fake HTTP layer that
answers with n8n's own JSON shape and raises the same `urllib` errors — so the
instance-down and 401 tests are honest tests, not pretend ones. Nothing opens a
socket. It covers: healthy run; first silence alert; the same silence NOT
re-alerting; re-alert exactly at the cooldown and not one second early; recovery
exactly once; error detection by execution id; no re-alert on the same failed
id; **no false recovery while the newest run is still the failure**; instance
down and 401 and 403 and 500 and non-JSON all classified PROBE-FAULT exit 2 and
never as silence; a probe fault leaving workflow verdicts untouched and not
resolving alerts already open; DST both directions; UTC day rollover; boundary
seconds; a `startedAt` of null (n8n: "null until the execution leaves the
queue") not counting as a run; `canceled` not a failure and `crashed` one;
`startedAfter` reaching back the full lookback; cursor pagination; 28 distinct
config failures each with its own message; a corrupt, wrong-shaped or
wrong-versioned state file recovered without crashing; webhook and SMTP
delivery; and the shipped demo command itself.

**The suite was proved to be able to fail**, by planting three breaks into
`deadman.py` and restoring the file byte-exact after each one (it hashed
`9d32d2314e305269e692b42810ca602e76649d9c36a698d901879060053c10c3` before the
first plant and after all three restores):

| planted break | what the suite said |
|---|---|
| dedupe guard removed (same silence re-alerts every poll) | 4 failed, naming `test_same_silence_does_not_realert_on_the_next_poll` |
| exit-2 classification inverted (probe fault returns 1) | 6 failed, naming `test_401_is_probe_fault_and_not_silence` and the other fault tests |
| cooldown `>=` → `>` | 1 failed, naming `test_realert_happens_exactly_at_the_cooldown_boundary` |

`selfcheck.sh` runs `py_compile`, the suite, the demo, a second demo poll for
idempotence, and a bad-config run, and prints one `PASS`/`FAIL` per step and a
final `RESULT PASS`/`RESULT FAIL` with its own exit code.

## What this will not do for you

Read this before you rely on it. These are real limits, not marketing hedges.

- **It cannot know a workflow ran and produced wrong data.** Zero rows
  exported, report emailed to nobody, `status: success`: deadman calls that
  healthy, correctly. Only a check *inside* the workflow can catch that — e.g.
  a final step that fails when the row count is zero, which deadman then sees as
  an ERROR.
- **It cannot see a downstream API that accepted a bad payload.** If your
  workflow POSTed garbage and the other side answered 200, everything looks
  fine, because everything *did* run.
- **It only watches workflows you declare.** An undeclared one can die tonight
  and deadman will never mention it. It also never queries undeclared ones.
- **It cannot tell you *why* nothing ran.** It reports the absence and when the
  last run was. Why (scheduler, credential, host, a bad deploy) is your triage.
- **If the whole instance is unreachable it will not tell you which workflows
  died** — it could not look, so it refuses to guess. Exit 2, one `probe` alert.
- **Nothing watches the watcher.** If the box running deadman is down, it does
  not alert. Cover that with cron's `MAILTO`, or a healthchecks-style ping on
  the exit code, or a second deadman on another host watching the first's cron
  log. Do not assume silence from deadman means good news — with this tool,
  unlike your workflows, silence is only meaningful if you know it is alive.
- **It is read-only and cannot fix anything.** No restarts, no retries.
- **The cadence is what you type.** If you move a workflow from hourly to
  daily, deadman keeps expecting it hourly until you edit the config — which
  means a false alert, not a missed one.
- **n8n's own retention applies.** If executions are pruned (or the instance was
  rebuilt), a gap older than what n8n still has is invisible. `max_pages` also
  bounds how far back the error class reads on a very busy workflow; when that
  happens deadman says so on stderr rather than guessing.
- **It assumes n8n's public API is enabled.** On a Cloud trial with it disabled
  you get PROBE-FAULT (404), which is the honest answer, not a workaround
  target.

## What I could not verify, said plainly

- **This has never run against a live n8n instance.** Every test drives a fake
  HTTP layer. The API contract is read from n8n's own generated endpoint
  reference for `GET /api/v1/executions` (public API v1.1.1) at
  `https://docs.n8n.io/connect/n8n-api/build-and-run/executions` and the OpenAPI
  spec it is generated from (`https://n8n-openapi-dfb.pages.dev/v1/openapi.yml`),
  and each field used carries its URL in a code comment. Verified from that page:
  the `X-N8N-API-KEY` header, the query params (`limit` ≤ 250, `cursor`,
  `workflowId`, `startedAfter`), the `{data, nextCursor}` response, the required
  item fields `id`/`status`/`startedAt`/`workflowId`, the `status` enum, and that
  `startedAt` is null until a run leaves the queue.
- **Three things the docs do not settle, and what I did instead** (each is
  marked `NOT VERIFIED` beside the code that touches it): the sort order of
  `data` — nothing depends on it, rows are compared client-side; whether
  `startedAfter` accepts sub-second precision — queries are sent at whole
  seconds and every window test is re-applied locally, so a strict or lenient
  server cannot change a verdict; and which column `startedAfter` is actually
  compared against — the client-side filter is authoritative and the server-side
  one is treated as a transfer optimisation.
- **SMTP is tested with an injected mailer**, so the message construction and
  the routing are covered, but no real provider handshake has been exercised.
  Gmail/Office365 app-password quirks are therefore untested.
- The Slack-shaped `text` field is a **compatibility courtesy, not a
  certification**: I have not posted to a real Slack or Discord webhook.

## Files

| file | what it is |
|---|---|
| `deadman.py` | the program. `--config`, `--simulate`, `--now`, `--state`, `--help` |
| `test_deadman.py` | 51 rules, 270 assertions, no pytest needed |
| `config.example.toml` | every field present and commented; copy to `config.toml` |
| `demo_fixture.json` | the stand-in instance for `--simulate` |
| `selfcheck.sh` | prove this copy is sound on your machine, in one command |