We Built Durable Workflows on Postgres, and Skipped Temporal

When we started building Kestrel Workflows— deterministic workflows to automate incident response, cloud provisioning, CI/CD, and developer requests — we hit the problem every automation system eventually hits: our workflows had to be durable and survive all types of failures. A workflow can be defined to trigger on failing Kubernetes workloads, run an AI root cause analysis, generate a fix, wait for a human to approve the fix via Slack, and then open a GitOps pull request. If any of our systems get rescheduled in the middle of such a workflow — and in Kubernetes, it most likely will — we couldn't just lose the workflow execution. We needed durable execution.

The idea behind durable execution is quite simple and powerful. As a program runs, it checkpoints its progress to an external system like a database. If the process dies, another one reloads the last checkpoint from the external system and continues from the last completed step. It's like autosave from a video game, but for backend code.

The default: using an external orchestrator

The usual advice for implementing durable execution is to use an external orchestrator like Temporal or AWS Step Functions, which act as central orchestrators that coordinate steps in a workflow. In this model a client submits a workflow, the orchestrator persists it in its own data store and dispatches it to a worker. For every completed step, the workflow sends its results back to the orchestrator, which checkpoints the progress and then dispatches the next step. If a worker dies, the orchestrator reassigns its steps to a healthy worker.

Click to expand

We almost went this way, but reconsidered when we looked at what it would cost us. Using an external orchestrator means having another stateful system to deploy, secure, monitor, and upgrade. It sits in the critical path of every workflow, so it becomes a single point of failure. It needs its own access controls and audit handling because workflow step payloads — which for us include things like infrastructure topology, source code, and logs — now transmit to an external system. And it brings its own data model, which is usually a key-value store that is not easy to query.

But here's the thing: durable execution is fundamentally about a database since it's all about checkpointing state to a durable external system. We already run Postgres as our system of record and have invested heavily in making sure it performs at scale, so we asked ourselves: why use an external orchestrator at all? Why not make Postgres the orchestrator?

Postgres as the orchestrator

In Kestrel, there is no orchestrator process. Each application server runs an embedded durable workflow library and talks straight to Postgres. Every trigger — from PagerDuty alerts to GitHub webhooks — inserts a row into a workflow_executions table:

CREATE TABLE workflow_executions (
  id             BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  workflow_name  TEXT NOT NULL,
  status         TEXT NOT NULL DEFAULT 'enqueued',
  input          JSONB NOT NULL,
  owner_id       TEXT,          -- which worker holds the lease
  lease_expires  TIMESTAMPTZ,   -- when the lease lapses
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

Servers poll this table to claim work, and for each completed step, the server checkpoints the output to Postgres itself.

Click to expand

One clause that makes everything safe

All coordination happens through the Postgres database, and the trick that makes it safe is one clause:

SELECT id, input
FROM workflow_executions
WHERE status = 'enqueued'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 1;

FOR UPDATE SKIP LOCKED is the quiet hero of most Postgres-backed queues — it locks the rows a worker grabs and tells every other worker to skip past them rather than block, giving the system exactly-once processing guarantees. Two servers can poll the same table at the same time and never hand the same workflow to two executors.

This claim happens inside a short transaction that also flips the status and writes the lease, so the worker owns the row before it does any work:

BEGIN;

WITH claimed AS (
  SELECT id
  FROM workflow_executions
  WHERE status = 'enqueued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE workflow_executions e
SET status        = 'running',
    owner_id      = $1,                          -- the worker's id
    lease_expires = now() + INTERVAL '30 seconds',
    updated_at    = now()
FROM claimed
WHERE e.id = claimed.id
RETURNING e.id, e.input;

COMMIT;

This transaction is the entire dispatcher. You don't need a broker, leader election, or Redis locks with TTLs. A few things that should be called out for production implementations:

• The claim transaction should be tiny. It should just lock the row, flip the status, and commit. If you hold the FOR UPDATE transaction open while the step runs, you will pin a row lock for the duration of potentially long running, external work.
• LIMIT can be more than 1 to claim a batch of executions and amortize round-trips, but the larger the batch is, the larger the blast radius if the worker dies before it finishes them.
• SKIP LOCKEDdoes not preserve FIFO under contention; for our workflows that is fine and actually desirable, but if you need strict ordering, this pattern won't work.

Let the database enforce idempotency

Step checkpoints make use of a second Postgres primitive: integrity constraints. Each step writes its output to an operation_outputs table keyed by (execution_id, step_id) as the primary key:

CREATE TABLE operation_outputs (
  execution_id  BIGINT NOT NULL REFERENCES workflow_executions(id),
  step_id       TEXT NOT NULL,
  output        JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (execution_id, step_id)
);

When a step finishes, the worker records the result with an upsert that never overwrites:

INSERT INTO operation_outputs (execution_id, step_id, output)
VALUES ($1, $2, $3)
ON CONFLICT (execution_id, step_id) DO NOTHING
RETURNING output;

So if a recovering worker reruns a step that has already committed, the duplicate INSERT would violate the unique constraint, and it would read the prior result instead. The database — not the application code — enforces idempotency. Before executing a step, the Orchestrator SDK running on the worker just checks operation_outputs for the (execution_id, step_id), and if a row exists, it skips execution and returns the stored output downstream. Here again, we express an invariant of durable execution as a constraint and let Postgres primitives uphold it, instead of writing distributed systems bookkeeping from scratch.

Crash recovery: leases and a sweeper

The interesting failures are the ones where a worker claims an execution, sets status to running, and then dies — OOM kills, node drains, pod evictions, or a dozen other Kubernetes failure reasons. These rows get stuck in running state without a live owner; this is where the lease columns come into play.

A worker that owns a running execution heartbeats it on an interval, pushing the lease forward:

UPDATE workflow_executions
SET lease_expires = now() + INTERVAL '30 seconds',
    updated_at    = now()
WHERE id = $1 AND owner_id = $2;

Implementing recovery with Postgres is just as simple as enforcing concurrency and idempotency: a sweeper resets executions whose worker has not renewed its lease, and a healthy worker reloads the checkpoints and resumes from the most recently completed steps:

UPDATE workflow_executions
SET status = 'enqueued', owner_id = NULL
WHERE status = 'running'
  AND lease_expires < now();

There are two important details that matter when implementing this for production use:

1. The lease duration is a tradeoff. If it's too short, a slow-but-alive worker can get its work stolen, resulting in two workers running the same execution (this is safe thanks to the idempotent checkpoints but wasteful). If it's too long, then recovery after a crash is delayed.
2. Without idempotent checkpoints — which we get for free from Postgres — we wouldn't be able to use aggressive leases because a false-positive sweep would cause double-execution with side effects, not just duplicated work.

Durable sleeps and human approvals

Even waiting for a human to approve workflows from Slack — which can take an arbitrary amount of time — is durable: it's simply a row inserted into a workflow_waits table with a wake_attimestamp, instead of goroutines praying the pod it's running on lives long enough to not miss the approval decision:

CREATE TABLE workflow_waits (
  execution_id  BIGINT NOT NULL REFERENCES workflow_executions(id),
  step_id       TEXT NOT NULL,
  wake_at       TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (execution_id, step_id)
);

When a workflow execution hits a sleep or approval gate, it sets its own status to waiting and inserts a workflow_waits row with a wake_at timestamp, while a periodic sweeper re-enqueues anything due:

UPDATE workflow_executions e
SET status = 'enqueued'
FROM workflow_waits w
WHERE w.execution_id = e.id
  AND e.status = 'waiting'
  AND w.wake_at <= now();

This way, wait and approval workflow steps survive any number of restarts or reschedules, since their state is stored durably in Postgres, not a thread in memory. And no matter how long a wait or approval window is — be it 2 hours or 2 weeks — the cost is the same: just one row.

Why this scales

Scalability and availability are the hard problems when you push orchestration into Postgres, which is a good thing because they're well understood with proven solutions, and in Kestrel's case, problems we previously tackled. We scale throughput by adding stateless workers; the bottleneck becomes how fast Postgres can clear the queue, and we've seen that a single instance can handle tens of thousands of workflows per second without adding read replicas or using tools like Citus. Availability becomes whatever your Postgres HA setup already gives — things like streaming replication and managed multi-AZ failover — since workers are fungible and can recover each other's state.

Connection pressure is something to look out for because hundreds of polling workers, each with a handful of connections, can exhaust Postgres' connection limit quickly. A solution to this is to use PgBouncer transaction pooling in front of Postgres and have idle workers back off with a poll interval and jitter. But if you need fan-out across thousands of workers and sub-millisecond dispatch latency, an external durable execution engine like Temporal may be the better architectural choice. Because our workloads are I/O-bound automation pipelines — which can run from seconds to hours, need only modest concurrency, and are heavy on external calls — using Postgres is not a compromise, it's actually a better fit.

Another thing to watch for is dead tuples becoming a bottleneck at high churn. A busy queue table can become a hotspot of UPDATEs and DELETEs, which generates bloat. Thankfully, tuning autovacuum on these tables, keeping the hot row set small with partial indexes, and partitioning completed executions away from the live queue will keep the working set small and avoid this problem entirely.

Observability for free

You also get observability for free, which is what I like most about using Postgres here. Every workflow and every step is a row in a table, so monitoring is just SQL. For example, if you want every run that errored this month, you just execute:

SELECT * FROM workflow_executions
WHERE created_at > now() - INTERVAL '1 month'
  AND status = 'error';

This looks extremely trivial, but it's only possible because Postgres is relational. You can do a lot more powerful things, like joining executions against steps and approvals to answer questions like "which fixes got rejected, and at which step?" And with a few secondary indexes, you can build a real-time workflow observability dashboard with no extra infrastructure. Key-value stores that most external orchestrators expose cannot do this.

Reliability and security collapse to one dependency

Another benefit of using Postgres for durable workflows is that reliability and security collapse into a single dependency. The only thing that can take down Kestrel workflows is Postgres failing, but as our primary system of record, if Postgres fails, so does everything else. And because no workflow data ever leaves the database, we implemented durable execution without adding new attack surface and systems to audit.

Databases like Postgres are already part of so many companies' infrastructure, including Kestrel's. So, we decided it made more sense to reuse ours instead of bolting on an external orchestrator on top of it. If you already run Postgres, you have everything you need for durable workflows too.