Chat-style agents finish in seconds. Enterprise agents reconcile invoices, triage incidents, or migrate records, and run for minutes or hours. Over that duration, infrastructure will interrupt them. The question is whether the interruption costs you a retry or the whole task.
The core idea
Model the run as a sequence of steps, and persist the result of each step before moving on. On restart, replay from the last completed step instead of from the beginning. This is the same idea behind workflow engines, applied to agents.
What to checkpoint
- The plan or task list, not only the conversation.
- Tool results, keyed by step, so a replay does not re-call the tool.
- Budget consumed, so retries cannot silently double-spend.
- The agent version that started the run, so a mid-run deploy does not change behavior halfway through.
Idempotency is the second half
Checkpoints help you resume. They do not help if the interruption happened *during* a side effect. Did the email send before the process died? Make every writing tool accept an idempotency key derived from run id and step id. A repeated call with the same key returns the original result instead of repeating the action.
key = hash(run_id, step_id)
tools.send_email(to, body, idempotency_key=key)For systems you do not control and that offer no such key, record intent first, perform the action, then record completion, and reconcile on resume.
Separate waiting from working
Many steps are waiting: for an approval, a webhook, or a batch job. A durable runtime should let a run sleep for days at zero compute cost and wake on an event. Do not hold a process open for a human who replies tomorrow.
Design for the failed middle
Test the ugly cases deliberately:
- Kill the worker between a tool call and its checkpoint.
- Deploy a new agent version during a live run.
- Return a rate limit from a tool on step seven of twelve.
- Let an approval time out.
If each of those has a defined, tested outcome, you have durable execution. If the answer is "it usually works," you have a demo.