A Small Task Queue That Survives Crashes

Oct 2025 · 3 min read

A system where you submit jobs, workers run them, and every job always ends as “completed” or “failed”, even if a worker or the coordinator crashes. If a job runs twice, you still get one correct result.

I wanted to build a small task queue with one clear promise: no matter what crashes, every job finishes in a known state, and running a job twice never gives you two results.

Not a giant distributed framework. Just a small, dependable system where crashes are expected and handled cleanly.

The job

For testing, the job is simple:

“Count the words in a text file and save the count.”

It is deterministic, so running it twice gives the exact same answer. If a retry happens, the result remains correct.

Parts

The system is split into four parts:

• API: Accepts a job and returns a job ID.
• Coordinator: Keeps every job's status in a database (SQLite) on disk.
• Workers: Ask for a job, run it, save the result, then tell the coordinator.
• Result store: A table (or folder) where each result is saved under its job ID.

Rules

These rules make the system resilient:

No double jobs. Each submit carries a unique key. If the same key arrives again, the API returns the old job ID instead of making a new job.

Time limit (lease). When a worker takes a job, it gets a time limit. A worker that is still working sends a “still alive” signal (heartbeat) to extend it. Only the coordinator's clock decides when time is up.

Retries. If time runs out, the job goes back to the queue after a wait that grows each time. Count a try when the job is picked up, not when it fails. After the max tries (3), mark it “failed”.

No duplicate results. The result is saved under the job ID. Marking “completed” works only if the job isn't already completed. The first result wins and later ones are ignored.

Too many jobs (backpressure). If the waiting queue is over a limit, the API answers “429, try later” instead of accepting more.

Security. HTTPS plus a secret token for each worker.

Tests (the success checklist)

To verify that it actually works under failure, every test on this checklist had to pass:

• Kill a worker mid-job. The job ends as completed or failed.
• Kill a worker after it saved the result but before it told the coordinator. The retry runs, and there is still one result.
• Slow worker: it finishes after its time ran out, and a second worker already has the job. There is still one result.
• Kill and restart the coordinator. All job states are still correct.
• Submit the same job key twice. Only one job exists.
• A job that always crashes its worker ends as “failed” after the max tries.
• Send jobs faster than workers finish. The API starts answering 429.

Building it confirmed something simple: reliability is not about avoiding crashes. It is about making sure that when crashes happen, the rules keep the state clean.

Prem signing off. ✍️