Background Jobs and Real-Time Progress

Background work is reliable only when intent and outcome are durable, delivery can repeat safely, retries are bounded, poison messages are isolated, and operators can inspect or replay failures. A queue transports work; PostgreSQL job and AI-run records tell the product truth. Server-sent events can project progress but never replace that truth.

What will you be able to do?

• decide whether work belongs inline or in a worker;
• design job states, leases, heartbeats, retries, and dead letters;
• make handlers idempotent under at-least-once delivery;
• stream resumable progress with authorization and backpressure.

What should become a job?

Move work when it exceeds request latency budgets, is CPU intensive, needs independent concurrency, can retry after client disconnect, or fans out. Tiny post-response tasks may use FastAPI background tasks, but they are process-local and should not own important business work.

Create a durable job record:

Record tenant, type, versioned payload reference, idempotency key, attempt, maximum attempts, schedule time, lease owner/expiry, progress, outcome, trace, and timestamps. Avoid placing secrets or large documents directly in queue messages.

How should a worker behave?

1. Claim a ready job atomically.
2. Re-load authoritative tenant and resource state.
3. Check cancellation and idempotent outcome.
4. Execute with timeout and bounded concurrency.
5. Heartbeat or checkpoint long work.
6. Persist result before acknowledging delivery.
7. Classify errors as retryable, permanent, safety, or cancellation.
8. Retry with exponential backoff and jitter; dead-letter after the limit.

Use an outbox if the database commit and queue publish must agree. A broker can be Redis-backed, PostgreSQL-backed, or managed; choose from guarantees and operations, not fashion.

How does progress reach the client?

Persist coarse progress events or sequence numbers. An SSE endpoint authenticates the requester, verifies job ownership, sends bounded events with IDs, heartbeats periodically, and resumes after Last-Event-ID. If the stream fails, GET /v1/jobs/{id} remains authoritative.

Why does this matter in AI-native systems?

Document ingestion, batch evaluation, long generations, and tool workflows are natural jobs. Track monetary/token budgets and provider request IDs per attempt. Cancellation stops future application work, but a provider may still finish or charge; reconcile late outcomes without overwriting terminal policy decisions.

Common mistakes

• Acknowledging before durable result commit loses work on crash.
• Retrying invalid input wastes capacity indefinitely.
• One poison job blocks an ordered queue.
• Worker trusts stale authorization embedded in the message.
• Progress percentage is fabricated when total work is unknown; report stages instead.
• No backpressure allows one tenant to exhaust workers.

AI Pair-Programmer Prompt

Exercise

Design the knowledge-ingestion job and an SSE progress endpoint. Simulate a crash after provider success but before acknowledgment, duplicate delivery, cancellation, and a poison document.

Acceptance criteria: no duplicate published chunks or charges are accepted as new outcomes, terminal states are monotonic, poison work is isolated, and clients recover without an open stream.

Knowledge check

1. Is a queue the source of product truth?
2. When should a message be acknowledged?
3. Why reload authorization state in the worker?

---

Answers

1. No; durable application records own truth.
2. After the result or idempotent outcome is durably recorded.
3. Roles, membership, resource state, or cancellation may have changed since enqueue.

Completion checklist

• Job states and terminal semantics are explicit.
• Duplicate delivery is harmless.
• Failed streams do not lose progress truth.

Primary references

• FastAPI background tasks
• PostgreSQL SKIP LOCKED
• Redis Streams