Resilience is the ability to deliver a defined reduced service and recover when components fail. Every remote call needs a deadline, retry eligibility, attempt budget, idempotency, concurrency bound, and fallback. Circuit breakers and bulkheads can contain failures, but durable recovery also requires backups, restore tests, runbooks, and known recovery objectives.
If the client budget is 10 seconds, the route cannot give each of three dependencies 10 seconds. Reserve response overhead, allocate smaller timeouts, and propagate the remaining deadline. Database statement timeout, HTTP connect/read/write/pool timeouts, queue lease, and model timeout need intentional values.
Retry at one layer when possible. Three layers with three attempts can create 27 downstream calls. Add exponential backoff with full jitter, cap total attempts/time, respect Retry-After, and stop when the caller deadline or cost budget is exhausted.
SupportDesk AI can preserve ticket CRUD while:
Fallbacks are product behavior and need tests and evaluations. A lower-quality model is not automatically safe.
Define RPO (maximum tolerable data loss) and RTO (maximum tolerable recovery time) per capability. Back up PostgreSQL and object storage according to those objectives; protect encryption keys and configuration; document Redis reconstruction; and test restoration into an isolated environment. A backup that has never been restored is an assumption.
AI systems depend on providers, indexes, queues, policy stores, and evaluation pipelines. Recover the version mapping between source documents, chunks, embeddings, prompts, and models. After restoring durable truth, rebuild derived indexes rather than treating them as irreplaceable mystery state.
Write a game-day plan for simultaneous Redis loss and model-provider rate limiting, followed by a PostgreSQL restore scenario.
Acceptance criteria: core ticket operations remain defined, no retry storm occurs, quotas follow a declared policy, ephemeral state is rebuilt, and restored data meets measured RPO/RTO in a drill.