Kubernetes Judgment and Production Operations
Kubernetes is a platform for scheduling and operating containers, not a prerequisite for a production FastAPI backend. Adopt it when workload diversity, scale, portability, policy, or platform-team leverage justifies cluster complexity. Production readiness still depends on correct applications, data services, security, observability, incident response, and cost ownership.
What will you be able to do?
- decide whether Kubernetes is warranted;
- define safe Deployment, Job, autoscaling, disruption, and network baselines;
- operate incidents with runbooks and evidence;
- review reliability, security, AI quality, and cost continuously.
When is Kubernetes justified?
Positive evidence includes many independently operated workloads, custom scheduling, GPU/local inference, consistent multi-environment platform APIs, advanced traffic/policy requirements, and a team able to operate clusters. Negative evidence includes one API and worker, no platform ownership, limited observability, and a managed container service that already meets requirements.
What does a baseline require?
- Deployment for stateless API; separate worker Deployment; Jobs for migrations/backfills.
- Requests and limits informed by profiles; non-root security context; minimal service account.
- Startup/readiness/liveness probes; graceful termination; disruption budget.
- Horizontal scaling on meaningful signals; hard maximum replicas.
- Network policies and controlled egress; external secret/workload identity integration.
- Spread/affinity based on failure domains; rolling-update surge/unavailable policy.
- Managed PostgreSQL/object storage and normally managed Redis/queue outside pods.
- GitOps or equivalent reviewed reconciliation, admission policy, image verification, and audit.
Do not store durable database state in an ephemeral pod because Kubernetes can restart it. Stateful systems require specialized ownership and recovery expertise.
How should incidents be handled?
- Declare severity, incident commander, channel, and timestamped log.
- Stabilize users: rollback, shed load, disable a feature, or fail over.
- Preserve evidence and communicate known facts, impact, and next update.
- Diagnose with hypotheses and disconfirming tests.
- Recover and verify user plus AI-quality signals.
- Follow with a blameless review, owners, deadlines, and effectiveness checks.
Create runbooks for database saturation, stuck migrations, queue backlog, Redis loss, provider outage, cost spike, prompt/model regression, cross-tenant alert, and credential compromise.
Why does this matter in AI-native systems?
Kubernetes can scale workers but cannot make provider quotas, evaluation, or tools safe. Local model serving introduces GPU utilization, model artifact provenance, batching, cold start, and safety-patch operations. Treat model deployments as versioned production services.
Common mistakes
- CPU limits throttle latency-sensitive Python unexpectedly without measurement.
- Readiness depends on a volatile provider and removes every pod.
- HPA scale-up exhausts database connections.
- Migration is an init container on every replica.
- Operators can exec broadly into pods with production secrets.
- Incident review stops at “human error” and changes no system.
AI Pair-Programmer Prompt
Exercise
Write an architecture decision record comparing managed containers and managed Kubernetes for SupportDesk AI at its current stage and at 50 independently owned services with GPU inference.
Acceptance criteria: the current decision is supported by evidence, future triggers are measurable, total operational ownership is named, and the incident runbook covers AI configuration rollback as well as image rollback.
Knowledge check
- Is Kubernetes required for production?
- Where should a one-time migration run?
- What is the first incident priority?
Answers
- No.
- A controlled singleton Job or release job.
- Stabilize user impact while preserving evidence and communication.
Completion checklist
Primary references