We built a serverless GPU inference platform with predictable latency

We’ve been working on a GPU-first inference platform focused on predictable latency and cost control for production AI workloads. Some of the engineering problems we ran into: - GPU cold starts and queue scheduling - Multi-tenant isolation without wasting VRAM - Model loading vs container loading tradeoffs - Batch vs real-time inference routing - Handling burst workloads without long-term GPU reservation - Cost predictability vs autoscaling behavior We wrote up the architecture decisions, what failed, and what worked. Happy to answer technical questions - especially around GPU scheduling, inference optimization, and workload isolation.