Cold Start Latency of Quantized Small Language Models on Serverless CPU Infrastructure

Serverless platforms offer compelling economics for low-traffic inference of small language models (SLMs), but cold start latency remains a critical barrier. We present a systematic empirical study of cold start performance for quantized SLMs served via llama.cpp on Google Cloud Run’s CPU-only infrastructure. We benchmark five models (270M– 3.8B parameters) across two memory tiers (4 GiB and 8 GiB), with a quantization sweep spanning five GGUF formats (Q2_K through Q8_0) for LLaMA 3.2 1B and three-point validation on Gemma 3 270M and Qwen3 0.6B. Our results reveal that model loading dominates cold start time (55–70%), the 8 GiB tier provides a hidden 2× vCPU advantage that nearly halves warm inference time, and Q4_K_M quantization achieves a Pareto-optimal tradeoff across latency, throughput, and model quality (confirmed via WikiText-2 perplexity). A threepredictor latency model (R 2 = 0.97) and break-even cost analysis provide practitioners a quantitative framework for SLM deployment decisions. Index Terms—small language models, serverless computing, cold start, quantization, llama.cpp, Cloud Run, GGUF, inference optimization

• Machine Intelligence

• Distributed Systems and Parallel Computing