What are the industry standards for scaling open-source LLM inference infrastructure using Kubernetes as of today, and what are the best tools to automate it?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
What are the industry standards for scaling open-source LLM inference infrastructure using Kubernetes as of today, and what are the best tools to automate it?
3 real searches · 1 real-search pass
2026 Kubernetes open source LLM inference production best practices vLLM llm-d KServe NVIDIA Dynamo autoscalingGoogle ↗
OBSERVEDInstruction
official docs Kubernetes GPU scheduling autoscaling KEDA Prometheus DCGM vLLM production serving llm-d 2026Google ↗
OBSERVEDOther / navigational
CNCF Kubernetes AI inference serving gateway API inference extension llm-d KServe 2026Google ↗
OBSERVEDOther / navigational
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (3)
Classes come from the query-classifier (a local Laya model). 3 of 3 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT3 real searches · gpt-6
2026 Kubernetes open source LLM inference production best practices vLLM llm-d KServe NVIDIA Dynamo autoscaling Google ↗
official docs Kubernetes GPU scheduling autoscaling KEDA Prometheus DCGM vLLM production serving llm-d 2026 Google ↗
CNCF Kubernetes AI inference serving gateway API inference extension llm-d KServe 2026 Google ↗
- llm-d · cncf.io
- Welcome llm-d to the CNCF: Evolving Kubernetes into SOTA AI infrastructure · cncf.io
- Kubernetes WG Serving concludes following successful advancement of AI inference support · cncf.io
- Official documentation · docs.vllm.ai
- Official documentation · docs.sglang.ai
- Documentation · kserve.github.io
- LLMInferenceService with Inference Gateway Extension (IGW) · kserve.github.io
- Autoscaler · kserve.github.io
- Autoscale Inference Pools · llm-d.ai
- Project website · llm-d.ai
- Documentation · llm-d.ai
- Documentation · docs.nvidia.com
- Introduction to Dynamo · docs.nvidia.com
- Deploying Disaggregated LLM Inference Workloads on Kubernetes · developer.nvidia.com
- Documentation · gateway-api-inference-extension.sigs.k8s.io
- GitHub - kubernetes-sigs/gateway-api-inference-extension: Gateway API Inference Extension · github.com
- NVIDIA GPU Operator · docs.nvidia.com
- Kueue · kueue.sigs.k8s.io
- Kubernetes Dynamic Resource Allocation · kubernetes.io
- LeaderWorkerSet · github.com
- KAI Scheduler · github.com
- Documentation · keda.sh
- Argo CD · argo-cd.readthedocs.io
- Helm · helm.sh
- Prometheus · prometheus.io
- Grafana · grafana.com
- OpenTelemetry · opentelemetry.io
- External Secrets Operator · external-secrets.io
- Kyverno · kyverno.io
- GenAI-Perf · github.com
- vLLM benchmarks · github.com
- vLLM production stack · github.com
Queries
3 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
2026 Kubernetes open source LLM inference production best practices vLLM llm-d KServe NVIDIA Dynamo autoscaling OBSERVED | Google ↗ AI Mode ↗ | — | Instruction | — | — | 1x of 1 | |
official docs Kubernetes GPU scheduling autoscaling KEDA Prometheus DCGM vLLM production serving llm-d 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
CNCF Kubernetes AI inference serving gateway API inference extension llm-d KServe 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 |