What are the latency bottlenecks when running vLLM on AWS EC2 instances with NVIDIA H100 GPUs compared to A100s?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
What are the latency bottlenecks when running vLLM on AWS EC2 instances with NVIDIA H100 GPUs compared to A100s?
3 real searches · 1 real-search pass
vLLM H100 A100 performance latency bottlenecks paged attention chunked prefill FP8 documentationGoogle ↗
OBSERVEDReason
AWS EC2 P5 H100 P4d A100 instance networking GPU memory bandwidth specificationsGoogle ↗
OBSERVEDComparison
NVIDIA H100 vs A100 tensor core memory bandwidth NVLink specificationsGoogle ↗
OBSERVEDComparison
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (3)
Classes come from the query-classifier (a local Laya model). 1 of 3 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT3 real searches · gpt-6
vLLM H100 A100 performance latency bottlenecks paged attention chunked prefill FP8 documentation Google ↗
AWS EC2 P5 H100 P4d A100 instance networking GPU memory bandwidth specifications Google ↗
NVIDIA H100 vs A100 tensor core memory bandwidth NVLink specifications Google ↗
- H100 GPU · nvidia.com
- NVIDIA A100 Tensor Core GPU Datasheet · resources.nvidia.com
- New – Amazon EC2 P5 Instances Powered by NVIDIA H100 Tensor Core GPUs for Accelerating Generative AI and HPC Applications · aws.amazon.com.cdn.amazon.com
- Optimization and Tuning · docs.vllm.ai
- Optimization and Tuning · docs.vllm.ai
- Amazon EC2 P5 Instances · aws.amazon.com
- Amazon EC2 P4d Instances · aws.amazon.com
Queries
3 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
vLLM H100 A100 performance latency bottlenecks paged attention chunked prefill FP8 documentation OBSERVED | Google ↗ AI Mode ↗ | — | Reason | — | — | 1x of 1 | |
AWS EC2 P5 H100 P4d A100 instance networking GPU memory bandwidth specifications OBSERVED | Google ↗ AI Mode ↗ | — | Comparison | — | — | 1x of 1 | |
NVIDIA H100 vs A100 tensor core memory bandwidth NVLink specifications OBSERVED | Google ↗ AI Mode ↗ | — | Comparison | — | — | 1x of 1 |