SGLang 还是 vLLM？大模型推理框架选型，别只看 GitHub 星星

你要部署一个大模型服务。打开 GitHub，搜推理框架，跳出来两个项目：一个 89.8k 星，一个 32.3k 星。正常人第一反应：选星星多的。

先别急。这两个项目都是 UC Berkeley 出品——同一个学校，同一批方向的师兄弟。星星差三倍，不是因为差三倍，是因为一个出生早、一个定位野。

vLLM 靠 PagedAttention 起家，现在 89.8k 星，是生产部署的事实标准，Anyscale 在背后商业化。SGLang 靠 RadixAttention 出圈，现在 32.3k 星，性能上经常压着 vLLM 打，跟 DeepSeek、Qwen 绑得比谁都紧。

选哪个，取决于你的场景，不取决于星星数。这篇文章把两者的核心机制、实测性能、使用场景拆开讲清楚，最后给选型结论。

一、出身：同门师兄弟，两条路线

vLLM：2023 年出自 UC Berkeley Sky Computing Lab，靠 PagedAttention 论文一战成名，Anyscale 主导开发，背靠商业公司。生态最广：HuggingFace、Ray、LangChain、LlamaIndex 全部深度集成。Together、Perplexity、RunPod 这些服务商的底层引擎基本都是它。最新版本 v0.27.1（2026-08-11 发布）。

SGLang：2024 年 1 月出自 LMSYS（也是 UC Berkeley 的实验室，搞 Chatbot Arena 的那个），靠 RadixAttention 论文出圈。没有商业公司背书，但跟模型厂商绑得极深——DeepSeek、Qwen 的官方部署文档里都有它的位置。特性激进，经常是第一个支持新模型架构的框架。最新版本 v0.5.18（2026-08-22 发布）。

一句话定位：vLLM 是稳，SGLang 是快。

二、核心机制：一个管内存，一个管缓存

这是两者最本质的差异，理解了这一个点，选型逻辑就通了一半。

vLLM：PagedAttention —— 管 KV Cache 怎么放

LLM 推理时，每个请求的注意力矩阵（KV Cache）会占大量显存。早期框架的做法是预分配一大块连续显存，浪费严重——一个请求只用了 60% 的块，剩下的 40% 谁也动不了，这就是显存碎片。

vLLM 的 PagedAttention 把 KV Cache 切成固定大小的块，像操作系统虚拟内存一样分页管理，块可以散落在显存任意位置。碎片没了，显存利用率高了，同一个 GPU 能塞进更多并发请求。这套思路直接抄的 OS 内存管理，妙就妙在它是第一个这么干的。

配合连续批处理（continuous batching）：一个请求生成完一个 token 就可以出列，新请求立刻补位，不用等整批跑完。这是高并发吞吐的基础。

SGLang：RadixAttention —— 管 KV Cache 怎么不重复算

SGLang 的出发点不同：它问的是——为什么每个请求都要从头算一遍 KV？

现实里请求之间有大量共享前缀：RAG 场景，所有请求都带着同一个系统提示词和同一份文档；多轮对话，每一轮都在重复前面所有轮次的输入；Agent 场景，工具定义、角色设定全是重复的。

RadixAttention 把这些 KV Cache 存成一棵前缀树（radix tree），新请求来了，先查树：前缀跟我一样的，直接复用树上已有的 KV，只算新增的部分。论文数据：吞吐最高提升 5 倍以上，延迟最高降低 3.7 倍——当然这是理想场景，真实收益取决于你的前缀重复率。

一句话总结：vLLM 解决"KV Cache 怎么放得下"，SGLang 解决"KV Cache 怎么不重复算"。 前者对一切请求生效，后者对带共享前缀的请求效果爆炸。

三、性能：实测数据说话

性能是选型最关心的，但也是最容易被忽悠的。看数据，注意版本和时间。

阿里云官方压测（2025 年 6 月，最有参考价值的一组）

阿里云函数计算用 SGLang v0.4.6 和 vLLM v0.8.5 部署 Qwen 系列模型，evalscope 压测：

模型	部署	指标	SGLang vs vLLM
Qwen2.5-1.5B-Instruct	单卡	TTFT/TPOT	快 20% ~ 50%
Qwen2.5-1.5B-Instruct	单卡	吞吐	高 20% ~ 40%
Qwen2.5-1.5B-Instruct	双卡	吞吐	SGLang 提升 25%，vLLM 提升 15%
QWQ-32B-AWQ	单卡	TTFT（并发1）	快 18%
QWQ-32B-AWQ	单卡	TPOT	快 10% ~ 15%
QWQ-32B-AWQ	单卡	吞吐	高 20%（约 35 tok/s）
QWQ-32B-AWQ	双卡	吞吐	较单卡提升约 50%（约 50 tok/s）
QWQ-32B	双卡	TTFT	快 15% ~ 50%（单卡显存不足，跑不起来）

结论：大部分场景 SGLang 优于 vLLM，且 SGLang 启动速度快约 30%。

这些数字是什么概念？拿 QWQ-32B-AWQ 单卡约 35 tok/s 来说——一秒生成 35 个 token，一篇 1000 token 的回答大约要 30 秒；双卡提到约 50 tok/s，快了四成多。而 1.5B 小模型单卡能跑到几百 tok/s，差距主要体现在延迟和并发承载上。这就是为什么选型不能只看星星——同样一张卡，框架选对了能白赚 20% 到 50%。

社区测试（2025 年 2 月，Medium）

有人用同样配置实测：SGLang 吞吐 1532 tok/s，vLLM 661 tok/s——两倍以上差距。这个数字被引用了无数次，但要注意：这是特定场景特定版本的快照，不是永恒真理。

时间线变化（Reddit 实测，2025 年 3 月~12 月）

• DeepSeek-V3 刚发布时：SGLang 明显比 vLLM 快——因为 SGLang 对 DeepSeek 的 MLA 架构优化得最早
• 2025 年下半年：两者并驾齐驱。vLLM 追上了 MLA 优化，还推出了 Wide-EP 方案，官方博客宣称 DeepSeek 模型跑到 2.2k tok/s/H200
• 2025 年 12 月：RAG 高并发场景（并发 20-50）下，vLLM 反而能赢 10-15%；低并发下 SGLang 依旧更快（快 3.5 倍）

结论：性能差距是场景、版本、时间的函数，不是恒定的。 你抄任何一篇 benchmark 当结论，都可能已经过期。正确姿势：用你的模型、你的硬件、你的流量模式，自己压一遍。

四、特性对比

维度	vLLM	SGLang
出身	UC Berkeley Sky Computing Lab	UC Berkeley LMSYS
核心机制	PagedAttention（分页管 KV）	RadixAttention（前缀树缓存）
GitHub Stars	89.8k	32.3k
最新版本	v0.27.1（2026-08）	v0.5.18（2026-08）
前缀缓存	V1 引擎已支持	原生核心能力
结构化输出	支持	支持且更快
DeepSeek MLA	已优化	最早深度优化
并行方式	TP / PP / EP	TP / PP / EP / DP（数据并行一条参数搞定）
生态集成	最广（HF/Ray/LangChain/LlamaIndex）	聚焦性能，集成靠标准 API
商业支持	Anyscale 商业化	社区驱动
OpenAI 兼容 API	✅	✅
多模态	支持	支持（LLaVA、Qwen-VL 等）

注意 vLLM 的版本号：现在还是 v0.27.1，不是 1.0。V1 引擎 2025 年 1 月发布 alpha，0.8.1 起默认启用，团队一直在往 1.0 推。别被 0.x 的版本号吓到——它已经是最成熟的生产引擎之一。

五、使用场景：怎么选

生产 API 服务、要稳、团队要省心 → vLLM 生态最全，文档最厚，踩坑的人最多意味着坑都被踩平了。OpenAI 兼容 API 成熟，接任何框架都顺。Anyscale 有企业级商业支持，出问题有人兜底。如果你的需求是"跑起来别出幺蛾子"，选它。

RAG / Agent / 多轮对话、前缀重复高 → SGLang 这是 RadixAttention 的主场。你的请求共享系统提示词、共享文档、共享工具定义——前缀复用直接省掉大量重复计算。这个场景下 SGLang 的收益是实打实的，不是纸面数据。

DeepSeek 系模型 → 两者都行，SGLang 血统更纯 DeepSeek 官方对 MLA 架构的优化最早是跟 SGLang 合作的，V3/R1 刚出那阵 SGLang 是唯一能流畅跑的框架。现在 vLLM 也追上来了。追求极致就 SGLang，求稳就 vLLM。

研究、尝鲜新模型 → SGLang 特性激进，新模型架构支持最快，经常第一个上。你的需求是"最新最强"，它最合适。

追求吞吐上限、硬件吃紧 → SGLang 单卡显存紧张、想把每一分显存榨干，SGLang 的激进优化通常能多榨出 10-30%。它的 --dp（数据并行）参数一条命令就能把多卡吞吐拉满，比 vLLM 配置 TP 简单。

企业合规、需要商业支持 → vLLM Anyscale 提供 SLA 和商业支持，这是 SGLang 没有的。政企项目、客户要求供应商兜底，没得选。

六、部署与坑

两个框架部署都简单到令人发指，一行 pip + 一行命令：

# vLLM
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --tensor-parallel-size 1

# SGLang
pip install sglang[all]
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct

都是 OpenAI 兼容 API，换框架几乎不用改业务代码。

坑清单：

1. 版本锁死再测性能：不同版本性能天差地别（vLLM 切 V1 引擎前后差距很大）。线上压测必须记录版本号，不然 benchmark 就是废纸。
2. 国内网络：pip 用镜像，模型权重从 ModelScope 拉，比 HuggingFace 快一个数量级。
3. 显存评估：别拍脑袋。先按模型参数量估算（7B 权重约 14GB FP16），再用官方文档的显存计算器；KV Cache 会吃掉剩余显存，并发越高吃得越多。
4. RAG 场景别忘开前缀缓存：vLLM 默认开，SGLang 默认开，但如果你用旧版本或改了配置，关掉前缀缓存的 SGLang 优势就没了。
5. DeepSeek 671B 这种巨无霸：单机 8×H100 起步，走 EP（专家并行）而不是 TP，两框架都有现成方案，跟着官方部署文档走，别自己发明。

七、自己动手压一遍

看到这里你可能会问：你给的 benchmark 都是别人的，我到底该信谁？

都不该全信。benchmark 是场景快照：模型、硬件、版本、并发、prompt 长度、前缀复用率，任何一项不同，结论都可能反转。别人的 20% 优势，换到你的场景可能是 -10%。

所以选型的最后一步，是自己压。不用花多少钱，半天时间，照着下面做：

第一步：锁版本，起服务

两个框架各起一个服务，用同一个模型、同一张卡，记录版本号：

# vLLM（端口 8000）
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000

# SGLang（端口 8001）
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --port 8001

第二步：用 evalscope 压

魔搭官方的 evalscope 压测工具，一行命令自动出报告：

pip install evalscope[perf] -U

evalscope perf \
  --parallel 1 10 50 100 200 \
  --number 10 20 100 200 400 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api openai \
  --dataset random \
  --max-tokens 512 \
  --min-tokens 512 \
  --min-prompt-length 1024 \
  --max-prompt-length 1024 \
  --tokenizer-path Qwen/Qwen2.5-7B-Instruct

对 SGLang 的 8001 端口再跑一遍，参数完全一样。--parallel 是并发档位，--number 是每个档位的请求数，可以按你的预期流量改。

第三步：看四个指标

报告里指标很多，先看这四个：

• TTFT（首字延迟）：用户点发送到看到第一个字的等待时间。对话产品盯这个，超过 2 秒体验就崩
• TPOT / ITL（每 token 延迟）：生成速度，决定打字机效果的流畅度
• Output throughput（输出吞吐 tok/s）：每秒吐多少 token，决定单卡能扛多少并发、每个请求的算力成本
• 并发拐点：并发从 1 加到 200，吞吐一路涨到某个点开始掉——那个点就是服务拐点。你的预期并发必须低于它

第四步：你的场景特殊，就测特殊的

• 做 RAG/Agent：prompt 里放一段共享的文档或工具定义（1000+ token），测前缀复用的真实收益——这是 SGLang 的 RadixAttention 最能拉开差距的场景
• 多轮对话：用真实对话历史当输入，测长上下文表现
• 时长：每个并发档位跑 2-3 分钟，别用几十秒的快测，预热和波动会骗人
• 显存：nvidia-smi 盯着跑，确认没 OOM，顺便看两个框架的显存利用率差异

两个框架各跑一遍，同一张表填两次，谁赢你的场景，一目了然。

参考：evalscope 官方文档的示例输出长这样（200 并发、400 请求）——输出吞吐 6501 tok/s，平均 TTFT 1.3 秒，TPOT 0.029 秒。你的数字会跟它差很多，但指标长这样，你就知道自己在看什么了。

最后一句：别人的 benchmark 是别人家的孩子，你的场景才是你的成绩单。

结论

选型不是选"最好的框架"，是选"匹配你场景的框架"。

• 星星多 ≠ 适合你。vLLM 的 89.8k 星是生态积累，SGLang 的 32.3k 星是性能口碑，两者都不是选型依据
• 要稳、要生态、要商业支持 → vLLM
• 要性能、要前缀复用、要跟 DeepSeek 绑 → SGLang
• 拿不准 → 先用 vLLM 跑通，再用 SGLang 压一遍，用你自己的数据说话

最后一句：大模型推理框架的格局三个月一变。今天 SGLang 快，明天 vLLM 追平，后天可能有新选手入场。别把某篇 benchmark 当信仰——把你的场景当标准。

---

You're about to deploy an LLM service. You search GitHub for inference frameworks and two projects pop up: one with 89.8k stars, the other with 32.3k. Normal instinct: pick the one with more stars.

Hold on. Both projects come from UC Berkeley—same school, sibling labs. Three times the stars doesn't mean three times the quality. It means one was born earlier and one plays a wilder game.

vLLM built its name on PagedAttention, now at 89.8k stars—the de facto standard for production deployment, backed commercially by Anyscale. SGLang broke out with RadixAttention, now at 32.3k stars—and it regularly outperforms vLLM, with ties to DeepSeek and Qwen tighter than anyone else's.

Which one you pick depends on your scenario, not the star count. This article breaks down the core mechanisms, real benchmarks, and use cases, then gives you a selection framework.

1. Origins: Same School, Two Paths

vLLM: Born 2023 at UC Berkeley's Sky Computing Lab, famous from the PagedAttention paper, led by Anyscale. The widest ecosystem: deep integrations with HuggingFace, Ray, LangChain, and LlamaIndex. It's the engine behind Together, Perplexity, RunPod, and many more. Latest: v0.27.1 (Aug 11, 2026).

SGLang: Born January 2024 at LMSYS (also UC Berkeley—the lab behind Chatbot Arena), famous from the RadixAttention paper. No commercial backer, but deeply embedded with model vendors—it has a permanent seat in DeepSeek's and Qwen's official deployment docs. It's aggressive: often the first to support new model architectures. Latest: v0.5.18 (Aug 22, 2026).

One-line positioning: vLLM is stable. SGLang is fast.

2. Core Mechanism: One Manages Memory, One Manages Caching

This is the fundamental difference. Get this and half your selection logic is done.

vLLM: PagedAttention—How the KV Cache Fits

During LLM inference, every request's attention matrices (KV Cache) eat huge amounts of VRAM. Early frameworks pre-allocated one contiguous block per request—wasteful. A request uses 60% of its block and the other 40% is untouchable. That's fragmentation.

vLLM's PagedAttention cuts the KV Cache into fixed-size pages and manages them like OS virtual memory. Pages can scatter anywhere in VRAM. Fragmentation gone, utilization up, more concurrent requests per GPU. The idea is literally borrowed from OS memory management—and it was the first to do it.

Combined with continuous batching: a request leaves the batch the moment it finishes a token, and a new one takes its place. No waiting for the whole batch. This is the foundation of high-concurrency throughput.

SGLang: RadixAttention—How the KV Cache Doesn't Get Recalculated

SGLang asks a different question: why does every request recompute the KV from scratch?

In reality, requests share massive prefixes. In RAG, every request carries the same system prompt and the same documents. In multi-turn chat, every turn repeats all previous turns. In agent scenarios, tool definitions and role settings are identical across calls.

RadixAttention stores KV Cache in a radix tree. A new request walks the tree: if my prefix matches something stored, reuse the existing KV and only compute the new part. Paper numbers: up to 5x higher throughput, up to 3.7x lower latency—in ideal scenarios. Real-world gain depends on how much prefix you share.

One-line summary: vLLM solves "how to fit the KV Cache"; SGLang solves "how not to recompute the KV Cache." The first helps every request. The second explodes in value when requests share prefixes.

3. Performance: Let the Data Talk

Performance is what everyone cares about—and what gets most easily hyped. Read the numbers, note the versions and dates.

Alibaba Cloud official benchmark (June 2025, the most credible set)

Alibaba Cloud Function AI deployed Qwen models with SGLang v0.4.6 vs vLLM v0.8.5, stress-tested with evalscope:

Model	Deployment	Metric	SGLang vs vLLM
Qwen2.5-1.5B-Instruct	1 GPU	TTFT/TPOT	20% ~ 50% faster
Qwen2.5-1.5B-Instruct	1 GPU	Throughput	20% ~ 40% higher
Qwen2.5-1.5B-Instruct	2 GPU	Throughput	SGLang +25%, vLLM +15%
QWQ-32B-AWQ	1 GPU	TTFT (c=1)	18% faster
QWQ-32B-AWQ	1 GPU	TPOT	10% ~ 15% faster
QWQ-32B-AWQ	1 GPU	Throughput	20% higher (~35 tok/s)
QWQ-32B-AWQ	2 GPU	Throughput	~50% over single GPU (~50 tok/s)
QWQ-32B	2 GPU	TTFT	15% ~ 50% faster (won't fit on 1 GPU)

Verdict: SGLang wins in most scenarios, and boots ~30% faster.

What do these numbers mean? Take QWQ-32B-AWQ at ~35 tok/s on a single GPU: 35 tokens per second means a 1,000-token answer takes about 30 seconds. Two GPUs push it to ~50 tok/s—over 40% faster. Small models like 1.5B reach hundreds of tok/s on one card, where the gap shows up in latency and concurrency headroom instead. That's why selection isn't about star counts—on the same GPU, the right framework buys you a free 20% to 50%.

Community benchmark (Feb 2025, Medium)

Same hardware, same model: SGLang 1532 tok/s vs vLLM 661 tok/s—more than 2x. This number gets quoted everywhere. It's a snapshot of a specific scenario and specific versions, not an eternal truth.

The timeline shift (Reddit tests, Mar-Dec 2025)

• When DeepSeek-V3 launched: SGLang was clearly faster—it optimized DeepSeek's MLA architecture first
• Second half of 2025: neck and neck. vLLM caught up on MLA and shipped Wide-EP, claiming 2.2k tok/s/H200 for DeepSeek models
• December 2025: under RAG workloads at concurrency 20-50, vLLM wins by 10-15%; at low concurrency, SGLang is still faster (3.5x)

Conclusion: the performance gap is a function of scenario, version, and time—not a constant. Copying any single benchmark as your conclusion will likely be stale. The right move: benchmark your model, your hardware, your traffic pattern.

4. Feature Comparison

Dimension	vLLM	SGLang
Origin	UC Berkeley Sky Computing Lab	UC Berkeley LMSYS
Core mechanism	PagedAttention	RadixAttention
GitHub stars	89.8k	32.3k
Latest version	v0.27.1 (Aug 2026)	v0.5.18 (Aug 2026)
Prefix caching	Supported since V1	Native core feature
Structured output	Supported	Supported, faster
DeepSeek MLA	Optimized	Optimized first, deepest
Parallelism	TP / PP / EP	TP / PP / EP / DP (one flag)
Ecosystem	Broadest (HF/Ray/LangChain/LlamaIndex)	Performance-focused
Commercial support	Anyscale	Community-driven
OpenAI-compatible API	✅	✅
Multimodal	Supported	Supported (LLaVA, Qwen-VL, etc.)

Note the vLLM version number: still v0.27.1, not 1.0. The V1 engine went alpha in January 2025, became default in 0.8.1, and the team keeps pushing toward 1.0. Don't let the 0.x scare you—it's already one of the most mature production engines out there.

5. Use Cases: How to Choose

Production API service, stability first, team wants zero fuss → vLLM Biggest ecosystem, thickest docs, and the most people stepping on the same rake means most rakes are already flattened. OpenAI-compatible API is battle-tested. Anyscale offers enterprise support when things break. If your requirement is "just run without drama," pick this.

RAG / Agent / multi-turn chat with heavy prefix sharing → SGLang This is RadixAttention's home turf. Shared system prompts, shared documents, shared tool definitions—prefix reuse eliminates a lot of redundant compute. The gains here are real, not paper numbers.

DeepSeek-family models → either works, SGLang has the deeper bloodline DeepSeek's MLA optimization was originally built with SGLang; for a while after V3/R1 launched, SGLang was the only engine that ran it smoothly. vLLM has since caught up. Chase peak performance? SGLang. Chase stability? vLLM.

Research, adopting new models early → SGLang It's aggressive, ships support for new architectures first. If your requirement is "newest and fastest," this is your pick.

Pushing throughput limits, tight on hardware → SGLang When VRAM is tight and you want every last bit squeezed, SGLang's aggressive optimizations typically extract 10-30% more. Its --dp (data parallelism) flag maxes out multi-GPU throughput with one parameter, simpler than vLLM's TP setup.

Enterprise compliance, needs commercial support → vLLM Anyscale provides SLAs and paid support—SGLang doesn't have that. For government or enterprise deals where the vendor must back you up, there's no choice.

6. Deployment and Pitfalls

Both frameworks deploy almost trivially: one pip line plus one command.

# vLLM
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --tensor-parallel-size 1

# SGLang
pip install sglang[all]
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct

Both expose OpenAI-compatible APIs; switching frameworks rarely touches your business code.

Pitfalls:

1. Pin your versions before benchmarking: performance varies wildly between versions (vLLM's V1 engine switch was a big jump). Record version numbers in every benchmark, or the numbers are worthless.
2. China network: use pip mirrors; pull model weights from ModelScope—an order of magnitude faster than HuggingFace from mainland networks.
3. VRAM estimation: don't guess. Estimate from parameter count (7B weights ≈ 14GB in FP16), then use the official memory calculators; KV Cache eats whatever's left, and more concurrency eats more.
4. Keep prefix caching on: both default to on, but if you're on an old version or changed configs, SGLang's advantage silently disappears when caching is off.
5. For 671B-class models like DeepSeek: start with 8×H100, use EP (expert parallelism) not TP. Both frameworks have ready-made solutions—follow the official deployment docs, don't invent your own.

7. Benchmark It Yourself

By now you might be asking: these benchmarks are all someone else's. Who do I trust?

No one, fully. A benchmark is a snapshot of a scenario: model, hardware, version, concurrency, prompt length, prefix reuse—change any one and the conclusion can flip. Someone else's 20% edge can become a -10% loss in your setup.

So the last step of selection is to test it yourself. It doesn't cost much. Half a day. Here's how:

Step 1: Pin versions, start the services

Run both frameworks with the same model, the same GPU, and record the versions:

# vLLM (port 8000)
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000

# SGLang (port 8001)
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --port 8001

Step 2: Hammer it with evalscope

ModelScope's official load-testing tool. One command, auto-generated report:

pip install evalscope[perf] -U

evalscope perf \
  --parallel 1 10 50 100 200 \
  --number 10 20 100 200 400 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api openai \
  --dataset random \
  --max-tokens 512 \
  --min-tokens 512 \
  --min-prompt-length 1024 \
  --max-prompt-length 1024 \
  --tokenizer-path Qwen/Qwen2.5-7B-Instruct

Run the same command against SGLang's port 8001 with identical parameters. --parallel sets the concurrency steps, --number the request count per step—adjust both to your expected traffic.

Step 3: Read four metrics

The report has many metrics. Start with these four:

• TTFT (time to first token): how long until the first character appears after the user hits send. Chat products live and die by this—past 2 seconds, the experience collapses
• TPOT / ITL (per-token latency): generation speed. Determines how smooth the typewriter effect feels
• Output throughput (tok/s): how many tokens per second. Determines how much concurrency one GPU can carry and the per-request compute cost
• Concurrency knee: push concurrency from 1 to 200—throughput rises until a point where it starts falling. That point is your service's knee. Your expected concurrency must stay below it

Step 4: If your scenario is special, test the special part

• RAG/Agent workloads: put a shared document or tool definition in the prompt (1000+ tokens) and measure the real prefix-reuse gain—this is the scenario where SGLang's RadixAttention pulls farthest ahead
• Multi-turn chat: use real conversation history as input to test long-context behavior
• Duration: run each concurrency level for 2-3 minutes. A 30-second quick test lies—warmup and jitter will fool you
• VRAM: keep nvidia-smi open to confirm no OOM and compare memory utilization between frameworks

Run both frameworks, fill in the same table twice. Whoever wins your scenario is obvious.

Reference: the sample output in evalscope's official docs looks like this (200 concurrency, 400 requests)—output throughput 6501 tok/s, average TTFT 1.3s, TPOT 0.029s. Your numbers will differ wildly, but once you know what the metrics look like, you know what you're reading.

Last word: other people's benchmarks are other people's children. Your scenario is your report card.

Conclusion

Selection isn't about picking "the best framework." It's about picking the one that matches your scenario.

• More stars ≠ right for you. vLLM's 89.8k stars are ecosystem maturity; SGLang's 32.3k are performance reputation. Neither is a selection criterion.
• Want stability, ecosystem, commercial support → vLLM
• Want performance, prefix reuse, deep DeepSeek ties → SGLang
• Unsure → get it running on vLLM first, then benchmark SGLang, and let your own data decide

Last word: the inference framework landscape changes every quarter. SGLang is faster today, vLLM catches up tomorrow, a new player may appear the day after. Don't treat any benchmark as gospel—make your scenario the standard.

---

© cn-res.vip — Grout