FreeToken：让游戏本跑 284B MoE 的本地推理引擎（安装使用 + vLLM/Ollama 对比）

你手里有一张 8GB 显存的笔记本显卡，想跑 DeepSeek-V4-Flash（284B 参数）——正常人告诉你别做梦。Ollama 直接说显存不够，llama.cpp 硬塞进去速度掉到没法用，vLLM 压根是给机房准备的。

FreeToken 换了个思路：不把显存当模型仓库，把它当工作台。完整权重躺在系统内存里，GPU 用到哪个专家就现取哪个。于是 8GB 笔记本能跑 35B MoE，游戏台式机能跑 284B，一张 RTX PRO 6000 能跑 753B 的 GLM-5.2。这不是 PPT，是 2026 年 8 月刚发的论文（arXiv:2608.16157）+ 能装的 PyPI 包（freetoken 0.1.2，Apache-2.0）。

下面先讲它是什么，重点给你安装和使用的完整流程，最后和 vLLM、Ollama 摆在一起对比——结论先放这：三个工具根本不抢同一批用户。

一、FreeToken 是什么：把显存当工作台，不是当仓库

一句话定位：面向消费级硬件的 MoE 专用本地推理引擎。不是大模型，不是聊天客户端，是服务引擎。

出处：论文《FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution》（arXiv:2608.16157，2026-08-17）；GitHub 仓库 FlashML-org/FreeToken（2026 年 7 月建仓，现在 5500+ star）；PyPI 包 freetoken 0.1.2（2026-08-19 上传）。作者名单里有韩松（Song Han）、Ion Stoica、Matei Zaharia、Kurt Keutzer，Berkeley/MIT 血统。

它赌的是 MoE 的稀疏性：284B 的模型，每个 token 只激活约 40B 参数，剩下 240B 权重闲着。传统框架的思路是把所有权重塞进显存——塞不下就完蛋。FreeToken 反着来：权重默认住内存，GPU 显存只留高频专家的缓存，缺哪个专家就动态调哪个。

六个核心机制（都写在论文和 README 里）：

1. 带宽自适应 CPU-GPU 协同执行（q 策略）*：不写死卸载策略，实时看 PCIe 带宽和 CPU/GPU 负载，决定哪些专家放显存缓存、哪些直接在 CPU 上算。llama.cpp 那种固定 offload 方案做不到。
2. 全层双缓冲预取流水线：GPU 算当前层，后台异步预取下一层的专家权重，把内存→GPU 的搬运延迟藏起来。
3. 全局 LRU 专家缓存 + FTW 权重格式：高频专家驻显存，冷门专家回内存；FTW 是自定义快速加载格式，ft checkpoint 可以预转换。
4. 运行时动态显存重分配：不重启、不重载模型，运行中动态划分显存——专家缓存和 KV 缓存互相调剂，适配长对话和 Agent 场景。
5. 语义感知 KV 缓存：Agent 改上下文（删工具调用、改思考块）时只重算变化的部分，不复算整段。这是冲着 Claude Code / Codex 这类工具链去的。
6. OpenAI + Anthropic 双兼容 API：/v1/chat/completions、/v1/responses、/v1/messages 全都有，客户端把 base URL 指过来就能用。

二、能跑什么：三档硬件，三个天花板

官方文档列了「known-good」模型清单（完整列表见仓库 docs/models.md）：

支持 MXFP4、NVFP4、FP8、BF16 量化格式。注意：稠密模型也能跑（Qwen3.6-27B dense 在列表里），但 FreeToken 的加速逻辑全在 MoE 上，跑稠密模型没有增益。

硬件门槛按论文摘要的结论：

• 8GB 笔记本（如 RTX 4060 Laptop） → Qwen3.6-35B-A3B 这类 35B 级，交互可用
• 游戏台式机（32GB 显存，如 RTX 5090） → DeepSeek-V4-Flash 284B 级
• 单张工作站卡（RTX PRO 6000） → GLM-5.2 753B 级

论文的基准线很有意思：图里那条虚线是 Codex 生产环境轨迹的中位数速度——33 token/s。换算成体感：一篇 1000 token 的回答大约 30 秒。FreeToken 的目标就是让消费级硬件跑 Agent 工作负载时达到或超过这个线。

三、安装：Linux 是主场，Windows 只有桌面版

先说硬要求，别装一半发现不行：

• Linux x86_64（CLI 引擎没有原生 Windows 版！Windows 用户只能从 flashml.ai 下载桌面 GUI app，或走 WSL2）
• NVIDIA GPU（RTX 30/40/50 系列），驱动 r580+（要求 CUDA 13）
• Python ≥ 3.10，推荐用 uv
• CUDA kernel 首次使用时 JIT 编译，需要 PATH 上有 nvcc（CUDA 13 toolkit）
• 主机内存要大，后面讲

方法一：uv 安装（推荐）

uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"

[accel] 带上 flashinfer 和 sglang kernel 加速包。装完跑 ft --version 验证。

方法二：pip

python -m venv .venv && source .venv/bin/activate
pip install "freetoken[accel]"

方法三：源码安装

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

验证一条龙：

ft --version
ft serve --model ~/models/Qwen3.6-35B-A3B
# 看到 "API server is ready to serve on 127.0.0.1:1919" 即成功
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.6-35B-A3B","messages":[{"role":"user","content":"hi"}]}'

模型从哪来：--model 支持本地路径或 Hugging Face repo id（如 deepseek-ai/DeepSeek-V4-Flash-0731），官方一键拉。国内下载慢就用 ModelScope 下好、指本地路径——PyPI 依赖里确实带着 modelscope 库。

四、使用：一个命令起服务，一个命令接 Claude Code

CLI 一共六个命令，核心就三个。

1. 起服务

ft serve --model ~/models/Qwen3.6-35B-A3B

--model 是唯一必填参数。dtype、attention 后端、MoE 后端、缓存大小、KV 容量、工具调用解析器——全部根据模型和显卡自动决定。默认端口 1919，默认绑 127.0.0.1。多卡机器用 --gpu 1 或 UUID 选卡。

2. 请求（OpenAI / Anthropic API 双兼容）

curl http://127.0.0.1:1919/v1/models

curl http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.6-35B-A3B",
    "messages": [{"role": "user", "content": "什么是混合专家模型？"}],
    "max_tokens": 256,
    "stream": true
  }'

Anthropic 客户端把 base URL 指到 http://127.0.0.1:1919 就能打 /v1/messages。所以 Cursor、各种本地 Agent 客户端都能直接接。

3. 终端聊天

ft shell                                    # 连上正在跑的服务
ft shell --model ~/models/Qwen3.6-35B-A3B   # 一步到位：起引擎 + 聊天

ft shell 连远程服务不需要 GPU（--server URL 可连别的机器），/help 看内置命令。

4. 接编码 Agent（重点卖点）

ft launch claude   # claude / codex / dsh / hermes / openclaw / opencode

它会写对应 Agent 的 provider 配置、缺 CLI 自动装、然后直接对着你的本地服务启动。--dry-run 先预览改动。也就是说：一条命令，Claude Code 就跑在本地 284B 模型上。

5. 其他命令

ft checkpoint --model ...   # 预转换为 FTW 快速加载格式（可选，加快启动）
ft bench bw                 # 校准 PCIe/内存带宽，为 hybrid 后端提供依据
ft ctl                      # 查询/管理运行中的服务

常用参数速查：

MoE 后端四个模式值得知道：fused（全住显存，需要显存够）、offload（专家住内存，显存留 LRU 缓存，PCIe 搬运）、cpu（未命中直接 CPU 算）、hybrid（搬运和 CPU 算重叠，ft bench bw 校准分配比例）。默认 auto：稠密模型用 fused，MoE 用 offload，检测到带宽 profile 就升 hybrid。

五、实战：用 FreeToken 跑 Gemma-4——但先分清 Dense 和 MoE

先纠正一个容易踩的认知坑：Gemma 这个名字，前几代（Gemma 1/2/3）全是稠密模型，很多人默认 Gemma-4 也是稠密的——不是。Gemma-4 分两个形态：

• Gemma-4-26B-A4B：MoE，26B 总参数、每 token 激活 4B（A4B = Active 4B）。这是 FreeToken 的主场，能吃到全部加速逻辑，8GB 笔记本就能跑。
• Gemma-4-12B：稠密模型。FreeToken 能加载，但加速机制对它全部空转。

跑 MoE 版 Gemma-4，套路和前面完全一样：

# 起服务（--model 支持本地路径或 HF repo id；国内可用 ModelScope 下好再指本地路径）
ft serve --model ~/models/gemma-4-26B-A4B-it

# 请求
curl http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma-4-26B-A4B-it","messages":[{"role":"user","content":"用三句话解释 MoE 和 Dense 的区别"}],"max_tokens":256}'

一个 Gemma-4 特有的细节：它是官方支持列表里唯一原生支持 GGUF 的模型，其它模型都要 safetensors checkpoint。如果你手上已经有一份 Gemma-4 的 GGUF 量化文件，直接指路径就能用，不用转换格式。

特别注明：FreeToken 的全部加速逻辑只针对 MoE 混合专家模型。

Dense 稠密模型和 MoE 的根本区别：稠密模型每个 token 都要激活全部参数，没有「冷专家」可以闲置；而 FreeToken 的带宽自适应调度、LRU 专家缓存、CPU-GPU 协同执行，全部建立在「大部分专家可以不上 GPU」这个前提上。前提不存在，加速就是空转——稠密模型跑 FreeToken，效果和跑通用引擎差不多，成熟度还不如 llama.cpp。所以选模型之前先确认它是 MoE：看名字里的 AxxB 后缀（A3B/A4B = 每 token 激活 3B/4B），没有这个后缀的通常是稠密模型。

六、和 vLLM、Ollama 怎么选：不是对手，是分工

怎么选，三条：

1. 你要在本地跑 DeepSeek-V4-Flash / GLM-5.2 这种超大 MoE，还想接 Claude Code → FreeToken。这是它唯一的主场，Ollama 直接加载不了 284B，vLLM 在单卡消费级上没优化。
2. 你要开生产 API 服务、几十路并发 → vLLM。FreeToken 的 --max-running-requests 默认才 4，它压根没往服务端吞吐设计。
3. 你要傻瓜式聊天、跑跑 7B/14B 小模型 → Ollama。杀鸡不用牛刀。

注意 vLLM 那条不是贬低：两个项目解决的是不同规模的问题。vLLM 的目标是「一个 GPU 集群服务几百个用户」，FreeToken 的目标是「一张消费卡服务一个 Agent」。硬要用 FreeToken 扛多用户并发，和硬用 vLLM 在 4060 上跑 284B，一样是错配。

七、别急着上车：五个现实局限

1. 主机内存是硬门槛。权重默认住内存：284B 的 NVFP4 权重约 150GB+，GLM-5.2 的 checkpoint 是 433GB。内存不够直接 OOM，显存只是缓存。小内存机器别碰大模型。
2. PCIe 带宽决定下限。专家频繁换入换出，PCIe 4.0 x8 的笔记本比台式机慢一大截。q* 策略能优化，但物理带宽骗不了人。
3. 只对 MoE 有显著增益。稠密模型（Llama 系、Qwen3.6-27B dense）没有稀疏性可用，加速机制全空转。
4. 环境要求苛刻：Linux x86_64 + NVIDIA + 驱动 r580+（CUDA 13）+ Python 3.10+ + 首次 JIT 编译要 nvcc。Windows 没有原生 CLI。AMD 卡？不支持。
5. 太新：repo 2026 年 7 月才建，PyPI 0.1.2。生态在快速迭代，小众模型适配靠社区，出问题要自己看 GitHub issues。

八、小结

FreeToken 解决的问题很具体：MoE 模型的权重稀疏和显存稀疏之间的错配。它不追求推翻 llama.cpp，也不跟 vLLM 抢服务器市场，就是给「手里有游戏本、想跑前沿 MoE、还想接 Agent」的人一条路。

装之前先算三笔账：显卡是不是 NVIDIA 30/40/50 系、系统是不是 Linux、内存够不够装下权重。三笔都过，值得试。过不了，老老实实 Ollama 或云端 API。

数据来源说明：本文信息来自论文摘要与正文（arXiv:2608.16157）、GitHub README 与 docs（install/quickstart/cli/models）、PyPI 元数据（freetoken 0.1.2）。论文图表包含各硬件档位的具体吞吐数字，正文未逐字引用；网上流传的「2225 token/s」等精确数值未经官方文本源确认，本文不采信，以你机器实测为准。

版权没有，随意转载。过意不去就加一句「原文发表于 cn-res.vip」，不加也无所谓。

Your GPU has 8GB of VRAM. You want to run DeepSeek-V4-Flash (284B parameters) locally. Normal people will tell you to stop dreaming. Ollama says "not enough VRAM", llama.cpp grinds to a crawl, and vLLM is built for datacenter racks, not your desk.

FreeToken flips the assumption: don't treat VRAM as a warehouse for the model — treat it as a workbench. The full weight set lives in system RAM; the GPU pulls in whichever experts it actually needs, on demand. Result: an 8GB laptop runs a 35B MoE, a gaming desktop runs 284B, a single RTX PRO 6000 runs GLM-5.2 at 753B. This isn't a slide deck — it's a paper published this month (arXiv:2608.16157) plus a real PyPI package (freetoken 0.1.2, Apache-2.0).

Here's what it is, a full install-and-use walkthrough, and an honest comparison with vLLM and Ollama. Spoiler: these three tools are not fighting over the same users.

One. What FreeToken is: VRAM as a workbench, not a warehouse

One-line positioning: an edge-native inference engine purpose-built for MoE models on consumer hardware. Not a model, not a chat client — a serving engine.

Credentials: paper at arXiv:2608.16157 (Aug 2026), repo FlashML-org/FreeToken (created July 2026, now 5.5k+ stars), PyPI freetoken 0.1.2 (uploaded 2026-08-19). The author list includes Song Han, Ion Stoica, Matei Zaharia, Kurt Keutzer — Berkeley/MIT pedigree.

Its bet is MoE sparsity: a 284B model activates only ~40B parameters per token; the other 240B of weights sit idle. Traditional frameworks try to stuff everything into VRAM — and fail when it doesn't fit. FreeToken inverts this: weights default to host RAM, VRAM holds an LRU cache of hot experts, and missing experts are streamed in dynamically.

Six core mechanisms (all in the paper and README):

1. Bandwidth-adaptive CPU–GPU co-execution (q* policy): no hardcoded offload strategy. It watches PCIe bandwidth and CPU/GPU load in real time and decides which experts live in VRAM cache and which get computed on CPU. llama.cpp's fixed offload can't do this.
2. Full-layer double-buffered prefetch pipeline: while the GPU computes layer N, the engine prefetches layer N+1's experts over PCIe, hiding memory-to-GPU transfer latency.
3. Global LRU expert cache + FTW weight format: hot experts stay in VRAM, cold ones return to RAM; FTW is a custom fast-load format (ft checkpoint pre-converts checkpoints).
4. Runtime VRAM reallocation: no restart, no reload — the engine reshuffles memory between expert cache and KV cache on the fly for long chats and agent sessions.
5. Semantic-aware KV caching: when an agent edits context (removes a tool call, rewrites a thinking block), only the changed segment is recomputed, not the whole prompt.
6. OpenAI + Anthropic dual-compatible APIs: /v1/chat/completions, /v1/responses, /v1/messages — point any client's base URL at it and it works.

Two. What it can run: three hardware tiers, three ceilings

The official "known-good" model list (full list in docs/models.md):

Quantization: MXFP4, NVFP4, FP8, BF16. Note: dense models do load (Qwen3.6-27B dense is listed), but FreeToken's entire acceleration story is MoE-specific — running dense models gains you nothing.

Hardware tiers (per the paper's abstract):

• 8GB laptop (e.g. RTX 4060 Laptop) → 35B-class (Qwen3.6-35B-A3B), interactive
• Gaming desktop, 32GB VRAM (e.g. RTX 5090) → DeepSeek-V4-Flash 284B-class
• Single workstation card (RTX PRO 6000) → GLM-5.2 753B

The paper's benchmark baseline is telling: a dashed line at the median Codex production decode speed — 33 tokens/s. In felt terms: a 1,000-token reply takes about 30 seconds. FreeToken's goal is consumer hardware hitting or beating that line on real agent workloads.

Three. Install: Linux is the native home; Windows gets a desktop app

Hard requirements first — check before you start:

• Linux x86_64 (the CLI engine has no native Windows build; Windows users get the desktop GUI app from flashml.ai, or run WSL2)
• NVIDIA GPU (RTX 30/40/50 series), driver r580+ (CUDA 13)
• Python ≥ 3.10, uv recommended
• CUDA kernels JIT-compile on first use — nvcc (CUDA 13 toolkit) must be on PATH
• Generous host RAM (more below)

Method 1: uv (recommended)

uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"

[accel] pulls in flashinfer and sglang kernels. Verify with ft --version.

Method 2: pip

python -m venv .venv && source .venv/bin/activate
pip install "freetoken[accel]"

Method 3: from source

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

End-to-end verification:

ft --version
ft serve --model ~/models/Qwen3.6-35B-A3B
# success: "API server is ready to serve on 127.0.0.1:1919"
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.6-35B-A3B","messages":[{"role":"user","content":"hi"}]}'

Where models come from: --model accepts a local path or a Hugging Face repo id (e.g. deepseek-ai/DeepSeek-V4-Flash-0731). If HF is slow from where you are, download via ModelScope and point at the local path — the PyPI dependencies do include modelscope.

Four. Usage: one command to serve, one command to launch Claude Code

Six CLI commands; three matter.

1. Serve

ft serve --model ~/models/Qwen3.6-35B-A3B

--model is the only required flag. dtype, attention backend, MoE backend, cache sizes, KV capacity, tool-call parser — everything resolves from the checkpoint and your GPU. Default port 1919, bound to 127.0.0.1. Multi-GPU: --gpu 1 or a UUID.

2. Request (OpenAI / Anthropic dual API)

curl http://127.0.0.1:1919/v1/models

curl http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.6-35B-A3B",
    "messages": [{"role": "user", "content": "What is a Mixture-of-Experts model?"}],
    "max_tokens": 256,
    "stream": true
  }'

Anthropic clients just point their base URL at http://127.0.0.1:1919 and hit /v1/messages. Cursor and local agent clients plug straight in.

3. Terminal chat

ft shell                                    # attach to the running server
ft shell --model ~/models/Qwen3.6-35B-A3B   # all-in-one: engine + chat

Attach mode needs no GPU and can drive a remote server (--server URL). /help lists in-shell commands.

4. Hook up a coding agent (the headline feature)

ft launch claude   # claude / codex / dsh / hermes / openclaw / opencode

It writes that agent's provider config, installs the CLI if missing, and launches it against your local server. --dry-run previews the changes. Meaning: one command and Claude Code is running on a local 284B model.

5. The rest

ft checkpoint --model ...   # optional: pre-convert to FTW fast-load format
ft bench bw                 # calibrate PCIe/RAM bandwidth for the hybrid backend
ft ctl                      # inspect/manage a running server

Quick reference:

Four MoE backends worth knowing: fused (all experts on GPU — needs the VRAM), offload (experts in RAM, LRU cache on GPU, streamed over PCIe), cpu (misses computed on CPU instead of fetched), hybrid (fetch some misses over PCIe while computing others on CPU, overlapped; calibrate with ft bench bw). Default auto: dense → fused, MoE → offload, upgraded to hybrid when a bandwidth profile recommends it.

Five. Hands-on: run Gemma-4 — but first, dense vs MoE

Quick correction of a common assumption: Gemma 1/2/3 were all dense models, so most people assume Gemma-4 is dense too — it isn't. Gemma-4 comes in two flavors:

• Gemma-4-26B-A4B: MoE — 26B total, 4B active per token (A4B = Active 4B). FreeToken's home turf, full acceleration, runs on an 8GB laptop.
• Gemma-4-12B: dense. FreeToken can load it, but every acceleration mechanism idles.

Running the MoE variant is the same pattern as before:

# serve (--model takes a local path or an HF repo id)
ft serve --model ~/models/gemma-4-26B-A4B-it

# request
curl http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma-4-26B-A4B-it","messages":[{"role":"user","content":"Explain MoE vs dense in three sentences"}],"max_tokens":256}'

Gemma-4-specific detail: it's the only family in the official support list with native GGUF support — every other model needs safetensors checkpoints. If you already have a quantized Gemma-4 GGUF, point --model at it directly; no conversion needed.

Important: every FreeToken acceleration mechanism is MoE-only.

A dense model activates all its parameters for every token — there are no "cold experts" to leave idle. Bandwidth-adaptive scheduling, LRU expert caching, CPU–GPU co-execution: all of it assumes "most experts can stay off the GPU". No sparsity, no gain. A dense model on FreeToken behaves like it would on any generic engine — with less maturity than llama.cpp. So check before you download: is it actually MoE? The AxxB suffix (A3B/A4B = 3B/4B active per token) is the tell; no suffix usually means dense.

Six. vLLM vs Ollama vs FreeToken: division of labor, not competition

Three-way decision:

1. Local DeepSeek-V4-Flash / GLM-5.2-class MoE, plus Claude Code integration → FreeToken. That's its only home turf. Ollama can't even load 284B; vLLM has no single-consumer-card story.
2. Production API serving, dozens of concurrent users → vLLM. FreeToken's --max-running-requests defaults to 4; it was never designed as a throughput server.
3. Frictionless chat with small models → Ollama. Don't bring a scalpel to cut bread.

That vLLM line isn't a knock — the two projects solve different scales. vLLM serves hundreds of users from a cluster; FreeToken serves one agent from one card. Using FreeToken for multi-user concurrency is as wrong a fit as using vLLM to run 284B on a 4060.

Seven. Before you jump in: five real limitations

1. Host RAM is the hard wall. Weights default to memory: 284B at NVFP4 is ~150GB+, and GLM-5.2's checkpoint is 433GB. Not enough RAM → OOM. VRAM is just a cache. Small-memory machines should not touch big models.
2. PCIe bandwidth sets the floor. With experts constantly swapping, a PCIe 4.0 x8 laptop loses badly to a desktop. The q* policy optimizes, but physics doesn't bend.
3. MoE-only gains. Dense models (Llama-family, Qwen3.6-27B dense) have no sparsity to exploit — the whole acceleration mechanism idles.
4. Pickier environment than most: Linux x86_64 + NVIDIA + driver r580+ (CUDA 13) + Python 3.10+ + nvcc for first-use JIT. No native Windows CLI. AMD GPUs: not supported.
5. Very young. Repo created July 2026, PyPI 0.1.2. Ecosystem is moving fast, niche model support depends on community, and when something breaks you read GitHub issues.

Eight. Bottom line

FreeToken attacks one specific problem: the mismatch between MoE's weight sparsity and VRAM's size. It doesn't try to dethrone llama.cpp or take vLLM's server market. It exists for people who own a gaming PC, want frontier MoE weights, and want an agent hooked to it.

Three checks before installing: NVIDIA 30/40/50 GPU? Linux? Enough RAM to hold the weights? Pass all three — try it. Otherwise, Ollama or cloud APIs will serve you better.

Sources: paper abstract and body (arXiv:2608.16157), GitHub README and docs (install/quickstart/cli/models), PyPI metadata (freetoken 0.1.2). The paper's figures contain per-tier throughput numbers; I haven't reproduced exact figures here. Round-number claims floating around online (e.g. "2225 tokens/s") aren't confirmed by any official text source — benchmark on your own machine.

No copyright reserved — share it freely. If you feel like it, a credit line to cn-res.vip is appreciated. Or don't. Either way.