手机跑本地大模型：那张流传的对比表，硬错五处
手机跑本地大模型这个话题，最近又火了一轮。流传最广的那篇对比长文我读完了——框架搭得不错，五个框架、三档芯片、四个场景，看着挺像那么回事。
然后我把五个框架的官方 README、后端支持表、后端源码目录全拉下来对了一遍。
五处硬错误，一处重大漏项，所有性能数字零出处。
最要命的那条：它把 MNN 官方评级表里标着 C（不支持）的项目，写成了"算子级异构调度最优解"。那张表就贴在 MNN 仓库 README 的正中间，谁都能看。
我今天不复述那张表。只做一件事：把官方账本摊开，告诉你哪些机型上什么方案是真能用的，哪些是二手文章互相抄出来的幻觉。
本文所有数据来自各项目官方仓库 README、官方后端文档与源码目录，抓取时间 2026-09-17。版本与 star 数全部锁定——推理框架是时间函数，不锁版本的对比都是耍流氓。
一、先破误区：手机能跑，但核心瓶颈不是"没有独显"
手机确实没有独显，SoC 里也确实有 GPU 和 NPU。这一段没问题。
但流传文章给的两个原因——"推理方案单一，只用了 CPU"和"不能异构协同"——只对了一半。它给出的解决方案（NPU+GPU 分工）听着像万能药，实际不是。
手机上真正的三道墙，一道比一道硬：
第一道：共享内存带宽。 手机 GPU、NPU、CPU 共用同一块 LPDDR，没有独立显存。旗舰机 LPDDR5X 带宽大约 60–120 GB/s，桌面独显 GDDR6X 起步就是 1 TB/s。大模型解码阶段是彻底的带宽瓶颈——每生成一个 token，都要把参与计算的权重从内存搬一遍。带宽差一个数量级，速度就注定差一个数量级，跟你用 CPU 还是 NPU 没关系。
第二道：算子覆盖率。 NPU 不是"什么都能算的加速器"，它只认自己被编译过、被量化过的那些算子。Transformer 里恰好有一批算子在 NPU 上常年缺位：RMSNorm、SwiGLU、Softmax、KV 缓存读写。这不是厂商不用心，是这些算子对张量核心的利用效率太差，塞进去反而更慢。
第三道：热墙。 手机持续功耗预算大概是 3–8W 这个量级，跑满就降频。所以手机端跑大模型真正的用户体验不是"快不快"，是"能快多久"。
把这三道墙记住了，后面所有选型判断都顺了。异构协同的意义是"把能塞进 NPU 的算子塞进去，剩下的回退"——它是省电和分摊负载的手段，不是性能翻倍的魔法。
二、五个框架，官方账本长什么样
先给我自己的数据来源：ggml-org/llama.cpp、alibaba/MNN、ml-explore/mlx、pytorch/executorch、google-ai-edge/LiteRT-LM、mlc-ai/mlc-llm 六个仓库的官方 README 与文档。后面每个结论我都会把门牌号给你。
llama.cpp：兼容性天花板，但没有 ANE
官方 README 里有一张后端支持表，实际内容是这样：
| 后端 | 目标硬件 |
|---|---|
| Metal | Apple Silicon |
| OpenCL | Adreno GPU（高通） |
| Vulkan | 通用 GPU |
| Hexagon | Snapdragon（高通 NPU/DSP） |
| CANN | Ascend NPU（昇腾） |
| SYCL | Intel GPU |
| OpenVINO（进行中） | Intel CPU / GPU / NPU |
看清楚：苹果那一栏只有 Metal，没有 Core ML，没有 ANE。
再补一刀——我在仓库里搜了 CoreML 相关的 issue 和 PR，只有两条还开着的：一条是给视觉编码器加 Core ML 后端，一条是"能不能像 whisper.cpp 那样支持 CoreML"。文本推理的 ANE 后端，不存在。
所以"llama.cpp 底层自动联动苹果 ANE 算子优化"这句话，是编的。
顺带一个多数人不知道的事实：llama.cpp 官方主线已经有 NPU 后端——高通的 Hexagon，和昇腾的 CANN。这一点流传文章也写错了，第四节细讲。
MNN：后端最全，但官方评级表说明了一切
MNN 引擎的后端覆盖确实夸张，源码目录 source/backend/ 下面有：coreml、hiai、qnn、nnapi、neuropilot、hexagon、rknn、tensorrt、opencl、vulkan、metal、cuda、musa。
看着是无敌的全机型覆盖。但 README 里那张评级表，是自己给自己打的分。先把评级定义抄下来，这是 MNN 官方原文：
- S：支持且深度优化，推荐使用
- A：支持且能正常工作
- B：支持但有 bug 或未优化，不推荐使用
- C：不支持
然后看 NPU 那一栏的实际评级：
| NPU 后端 | Normal | FP16 | Int8 | 说明 |
|---|---|---|---|---|
| Core ML | A | C | C | 苹果 |
| HIAI | A | C | C | 华为 |
| NNAPI | B | B | B | 安卓通用，官方明确不推荐 |
| QNN | C | B | C | 高通，Normal 精度下不支持 |
翻译一下：MNN 的 NPU 后端，最好的评级是 A，而且只支持 Normal 精度——FP16 和 Int8 全部标 C。高通的 QNN 在 Normal 精度下直接标不支持，FP16 勉强 B 级（官方原话：有 bug 或未优化，不推荐使用）。
再看 LLM 这条路。MNN-LLM 的官方文档里，运行时参数 backend_type 只有三个合法值：
cpu（默认） / opencl（安卓 GPU） / metal（macOS 与 iOS GPU）
没有 npu 这个选项。
结论：MNN 引擎层确实有极广的 NPU 后端，但 LLM 推理这条路，官方运行时根本没有把 NPU 暴露出来。谁要写"MNN 是唯一支持麒麟 NPU 异构、骁龙全版本异构的最优解"，先解释一下官方评级表里的 C 是什么意思。
顺便纠正流传文章的另一句"MNN 没有普通用户成品 App"——MNN 官方有 App。安卓端是 MNN Chat App（多模态，文字/图像/音频），iOS 端有 MNN LLM iOS，两个都在 README 首页挂着。你在 App 里就能浏览、下载模型、直接对话。说它没有成品 App，说明作者没打开过 README。
ExecuTorch：官方平台表，麒麟不在里面
Meta 的端侧框架，官方 README 的平台支持表：
| 平台 | 后端 |
|---|---|
| Android | XNNPACK（CPU）、Vulkan（GPU）、Qualcomm（QNN NPU） |
| iOS / iPadOS | XNNPACK（CPU）、Core ML、MLX |
| macOS | XNNPACK、Core ML、MLX（实验） |
| Linux | XNNPACK、OpenVINO、CUDA（实验） |
| Windows | XNNPACK、CUDA / Vulkan（实验） |
| 浏览器 | WebAssembly、WebGPU（均实验） |
| 嵌入式 | Cortex-M、Ethos-U、NXP NPU |
麒麟确实不在表里。流传文章这一段的判断方向是对的。
但有个它漏掉的新东西：iOS 和 macOS 那一行有 MLX 后端。Meta 官方把苹果自家的推理框架做成了 ExecuTorch 的后端之一，这个组合值得留意。
LiteRT-LM：别再说 NNAPI 了
Google 的 LiteRT-LM（原 TFLite 那条线），官方定位是"production-ready、高性能的端侧 LLM 推理框架"，支持 Android、iOS、Web、桌面、IoT。语言绑定：Kotlin 标稳定，Swift 标早期预览。
加速方面官方写的是"通过 GPU 和 NPU 加速器获得峰值性能"，另有一个实验性的 YNNPACK delegate，只在 linux arm64 上开放。
这里必须纠正流传文章的一个说法：它说 LiteRT 依赖安卓 NNAPI 自动分配算子。但 NNAPI 已经被 Google 放上废弃通道了——现在的主流路径是厂商 delegate（高通、联发科、Google Tensor 各自的 NPU 通路），NNAPI 正在退场。一个还在拿 NNAPI 当核心卖点的对比文，说明它抄的是几年前的材料。
MLX：不碰 ANE，而且早就不只跑苹果了
MLX 官方 README 的原话，关于它跑在哪：
currently the CPU and the GPU
CPU 和 GPU。没有 ANE。 流传文章说"MLX 底层自动实现 Metal GPU+ANE 异构协同，是 iOS 端性能功耗天花板"——前半句是编的。
而且"仅适用于苹果设备"这条也过期了。2026 年的 MLX 官方安装说明里有：
pip install mlx[cuda] # Linux 上的 CUDA 后端
外加一个 CPU-only 的 Linux 包。MLX 确实不支持安卓，但它已经不是苹果专属了。
MLC-LLM：流传文章整篇没提的那个
mlc-ai/mlc-llm，23k star，活跃度不低。官方平台表：安卓走 Adreno 或 Mali 的 OpenCL，iOS 走 A 系列 GPU 的 Metal，另外还有 WebGPU / WASM 全平台覆盖。
定位很清晰：编译器路线。它把模型编译成各平台的原生产物，跨平台覆盖是六个方案里最广的（连浏览器都能跑）。成品 App 有 MLCChat。
纯 GPU 路线，不碰 NPU。一篇讲"手机跑大模型终极方案"的文章漏掉它，等于写中国互联网发展史漏掉腾讯。
三、五处硬错误和一处漏项，逐条对账
| 流传的说法 | 官方文档实际怎么说 | 结论 |
|---|---|---|
| llama.cpp 在苹果上"底层自动联动 ANE 算子优化" | 官方后端表 Apple 只有 Metal；CoreML 相关仅两个未合并的 open issue，且是视觉编码方向 | 编的 |
| MLX "自动实现 Metal GPU+ANE 异构协同" | 官方 README：跑在 CPU 和 GPU 上，无 ANE | 编的 |
| "MNN 无普通用户成品 App，需自行编译" | 官方 MNN Chat App（安卓多模态）+ MNN LLM iOS，均在 README 首页 | 编的 |
| "MNN 支持 QNN NPU+Vulkan GPU 算子级异构调度" | 官方评级表：QNN 在 Normal 精度下 C（不支持），FP16 B（官方明确不推荐） | 查无实据 |
| "llama.cpp 主线不支持 GPU+NPU 异构，只有第三方魔改支持" | 官方 docs/backend/snapdragon/README.md 明确实现 Hexagon NPU 与 CPU/GPU 的算子级回退协同 | 编的 |
| （整篇未提及 MLC-LLM） | 23k star，安卓 OpenCL / iOS Metal / WebGPU 全覆盖 | 重大漏项 |
| "NPU+GPU 分工降低 30% 以上发热，提升 20%–40% 速度" | 无任何来源。MNN、llama.cpp、ExecuTorch 官方文档均无此类数据 | 零出处 |
七个里六条站不住。这不是"观点不同"，这是没查。
四、异构协同：真有人做到了，但不是你想的那样
先把机制说清楚。所谓"NPU+GPU 异构协同"，实际做法是把大矩阵乘这类高负载、低功耗友好的算子丢给 NPU，把 NPU 不支持的算子（归一化、激活、注意力、KV 读写）回退给 GPU 或 CPU。
目前官方主线里真的把这套做出来的，是 llama.cpp 的骁龙后端。
证据在 docs/backend/snapdragon/README.md 里，我摘几条原文要点：
- 支持把模型按**层拆分（layer-split）或行拆分（row-split）**分布到多个物理 NPU 上，例如
--devices 'HTP0[0-1],HTP1[2-3]'表示跨 4 个物理 NPU - 支持虚拟 session，单个物理 NPU 上开多个会话（
HTP0:0,HTP0:1） - 有算子过滤器
GGML_HEXAGON_OPFILTER，可以精确指定哪些算子跑在 Hexagon 上，其余的自动回退到 CPU 或 GPU。文档里的示例甚至直接写着：关掉 Hexagon 上的 Flash Attention，让它回退 - 官方构建支持
GGML_HEXAGON=ON和GGML_OPENCL=ON同时打开 - 调试输出里能看到真实的 LLM 算子被卸载到 NPU 上跑，例如第 27 层的 FFN 权重矩阵乘，q4_0 × f32，单元是
HTP0
这是实打实的算子级异构。但它有几条硬限制，流传文章一句没提：
- 平台锁死在高通骁龙。 HTP 是 Hexagon Tensor Processor，别家芯片没有这个。
- 构建门槛很高。 要走 Docker 交叉编译工具链镜像，再通过 ADB 推到设备上。官方文档主要围绕开发者设备与骁龙 X 系列笔记本，不是给人装个 App 就完事的场景。
- 回退是常态，不是例外。 文档里明确写着 Flash Attention、ADD、SUB 这些算子在 Hexagon 上是关掉的，要回退。所以别指望"全模型跑在 NPU 上"。
至于 MNN：引擎里有 hiai、qnn、neuropilot、nnapi、coreml、hexagon 六个 NPU 相关后端目录，但 LLM 运行时的 backend_type 只认 cpu / opencl / metal。引擎支持 ≠ 你的手机能用。 这是所有端侧推理文章最容易偷换的一步——把"源码里有这个目录"写成"这个方案支持你"。
五、修正版选型：按你的芯片对号入座
| 你的设备 | 首选 | 说明 |
|---|---|---|
| 苹果（A/M 系列） | MLX（框架层）或 llama.cpp 走 Metal | ANE 不参与，别指望。成品 App 有 PocketPal 等 |
| 高通骁龙（三星/台积电代工都一样） | llama.cpp 走 Vulkan 或 OpenCL（Adreno） | 最稳的路径。想折腾 NPU 走 Hexagon 交叉编译，门槛高 |
| 华为麒麟 | CPU + 小模型，降低预期 | 达芬奇 NPU 无可用 LLM 路径。NPU 相关后端只覆盖通用模型且评级一般 |
| 联发科天玑 | llama.cpp Vulkan / OpenCL，或 MLC-LLM OpenCL(Mali) | 纯 GPU 路线，够用 |
| 想装 App 就用 | PocketPal（llama.cpp 血统）、MNN Chat（官方）、MLCChat | 三个都是真的成品，不是"需要自己编译" |
补一句流传文章说对的：骁龙芯片的三星代工版和台积电代工版，在框架适配上确实没有任何差异。 框架只识别 ARM 架构、GPU 型号、NPU 架构，不识别晶圆在哪家工厂流片。唯一区别是三星 4nm 版（8 Gen 1）漏电率更高、更容易触发降频，速度波动大——那是芯片物理特性的问题，跟适配无关。这条判断是对的。
六、内存和量化的账，自己算一遍
流传文章给的"6G 内存配 1B、8G 配 3B、12G 才能碰 7B"过于保守，而且它没算真正的大头。
权重账：Q4_K_M 量化大约每 10 亿参数占 0.55–0.6GB。
| 模型 | Q4_K_M 权重 |
|---|---|
| 1B | ≈ 0.7 GB |
| 3B | ≈ 1.9 GB |
| 7B | ≈ 4.4 GB |
| 8B | ≈ 4.9 GB |
KV 缓存账——这才是真正的杀手，公式是：
每 token 字节数 = 2 × 层数 × KV 头数 × head_dim × 2（fp16）
拿一个 4B 级模型（36 层、8 个 KV 头、head_dim 128）算：单 token 约 144 KB。
- 4K 上下文 → 约 576 MB
- 16K 上下文 → 约 2.3 GB
- 32K 上下文 → 约 4.6 GB
一个 4.9GB 的 8B 模型加 32K 上下文，光内存就要 9.5GB。这就是为什么"12G 手机能跑 7B"这句话必须加个"上下文压到 4K"的条件——上下文比模型大小更容易把你的手机撑爆。
再叠上手机的两道硬墙：安卓有低内存杀手（LMK），后台一挤就把你的进程干掉；iOS 有 jetsam 限制，单个 App 可用内存通常被压到设备内存的一半左右，具体阈值随机型与系统版本浮动。
所以实际体验档位是：8GB 机器跑 3B Q4 + 4K 上下文很舒服；12GB 跑 7B Q4 必须压上下文；16GB 才有余量。
量化后缀怎么选（Q4_K_M 还是 Q5_K_M，IQ2 能不能碰），我上个月单独写过一篇，不在这里重复：/gguf-quant-guide/
七、避坑清单（修正版）
- 别指望苹果 ANE。 llama.cpp 用 Metal，MLX 用 Metal，ANE 在端侧 LLM 这条路上目前不参与。谁跟你说"ANE 加速"，让他拿后端表出来。
- 麒麟机型先降低预期。 现实路径是 CPU（ARM NEON）+ 小模型。MNN 的 HIAI 后端是给通用模型用的，LLM 运行时不给这个选项。
- 骁龙机型要手动开后端。 安卓上 GPU 加速默认未必启用，自己确认 Vulkan 或 OpenCL 有没有生效，别装完就以为跑在 GPU 上。
- 先算 KV 缓存，再看权重。 大多数人爆内存不是模型选大了，是上下文开长了。
- 看到"NPU 加速"先问三句：哪个后端、什么精度、官方评级几级。答不上来的一律当营销话术处理。
- 别信没有版本号和日期的 benchmark。 这个领域半年就换一代，2024 年的数据今天可能完全反向。
- "全机型兼容 + 异构最优 + 零门槛"三合一的东西不存在。 全兼容的（llama.cpp）异构弱，异构强的（骁龙 Hexagon）门槛高且锁平台，零门槛的（成品 App）背后是别人替你做的取舍。
八、我把门牌号给你，自己去看
我不想让你信我。这个领域的二手文章多到能把人淹死，包括我这篇——所以最后一步是你自己打开官方表。
- llama.cpp 后端表：
github.com/ggml-org/llama.cpp（README 里的 Backends 一节）+docs/backend/snapdragon/README.md - MNN 后端评级表：
github.com/alibaba/MNN（README 里那张 Architecture / Precision 表） - MNN-LLM 运行时参数：
alibaba/MNN仓库transformers/README.md - ExecuTorch 平台与后端：
docs.pytorch.org/executorch/main/backends-overview.html - LiteRT-LM：
github.com/google-ai-edge/LiteRT-LM - MLX：
github.com/ml-explore/mlx - MLC-LLM：
github.com/mlc-ai/mlc-llm
看表的方法很简单：找到那行表格，看它的精度支持矩阵，看它的评级字母，看它的平台那一列写的是谁。所有二手文章的破绽，都藏在官方表格那几个格子里。
收尾
"手机没有独显，跑不动大模型"这个问题，在 2026 年已经过时了。手机能跑，一直都能跑，只是慢、只是热、只是别指望它跟台式机比。
真正过时不了的是另一个问题：
你是打算看别人替你总结的表格，还是自己打开官方那一页？
第一个选项免费、快、舒服，代价是它可能错了五处，而你永远不会知道。
版权没有，随意转载。过意不去就加一句「原文发表于 cn-res.vip」，不加也无所谓。
Running an LLM on your phone has become a trending topic again. The most widely circulated comparison piece covers five frameworks, three chip families, four use cases. It reads well.
Then I pulled the official READMEs, backend support tables, and backend source directories for all five frameworks and checked it line by line.
Five hard errors. One major omission. Zero cited sources for any performance number.
The worst one: it takes a row that MNN's own README grades as "C — not supported" and presents it as "the optimal solution for operator-level heterogeneous scheduling." That table sits in the middle of the MNN README. Anyone can look at it.
So I'm not going to re-narrate that piece. I'm going to lay out the official ledgers and tell you which framework actually works on which chip, and which claims are just secondhand articles copying each other.
All data here comes from official repository READMEs, official backend documentation, and source directory listings, retrieved 2026-09-17. Versions and star counts are pinned. Inference frameworks are a function of time — a comparison that doesn't pin versions is worthless.
1. First, kill the myth: phones can run them — but "no discrete GPU" is not the real bottleneck
Phones don't have discrete GPUs. They do have an integrated GPU and an NPU in the SoC. That part is fine.
But the two causes the popular article gives — "single inference path, CPU only" and "no heterogeneous cooperation" — are only half right. And its proposed fix (split work between NPU and GPU) is not the magic bullet it sounds like.
There are three hard walls on a phone, each harder than the last.
Wall one: shared memory bandwidth. The phone's GPU, NPU, and CPU share one LPDDR pool. There is no dedicated VRAM. A flagship's LPDDR5X delivers roughly 60–120 GB/s. A desktop GPU's GDDR6X starts around 1 TB/s. Token decoding is a pure bandwidth problem — every token requires streaming the participating weights out of memory again. An order-of-magnitude bandwidth gap means an order-of-magnitude speed gap, regardless of whether you're using CPU or NPU.
Wall two: operator coverage. An NPU is not a general-purpose accelerator. It only runs operators that were compiled and quantized for it. Transformers happen to have a set of operators that are chronically missing on NPUs: RMSNorm, SwiGLU, Softmax, and KV-cache reads and writes. This isn't vendor laziness — those operators use tensor cores so poorly that offloading them makes things slower.
Wall three: the thermal budget. A phone sustains somewhere in the 3–8W range. Run wide and it throttles. The real user experience question on mobile isn't "how fast," it's "fast for how long."
Hold these three walls in mind and every selection decision downstream becomes obvious. Heterogeneous execution means "put the operators that fit onto the NPU, fall back for the rest." It's a way to spread the load and save power. It is not a 2x performance trick.
2. Five frameworks, and what their official ledgers actually say
My sources: the official READMEs and docs of ggml-org/llama.cpp, alibaba/MNN, ml-explore/mlx, pytorch/executorch, google-ai-edge/LiteRT-LM, and mlc-ai/mlc-llm. I'll give you the address for every claim below.
llama.cpp: the compatibility ceiling — with no ANE
The official README carries a backend table. Here is what's actually in it:
| Backend | Target hardware |
|---|---|
| Metal | Apple Silicon |
| OpenCL | Adreno GPU (Qualcomm) |
| Vulkan | Generic GPU |
| Hexagon | Snapdragon (Qualcomm NPU/DSP) |
| CANN | Ascend NPU |
| SYCL | Intel GPU |
| OpenVINO (in progress) | Intel CPU / GPU / NPU |
Look carefully: the Apple row says Metal. Not Core ML. Not ANE.
One more cut — I searched the repo's CoreML issues and PRs. There are exactly two, both still open: one adds a Core ML backend for vision encoding, the other asks whether it could work "like whisper.cpp." A text-inference ANE backend does not exist.
So the claim that "llama.cpp automatically hooks into Apple's ANE for operator optimization" is fabricated.
And here's something most people don't know: llama.cpp's mainline already ships NPU backends — Hexagon for Qualcomm and CANN for Ascend. The popular article gets that wrong too. More in section 4.
MNN: the broadest backend coverage, and a grading table that settles the argument
MNN's engine backend coverage is genuinely wild. Under source/backend/: coreml, hiai, qnn, nnapi, neuropilot, hexagon, rknn, tensorrt, opencl, vulkan, metal, cuda, musa.
Looks like total device coverage. But the README contains a grading table where the project grades itself. Here is MNN's own legend, verbatim:
- S: supported and deeply optimized, recommended
- A: supported and works well, usable
- B: supported but buggy or unoptimized, not recommended
- C: not supported
Now the NPU rows:
| NPU backend | Normal | FP16 | Int8 | Notes |
|---|---|---|---|---|
| Core ML | A | C | C | Apple |
| HIAI | A | C | C | Huawei |
| NNAPI | B | B | B | Android generic; officially not recommended |
| QNN | C | B | C | Qualcomm; unsupported at Normal precision |
Translation: MNN's best NPU grade is A, and only at Normal precision — FP16 and Int8 are all marked C. Qualcomm's QNN is marked unsupported at Normal precision, and only reaches "B — buggy, not recommended" at FP16.
Now the LLM path. MNN-LLM's official documentation gives exactly three legal values for the runtime parameter backend_type:
cpu (default) / opencl (Android GPU) / metal (macOS and iOS GPU)
There is no npu option.
Conclusion: MNN's engine layer has broad NPU backend coverage, but on the LLM path the official runtime does not expose the NPU at all. Anyone writing "MNN is the only solution supporting Kirin NPU heterogeneity and full Snapdragon heterogeneity" needs to explain what that C grade means.
One more correction: the popular article claims MNN "has no end-user app, requires compiling it yourself." MNN ships official apps. MNN Chat App on Android (multimodal: text, image, audio), MNN LLM iOS on the Apple side. Both are linked on the README front page. You can browse models, download them, and chat — inside the app. Claiming otherwise means the author never opened the README.
ExecuTorch: the official platform table, and Kirin isn't on it
Meta's on-device framework. From the official README:
| Platform | Backends |
|---|---|
| Android | XNNPACK (CPU), Vulkan (GPU), Qualcomm (QNN NPU) |
| iOS / iPadOS | XNNPACK (CPU), Core ML, MLX |
| macOS | XNNPACK, Core ML, MLX (experimental) |
| Linux | XNNPACK, OpenVINO, CUDA (experimental) |
| Windows | XNNPACK, CUDA / Vulkan (experimental) |
| Browser | WebAssembly, WebGPU (both experimental) |
| Embedded | Cortex-M, Ethos-U, NXP NPU |
Kirin is indeed absent. The popular article gets the direction right here.
But it misses something new: the iOS and macOS rows list MLX as a backend. Meta made Apple's own inference framework into one of ExecuTorch's backends. Worth watching.
LiteRT-LM: stop talking about NNAPI
Google's LiteRT-LM (the former TFLite line) positions itself as a "production-ready, high-performance" on-device LLM inference framework covering Android, iOS, web, desktop, and IoT. Language bindings: Kotlin labeled stable, Swift labeled early preview.
On acceleration, the official text says "peak performance via GPU and NPU accelerators," plus an experimental YNNPACK delegate available only on linux arm64.
One correction that matters: the popular article says LiteRT relies on Android NNAPI to auto-assign operators. But NNAPI has been put on Google's deprecation track. The current path is vendor delegates — Qualcomm, MediaTek, and Google Tensor each with their own NPU route. A comparison piece still selling NNAPI as the core mechanism is copying material from years ago.
MLX: no ANE, and no longer Apple-only
From the MLX README, on what it runs on:
currently the CPU and the GPU
CPU and GPU. No ANE. The claim that "MLX automatically delivers Metal GPU + ANE heterogeneous cooperation" is fabricated.
And "Apple devices only" has expired. MLX's current install docs include:
pip install mlx[cuda] # CUDA backend on Linux
Plus a CPU-only Linux package. MLX still doesn't run on Android, but it is no longer Apple-exclusive.
MLC-LLM: the one the popular article never mentions
mlc-ai/mlc-llm, 23k stars, actively maintained. Its official platform table: Android via OpenCL on Adreno or Mali, iOS via Metal on A-series GPUs, plus WebGPU/WASM across the board.
Its positioning is clear: a compiler route, compiling models into native artifacts per platform. Its cross-platform reach is the widest of the six — it even runs in a browser. MLCChat is the companion app.
Pure GPU. No NPU. An article titled "the ultimate guide to running LLMs on phones" that omits it is like writing a history of the Chinese internet and leaving out Tencent.
3. Five hard errors and one omission, itemized
| The popular claim | What the official docs say | Verdict |
|---|---|---|
| llama.cpp "automatically hooks into ANE on Apple silicon" | Backend table lists Metal only for Apple; the only CoreML items are two unmerged open issues, vision-encoding scoped | Fabricated |
| MLX "delivers Metal GPU + ANE heterogeneous cooperation" | Official README: runs on CPU and GPU. No ANE | Fabricated |
| "MNN has no end-user app; compile it yourself" | Official MNN Chat App (Android, multimodal) + MNN LLM iOS, both on the README front page | Fabricated |
| "MNN supports QNN NPU + Vulkan GPU operator-level heterogeneous scheduling" | Official grading table: QNN is C (unsupported) at Normal precision, B at FP16 (officially not recommended) | No evidence |
| "llama.cpp mainline can't do GPU+NPU heterogeneity; only third-party forks can" | Official docs/backend/snapdragon/README.md implements operator-level fallback between Hexagon NPU and CPU/GPU | Fabricated |
| (MLC-LLM never mentioned) | 23k stars; Android OpenCL, iOS Metal, WebGPU across platforms | Major omission |
| "NPU+GPU splitting cuts heat by 30%+ and boosts speed 20–40%" | No source anywhere. Not in MNN, llama.cpp, or ExecuTorch docs | Zero provenance |
Six of seven don't survive contact with the docs. This isn't a matter of opinion. It's a matter of not having looked.
4. Heterogeneous execution: someone actually built it, but not the way you think
Here's the mechanism. "NPU + GPU cooperation" means sending big matrix multiplies — heavy, power-friendly — to the NPU, and falling back to GPU or CPU for whatever the NPU can't run: normalization, activations, attention, KV reads and writes.
The one place this is genuinely implemented in an official mainline today is llama.cpp's Snapdragon backend.
The evidence is in docs/backend/snapdragon/README.md. Key points, paraphrased from the source:
- Models can be split layer-wise or row-wise across multiple physical NPUs, e.g.
--devices 'HTP0[0-1],HTP1[2-3]'spanning four physical NPUs - Virtual sessions are supported, letting a single physical NPU host multiple sessions (
HTP0:0,HTP0:1) - An operator filter,
GGML_HEXAGON_OPFILTER, controls exactly which operators run on Hexagon, with everything else falling back to CPU or GPU. The docs even show disabling Flash Attention on Hexagon so it falls back - The official build supports
GGML_HEXAGON=ONandGGML_OPENCL=ONat the same time - Debug output shows real LLM operators offloaded to the NPU — for example layer 27's FFN weight matmul, q4_0 × f32, executing on
HTP0
That is operator-level heterogeneity, for real. But it comes with hard limits the popular article never mentions:
- Qualcomm only. HTP is the Hexagon Tensor Processor. No other vendor has it.
- High build barrier. It requires Docker cross-compilation toolchain images, then ADB deployment to the device. The official docs target developer devices and Snapdragon X-series laptops, not "install an app and go."
- Fallback is the norm, not the exception. The docs explicitly disable Flash Attention, ADD, and SUB on Hexagon, falling back to CPU or GPU. Don't expect the whole model on the NPU.
As for MNN: the engine contains six NPU-related backend directories (hiai, qnn, neuropilot, nnapi, coreml, hexagon), yet the LLM runtime accepts only cpu / opencl / metal. "The engine supports it" is not the same as "your phone can use it." That sleight of hand — turning "there's a directory in the source tree" into "this solution supports you" — is the single most common trick in on-device inference writing.
5. Corrected selection guide: match your chip
| Your device | First choice | Notes |
|---|---|---|
| Apple (A/M series) | MLX at the framework level, or llama.cpp via Metal | ANE is not in play. Don't count on it. PocketPal and similar apps are the consumer route |
| Qualcomm Snapdragon (Samsung or TSMC fab, same answer) | llama.cpp via Vulkan or OpenCL (Adreno) | The most reliable path. NPU means Hexagon cross-compilation — high barrier |
| Huawei Kirin | CPU plus a small model. Lower your expectations | No usable LLM path to the Da Vinci NPU. NPU-related backends cover general models only, at middling grades |
| MediaTek Dimensity | llama.cpp Vulkan / OpenCL, or MLC-LLM OpenCL (Mali) | Pure GPU route, good enough |
| You just want an app | PocketPal (llama.cpp lineage), MNN Chat (official), MLCChat | All three real, shipping apps — none require compiling anything |
One thing the popular article gets right: Samsung-fabbed and TSMC-fabbed Snapdragon chips show no compatibility difference in any framework. Frameworks recognize the ARM architecture, the GPU model, and the NPU architecture. They do not recognize which foundry produced the wafer. The only difference is physical: the Samsung 4nm Snapdragon 8 Gen 1 leaks more and throttles sooner, so speeds fluctuate — a silicon property, not an adaptation issue. That judgment is correct.
6. The memory and quantization math — run it yourself
The popular article's guidance ("6GB → 1B, 8GB → 3B, 12GB for 7B") is overly conservative, and it omits the actual dominant cost.
Weight budget: Q4_K_M quantization runs roughly 0.55–0.6GB per billion parameters.
| Model | Q4_K_M weights |
|---|---|
| 1B | ≈ 0.7 GB |
| 3B | ≈ 1.9 GB |
| 7B | ≈ 4.4 GB |
| 8B | ≈ 4.9 GB |
KV cache budget — this is the real killer. The formula:
bytes per token = 2 x layers x KV heads x head_dim x 2 (fp16)
Take a 4B-class model (36 layers, 8 KV heads, head_dim 128): about 144 KB per token.
- 4K context → about 576 MB
- 16K context → about 2.3 GB
- 32K context → about 4.6 GB
An 8B model at 4.9GB plus a 32K context needs 9.5GB of memory. This is why "a 12GB phone can run a 7B" must always carry the clause "with context capped at 4K." Context length blows up phones far more often than model size.
Then stack the two hard walls: Android's low-memory killer will reap your process when memory pressure rises, and iOS's jetsam limit typically caps a single app near half the device's RAM, with the exact threshold floating by model and OS version.
The realistic tiers: 8GB handles 3B Q4 with 4K context comfortably. 12GB runs 7B Q4 only with context trimmed. 16GB is where you get headroom.
For choosing between quantization suffixes (Q4_K_M vs Q5_K_M, whether IQ2 is usable), I wrote a separate piece last month — no need to repeat it here: /gguf-quant-guide/
7. Pitfall checklist (corrected)
- Don't count on Apple's ANE. llama.cpp uses Metal, MLX uses Metal, and the ANE is not part of on-device LLM inference today. Anyone claiming "ANE acceleration" should show you a backend table.
- Lower expectations on Kirin first. The realistic path is CPU (ARM NEON) with a small model. MNN's HIAI backend serves general models; the LLM runtime doesn't expose it.
- Verify your backend is actually active on Snapdragon. GPU acceleration isn't necessarily enabled by default on Android. Confirm Vulkan or OpenCL is running — don't assume you're on the GPU because you installed something.
- Compute the KV cache before the weights. Most people who blow their memory budget didn't pick too large a model. They set the context too long.
- When you see "NPU accelerated," ask three questions: which backend, what precision, what official grade. If none can be answered, treat it as marketing.
- Never trust a benchmark without a version number and a date. This field turns over every six months. 2024 data can be exactly backwards today.
- "Universal compatibility + optimal heterogeneity + zero setup" does not exist in one product. The all-compatible option (llama.cpp) is weak on heterogeneity. The heterogeneity-strong option (Snapdragon Hexagon) is high-barrier and platform-locked. The zero-setup option (consumer apps) hides someone else's trade-offs.
8. Here are the addresses — go look yourself
I don't want you to take my word for it. This field has enough secondhand writing to drown in, mine included. So the last step is yours: open the official tables.
- llama.cpp backend table:
github.com/ggml-org/llama.cpp(the Backends section) plusdocs/backend/snapdragon/README.md - MNN grading table:
github.com/alibaba/MNN(the Architecture / Precision table in the README) - MNN-LLM runtime parameters:
alibaba/MNN, filetransformers/README.md - ExecuTorch platforms and backends:
docs.pytorch.org/executorch/main/backends-overview.html - LiteRT-LM:
github.com/google-ai-edge/LiteRT-LM - MLX:
github.com/ml-explore/mlx - MLC-LLM:
github.com/mlc-ai/mlc-llm
How to read them: find the row, read the precision matrix, read the grade letter, and look at which vendor occupies the platform column. Every flaw in every secondhand article is hiding in those cells.
Closing
"Phones have no discrete GPU, so they can't run LLMs" expired as a question in 2026. Phones can run them, always could. Just slowly, just hotly, and don't expect desktop performance.
What doesn't expire is a different question:
Are you going to read someone else's summary table, or open the official page yourself?
The first option is free, fast, and comfortable. The cost is that it might be wrong in five places, and you will never find out.
No copyright reserved — share it freely. If you feel like it, a credit line to cn-res.vip is appreciated. Or don't. Either way.