Flash-MoE claims Qwen3.5-397B runs on iPhone at 0.6 tokens/sec via SSD streaming

Anemll’s iPhone demo claims “Running 400B model on iPhone! 0.6 t/s” and ties it back to the Flash-MoE codebase. The post credits multiple collaborators and says the implementation uses “giant KV cache and SSD streaming,” extending the earlier laptop work rather than introducing a separate mobile stack.

The earlier Flash-MoE repo is more specific about the model family and engine: Qwen3.5-397B-A17B, a pure C/Metal inference path, and SSD-backed expert streaming on Apple hardware. Simon Willison’s thread summary captured the engineering leap cleanly: you can run “enormous Mixture-of-Experts” models without fitting the full model in RAM by streaming only the subset of experts needed for each generated token.

The core trick is explicit storage-to-memory tiering. The project repo says the 397B Qwen3.5 MoE is 209GB at 4-bit, but Flash-MoE avoids keeping all of that resident: non-expert weights are mmap’d at 5.5GB, while expert weights are fetched from SSD with parallel pread as tokens are generated. The same writeup says the laptop version was built in “24 hours” with hand-tuned Metal shaders and no Python frameworks.

That framing also explains why direct throughput comparisons are messy. The HN discussion notes this is “not an ordinary LLM benchmark” because the system is streaming weights from storage, so comparing it to fully resident local models can be misleading. Simon Willison’s earlier Mac hardware thread adds a useful MoE lens here: even trillion-parameter models can become plausible on Apple hardware when the active parameter count is much smaller than the total model size.

The most concrete new performance datapoint is in the fresh HN delta: one commenter highlighted “removing the 9.8 GB Metal LRU cache” for a 38% speedup, then asked why the system still lands around 5.7 tok/s versus an 18.6 tok/s theoretical ceiling. That shifts the conversation from novelty to bottleneck analysis: cache overhead, compute saturation, and I/O scheduling still appear unresolved.

The usability debate is even harsher on phone. The HN discussion quotes one commenter arguing that “under 20t/s” is “unusable in any real workflow,” and another saying that at 6 tok/s a mistake can cost “20-30 minutes.” On the iPhone side, the iPhone discussion keeps returning to “the heat problem,” limited RAM for “any reasonable amount of context,” and the fact that Apple’s unified memory helps but does not remove those constraints. The engineering result is real; the deployment envelope still looks narrow.