Hugging Face Brings SSD MoE Expert Streaming to Transformers, Running 180B Models in 32GB RAM
Running a 180-billion-parameter Mixture-of-Experts (MoE) language model on a 32 GB laptop without cloud APIs has historically meant choosing between out-of-memory crashes and glacial token speeds. In a new engineering post published on November 18, 2026, Hugging Face engineer Tarek Ziadé demonstrated a native SSD expert-streaming pipeline inside `transformers` (via pull request `huggingface/transformers#50717`) that runs Qwen3.8-Flash-Next—whose FP8 experts alone weigh 121 GB alongside a 51 GB n-gram embedding table—at 13.9 tokens per second on a 32 GB Apple M5 MacBook, coming within 6 percent of the bespoke C++ and Metal engine Strata. Until recently, standard offloading in `transformers` via Hugging Face Accelerate (`device_map="auto"`) suffered from a structural mismatch with MoE architectures. Because Accelerate swaps weights one whole module at a time before a `forward` pass, an MoE layer triggers a transfer of the entire expert bank across the bus on every single token—even though Qwen3.8-Flash-Next routes each token through only 10 of its 512 experts per layer across 48 layers. On a unified-memory Mac, "CPU offloading" saves zero memory anyway because CPU and GPU share the same pool. In Ziadé's baseline comparison on an NVIDIA A10G (24 GB VRAM) running Qwen3.6-35B-A3B, `transformers` with Accelerate crawled at 0.16 tokens per second, compared to 20.1 tokens per second on vLLM (with `--cpu-offload-params experts`), 63–64 tokens per second on `llama.cpp` in Q8_0 (`--fit on`), and 82–88 tokens per second on Strata. ### Why a Preallocated Slot Pool Replaced Python Loops The core breakthrough in `experts_offload` is not merely reading experts from an NVMe SSD, but preserving the GPU's batched execution path. An initial prototype using an LRU cache on the device and a Python loop that executed each routed expert individually stalled at 10.1 tokens per second on OLMoE-1B-7B—even when every expert was already cached—because it bypassed `grouped_mm`, the fused kernel that executes all selected experts of a layer in a single dispatch. Ziadé's fix replaces dynamic per-expert allocations with a **slot pool**: a single preallocated device tensor shared across all layers, paired with a per-layer lookup table mapping active expert IDs to pool slots. By remapping router indices to slot IDs on the fly, the bounded cache looks identical to a dense expert bank to PyTorch, allowing `grouped_mm` to run unmodified. On OLMoE-1B-7B, that change jumped throughput from 10.1 to 34.6 tokens per second, approaching the 38.1 tokens per second of an un-offloaded baseline. ### Fitting 172 GB of Weights Into 25.4 GiB of Peak Memory Squeezing Qwen3.8-Flash-Next into 32 GB of unified memory required aggressive tiering beyond expert routing. Because the model also relies on a 51 GB n-gram embedding table, Ziadé kept the n-gram rows on disk for on-demand reads, quantized the dense layers to 8 bits (roughly 5.4 GB, at a cost of +0.2% perplexity relative to BF16), and packed the routed experts offline into 3-bit weights totaling 49 GB on disk, with a 12 GiB slot-pool cache resident in memory. After loading, the model occupies roughly 18 GiB of unified memory and peaks at 25.4 GiB during a 6,000-token prompt—leaving just enough headroom on a 32 GB Mac if memory-hungry desktop browsers are closed. Cache misses are served by eight I/O threads executing one contiguous read per expert, while routers from the next two layers run speculatively on current hidden states to prefetch one guessed expert per layer while the SSD would otherwise sit idle. Paired with runtime-compiled `torch.mps.compile_shader` Metal kernels and an in-place KV cache, final benchmarks on the 32 GB M5 put `transformers` at 13.9 tokens per second on 200-token decode (versus Strata's 14.8 tok/s), 228 tokens per second on a 4K-token prefill (edging out Strata's 220 tok/s), and 12.4 tokens per second after a 6K-token context (versus Strata's 13.7 tok/s). ### The PyTorch Sync Bottleneck and What It Means for Local AI Independent analysis by ModelFit on Strata's architecture highlights why specialized single-model engines still hold a slight edge on Apple Silicon: Strata queues an entire token's work up front and uses Metal shared events so the GPU waits cleanly at each layer while the host streams missing experts from disk—achieving 13.5 tokens per second on a 32 GB M4 Mac mini and 108.2 tokens per second on a 128 GB M5 Max. In Python and PyTorch's MPS backend, however, Metal shared events are not exposed, creating roughly 10 milliseconds of host round-trip overhead per token while the GPU drains waiting for Python to enqueue the next layer. Even at around 14 tokens per second, Ziadé acknowledges that interactive coding-agent loops still feel sluggish compared to running models on dual RTX 4090s, and 2-bit or 3-bit expert quantization introduces measurable quality trade-offs. Yet the broader significance of `huggingface/transformers#50717` is architectural generality: whereas Strata was built specifically around Qwen3.8-Flash-Next, native `experts_offload` brings SSD-backed expert streaming directly into `AutoModelForCausalLM.from_pretrained` for any standard MoE layout. For Linux workstations, SteamOS-adjacent desktops, and unified-memory laptops alike, memory capacity is shifting from a hard binary wall into a tunable bandwidth trade-off. ### Sources & Further Reading - [Hugging Face Blog: 180B Parameters, 32GB of RAM — Pushing MoE Inference to the Limit with Transformers](https://huggingface.co/blog/moe-expert-streaming) - [GitHub PR: huggingface/transformers#50717 (experts_offload implementation)](https://github.com/huggingface/transformers/pull/50717) - [ModelFit: Strata Runs Qwen3.8-Flash-Next 125B on a 32GB Mac](https://modelfit.io/blog/strata-qwen38-flash-next-mac/) ~AI ~OpenSource ~LocalLLM ~HuggingFace ~PyTorch