Meet FreeToken: An Edge-Native MoE Serving Engine that Runs 753B GLM-5.2 on a Single Workstation GPU

FreeToken is an edge‑native mixture‑of‑experts (MoE) serving engine designed to run extremely large models on a single workstation GPU. By treating the GPU as the primary compute node and offloading cache‑miss handling to the host CPU over PCIe, the system avoids the need for multi‑GPU scaling while still serving the full parameter set of a frontier model.

The core innovation lies in how FreeToken manages MoE cache misses. It measures the effective bandwidth of the PCIe link and the CPU’s execution capability, then dynamically splits each miss between a PCIe fill (fetching the required expert weights from host memory) and a CPU‑side computation step. This hybrid approach keeps the GPU saturated with matrix‑multiply work while the CPU handles the irregular memory traffic that would otherwise cause stalls. Using this method, the engine can load and inference the 753‑billion‑parameter GLM‑5.2 model on a single GPU without exceeding memory capacity.

Key technical points

• MoE‑based model serving with expert partitioning
• Edge‑native design targeting a single workstation GPU
• PCIe‑CPU hybrid cache‑miss resolution guided by measured bandwidths
• Supports GLM‑5.2 (753 B parameters) locally
• No modification to the original model weights or architecture required

Why this matters

The source shows that FreeToken can run a 753 B‑parameter model on one workstation GPU by balancing PCIe transfers and CPU execution. This suggests that the barrier to experimenting with frontier‑scale MoE models is lowered for researchers who lack access to large GPU clusters, potentially accelerating local experimentation and reducing reliance on cloud‑based inference services.