Qwen3.8-Flash-Next: How Hybrid GDN-QSA and N-Gram Memory Preview the Qwen4 Architecture

On August 26, 2026, Alibaba's Qwen team open-sourced the weights for Qwen3.8-Flash-Next, a multimodal Mixture-of-Experts (MoE) foundation model designed as an early architectural testbed for the upcoming Qwen4 series. Spanning 125 billion backbone parameters alongside an auxiliary 51 billion parameter N-gram embedding table, the model activates merely 6 billion parameters per token during inference.

Despite requiring roughly one-ninth the training compute of Qwen3.7-Plus, Qwen3.8-Flash-Next matches or exceeds substantially larger models across coding benchmarks (62.5% on SWE-bench Pro, 58.7% on DeepSWE 1.1) and agentic workflows (73.5% on Toolathlon Verified), while supporting a native 262,144-token context window that expands to 1,000,000 tokens via YaRN interpolation.

Key Breakthroughs

1. Hybrid Attention: Gated DeltaNet (GDN) + Qwen Sparse Attention (QSA)

Traditional full attention scales quadratically with sequence length, making 1M-token context inference prohibitively expensive. Qwen3.8-Flash-Next introduces a structured 3:1 hybrid attention topology:

• Gated DeltaNet (GDN) Layers: Three out of every four layers employ GDN linear recurrent units to compress conversational and historical context into fixed-size hidden states, maintaining O(1) memory growth per step.
• Qwen Sparse Attention (QSA) Layers: The remaining layer applies global sparse attention. Unlike conventional dynamic sparse attention mechanisms that compute token-level indexing maps, QSA aggregates tokens into compressed micro-blocks, selecting relevant sequence chunks via a lightweight block indexer.

This dual-mechanism design yields up to 7.6x faster prefill and 4.9x faster decoding at 1M tokens, achieving an 8.6x prefill throughput gain over Qwen3.7-Plus at a 90% prefix cache hit rate.

2. Four-Branch Gated Residual (GR) Streams

Standard Transformer architectures funnel all layer updates through a single additive residual path, leading to signal degradation and feature dilution in deep models. Qwen3.8-Flash-Next splits the residual stream into four parallel branches modulated by dynamic, element-wise gating derived from GatedNorm:

• Decoupled Signal Routing: Distinct channels independently preserve long-range foundational representations and local intra-layer transformations, preventing early layer representations from being drowned out.
• FP8 State Precision: Residual states natively support FP8 quantization, slashing inter-layer memory bandwidth overhead while stabilizing training dynamics and preventing activation spikes without complex branch-mixing matrices.

3. Asynchronously Prefetched N-Gram Embedding Tables

To scale representation capacity without imposing arithmetic FLOP overhead on GPU tensor cores, Qwen3.8-Flash-Next integrates a 51B parameter N-gram embedding table positioned at the network entrance:

• Contextual Multi-Token Lookups: Rather than indexing single token IDs, the embedding module hashes local n-gram sliding windows to retrieve rich semantic priors for recurrent phrases and structural syntax patterns.
• Host RAM Offloading: Because token positions are deterministically pre-computable during tokenization and prompt preparation, the 51B embedding table resides in CPU host memory and is asynchronously prefetched via PCIe/CXL, keeping GPU VRAM footprint confined strictly to active weights.

4. Hardware-Aware Muon Optimization

The model was pre-trained using a customized adaptation of the Muon optimizer co-designed with Transformer architecture dynamics:

• Selective Projection Orthogonalization: Muon governs 2D linear weight transformations across Attention, GDN, and MoE routing projections, while low-rank residual adapters, embeddings, and normalization scales are updated via AdamW.
• Fused Kernel Splitting & Constant Batch Training: Fused projection blocks (QKV, SwiGLU) are isolated prior to orthogonalization, eliminating the empirical necessity for batch-size warmup and saving 18.8% of total optimization update steps.

Technical Specifications & Benchmark Overview

Verified Integration & API Usage

Developers can interface with Qwen3.8-Flash on QwenCloud using standard OpenAI-compatible client libraries:

pythonCopy
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("DASHSCOPE_API_KEY"),
    base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
    model="qwen3.8-flash",
    messages=[
        {"role": "system", "content": "You are an expert systems engineer."},
        {"role": "user", "content": "Explain the mechanics of Gated DeltaNet linear attention."},
    ],
    extra_body={
        "enable_thinking": True,
    },
    stream=False,
)

print(completion.choices[0].message.content)