Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

Quantization‑Aware Healing (QAH) introduces a training procedure that directly optimizes a 4‑bit quantized version of a transformer‑based language model while simultaneously correcting the induced quantization error. Rather than applying post‑training quantization, the method jointly learns the scaling factors, zero‑points, and a small compensatory residual network (the “healing” module) that is added to each layer’s output during forward passes. The base architecture—number of layers, hidden size, attention heads, and context window—remains unchanged, so the model can be swapped into existing inference pipelines without modification.

Experiments on a 7‑billion‑parameter LLM show that the QAH‑produced 4‑bit model achieves perplexity scores on WikiText‑103 and C4 that are within 0.2 points of the FP16 baseline and, on several zero‑shot reasoning benchmarks, matches or slightly exceeds the full‑precision counterpart. Training overhead is limited to a few extra epochs, and the healed model retains the original license (Apache 2.0) of the source model.

• 4‑bit uniform weight and activation quantization (symmetric, per‑channel)
• Healing module: a lightweight bias‑only residual added post‑quantization
• No change to depth, width, or context length (e.g., 4 k tokens)
• Training cost ≈ 1.2× standard fine‑tuning
• Released under Apache 2.0, compatible with the original model’s license

Why this matters

Based on the reported results, we can infer that deploying a 4‑bit model with QAH reduces memory footprint and inference latency without sacrificing accuracy, making large models more viable on edge hardware. The preservation of the original architecture and license means existing tooling and downstream pipelines remain applicable, while the modest training overhead suggests the technique can be integrated into standard fine‑tuning workflows. However, the long‑term stability of the healing module across diverse datasets and quantization schemes remains an open question requiring further study.