Huawei Quantizes 122B Model in 64 Minutes

A Huawei research group has open-sourced MACaQ, a quantization toolkit it says can compress a 122-billion-parameter model to 4-bit precision in 64 minutes while using only 32.5GB of GPU memory. The code went up on GitHub under the Apache 2.0 license, alongside a paper first posted to arXiv on Nov. 7, and the AI newsletter AI Weekly reported the release on Nov. 12. Every number below comes from the group's own paper and README; none has yet been reproduced outside the lab.

Key takeaways

• MACaQ quantized Qwen3.5-122B-A10B, whose 16-bit weights take about 244GB, on a single 96GB GPU, and the 4-bit result fits back on that same card at 74GB.
• On Llama-3-8B, the authors report lower perplexity than the MR-GPTQ and DuQuant++ methods, and say MACaQ matches their best scores in 11 times less quantization time.
• One codebase targets llama.cpp, vLLM, Diffusers and PyTorch, covering text, image, video and speech models.

How MACaQ quantizes a model bigger than the GPU

MACaQ, short for Model Agnostic Calibrated Quantization, converts both weights and activations to MXFP4, a 4-bit floating-point format, a setup known as W4A4. Most competing methods push their math transforms across neighboring layers to avoid any cost at inference time. MACaQ instead keeps a small scaling step and a block-Hadamard rotation inside each linear layer and runs them live.

That choice means each layer can be calibrated and quantized on its own, so only one layer has to sit on the GPU at a time while the rest wait in system memory. A refinement step the authors call Flip-Refinement then nudges individual 4-bit codes to neighboring values when that lowers the layer's output error.

In the paper, written by Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang and Lukas Cavigelli, the 122B run used 512,000 calibration tokens. WikiText-2 perplexity rose from 5.06 at full precision to 5.14 after quantization.

MACaQ benchmark results vs. GPTQ-style methods

The authors say keeping the transforms live adds only 2.3% overhead on Qwen3-32B in vLLM. For the image model, they report MACaQ output stayed closer to the original than the model's official NVFP4 version on all four similarity metrics they used.

Context: why 4-bit tooling matters for local AI

The GitHub repository splits the work by backend: 2-, 3- and 4-bit GGUF files for llama.cpp, which also runs on CPUs; W4A4 or weight-only checkpoints for vLLM serving; and W4A4 for diffusion and speech models. A companion tool, MACaQ-R, refines existing GGUF files while keeping them compatible with standard llama.cpp.

Recipes target popular open models, including Alibaba's Qwen3.8-27B, which became a common single-GPU choice after Alibaba open-sourced the family. Speed claims, however, depend on hardware with native 4-bit math; the paper's tests ran on an unnamed GPU with 96GB of memory.

The paper also lists limits. Because every layer is handled alone, errors cannot be compensated across layers. For mixture-of-experts models only the expert layers were quantized, and on a small 0.6B model the live transforms cost about 8% in speed.

What's next

The repository had 35 stars and 16 commits when AI Newsway checked it, so outside benchmarks are still thin. Huawei's own Ascend chips are only partly covered: W4A4 kernels sit in pending pull requests, and full-model runs still need integration with an Ascend inference runtime, the README says.

FAQ

Can MACaQ models run on a regular PC?

The llama.cpp path produces standard-style GGUF files and supports CPU execution, according to the README. The 122B model still needs 74GB of GPU memory after quantization.

Is MACaQ free for commercial use?

The code is released under Apache 2.0, which permits commercial use. Bundled third-party components keep their own licenses, the README says.

References

• GitHub - huawei-csl/MACaQ — GitHub
• MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations — arXiv (Nov. 7, 2026)
• Huawei Open-Sources MACaQ, Quantizes 122B Model in 64 Minutes — AI Weekly (Nov. 12, 2026)