A research team has released APEX4, a pure W4A4 quantization system that finally makes full INT4 inference practical — but only on certain GPU architectures.
The core problem: running both weights and activations at 4-bit precision (W4A4) should unlock fast INT4 Tensor Core math, but dequantization work on CUDA Cores has historically eaten those gains. APEX4 traces this bottleneck to a single hardware metric — the ratio of Tensor Core to CUDA Core throughput, which they call ρ. On GPUs where ρ is low (RTX 3090 at ρ=16, A40 at ρ=16), the W4A4 kernel delivers 2.0–2.5x speedup at the kernel level. On the A100, where ρ hits 64, the same kernel degrades performance to 0.43–0.47x of baseline — meaning it runs more than twice as slow. That is not a rounding error; that is a fundamental mismatch between the workload and the chip.
The "why it matters" is what APEX4 does with that finding. Rather than accepting the A100's penalty, the system adds a mixed-granularity mode that recovers A100 performance to a 1.20–1.40x end-to-end speedup in vLLM. On friendlier hardware, end-to-end gains in unmodified vLLM reach 1.66x on the L40S (ρ=8), 1.78x on the RTX 3090, and 2.09x on the A40. Accuracy holds up: perplexity on LLaMA-2-70B stays within 0.63 of FP16, and zero-shot accuracy beats the competing W4Ax Atom-g128 baseline by 4.0–4.4%.
The honest caveat is that the biggest speedups land on prosumer and workstation GPUs — the RTX 3090 and A40 — not the data-center A100s that run most production inference today. Teams betting their stack on H100s or A100s should read the ρ analysis carefully before assuming these numbers apply to them.