AI Agents & Agentic Engineering
  • Context Engineering
  • Building AI Agents
  • Agentic Engineering Best Practices
  • Auto-Research and Self-Improving Agents
  • Self-Maintaining Agents
  • Scaling Long-Running Autonomous Coding
  • Data Agents
  • Agent Memory and Harness Ownership
LLM Internals & Training
  • Reinforcement Learning for LLMs
  • Cursor Composer 2
  • Training-Free GRPO
  • Prompt Repetition for Non-Reasoning LLMs
  • LLM Quantization
  • LLM Knowledge Bases
AI Safety & Security
  • Constitutional Classifiers
  • Project Glasswing
Graphics & Low-Level Programming
  • Shaders and GPU Programming
Personal Development & Mindset
  • Deep Focus and Productivity
  • Identity and Life Design
  • Self-Competition and Entrepreneurship

LLM Quantization

Quantization (reducing the precision of model weights and activations) is one of the most impactful techniques for making LLMs practical. This page covers two approaches: TurboQuant for KV cache compression on cutting-edge hardware, and quantization strategies from OpenAI's Parameter Golf challenge.

TurboQuant (Google Research)

An implementation of Google Research's TurboQuant as a CUDA-native compression engine on NVIDIA's Blackwell B200 GPU, achieving 5x KV cache compression with near-lossless attention scores.

TurboQuant results and profiling

TurboQuant kernel iterations

Source: Anirudh BV (@anirudhbv_ce)

How It Works

TurboQuant compresses the KV cache (the key-value pairs that grow linearly with sequence length during inference) from FP16 down to ~3 bits per coordinate.

Random rotation spreads outlier values into a uniform bell curve, making them amenable to Lloyd-Max quantization (optimal scalar quantization for a given distribution). The result: 5.02x compression at ~0.985 cosine similarity vs FP16 attention.

Implementation Details

Five custom cuTile CUDA kernels:

  • Fused attention (with QJL corrections)
  • Online softmax
  • On-chip cache decompression
  • Pipelined TMA loads
  • Additional optimizations

Performance on B200:

  • 144.7 tok/s on Qwen 2.5-1.5B
  • Fused attention kernel beats FP16 matmul at 4K+ sequence length
  • 6% compute time savings from fused V cache decompression and GPU memory swizzling

Built in ~2,000 lines of cuTile over about a week, starting from pseudocode and iterating through profiling.

Parameter Golf Quantization Strategies

OpenAI's Parameter Golf challenge (train the best language model in 16MB) forced creative quantization under extreme constraints. The RLLM Project's agent swarm discovered several key techniques.

Int5 MLP / Int6 Attention Split

Everyone used int6 quantization for all weights. But MLP weights are more tolerant of quantization noise than attention weights (ReLU² sparsity helps).

The innovation: drop MLP precision to int5, keep attention at int6.

  • Int5 leaves 3 zero high bits per byte, so zstd compression ratio jumps from 1.51x to 1.88x
  • Saves ~1.86 MB in a 16MB budget, enough for an extra transformer layer
  • Quality cost: +0.008 BPB. Extra layer gain: -0.01 BPB. Net: -0.002 BPB for free.

This became widely adopted across the competition and part of the consensus baseline stack.

GPTQ-lite

Standard int6 uses naive min/max clipping. GPTQ-lite tries 5 different clip percentiles per row and picks the one with minimum reconstruction error. Strictly better quantization at the cost of a slower export step. Operates entirely at export time (doesn't touch training, architecture, or optimizer) so it composes cleanly with everything else.

What Didn't Work

  • Int4 MLP: Quality cliff is steep. Int5 is a sweet spot, not a point on a smooth curve.
  • True 6-bit packing: Should save 25% raw, but zstd already exploits the unused high bits. Compressed savings: nearly zero.
  • Fancier embedding tables: Trigram hash, multi-gram hash, adaptive bigram. None beat simply having more rows in a standard BigramHash table.

Source: RLLM Project

Related Topics

  • auto-research - Agent swarms that discovered these quantization strategies
  • reinforcement-learning-for-llms - Quantization interacts with weight update sparsity
  • shaders-and-gpu-programming - GPU architecture fundamentals relevant to CUDA kernel design
  • cursor-composer-2 - Uses optimized inference infrastructure