Reinforcement Learning for LLMs

RL post-training has become a critical phase in building frontier LLMs, but the infrastructure requirements are distinct from pretraining. This page covers the systems engineering of RL at scale, with a focus on Fireworks' approach to making frontier RL accessible without mega-clusters.

Source: Fireworks

RL Infrastructure in 30 Seconds

An RL training run has two jobs:

• The trainer: Forward pass, reward computation, backward pass, parameter updates. Needs dense, tightly coupled hardware.
• The rollout fleet: Samples trajectories from the current policy (runs inference on the latest model). Needs inference throughput across many parallel requests.

Pretraining only has the first job. RL has both, which is why the infrastructure question is fundamentally different.

The 1 TB Problem

A typical frontier checkpoint is ~1 TB. If every policy refresh required shipping the full checkpoint to the rollout fleet, RL would need one giant co-located cluster with RDMA-class networking. This makes frontier RL look like a market only a handful of companies can enter.

But the premise is wrong. You don't need to move the full 1 TB on every update.

The Key Insight: 98% Sparsity

Between consecutive RL checkpoints, more than 98% of weights remain bit-equivalent in bf16 format. This happens because:

• Post-training updates are fine-grained
• RL provides very sparse information signal (just a few bits per rollout)
• Small learning rates mean most parameters move only slightly in fp32, often not enough to alter their 16-bit representation

Practical impact: A 1024 GiB full checkpoint has an average delta of only 20.3 GiB (1.98%). Over a 50-step window, this cuts cross-region transfer by ~94% compared to moving the full model every time.

A published paper, "Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL," provides theoretical foundation and reports similarly high sparsity (~99%).

Async RL and Pipelining

Running trainer and rollout fleet asynchronously means the fleet always serves a policy a few steps behind the trainer. This gap (staleness) is a real tradeoff.

The systems layer doesn't eliminate staleness. The algorithm must tolerate some off-policy data. But delta-compressed updates keep the gap bounded and predictable, making it a routine background operation rather than a stop-the-world event.

Multi-Region Rollout Capacity

Most teams don't have one giant contiguous cluster for rollouts. They have capacity scattered across regions, clouds, and availability zones. Once weight updates are small, that fragmented capacity becomes usable for RL.

Each rollout cluster independently downloads and reconstructs weights from the same delta chain, without direct connection to the training cluster. This makes the fleet elastic: add/remove/rebalance clusters as capacity and cost change.

Real-world validation: Fireworks used this architecture for cursor-composer-2's training run, which was "distributed across 3 (sometimes 4) different clusters around the world."

When This Approach Matters Less

• Model is small enough to fit trainer and rollout on one node, so bandwidth isn't the bottleneck
• Checkpoints are emitted so frequently that deltas can't keep up, and staleness becomes limiting. Tighter co-location makes sense.
• Rollout stack doesn't cleanly separate inference from training, so the disaggregated design doesn't fit

Fireworks Training Platform

Three deployment modes:

1. Fully managed RL: Provide dataset and evaluators; Fireworks runs everything
2. Tinker-compatible SDK: Customize the training flow; host GPU work on Fireworks
3. Bring your own trainer: Keep trainer in-house; use Fireworks for rollout serving and weight-update orchestration

Related Topics

• cursor-composer-2 - Used Fireworks' multi-region RL infrastructure
• training-free-grpo - An alternative: RL-like improvements without any training
• llm-quantization - Quantization interacts with weight update sparsity
• auto-research - RL as a form of automated self-improvement