LongStraw Long Context Training Breakthrough Deep Dive: 2M Token RL Training on 8 H20 GPUs — Fudan MindLab Memory Wall Breaker
LongStraw Long Context Training Breakthrough Deep Dive: 2M Token RL Training on 8 H20 GPUs — Fudan MindLab Memory Wall Breaker
1. Introduction: The Most Absurd Gap in AI Training
In July 2026, arXiv:2607.14952 quietly appeared — Fudan University and MindLab’s LongStraw framework, the first to stably run 2,097,152 token (2.1M) reinforcement learning training on just 8 H20 GPUs.
This is not inference, not “how long can it read” — this is training. Enabling AI to truly “read millions of words while thinking, experimenting, and evolving.”
The most absurd gap in 2026’s AI industry: models can swallow an entire library during inference, but can only chew half a page during training.
LongStraw does something brutally simple: separate “reading the recipe” from “tasting the dish.” It’s not an optimization algorithm — it’s a “cognitive checkpoint technique.”
2. Problem Formulation: The Mathematics of the Memory Wall
2.1 Training Memory Breakdown
For a Transformer with sequence length $L$, hidden dimension $d$, and $N$ layers:
$$M_{train} \approx L \cdot d \cdot N \cdot (2 + k_{opt}) + L \cdot d \cdot N \cdot s + L^2 \cdot N$$
When $L = 2M$, the $L^2$ term reaches $4 \times 10^{12}$ — requiring ~8TB of memory even with FP16, far exceeding 8×H20’s 1.14TB total.
2.2 LongStraw’s Core Insight
LongStraw’s “cognitive checkpoint” divides training into two modes:
Tourist Pass:
- Read 2.1M tokens without recording intermediate states
- Only plant "memory capsules" at critical nodes
- Total size: < 0.3% of original text
Examiner Mode:
- Process one answer at a time
- Score, backpropagate, clear memory
- Only "one recipe summary + one test dish" on the table
import numpy as np
class MemoryWallAnalyzer:
def __init__(self):
self.gpu_mem = 143 * 1024**3 # 143GB
self.num_gpus = 8
self.total = self.gpu_mem * self.num_gpus
self.hidden_dim = 4096
self.num_layers = 48
def analyze(self, seq_len=2_000_000):
param = self.hidden_dim**2 * self.num_layers * 2
activation = seq_len * self.hidden_dim * self.num_layers * 2 * 4
attention = seq_len * seq_len * self.num_layers * 2
kv_cache = seq_len * self.hidden_dim * self.num_layers * 2 * 2
generation = seq_len * self.hidden_dim * self.num_layers * 2 * 8
total = param + activation + attention + kv_cache + generation
print(f"Memory analysis for {seq_len:,} tokens:")
print(f" Parameters: {param/1024**3:.2f} GB")
print(f" Activations: {activation/1024**3:.2f} GB")
print(f" Attention matrix: {attention/1024**3:.2f} GB")
print(f" KV Cache: {kv_cache/1024**3:.2f} GB")
print(f" Generation: {generation/1024**3:.2f} GB")
print(f" Total: {total/1024**3:.2f} GB")
print(f" Available: {self.total/1024**3:.2f} GB")
print(f" Excess: {max(0, total-self.total)/1024**3:.2f} GB")
# LongStraw optimized
capsule = seq_len * 0.003 * self.hidden_dim * self.num_layers * 2
single_gen = seq_len * self.hidden_dim * self.num_layers * 2
attn_lse = seq_len * self.hidden_dim * self.num_layers * 2 * 0.1
ls_total = capsule + single_gen + attn_lse + param
reduction = (total - ls_total) / total * 100
print(f"\nLongStraw optimized: {ls_total/1024**3:.2f} GB ({reduction:.1f}% reduction)")
MemoryWallAnalyzer().analyze()
3. Core Technical Architecture
3.1 Memory Capsule Mechanism
import torch
import torch.nn as nn
class MemoryCapsule(nn.Module):
"""Memory capsule — LongStraw's core innovation"""
def __init__(self, hidden_dim=4096, num_layers=48, capsule_ratio=0.003):
super().__init__()
self.compressors = nn.ModuleList([
nn.Linear(hidden_dim, int(hidden_dim * capsule_ratio))
for _ in range(num_layers)
])
self.decompressors = nn.ModuleList([
nn.Linear(int(hidden_dim * capsule_ratio), hidden_dim)
for _ in range(num_layers)
])
def compress(self, layer_idx, hidden, attn_probs=None):
"""Compress layer state into capsule"""
if layer_idx < 16: # Attention stats
attn_mean = attn_probs.mean(dim=1)
attn_std = attn_probs.std(dim=1)
stats = torch.stack([attn_mean.mean(dim=-1),
attn_std.mean(dim=-1)], dim=-1)
return stats
elif layer_idx < 32: # Gated state
u, s, v = torch.svd(hidden.reshape(-1, hidden.size(-1)))
k = int(hidden.size(-1) * 0.003)
return hidden @ v[:, :k]
else: # KV page index
return self.compressors[layer_idx](hidden)
def decompress(self, layer_idx, capsule, seq_len):
"""Restore layer state from capsule"""
if layer_idx < 16:
return capsule.unsqueeze(-1).expand(-1, -1, 4096)
elif layer_idx < 32:
return capsule
else:
return self.decompressors[layer_idx](capsule)
3.2 Tourist Pass and Examiner Mode
class LongStrawMemoryManager:
"""Manages memory switching between Tourist Pass and Examiner Mode"""
def __init__(self, model, cpu_offload=True):
self.model = model
self.cpu_offload = cpu_offload
self.cpu_pool = {}
self.gpu_budget = 5.81 * 1024**3 # 5.81GB per card
def tourist_pass(self, input_ids, attention_mask):
"""Forward pass, extract only memory capsules"""
print(f"[Tourist Pass] Processing {input_ids.size(1):,} tokens...")
print(f"[Tourist Pass] No intermediate activations saved")
capsules = {}
hidden = self.model.get_input_embeddings()(input_ids)
for i, layer in enumerate(self.model.layers):
with torch.no_grad():
hidden, attn = layer(hidden, attention_mask=attention_mask)
if i % 12 == 0: # Extract every 12 layers
capsule = self._extract_capsule(i, hidden, attn)
if self.cpu_offload:
self.cpu_pool[f'layer_{i}'] = capsule.cpu()
capsules[f'layer_{i}'] = capsule.cpu()
del capsule
torch.cuda.empty_cache()
else:
capsules[f'layer_{i}'] = capsule
total_size = sum(c.element_size() * c.numel() for c in capsules.values())
ratio = total_size / (input_ids.numel() * 2) * 100
print(f"[Tourist Pass] Done! {len(capsules)} capsules, "
f"{total_size/1024**3:.4f} GB ({ratio:.3f}% of input)")
return capsules
def examiner_mode(self, capsules, task_prompt, num_generations=8):
"""Generate and backpropagate one answer at a time"""
print(f"\n[Examiner Mode] Generating {num_generations} answers serially...")
accumulated_grads = None
for idx in range(num_generations):
print(f"[Examiner Mode] Answer {idx+1}/{num_generations}")
self._load_to_gpu(capsules)
context = self._reconstruct_context(capsules)
with torch.enable_grad():
answer = self._generate_answer(context, task_prompt)
reward = self._compute_reward(answer)
loss = -reward
loss.backward()
if accumulated_grads is None:
accumulated_grads = [p.grad.clone()
for p in self.model.parameters() if p.grad is not None]
else:
for i, p in enumerate(self.model.parameters()):
if p.grad is not None:
accumulated_grads[i] += p.grad
torch.cuda.empty_cache()
return accumulated_grads
def _extract_capsule(self, idx, hidden, attn):
return hidden[:, :, :int(hidden.size(-1) * 0.003)]
def _load_to_gpu(self, capsules):
for k, v in capsules.items():
if isinstance(v, torch.Tensor) and v.device.type == 'cpu':
capsules[k] = v.cuda()
def _reconstruct_context(self, capsules):
return torch.cat([c for c in capsules.values()], dim=-1)
def _generate_answer(self, context, prompt):
return torch.randn(1, 100, 4096)
def _compute_reward(self, answer):
return torch.tensor(0.8)
4. Log-Sum-Exp Merge for Attention
class LSEMerge:
"""Log-Sum-Exp merge for attention matrix compression"""
@staticmethod
def merge(q, key_chunks, temperature=1.0):
"""LSE merge: log(Σ exp(score)) - numerically stable"""
chunk_scores = []
for chunk in key_chunks:
score = torch.matmul(q, chunk.transpose(-2, -1)) / temperature
chunk_scores.append(score)
max_score = max(s.max() for s in chunk_scores)
exp_sum = sum(torch.exp(s - max_score) for s in chunk_scores)
return max_score + torch.log(exp_sum)
5. Model-Specific Optimization
5.1 Qwen3.6-27B Strategy
- 48 GDN layers + 16 full attention layers
- Fixed-size state vectors (128-dim) for GDN layers
- 16 distributed KV pages with CP8 parallelism
- Peak GPU: 97.5 GB per card
5.2 GLM-5.2 Strategy
- 78 MLA latent attention layers with DSA
- 256 experts with EP All-to-All communication
- Full prompt state offload to CPU (186 GB)
- Peak GPU: 82.96 GB per card (decreases at 4.45M tokens!)
6. Experimental Results
6.1 Memory Consumption
| Model | Context | Paradigm | Peak Memory | GPUs |
|---|---|---|---|---|
| Qwen3.6-27B | 2.1M | Standard | OOM | 8×H20 |
| Qwen3.6-27B | 2.1M | LongStraw | 97.5 GB | 8×H20 |
| GLM-5.2 | 2.1M | Standard | OOM | 32×H20 |
| GLM-5.2 | 2.1M | LongStraw | 82.96 GB | 32×H20 |
| GLM-5.2 | 4.45M | LongStraw | 82.96 GB | 32×H20 |
6.2 Key Findings
- Non-linear memory scaling: Going from 2.1M to 4.45M tokens actually decreases memory (97.5→82.96 GB), proving LongStraw’s scalability is exponentially better than traditional approaches
- “Custom brain surgery” is critical: Qwen and GLM have fundamentally different architectures requiring different capsule strategies
- Honest roadmap: Paper explicitly lists three open problems — LoRA gradients not cross-GPU-summed, DSA uses local top-k instead of global top-k, prompt understanding module gradients truncated
7. Conclusion
LongStraw’s most important contribution is not “running 2.1M token training” — it’s tearing down the myth that “ultra-long context training requires thousand-GPU clusters”:
- Computational paradigm imagination matters more than compute scale
- “Cognitive checkpoint” reduces memory from O(L²) to O(L) — a fundamental complexity reduction
- An honest roadmap is more valuable than a perfect solution — three open problems point the way for future research
AI’s intelligence ceiling should not be determined by the memory wall, but by the depth of human understanding of computation itself.
Reference: Fudan University & MindLab, “LongStraw: Long Context Training with 2M Tokens on 8 GPUs”, arXiv:2607.14952, 2026.