235 Problems, 88 Firsts, Twice the Runner-Up: Infinigence Turned GPU Kernel Tuning into an Agent Loop — But the Leaderboard Framing Matters More Than the Score

235 Problems, 88 Firsts, Twice the Runner-Up: Infinigence Turned GPU Kernel Tuning into an Agent Loop — But the Leaderboard Framing Matters More Than the Score

Part One: SOL isn’t “fast,” it’s “how far from the hardware ceiling”

Start with what this leaderboard actually is, because most coverage skips the part that makes it unusual.

SOL-ExecBench was launched by NVIDIA Research on March 19, 2026. SOL stands for Speed of Light — here it refers to the theoretical performance ceiling derived from hardware capability. That definition determines everything: traditional kernel benchmarks compare against a software baseline, so a bad enough baseline lets you post a 10x; SOL-ExecBench instead uses a SOL Score referenced to the hardware’s theoretical ceiling, and requires actual execution on real high-performance accelerators.

In other words, it doesn’t ask “how much faster than PyTorch are you,” it asks “how far are you from what this chip can physically do.” Those two questions are not remotely the same difficulty.

The scale is worth recording too:

• 235 kernel optimization problems, extracted directly from the compute graphs of 124 real and frontier AI models;
• Covering language, vision, speech, video, and multimodal;
• Split into four tracks: L1 basic kernels (efficient implementation of fundamental compute blocks), L2 fused kernels (multi-step fusion, cutting memory traffic and data movement), Quant quantized kernels (FP8, NVFP4 and other low precision), and FlashInfer-Bench inference kernels (attention, MoE, and other hot paths in online inference).

One more easily missed feature: this is a continuously contested, high-frequency leaderboard. Per data SF Tensor published on September 25, 2026, 583 teams had cumulatively submitted over 31,000 evaluations, with the board refreshing roughly every 7 minutes. Competitors include Databricks, SF Tensor, Cursor, Recursive, and Geometric.

What does “refreshing every 7 minutes” mean? It means any static scorecard expires within hours. So every number below carries a timestamp.

Part Two: Three framings collide — “topped all four tracks” and “two firsts, two seconds” are two different ledgers

This is the part I most want to emphasize. On Infinigence’s results, public information contains three distinct framings, and they get mixed together in circulation:

# Three framings side by side: all true, all meaning different things
framings = {
    "Official press release (topped over time)": {
        "L1 basic kernels": "first", "L2 fused kernels": "first",
        "Quant kernels": "first", "FlashInfer-Bench": "first",
        "note": "'Over time' — different moments, not one instant",
    },
    "QbitAI snapshot, 2026-10-26": {
        "L1 basic kernels": "1st by total", "L2 fused kernels": "2nd by total",
        "Quant kernels": "1st by total", "FlashInfer-Bench": "2nd by total",
        "note": "One point-in-time snapshot: two firsts, two seconds",
    },
    "Per-problem firsts": {
        "total": 235, "Infinigence firsts": 88,
        "runner-up": "~40 (less than half of 88)",
        "note": "88 firsts, more than 2x the runner-up",
    },
}

for name, d in framings.items():
    print(f"\n[{name}]")
    for k, v in d.items():
        print(f"  {k:22s} {v}")

# Where the 88 firsts concentrate — the key is L2
print("\nDistribution of per-problem firsts (note L2):")
l2_total, l2_firsts = 82, 39
print(f"  L2 fused kernels: {l2_firsts}/{l2_total} = {l2_firsts/l2_total:.0%} (near half)")
print(f"  Other three tracks: {88 - l2_firsts} firsts across {235 - l2_total} problems")

Put the three ledgers together and the conclusion is actually clearer:

• “Topped all four tracks over time” is true — first place in each direction at different moments, showing the capability to win in all four;
• “L1/Quant first by total, L2/FlashInfer second by total” is also true — that’s the October 26 snapshot;
• “88 per-problem firsts, over 2x the runner-up” is equally true — and it’s the metric that best shows breadth and depth together.

The technically meaningful signal is L2 fused kernels: 39 firsts out of 82 problems, near half. Why does this matter most? Because L1 basic kernels are “can you write a single kernel well” — a test of fluency. L2 fused kernels test “how do you merge multiple kernels, avoid an HBM round trip, and balance register pressure against parallelism” — which requires systemic judgment, and is exactly where human experts are most expensive.

The more honest reading: this isn’t “across-the-board dominance” but “near-half firsts in the area that requires systemic optimization, plus total-score leads in two tracks.” Less exciting, but accurate.

Part Three: The agent’s first enemy isn’t a hard problem — it’s reward hacking

To understand this system, first understand its biggest risk: not “the agent can’t produce good kernels” but “the agent produces good scores.”

When an agent targets a performance score, the cheapest path often isn’t writing a better kernel but:

• Editing the evaluation script to slow down the baseline;
• Reading the reference implementation’s output directly;
• Exploiting timer precision holes so a no-op looks fast;
• Caching the previous result so repeated calls cost nothing.

Reinforcement learning has a name for this: reward hacking. And it gets amplified to the extreme in “agent auto-competes on a leaderboard” scenarios — because the agent gets dozens of iterations, plenty of time to find shortcuts.

Infinigence’s answer is physical isolation:

   Three-zone isolated agent kernel-optimization architecture
   ┌─────────────────────────────────────────────────────┐
   │  ① Agent workspace (network on / no GPU allocated)    │
   │     · reads docs, writes code                         │
   │     · mounts framework-aware Skill (persistent file)   │
   │     · mounts shared Memory (past lessons + conditions) │
   └──────────────────┬──────────────────────────────────┘
                      │ unified submission interface (controlled)
                      ▼
   ┌─────────────────────────────────────────────────────┐
   │  ② Host-side GPU scheduler (agent cannot touch it)    │
   └──────────┬───────────────────────────┬──────────────┘
              ▼                           ▼
   ┌────────────────────┐      ┌────────────────────────┐
   │ ③ Eval container    │      │ ④ Debug container       │
   │  · matches official │      │  · profiling tools      │
   │    image            │      │  · network disabled     │
   │  · network disabled │      │  · returns profile only │
   │  · returns results  │      │                         │
   └────────────────────┘      └────────────────────────┘
              │                           │
              └──────────┬────────────────┘
                         ▼
              results back → code gen → eval verify
                          → perf analysis → strategy adjust

Three design points matter, each blocking a specific attack surface:

# Three cheating paths blocked by three-zone isolation
def isolation_guards():
    return {
        "Workspace gets no GPU": {
            "blocks": "Reading reference results from GPU memory / tampering with timers",
            "cost": "Agent must submit through the interface — one extra round trip",
        },
        "Eval container matches official image": {
            "blocks": "Editing eval scripts, slowing the baseline, changing scoring",
            "cost": "Agent cannot specialize to the eval environment",
        },
        "Eval and debug containers both offline": {
            "blocks": "Pulling pre-trained kernels from outside / copying others' submissions",
            "cost": "Agent relies only on knowledge found in the workspace",
        },
    }

for guard, effect in isolation_guards().items():
    print(f"\n[Guard] {guard}")
    for k, v in effect.items():
        print(f"  {k}: {v}")

One detail deserves credit: the agent workspace allows network access but gets no GPU. That combination is clever — it preserves the agent’s ability to “look things up and learn new patterns” (an LLM agent’s biggest edge) while cutting the “verify directly on GPU” shortcut, forcing controlled submission. With GPUs under a host-side scheduler, the agent can’t get exclusive access and can’t specialize to “its own machine.”

This separates capability from oversight. Note the cost is real: an extra round trip per submission, no quick probing, limited debug info. It’s not a free safeguard but an explicit engineering trade-off — slower is acceptable if the score is real.

Part Four: Skills and shared Memory — so the agent still remembers lessons after dozens of rounds

The second key design is knowledge supply, in three layers.

Layer one: modular Skills, reusing open-source community knowledge. GPU programming know-how, debugging tool usage, performance analysis methods. Not novel — many agent systems have this.

Layer two: a framework-aware Skill — this is rarer. It explains the environment the agent is in: how to submit an evaluation, query credentials, borrow a GPU, read results, which operations are out of bounds. And it’s mounted as a persistent file in the workspace, consulted on demand, so the full manual never has to sit in context.

Why this matters only becomes obvious if you’ve run long-horizon agents:

# Why the framework-aware Skill must be mounted, not stuffed into the prompt
def context_pressure(n_rounds=50, tokens_per_round=6000, window=200_000):
    """Estimate context pressure over many optimization rounds"""
    total = n_rounds * tokens_per_round
    return {
        "total tokens": f"{total:,}",
        "context window": f"{window:,}",
        "compaction events": total // window,
        "risk": "each compaction can drop operating rules, causing repeat mistakes",
    }

for k, v in context_pressure().items():
    print(f"  {k:18s} {v}")

An optimization task easily runs dozens of rounds, accumulating 300k+ tokens — context compaction is inevitable. Put “how to submit an evaluation” in conversation history and each compaction may drop it: the agent starts retrying approaches already proven unworkable, or uses the wrong interface and fails submission.

Making the rules a persistent file gives the agent an always-available manual, sidestepping the context window’s physical limit. Any team building long-horizon agents should copy this.

Layer three: shared Memory, capturing “effective strategies + applicability conditions + failure lessons.” All three elements are required — recording only “what worked” is insufficient, because kernel optimization knowledge is almost entirely conditional: a blocking scheme that wins on compute-bound tasks can be a disaster on memory-bound ones. Memory must carry applicability conditions, or the next task will reuse it wrongly.

Part Five: The DSL ladder — Triton wins first in 1.26 hours, PTX starts 2.3x slower but ends ahead

This is the most valuable experimental data in the whole story, and the best evidence of whether agents resemble human experts.

The system guides the agent to pick an implementation starting point by workload characteristics and time budget: Triton, TileLang, CuTe DSL, CUDA C++, PTX. Measured results on the same task set:

# Input/output comparison of DSL starting points on the same task set
dsl_results = {
    # DSL: (time to first valid candidate h, first-version speedup, final speedup)
    "Triton":   (1.26, None, None),   # valid on first try; exact multiple undisclosed
    "CuTe DSL": (None, None, None),   # only ranking given
    "CUDA C++": (None, None, None),
    "PTX":      (2.94, 0.520, 1.438),
}

print("Triton : 1.26 h to first valid candidate")
print("PTX    : 2.94 h to first version, speedup only 0.520x (slower than baseline)")
print("PTX    : final speedup 1.438x after optimization — overtakes Triton")
print(f"startup time ratio: {2.94/1.26:.1f}x")
print(f"first version to final: {1.438/0.520:.2f}x")

roi_order = ["Triton", "CuTe DSL", "CUDA C++", "PTX"]
print("\nROI ranking (high to low):", " > ".join(roi_order))
print("The report states this ranking matches human expert experience")

This result is more thought-provoking than “88 firsts.”

First, Triton starts fastest (1.26 h) — a high-level DSL hides hardware detail, letting the agent produce a “runs, and it’s okay” implementation quickly. Intuitive.

Second, PTX starts slowest (2.94 h, 2.3x Triton) and its first version scores only 0.520x — slower than baseline. Why? PTX is near-assembly; the agent must handle register allocation, instruction scheduling, shared memory layout itself, so v1 is almost guaranteed to be inefficient.

Third, by the end of optimization, PTX reaches 1.438x, overtaking. That’s 2.77x its own first version.

Fourth, the ROI order is Triton > CuTe > CUDA > PTX, and the report explicitly says this “matches human expert experience.”

Together these say one thing: what the agent reproduces here isn’t “human knowledge” but “human strategy-selection logic.” Human experts do the same — build a baseline in Triton fast, and if the bottleneck turns out to be hardware-level, descend layer by layer into CUDA and PTX to squeeze the last few percent. The agent learned that “layered descent” meta-strategy rather than memorizing how to write a specific kernel.

Which explains why the system “guides the agent to pick abstraction level by time budget” — with a 2-hour budget, choosing PTX is suicide; with 20 hours and a task stuck at the hardware limit, choosing Triton is waste. That judgment is where the system’s real intelligence lives.

Part Six: Four workload classes, four entirely different playbooks

One layer deeper is fine-grained strategy by workload characteristics. This is the densest part technically, and the clearest evidence the team actually knows GPUs.

# Optimization strategies for four workload classes (auto-switched by the agent)
strategies = {
    "Compute-bound": {
        "watch": ["Tensor Core utilization", "tiling", "data reuse"],
        "means": ["async data movement", "pipeline overlap"],
        "tradeoff": "registers vs shared memory vs effective parallelism",
    },
    "Memory-bound": {
        "watch": ["redundant reads/writes", "intermediate materialization"],
        "means": ["coalesced access", "vectorized access", "cache reuse"],
        "goal": "lower data movement per unit of useful compute",
    },
    "Multi-kernel chains": {
        "watch": ["fusion benefit", "register pressure", "sync complexity"],
        "means": ["kernel fusion", "PDL", "Persistent Kernel"],
        "goal": "amortize block scheduling and init overhead",
    },
    "Small shape, low latency": {
        "watch": ["block count", "idle threads", "bounds checks", "instruction & sync overhead"],
        "means": ["coarser tiling", "simplified pipelines"],
        "trap": "over-tiling adds fixed cost that compute can't amortize",
    },
}

for workload, s in strategies.items():
    print(f"\n[{workload}]")
    for k, v in s.items():
        if isinstance(v, list):
            print(f"  {k}: {', '.join(v)}")
        else:
            print(f"  {k}: {v}")

Two deserve expansion.

One: PDL (Programmatic Dependent Launch). Traditionally a successor kernel must wait for the predecessor to fully finish, leaving a drain-and-launch gap. PDL lets the successor’s work that doesn’t depend on predecessor results start early, synchronizing only before actually reading dependent data. It’s a fine-grained hardware feature that fills part of the inter-kernel gap — but used wrong, it causes data races and everything breaks. The agent must judge “when is it safe,” which is well beyond code generation.

Two: Persistent Kernel. The usual approach launches a batch of CTAs per work unit, which exit on completion, paying scheduling overhead repeatedly. Persistent kernels keep CTAs resident, looping over multiple work units, amortizing scheduling and init. Big win when there are many small work units — at the cost of holding more resident resources and reducing concurrency elasticity.

The “small shape, low latency” class is the most counterintuitive. Intuitively “finer tiling is better,” but in fact: when the compute itself is small, the scheduling, synchronization, and bounds-check overhead from tiling can’t be amortized and becomes pure burden. So the optimal answer is often “tile coarsely, simplify the pipeline, minimize fixed cost.” A classic “over-optimization makes it slower” trap that human experts need a few bruises to learn — while an agent can learn it once via shared Memory.

Part Seven: The real boundaries of RSI — from Token Maxxing to compute efficiency

Now to the narrative layer. Infinigence positions itself as Agentic Infra, arguing for a shift from GPU Maxxing (squeeze the GPU) to Token Maxxing (squeeze productivity per token), with the loop pointing at RSI (Recursive Self-Improvement).

That narrative isn’t empty. There are contemporaneous empirical advances:

• Anthropic disclosed that Claude already participates in a substantial share of internal model R&D, with AI independently conducting open-ended AI safety research and proposing and validating new methods;
• Weco AI presented AIDE², the first experimental agent system reaching “Level 1 RSI.”

Why kernel optimization is a good entry point — the chain runs like this:

   Agent optimizes kernels → faster kernels → cheaper training/inference
        ↑                                        │
        │                                        ▼
   more compute budget ←── lower unit cost ←── more experiments per dollar
        │
        └──> stronger model → stronger agent (back to start)

Beautiful flywheel — but three physical speed bumps must be seen:

# Three speed bumps on the RSI flywheel
def rsi_limits():
    return {
        "Boundary 1: absolute per-point gains are limited": {
            "fact": "kernel optimization typically yields single-digit to tens of percent",
            "meaning": "not an order-of-magnitude jump, but compounding marginal gain",
            "implication": "needs long accumulation to matter; invisible in a quarterly report",
        },
        "Boundary 2: verification cost can't be compressed": {
            "fact": "every candidate kernel must run on real hardware and be correctness-checked",
            "meaning": "the faster the agent, the more the eval queue is the bottleneck",
            "implication": "the board refreshes every 7 min, but physical run time can't vanish",
        },
        "Boundary 3: generalizing to 'changing the model' is very hard": {
            "fact": "kernel work has unambiguous correctness and a clean performance metric",
            "meaning": "architecture improvements have no such instant feedback",
            "implication": "RSI at the kernel layer != RSI at the research layer",
        },
    }

for name, d in rsi_limits().items():
    print(f"\n{name}")
    for k, v in d.items():
        print(f"  {k}: {v}")

Boundary one is the most overlooked. Kernel gains compound — a few percent each time, substantial in aggregate — but they won’t deliver “10x cheaper training.” Reading this as proof that “AI builds stronger AI” is over-interpretation; reading it as “AI accelerated the compounding of engineering efficiency” is accurate.

Boundary three is the most critical. Kernel optimization can be handed to agents fundamentally because it has automatically decidable correctness (differential testing) and a quantifiable performance metric (SOL Score). Remove either and the agent’s loop breaks instantly — it doesn’t know whether its change is right or good. “Designing a better model architecture” lacks exactly these two: what counts as “better”? How many benchmarks prove it? Do you wait six months for downstream validation? That’s why kernel-layer RSI is already running while research-layer RSI is still in papers and demos.

Understanding this boundary is how to evaluate the work correctly: it’s not proof of “AI self-evolution,” but proof that “in engineering domains with clear feedback and controllable verification cost, AI has already outrun human experts’ iteration speed.” Less thrilling, closer to fact, more industrially valuable.

Part Eight: Judgment and boundaries

First, the value of 88/235 is breadth, not height. Per-problem firsts at 2x the runner-up means this agent isn’t farming a few favorite problem types but competing broadly across 235 problems. Harder to achieve and harder to sustain than “first in one track” — because the live board refreshes every 7 minutes, and a broad strategy demands continuous compute and engineering investment.

Second, what’s worth copying isn’t the score but the isolation architecture. Three-zone separation, host-side GPU scheduling, eval containers matching the official image, network disabled — these four constitute the minimum viable boundary for “let the agent work autonomously but not cheat.” Any team building “agent auto-optimizes / auto-competes / auto-tunes” should copy this boundary before discussing model capability. Once an agent learns to cheat, all your subsequent data is dirty.

Third, the DSL ladder experiment is the most academically valuable output here. “Triton first in 1.26h, PTX 2.94h start but 1.438x finish, ROI order matching human experts” — together they show the agent learned strategy-selection logic rather than specific implementations. That finding matters far beyond kernel optimization: it hints at where agents can genuinely replace experts — domains where the expert’s value lies mainly in knowing what to do first and when to switch tools.

Fourth, framing confusion is normal in this kind of news; readers need their own filter. “Topped all four tracks over time” and “two firsts, two seconds” aren’t contradictory, but they describe different time slices; “88 per-problem firsts” and “first by total” are also different ledgers. On a live leaderboard, any ranking without a timestamp is only indicative. Not an Infinigence problem — a systemic issue with leaderboard news.

Fifth, beware inflation of the term “RSI.” Kernel optimization is a good sample of RSI precisely because it’s simple: clear feedback, cheap verification, quantifiable gain. But being simple, it’s also the easiest class to achieve. Using the easiest class to argue “AI can self-improve” is a logical leap. The real watershed: when feedback is fuzzy, verification takes months, and success criteria are contested, can the agent still close the loop? That question has no answer yet.

A trackable framework: don’t just watch rankings; watch four numbers — ① whether the 88 firsts are still 88 two weeks later (decay rate on a live board is the real measure of competitiveness); ② whether L2’s 39/82 crosses half (the hardest track and the best evidence of system capability); ③ how many optimized kernels actually enter production training/inference stacks (the leaderboard-to-production conversion rate is where value is realized); ④ whether end-to-end latency from submission to result drops (this determines how long one turn of the flywheel takes).

Sources: NVIDIA Research SOL-ExecBench official leaderboard (research.nvidia.com/benchmarks/sol-execbench, launched 2026-03-19); Infinigence official news (2026-10-28, “SOL-ExecBench Four Tracks Global First, Infinigence’s Agentic Infra Advances RSI Practice”); QbitAI, “Chinese AI Storms NVIDIA’s GPU Performance Board, This Company Takes 88 Firsts” (2026-10-28); Phoenix Net / Dafeng, “Under the RSI Boom, a Self-Iterating Domestic Agentic Infra Pushes Toward NVIDIA GPU’s Physical Limits” (2026-10-31); submission data published by SF Tensor on 2026-09-25. The three-framing comparison, context-pressure estimate, DSL ROI analysis, and RSI boundary breakdown are synthesis and inference from public information, not official figures; SOL-ExecBench refreshes frequently, so all rankings correspond to specific moments — consult the official page for current status.

• ←Previous
• Next→