EvoLib Test-Time Learning Deep Dive: Microsoft Gradient-Free Knowledge Evolution with IG and Future IG Credit Assignment
EvoLib Test-Time Learning Deep Dive: Microsoft Gradient-Free Knowledge Evolution with IG and Future IG Credit Assignment
1. Introduction: The Post-Deployment Learning Dilemma
On July 30, 2026, Microsoft Research open-sourced EvoLib—a Test-Time Learning (TTL) framework that addresses a fundamental contradiction in large language model deployment: models with hundreds of billions of parameters become static artifacts once deployed, incapable of learning from real-world interactions.
The current dominant paradigm is essentially a “closed-book exam”: all knowledge comes from pre-training data, and once deployed, models face new tasks relying solely on memorized parameters—unable to learn from mistakes, extract lessons from successes, or improve with use.
EvoLib’s core insight is deceptively simple yet profoundly transformative: enable black-box LLMs to build a shared, evolving knowledge library during inference, distilling each inference trajectory into reusable “Skills” and “Insights”, and automatically applying them in subsequent tasks. No parameter updates, no gradients, no human-labeled supervision.
This article provides a deep technical analysis of EvoLib’s algorithm architecture, engineering implementation, and experimental results.
2. Problem Formulation: The Mathematics of Test-Time Learning
2.1 Problem Definition
Given a black-box LLM $f_{\theta}$ (parameters $\theta$ inaccessible), processing a sequence of task instances $\mathcal{T} = {t_1, t_2, …, t_N}$, each task $t_i$ generates a reasoning trajectory $\tau_i = (s_i, a_1, a_2, …, a_K)$.
In traditional approaches, each trajectory $\tau_i$ is discarded. EvoLib’s goal is to extract transferable knowledge from processed trajectories ${\tau_1, …, \tau_{i-1}}$ to improve subsequent task solving.
2.2 Knowledge Representation
EvoLib maintains two complementary knowledge types:
Skills: Reusable procedural knowledge represented as: $$skill = (name, body, signature, score)$$
Insights: Reflective declarative knowledge represented as: $$insight = (rule, context, score)$$
2.3 Knowledge Base Evolution
The knowledge base $\mathcal{K}$ evolves as a Markov Decision Process: $$\mathcal{K}{i} = \text{Update}(\mathcal{K}{i-1}, \tau_i, \text{Score}(\mathcal{K}_{i-1}, \tau_i))$$
3. Core Algorithm: Information Gain-Driven Credit Assignment
3.1 Information Gain (IG)
For a knowledge item $k \in \mathcal{K}$, its Information Gain on task $t$ is:
$$IG(k, t) = \log \frac{P(\text{success} | k \in \text{prompt}, t)}{P(\text{success} | k \notin \text{prompt}, t)}$$
3.2 Future Information Gain (FIG)
FIG captures the insight that some knowledge items, though seemingly useless now, serve as “seeds” for future useful knowledge:
$$FIG(k) = \sum_{k’ \in \text{descendants}(k)} \gamma^{\text{dist}(k, k’)} \cdot IG(k’)$$
3.3 Dynamic Weighting
The final weight combines IG and FIG: $$w(k) = \lambda \cdot IG(k) + (1 - \lambda) \cdot FIG(k)$$
import numpy as np
from typing import Dict, List, Any
class EvoLibCreditAssignment:
"""Core credit assignment combining IG and FIG"""
def __init__(self, lam: float = 0.6, gamma: float = 0.9):
self.lam = lam
self.gamma = gamma
self.knowledge_scores: Dict[str, float] = {}
self.knowledge_history: Dict[str, List[bool]] = {}
self.evolution_graph: Dict[str, List[str]] = {}
def compute_ig(self, k_id: str, success: bool, used: bool) -> float:
"""Compute Information Gain for a knowledge item"""
if not used:
return 0.0
if k_id not in self.knowledge_history:
self.knowledge_history[k_id] = []
self.knowledge_history[k_id].append(success)
recent = self.knowledge_history[k_id][-20:]
if len(recent) < 2:
return 0.0
success_rate = np.mean(recent)
all_successes = [s for hist in self.knowledge_history.values()
for s in hist]
base_rate = np.mean(all_successes) if all_successes else 0.5
success_rate = np.clip(success_rate, 0.01, 0.99)
base_rate = np.clip(base_rate, 0.01, 0.99)
return np.log(success_rate / base_rate)
def compute_fig(self, k_id: str, depth: int = 0, max_depth: int = 5) -> float:
"""Recursively compute Future Information Gain"""
if depth > max_depth:
return 0.0
direct_ig = self.knowledge_scores.get(k_id, 0.0)
children = self.evolution_graph.get(k_id, [])
if not children:
return direct_ig
child_figs = [self.compute_fig(c, depth + 1, max_depth)
for c in children]
max_child_fig = max(child_figs) if child_figs else 0.0
return direct_ig + self.gamma * max_child_fig
def update(self, k_id: str, task_result: Dict):
"""Update credit after task completion"""
used = k_id in task_result.get('used_knowledge', [])
success = task_result.get('success', False)
ig = self.compute_ig(k_id, success, used)
fig = self.compute_fig(k_id)
self.knowledge_scores[k_id] = self.lam * ig + (1 - self.lam) * fig
4. Knowledge Extraction and Consolidation
4.1 Skill Extraction Pipeline
class SkillExtractor:
"""Extracts reusable skills from inference trajectories"""
def extract_skills(self, trajectory: List[Dict]) -> List[Dict]:
steps = self._segment_trajectory(trajectory)
skills = []
for step in steps:
generalized = self._generalize_step(step)
signature = self._extract_signature(generalized)
code = self._generate_code(generalized, signature)
skills.append({
'name': signature['name'],
'body': code,
'signature': signature['description'],
'score': 0.0
})
return skills
def _segment_trajectory(self, trajectory: List[Dict]) -> List[Dict]:
"""Split trajectory into reusable segments"""
segments = []
current = {'steps': [], 'type': None}
for step in trajectory:
step_type = self._classify_step(step)
if step_type != current['type'] and current['steps']:
segments.append(current)
current = {'steps': [], 'type': step_type}
current['steps'].append(step)
current['type'] = step_type
if current['steps']:
segments.append(current)
return segments
def _classify_step(self, step: Dict) -> str:
content = str(step.get('content', '')).lower()
if any(kw in content for kw in ['import ', 'def ', 'class ']):
return 'code_definition'
elif any(kw in content for kw in ['error', 'exception', 'fail']):
return 'error_handling'
elif any(kw in content for kw in ['result', 'output', 'return']):
return 'result_processing'
return 'reasoning'
def _generalize_step(self, step: Dict) -> Dict:
"""Replace concrete values with template variables"""
import re
content = str(step.get('content', ''))
content = re.sub(r'https?://[^\s\'\"\)]+', '{API_ENDPOINT}', content)
content = re.sub(r'\b\d{4,}\b', '{NUMERIC_VALUE}', content)
step['generalized_content'] = content
return step
def _extract_signature(self, step: Dict) -> Dict:
body = str(step.get('generalized_content', ''))
words = body.split()[:5]
return {
'name': '_'.join([w.lower() for w in words if w.isalpha()][:3]),
'description': body[:100],
}
def _generate_code(self, step: Dict, signature: Dict) -> str:
template = f"""
def {signature['name']}(**kwargs):
\"\"\"{signature['description']}\"\"\"
# Auto-generated from past trajectory
{step.get('generalized_content', '')}
"""
return template.strip()
4.2 Knowledge Consolidation
Knowledge consolidation prevents the knowledge base from exploding. When two items exceed a semantic similarity threshold, they merge into one more general item:
class KnowledgeConsolidation:
"""Merges similar knowledge items to control growth and improve generality"""
def __init__(self, threshold: float = 0.75):
self.threshold = threshold
def consolidate(self, knowledge_base: Dict[str, Dict]) -> Dict[str, Dict]:
ids = list(knowledge_base.keys())
n = len(ids)
# Compute similarity matrix
sim_matrix = np.zeros((n, n))
for i in range(n):
for j in range(i + 1, n):
sim = self._similarity(knowledge_base[ids[i]],
knowledge_base[ids[j]])
sim_matrix[i][j] = sim_matrix[j][i] = sim
# Greedy clustering
assigned = [False] * n
clusters = []
for _ in range(n):
unassigned = [i for i in range(n) if not assigned[i]]
if not unassigned:
break
# Find item with highest average similarity to others
best = max(unassigned,
key=lambda i: np.mean([sim_matrix[i][j]
for j in unassigned if j != i]) if len(unassigned) > 1 else 0)
cluster = [ids[best]]
assigned[best] = True
for j in range(n):
if not assigned[j] and sim_matrix[best][j] >= self.threshold:
cluster.append(ids[j])
assigned[j] = True
clusters.append(cluster)
# Merge each cluster
new_base = {}
for cluster in clusters:
if len(cluster) == 1:
new_base[cluster[0]] = knowledge_base[cluster[0]]
else:
merged = self._merge([knowledge_base[k] for k in cluster])
new_base[merged['id']] = merged
return new_base
def _similarity(self, k1: Dict, k2: Dict) -> float:
body1 = str(k1.get('body', '')) + ' ' + str(k1.get('signature', ''))
body2 = str(k2.get('body', '')) + ' ' + str(k2.get('signature', ''))
words1 = set(body1.lower().split())
words2 = set(body2.lower().split())
if not words1 or not words2:
return 0.0
return len(words1 & words2) / len(words1 | words2)
def _merge(self, items: List[Dict]) -> Dict:
import hashlib
common_sig = items[0].get('signature', '')[:50]
merged_body = max([str(k.get('body', '')) for k in items], key=len)
avg_score = sum(k.get('score', 0.0) for k in items) / len(items)
return {
'id': f'merged_{hashlib.md5(common_sig.encode()).hexdigest()[:12]}',
'body': merged_body,
'signature': common_sig,
'score': avg_score,
'is_consolidated': True
}
5. EvoLib Main Loop
class EvoLibAgent:
"""Core EvoLib agent implementing the test-time learning loop"""
def __init__(self, llm_client, max_knowledge: int = 100):
self.llm = llm_client
self.skills: Dict[str, Dict] = {}
self.insights: Dict[str, Dict] = {}
self.credit = EvoLibCreditAssignment()
self.extractor = SkillExtractor()
self.consolidation = KnowledgeConsolidation()
self.max_knowledge = max_knowledge
self.iteration = 0
def solve(self, task: Dict) -> Dict:
"""Solve a single task while accumulating knowledge"""
self.iteration += 1
# Step 1: Select relevant knowledge
selected_skills = self._select_knowledge(self.skills, task['prompt'])
selected_insights = self._select_knowledge(self.insights, task['prompt'])
# Step 2: Build enhanced prompt
prompt = self._build_prompt(task['prompt'], selected_skills, selected_insights)
# Step 3: Execute inference
solution = self.llm.generate(prompt)
# Step 4: Evaluate
success = task.get('evaluator', lambda x: True)(solution)
# Step 5: Extract knowledge
trajectory = self._parse_trajectory(solution)
new_skills = self.extractor.extract_skills(trajectory)
new_insights = self._extract_insights(trajectory, success)
# Step 6: Update knowledge base
for skill in new_skills:
sid = f"skill_{self.iteration}_{len(self.skills)}"
self.skills[sid] = skill
for insight in new_insights:
iid = f"insight_{self.iteration}_{len(self.insights)}"
self.insights[iid] = insight
# Step 7: Credit assignment
result = {
'used_knowledge': selected_skills + selected_insights,
'success': success
}
for k in selected_skills + selected_insights:
self.credit.update(k, result)
# Step 8: Consolidate periodically
if self.iteration % 10 == 0:
self.skills = self.consolidation.consolidate(self.skills)
self.insights = self.consolidation.consolidate(self.insights)
# Prune if needed
if len(self.skills) + len(self.insights) > self.max_knowledge:
self._prune()
return {'solution': solution, 'success': success,
'kb_size': len(self.skills) + len(self.insights)}
def _build_prompt(self, prompt: str, skills: List[str], insights: List[str]) -> str:
parts = [prompt]
if skills:
skill_texts = []
for sid in skills:
s = self.skills.get(sid, {})
skill_texts.append(f"## Skill: {s.get('name', '')}\n"
f"```python\n{s.get('body', '')}\n```")
if skill_texts:
parts.append("### Available Skills:\n" + "\n".join(skill_texts))
if insights:
ins_texts = []
for iid in insights:
i = self.insights.get(iid, {})
ins_texts.append(f"- {i.get('rule', '')}")
if ins_texts:
parts.append("### Lessons Learned:\n" + "\n".join(ins_texts))
return "\n\n".join(parts)
def _select_knowledge(self, kb: Dict, context: str) -> List[str]:
scored = []
ctx_words = set(context.lower().split())
for k_id, entry in kb.items():
sig = str(entry.get('signature', '') + entry.get('rule', '')).lower()
sig_words = set(sig.split())
relevance = len(ctx_words & sig_words) / max(len(sig_words | ctx_words), 1)
score = entry.get('score', 0.0) * (1 + relevance)
scored.append((score, k_id))
scored.sort(reverse=True)
return [k for _, k in scored[:5]]
def _parse_trajectory(self, solution: str) -> List[Dict]:
return [{'id': f'step_{i}', 'content': line, 'context': ''}
for i, line in enumerate(solution.split('\n')) if line.strip()]
def _extract_insights(self, trajectory: List[Dict], success: bool) -> List[Dict]:
if success:
return []
errors = [s for s in trajectory if 'error' in str(s.get('content', '')).lower()]
return [{'rule': f"Avoid: {e['content'][:100]}",
'context': '', 'score': 0.0, 'type': 'negative'}
for e in errors[:3]]
def _prune(self):
threshold = np.percentile(
[k.get('score', 0.0) for k in self.skills.values()] +
[k.get('score', 0.0) for k in self.insights.values()], 10)
self.skills = {k: v for k, v in self.skills.items()
if v.get('score', 0.0) > threshold}
self.insights = {k: v for k, v in self.insights.items()
if v.get('score', 0.0) > threshold}
6. Experimental Results
6.1 Benchmark Performance
| Benchmark | Base Model | Baseline | EvoLib | Improvement |
|---|---|---|---|---|
| HMMT 2025-2026 (Math) | o4-mini | 57.0% | 77.4% | +20.4% |
| BigCodeBench Hard (Code) | GPT-4o | 29.7% | 40.8% | +11.1% |
| LiveCodeBench v6 Hard (Code) | o4-mini | - | 70.0% | Ties RSA |
6.2 Cost-Performance Analysis
def analyze_cost_performance():
"""Analyze EvoLib's cost-performance advantage"""
budgets = [1, 2, 4, 8, 16, 32, 64] # K weighted tokens
evo = [29.0, 33.5, 37.0, 39.0, 40.2, 40.6, 40.8]
bon = [29.0, 32.0, 34.5, 36.0, 37.0, 37.2, 37.2]
base = [29.0, 30.5, 31.0, 31.2, 31.3, 31.3, 31.3]
print("Key Findings:")
print(f"EvoLib final gain: {evo[-1] - base[-1]:.1f} pts")
print(f"Best-of-N final gain: {bon[-1] - base[-1]:.1f} pts")
print(f"EvoLib advantage: {(evo[-1] - base[-1]) - (bon[-1] - base[-1]):.1f} pts")
# Low-budget advantage
low_budget_gain = (evo[2] - base[2]) / (bon[2] - base[2]) if bon[2] > base[2] else float('inf')
print(f"Low-budget efficiency ratio: {low_budget_gain:.2f}x")
# Knowledge accumulation curve
print("\nKnowledge Accumulation S-Curve:")
print(" Phase 1 (0-20): Slow accumulation, ~15 skills + ~10 insights")
print(" Phase 2 (20-60): Accelerated growth, ~35 skills + ~25 insights")
print(" Phase 3 (60+): Saturation, consolidation kicks in")
print(f"\nKnowledge reuse rate: ~4.2x per item")
print(f"FIG ablation impact: -7% at 200 rounds")
analyze_cost_performance()
7. Conclusion
EvoLib represents a paradigm shift in test-time learning: from bigger models to smarter usage. It proves that even without updating model parameters, carefully designed knowledge accumulation and reuse mechanisms enable black-box LLMs to continuously improve through use.
Key Contributions:
- Dual knowledge representation: Skills (procedural) + Insights (reflective)
- FIG credit assignment: First to introduce future information gain in TTL
- Consolidation mechanism: Controls KB growth while improving generality
Future Directions:
- Multi-agent collaboration with per-agent knowledge bases
- Integration with continual learning for dual-level learning
- Knowledge graph structures for causal relationships between knowledge items
Reference: Microsoft Research, “Test-Time Learning with an Evolving Library”, arXiv:2605.14477, 2026. Code: https://github.com/microsoft/EvoLib (MIT License).