MiniMax H3 Omni-Modal Deep Dive: H3-Omni-Transformer Three-Module Architecture and 2K Video Regeneration
MiniMax H3 Omni-Modal Deep Dive: H3-Omni-Transformer Three-Module Architecture and 2K Video Regeneration
On August 3, 2026, MiniMax officially open-sourced MiniMax H3, a next-generation general-purpose omni-modal generation model. It is the first commercially viable flagship video model with open weights, ranking #1 globally in video editing on the Artificial Analysis leaderboard. This article dissects the H3 technical system across architecture design, core technologies, information flow, and code implementation.
I. From “Expert Models” to “General-Purpose System”: H3’s Design Philosophy
1.1 Industry Pain Points: Task Islands and Modality Barriers
Before H3, the AI video generation landscape was dominated by “task silos”:
- Image generation was fragmented into separate expert models for T2I, editing, subject reference, motion reference, and style reference
- Audio generation treated voice, sound effects, and music as largely independent domains
- Video generation was further splintered into text-to-video, image-to-video, first-and-last-frame, subject reference, motion reference, voice reference, video editing — with clear boundaries between image, video, and audio
These barriers constrained creative freedom in practice and capped the model’s ability to generalize on the training side. H3’s first design principle is cross-task, cross-modality unification and generalization.
1.2 Core Design Decisions
H3’s pretraining paradigm is built around the following pillars:
| Dimension | Design Choice |
|---|---|
| Data & Tasks | Text-to-image, text-to-video (with jointly generated audio & native multi-shot modeling), text-to-audio (voice/SFX/music jointly modeled), generalized reference & editing |
| Reference & Editing Definition | Built entirely from real, natural data; relationships expressed through natural language rather than a fixed task set |
| Architecture | Fuse diverse data types and tasks as early as possible — the right mixing ratio is key |
| Training Strategy | Fuse diverse data types and tasks at the earliest training stages |
In one sentence: Enable H3 to possess broad multimodal context understanding and generation capabilities from the pretraining stage onward.
II. Three-Module System Architecture: End-to-End Information Flow
2.1 System Overview
┌─────────────────────────────────────────────────────────────────────┐
│ MiniMax H3 Complete System Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────────┐ ┌─────────────────┐ │
│ │ H3-Context-IR│ │ H3-Base │ │H3-Regenerate-2K│ │
│ │ (Closed/API) │───▶│ (Open 33.1B) │───▶│ (Closed/API) │ │
│ │ │ │ │ │ │ │
│ │ · Instruction│ │ · H3-Encoder │ │ · In-context │ │
│ │ parsing │ │ (Qwen3-VL-32B) │ │ regeneration │ │
│ │ · Cross-modal│ │ · H3-VisualVAE │ │ · Original │ │
│ │ association│ │ · H3-AudioVAE │ │ context for │ │
│ │ · Temporal │ │ · Omni-Transform │ │ detail recovery│ │
│ │ understanding │ · 768p output │ │ · 2K resolution │ │
│ │ · Logic │ │ │ │ output │ │
│ │ reasoning │ │ │ │ │ │
│ │ · Serialization│ │ │ │ │ │
│ └──────────────┘ └──────────────────┘ └─────────────────┘ │
│ │ │ │ │
│ User multimodal 768p Audio-Video 2K Audio-Video │
│ input (text/img/ (short edge 768px) (up to 2560×1440) │
│ video/audio) │
│ │
│ 100K tokens ──compress──▶ ~4K tokens ──generate──▶ 768p ──regen──▶ 2K │
│ │
└─────────────────────────────────────────────────────────────────────┘
2.2 Three-Module Collaboration Pipeline
Phase 1: H3-Context-IR (Context Intermediate Representation)
H3-Context-IR is a managed preprocessing and orchestration system responsible for:
- Instruction Parsing: Understanding the creative intent in user text input
- Cross-Modal Association: Parsing relationships among text, images, audio, and reference videos
- Temporal Understanding: Processing timeline information from video materials
- Complex Logic Reasoning: Inferring implicit semantic relationships between materials
- Serialization & Compression: Converting understanding into structured representations that H3-Base can directly consume
Most source materials require ~100K tokens of inference, distilled down to an average of ~4K tokens. This 25:1 compression ratio is a key driver of system efficiency.
⚠️ Note: H3-Context-IR relies on multi-stage workflows and multiple hosted models/services. It is not included in this open-source release. An API and prompt-writing guide are provided.
Phase 2: H3-Base (Core Generation Model)
H3-Base is the open-sourced 33.1B-parameter core generation model:
- Receives structured representations from H3-Context-IR
- Encodes different modality information into a unified sequence via multi-encoders
- Jointly generates video and stereo audio through H3-Omni-Transformer
- Outputs audio-video results at short edge 768 pixels
Phase 3: H3-Regenerate-2K (2K Regeneration)
The innovation of H3-Regenerate-2K lies in not using a conventional super-resolution module. Instead:
- The 768p video from H3-Base serves as
base_video - Combined with the enhanced prompt and original multimodal context
- The base model regenerates a 2K version in an in-context manner
- Recovers small text, fine textures, and other details that low resolution cannot restore
⚠️ This module is also not yet open-sourced and is available via API.
III. H3-Omni-Transformer Architecture Deep Dive
3.1 Architecture Overview
H3-Omni-Transformer is the core component of H3-Base — a 33.1B-parameter Dense Single-Stream Transformer designed specifically for task generalization.
┌───────────────────────────────────────────────────────────────────────┐
│ H3-Omni-Transformer Internal Architecture │
├───────────────────────────────────────────────────────────────────────┤
│ │
│ Input Side: │
│ ┌──────────┐ ┌───────────────┐ ┌──────────────┐ ┌────────────┐ │
│ │ H3-Encoder│ │H3-VisualVAE │ │H3-AudioVAE │ │ Text │ │
│ │(Qwen3-VL │ │(32x spatial │ │(40Hz temporal│ │ Token │ │
│ │ -32B) │ │ compression │ │ frequency │ │ │ │
│ │ │ │ 4x temporal) │ │ stereo enc.) │ │ │ │
│ │ · Text │ │ · Image/video │ │ · Audio │ │ │ │
│ │ encoding│ │ spatial │ │ waveform │ │ │ │
│ │ · Visual │ │ compression │ │ → latent │ │ │ │
│ │ encoding│ │ · 4x sequence │ │ │ │ │ │
│ │ · Layer-50│ │ length gain │ │ │ │ │ │
│ │ hidden │ │ │ │ │ │ │ │
│ │ states │ │ │ │ │ │ │ │
│ └─────┬─────┘ └──────┬────────┘ └──────┬───────┘ └─────┬──────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Unified Packed Token Sequence │ │
│ │ │ │
│ │ [Text Tokens | Visual Tokens | Audio Tokens | Modality Tags] │ │
│ │ │ │
│ │ · Each token carries an AdaLN modality tag │ │
│ │ · MM-RoPE (3D Multimodal Rotary Position Embeddings) │ │
│ └─────────────────────────┬───────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ 52-Layer Dense Transformer Blocks │ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ │ │
│ │ │ Multi-Head │ │ FFN / │ │ AdaLN Modulation │ │ │
│ │ │ Attention │ │ SwiGLU │ │ Branch (~13B) │ │ │
│ │ │ │ │ │ │ · Precomputable │ │ │
│ │ │ · Full │ │ · Activation│ │ · Cacheable at │ │ │
│ │ │ attention│ │ function │ │ inference time │ │ │
│ │ │ · Cross- │ │ │ │ │ │ │
│ │ │ modal │ │ │ │ │ │ │
│ │ │ interact │ │ │ │ │ │ │
│ │ └─────────────┘ └──────────────┘ └────────────────────┘ │ │
│ │ │ │
│ │ Effective inference params: 33.1B - 13B ≈ 20B │ │
│ └─────────────────────────┬───────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Video Output │ │ Audio Output │ │
│ │ Head │ │ Head │ │
│ │ │ │ │ │
│ │ · Video latent │ │ · Audio latent │ │
│ │ · Decoded via │ │ · Decoded via │ │
│ │ H3-VisualVAE │ │ H3-AudioVAE │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
└───────────────────────────────────────────────────────────────────────┘
3.2 Key Architectural Decisions
Why abandon the Hailuo-02 architecture?
MiniMax explicitly stated in their blog: despite the significant architectural advantages Hailuo-02 once provided, “it would introduce unnecessary complexity for a model built around task generalization.” H3 chose a simpler path — a Dense Single-Stream Transformer — letting architecture serve the task rather than forcing tasks to adapt to the architecture.
Heterogeneous Compute Separation for Understanding and Generation
After introducing multimodal context, the variance in sequence length increased 3×, and the compute workloads for understanding and generation became markedly more heterogeneous. H3 adopted a training architecture that separates these workloads:
- Fine-tuning hardware utilization for each workload independently
- Jointly balancing per-sample heterogeneous compute against cross-sample load balancing
- End-to-end training throughput improvement of nearly 30%
3.3 Parameter Composition and AdaLN Optimization
Of H3-Omni-Transformer’s 33.1B parameters, approximately 13B reside in the AdaLN (Adaptive Layer Normalization) modulation branch. This design enables a critical optimization:
# AdaLN Modulation Branch Inference Optimization
# ~13B of 33.1B total params belong to the AdaLN branch
class AdaLNOptimization:
"""
The AdaLN modulation branch can be precomputed and cached
during inference, avoiding recomputation at each forward pass.
"""
total_params = 33.1e9 # Total parameter count
adaln_params = 13e9 # AdaLN modulation branch params
effective_inference = total_params - adaln_params # ≈ 20B
def precompute_adaln_curves(self, timestep_embed):
"""
Precompute AdaLN curve tables stored in cache.
At inference time, direct table lookup skips 13B param computation.
"""
# Timestep embedding → AdaLN modulation parameters
gamma, beta = self.adaln_mlp(timestep_embed)
# Cache curve tables
self.curve_cache = {gamma: gamma, beta: beta}
return self.curve_cache
The practical effect is visible in model file sizes: the pruned INT8 checkpoint is approximately 40% smaller than standard INT8 (19.5 GB vs 31.7 GB).
3.4 MM-RoPE: 3D Multimodal Rotary Position Embeddings
H3 employs MM-RoPE (Multimodal Rotary Position Embeddings) to handle positional information across multimodal sequences. Unlike standard RoPE, MM-RoPE must accommodate:
- Text dimension: Linear sequence positions
- Visual dimension: Spatial (H, W) + temporal (T) three-dimensional positions
- Audio dimension: Temporal frequency positions
import torch
import math
class MMRoPE3D(torch.nn.Module):
"""
3D Multimodal Rotary Position Embeddings (MM-RoPE)
Assigns different positional encoding dimensions per modality:
- Text tokens: 1D position
- Visual tokens: 3D (T, H, W) position
- Audio tokens: 1D temporal position
"""
def __init__(self, dim: int, base: int = 10000):
super().__init__()
self.dim = dim
self.base = base
self.dim_per_modality = dim // 3
inv_freq = 1.0 / (base ** (
torch.arange(0, self.dim_per_modality, 2).float() / self.dim_per_modality
))
self.register_buffer("inv_freq", inv_freq)
def _compute_rope(self, positions: torch.Tensor) -> torch.Tensor:
"""Compute rotary position embeddings"""
freqs = torch.einsum("i,j->ij", positions.float(), self.inv_freq)
emb = torch.cat([freqs, freqs], dim=-1)
return torch.stack([emb.cos(), emb.sin()], dim=-2)
def forward(
self,
text_positions: torch.Tensor, # (N_text,)
visual_positions: torch.Tensor, # (N_vis, 3) -> (T, H, W)
audio_positions: torch.Tensor, # (N_audio,)
) -> torch.Tensor:
"""
Compute positional encodings separately for each modality and concatenate
"""
# Text: 1D positional encoding
text_rope = self._compute_rope(text_positions)
# Visual: 3D positional encoding (temporal + height + width)
vis_t = self._compute_rope(visual_positions[:, 0])
vis_h = self._compute_rope(visual_positions[:, 1])
vis_w = self._compute_rope(visual_positions[:, 2])
visual_rope = torch.cat([vis_t, vis_h, vis_w], dim=-1)
# Audio: 1D temporal positional encoding
audio_rope = self._compute_rope(audio_positions)
# Concatenate all modalities
return torch.cat([text_rope, visual_rope, audio_rope], dim=0)
IV. Omni-Modal Context Encoding: Contextual Omni Representation
4.1 Core Concept
Traditional video generation models caption only the target video itself. H3 redefines “caption” as Contextual Omni Representation:
- Not just describing the target video, but the relationships between context materials
- Describing relationships between context and target video
- Jointly describing video and audio, including audio-visual relationships across multiple shots
Language as the generalization bridge: Unifying “tasks” into an open, descriptive form is the root of H3’s broad instruction-following ability.
4.2 Multimodal Encoding Pipeline
┌──────────────────────────────────────────────────────────────┐
│ Multimodal Encoding Pipeline │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌──────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Text │ │ Image │ │ Video │ │ Audio │ │
│ │ │ │ │ │ │ │ │ │
│ │ Tokenize│ │ Resize │ │ Sample │ │ Waveform │ │
│ │ ↓ │ │ Patch │ │ Frames │ │ ↓ │ │
│ │ │ │ ↓ │ │ ↓ │ │ STFT │ │
│ │ H3- │ │ H3- │ │ H3-Visual │ │ Audio │ │
│ │ Encoder │ │ Encoder │ │ VAE │ │ VAE │ │
│ │(Qwen3- │ │(Qwen3-VL │ │ (32x spat │ │ (40Hz │ │
│ │ VL-32B) │ │ Layer50)│ │ 4x temp) │ │ temporal)│ │
│ │ ↓ │ │ ↓ │ │ ↓ │ │ ↓ │ │
│ │ Hidden │ │ Visual │ │ Compressed│ │ Audio │ │
│ │ States │ │ Tokens │ │ Tokens │ │ Latents │ │
│ └────┬────┘ └────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Unified Sequence Assembly (w/ AdaLN Modality Tags)│ │
│ │ │ │
│ │ [text_emb | vis_emb | vid_emb | aud_emb | tags] │ │
│ │ │ │
│ │ Each token tagged with modality origin for │ │
│ │ Transformer-differentiated processing │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘
4.3 Text Encoder: Qwen3-VL-32B
H3 uses Qwen3-VL-32B as its text encoder, extracting layer-50 hidden states as the conditioning signal:
class TextEncoderWrapper:
"""
Text encoder wrapper based on Qwen3-VL-32B.
Extracts layer-50 hidden states as the conditioning signal.
"""
def __init__(self, model_path: str = "MiniMaxAI/MiniMax-H3"):
from transformers import AutoModel, AutoProcessor
self.processor = AutoProcessor.from_pretrained(model_path)
self.encoder = AutoModel.from_pretrained(
model_path,
subfolder="text_encoder",
torch_dtype=torch.bfloat16
)
self.extract_layer = 50 # Layer-50 hidden states
def encode_text(self, text: str) -> torch.Tensor:
"""Encode text, returning layer-50 hidden states"""
inputs = self.processor(text=text, return_tensors="pt")
outputs = self.encoder(
**inputs,
output_hidden_states=True
)
hidden = outputs.hidden_states[self.extract_layer]
return hidden # shape: (batch, seq_len, hidden_dim)
V. H3-VAE: A Major Breakthrough in Tokenizer Technology
5.1 Compression Efficiency
H3-VAE is the core technology enabling native 2K output, achieving a comprehensive tokenizer overhaul:
| Component | Compression | Effect |
|---|---|---|
| H3-VisualVAE spatial | 32× sampling factor | 32×32 = 1024× spatial compression |
| H3-VisualVAE temporal | 4× sampling factor | 4× temporal compression |
| Combined sequence gain | 4× | Effective sequence length improved 4× |
| H3-AudioVAE | 40Hz temporal frequency | Stereo compressed to unified latent space |
class H3VisualVAE:
"""
H3-VisualVAE: Video Variational Autoencoder
32× spatial compression, 4× temporal compression
"""
def __init__(self, model_path: str):
self.spatial_compression = 32 # Spatial sampling factor
self.temporal_compression = 4 # Temporal sampling factor
def encode(self, video_tensor: torch.Tensor) -> torch.Tensor:
"""
Encode pixel-space video to latent-space representation
Input: (B, C, T, H, W) — pixel space
Output: (B, C', T//4, H//32, W//32) — latent space
"""
B, C, T, H, W = video_tensor.shape
latent_T = T // self.temporal_compression
latent_H = H // self.spatial_compression
latent_W = W // self.spatial_compression
compression_ratio = (
self.temporal_compression *
self.spatial_compression ** 2
)
print(f"Pixel → Latent compression: {compression_ratio}× (volumetric)")
print(f"Sequence length gain: 4× (temporal dimension)")
return self.encoder(video_tensor)
Key significance: The 4× sequence length compression directly reduces training and inference costs, making native 2K resolution generation feasible — without this compression ratio, the token sequence for 2K video would be prohibitively long.
VI. In-Context Regeneration: A Revolutionary 2K Approach
6.1 Why Not Traditional Super-Resolution?
Traditional super-resolution modules (Real-ESRGAN, SwinIR, etc.) face fundamental limitations:
- Can only “guess” missing high-frequency information from low-resolution pixels
- Cannot recover semantic details from original multimodal context (small text, brand logos)
- Information disconnect between generative model and post-processing module
6.2 In-Context Regeneration Approach
┌──────────────────────────────────────────────────────────────┐
│ In-Context Regeneration Pipeline │
├──────────────────────────────────────────────────────────────┤
│ │
│ Input Combination: │
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────┐ │
│ │ 768p Video │ │ Original │ │ Enhanced │ │
│ │ (base_video)│ │ Context │ │ Prompt │ │
│ │ │ │ (text/img/ │ │ (from Context- │ │
│ │ │ │ video/ │ │ IR) │ │
│ │ │ │ audio) │ │ │ │
│ └──────┬───────┘ └──────┬───────┘ └────────┬────────┘ │
│ │ │ │ │
│ └─────────────────┼────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ H3-Base (Running Again) │ │
│ │ │ │
│ │ 768p result as condition + original multimodal │ │
│ │ context as reference → regenerate at 2K resolution │ │
│ │ │ │
│ │ Advantage 1: Maximally reuses base model capability │ │
│ │ Advantage 2: Original context supplements semantics │ │
│ │ Advantage 3: Recovers small text/fine textures that │ │
│ │ traditional SR cannot restore │ │
│ └──────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ 2K Output │ │
│ │ 2560×1440 │ │
│ │ 24FPS │ │
│ │ 32kHz Stereo │ │
│ └──────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘
The elegance of this approach: 2K generation is not “upscaling” — it is “regenerating.” The model leverages the rich information in the original context (text descriptions, fine details from reference images, audio features) to construct 2K resolution video from scratch — not guessing pixels, but reconstructing based on understanding.
VII. Dual Checkpoints and Input Specifications
7.1 FL2VA vs Ref2VA
| Feature | H3-Base-FL2VA | H3-Base-Ref2VA |
|---|---|---|
| Purpose | First/last-frame mode | Omni-reference mode |
| Image input | 0-2 (first/last frame) | Up to 9 (reference images) |
| Video input | None | Up to 3 clips (2-15s each, total ≤ 15s) |
| Audio input | None | Up to 3 clips (must accompany image/video) |
| Max files | 2 images | 12 files total |
| Use cases | T2V, first-frame, last-frame, first+last-frame | Identity preservation, scene retention, motion transfer, lip-sync, voice timbre |
7.2 Output Specifications
| Parameter | Value |
|---|---|
| Output duration | 4-15 seconds |
| Output frame rate | 24 FPS |
| Output audio | 32 kHz native stereo |
| Default resolution | Short edge 768px |
| 2K resolution | Short edge 1440px (via Regenerate-2K) |
| Aspect ratios | 21:9, 16:9, 4:3, 1:1, 3:4, 9:16 |
| Languages | 11 stably supported (ar, zh, en, fr, de, it, ja, ko, pt, ru, es) |
VIII. Open-Source Ecosystem and Deployment
8.1 Model Repositories
- HuggingFace:
MiniMaxAI/MiniMax-H3 - License: MiniMax H3 Community License
- ComfyUI Native Support: Comfy-Org/ComfyUI #15224 merged
8.2 Quantization Options
| Format | Model Size | Notes |
|---|---|---|
| BF16 | 61.7 GB | Full precision inference |
| INT8 | 31.7 GB | Standard quantization |
| Pruned INT8 | 19.5 GB | Precomputed AdaLN curves, 40% smaller |
| NVFP4 AWQ (text encoder) | 14.6 GB | Runs on any GPU |
8.3 Multi-Framework Deployment
16 ecosystem partners completed day-one adaptation, including Huawei Ascend, Moore Threads, AMD, Intel, and frameworks like Hugging Face, ComfyUI, SGLang, vLLM.
IX. Code Practice: Python Complete Pipeline Examples
9.1 Text-to-Video Pipeline with diffusers
"""
MiniMax H3 Text-to-Video Pipeline (diffusers)
Generate 768p video with native stereo audio from text description
"""
import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video
def text_to_video_audio(
prompt: str,
output_path: str = "output.mp4",
num_frames: int = 124,
height: int = 768,
width: int = 1344,
num_inference_steps: int = 30,
seed: int = 42,
):
"""
Text-to-Video-Audio Pipeline
Args:
prompt: Text description
output_path: Output video path
num_frames: Frame count (snapped to 17n+5)
height: Height (short edge 768px)
width: Width
num_inference_steps: Denoising steps
seed: Random seed
"""
# 1. Load model
pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")
# 2. Run inference
state = pipe(
prompt=prompt,
num_frames=num_frames, # snapped to 17n+5 at 24fps
height=height,
width=width,
num_inference_steps=num_inference_steps,
generator=torch.Generator("cpu").manual_seed(seed),
)
# 3. Encode output (video + audio muxed into one MP4)
encode_video(
state.get("videos")[0],
fps=24,
audio=state.get("audio")[0],
audio_sample_rate=state.get("sampling_rate"),
output_path=output_path,
)
print(f"✅ Video saved to: {output_path}")
print(f" Resolution: {width}x{height}")
print(f" Frames: {num_frames} ({num_frames/24:.1f}s)")
print(f" Audio: 32kHz stereo")
return output_path
if __name__ == "__main__":
text_to_video_audio(
prompt="A red fox trotting through a snowy pine forest, snow crunching underfoot",
output_path="fox.mp4",
num_frames=124,
height=768,
width=1344,
)
9.2 Omni-Reference Pipeline (Ref2VA)
"""
MiniMax H3 Omni-Reference Pipeline
Joint generation using reference images, video, and audio
"""
import torch
from diffusers.modular_pipelines import MiniMaxH3Ref2VABlocks
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Reference
from diffusers.utils import encode_video
def ref_to_video_audio(
prompt: str,
references: list,
output_path: str = "ref_output.mp4",
num_frames: int = 124,
num_inference_steps: int = 30,
):
"""
Omni-Reference Pipeline
Supports up to 9 images + 3 videos + 3 audios, max 12 files total
"""
pipe = MiniMaxH3Ref2VABlocks().init_pipeline("MiniMaxAI/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")
state = pipe(
prompt=prompt,
references=references,
num_frames=num_frames,
num_inference_steps=num_inference_steps,
)
encode_video(
state.get("videos")[0],
fps=24,
audio=state.get("audio")[0],
audio_sample_rate=state.get("sampling_rate"),
output_path=output_path,
)
print(f"✅ Reference-generated video saved: {output_path}")
if __name__ == "__main__":
ref_to_video_audio(
prompt="The subject walks toward camera, matching the reference video's shot rhythm",
references=[
MiniMaxH3Reference(video="motion_ref.mp4"),
MiniMaxH3Reference(image="subject.jpg"),
MiniMaxH3Reference(audio="voice.wav"),
],
num_frames=124,
)
9.3 SGLang Multi-GPU Deployment
#!/bin/bash
# MiniMax H3 SGLang Deployment (4 GPU parallel)
# FL2VA variant
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-H3 \
--num-gpus 4 \
--ulysses-degree 4 \
--performance-mode speed \
--host 0.0.0.0 \
--port 30011 \
--model-variant fl2va
# Ref2VA variant
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-H3 \
--num-gpus 4 \
--ulysses-degree 4 \
--performance-mode speed \
--host 0.0.0.0 \
--port 30011 \
--model-variant ref2va
X. Code Practice: Go Implementation of Full 2K Workflow
10.1 Complete 2K Workflow Client
package main
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
// MiniMaxH3Pipeline Full 2K workflow pipeline
type MiniMaxH3Pipeline struct {
SGLangEndpoint string
APIBaseURL string
APIKey string
}
// ContextIRRequest H3-Context-IR request
type ContextIRRequest struct {
Text string `json:"text"`
Images []string `json:"images,omitempty"`
Videos []string `json:"videos,omitempty"`
Audios []string `json:"audios,omitempty"`
}
// GenerationRequest H3-Base generation request
type GenerationRequest struct {
Prompt string `json:"prompt"`
NumFrames int `json:"num_frames"`
Height int `json:"height"`
Width int `json:"width"`
NumSteps int `json:"num_inference_steps"`
Seed int `json:"seed"`
}
// RegenerateRequest H3-Regenerate-2K request
type RegenerateRequest struct {
BaseVideo string `json:"base_video"`
Prompt string `json:"prompt"`
Resolution string `json:"resolution"`
Images []string `json:"images,omitempty"`
Videos []string `json:"videos,omitempty"`
Audios []string `json:"audios,omitempty"`
}
// NewPipeline creates a new H3 pipeline
func NewPipeline(sglangEndpoint, apiBaseURL, apiKey string) *MiniMaxH3Pipeline {
return &MiniMaxH3Pipeline{
SGLangEndpoint: sglangEndpoint,
APIBaseURL: apiBaseURL,
APIKey: apiKey,
}
}
// Step1: Context-IR preprocessing
func (p *MiniMaxH3Pipeline) ContextIR(req ContextIRRequest) (string, error) {
fmt.Println("📋 Step 1/3: H3-Context-IR multimodal understanding...")
body, _ := json.Marshal(req)
httpReq, _ := http.NewRequest("POST",
p.APIBaseURL+"/v1/h3-context-ir", bytes.NewReader(body))
httpReq.Header.Set("Authorization", "Bearer "+p.APIKey)
httpReq.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 120 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return "", fmt.Errorf("Context-IR request failed: %w", err)
}
defer resp.Body.Close()
var result struct {
EnhancedPrompt string `json:"enhanced_prompt"`
}
json.NewDecoder(resp.Body).Decode(&result)
fmt.Printf(" ✅ Context-IR complete, prompt enhanced\n")
fmt.Printf(" 📝 Enhanced prompt length: %d chars\n", len(result.EnhancedPrompt))
return result.EnhancedPrompt, nil
}
// Step2: H3-Base local 768p generation
func (p *MiniMaxH3Pipeline) Generate768p(prompt string) ([]byte, error) {
fmt.Println("🎬 Step 2/3: H3-Base local 768p audio-video generation...")
req := GenerationRequest{
Prompt: prompt,
NumFrames: 124,
Height: 768,
Width: 1344,
NumSteps: 30,
Seed: 42,
}
body, _ := json.Marshal(req)
httpReq, _ := http.NewRequest("POST",
p.SGLangEndpoint+"/v1/videos", bytes.NewReader(body))
httpReq.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 600 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return nil, fmt.Errorf("H3-Base generation failed: %w", err)
}
defer resp.Body.Close()
videoData, _ := io.ReadAll(resp.Body)
fmt.Printf(" ✅ 768p video generated (%d bytes)\n", len(videoData))
return videoData, nil
}
// Step3: Regenerate-2K
func (p *MiniMaxH3Pipeline) Regenerate2K(
baseVideo []byte, prompt string, refs ContextIRRequest,
) error {
fmt.Println("✨ Step 3/3: H3-Regenerate-2K regeneration to 2K...")
b64 := base64.StdEncoding.EncodeToString(baseVideo)
dataURL := "data:video/mp4;base64," + b64
req := RegenerateRequest{
BaseVideo: dataURL,
Prompt: prompt,
Resolution: "2k",
Images: refs.Images,
Videos: refs.Videos,
Audios: refs.Audios,
}
body, _ := json.Marshal(req)
httpReq, _ := http.NewRequest("POST",
p.APIBaseURL+"/v1/h3-regenerate-2k", bytes.NewReader(body))
httpReq.Header.Set("Authorization", "Bearer "+p.APIKey)
httpReq.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 900 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return fmt.Errorf("Regenerate-2K failed: %w", err)
}
defer resp.Body.Close()
outFile, _ := os.Create("output_2k.mp4")
defer outFile.Close()
io.Copy(outFile, resp.Body)
fmt.Println(" ✅ 2K video saved: output_2k.mp4")
return nil
}
// RunFullPipeline executes the complete 2K workflow
func (p *MiniMaxH3Pipeline) RunFullPipeline(userInput ContextIRRequest) error {
fmt.Println("🚀 MiniMax H3 Full 2K Workflow initiated")
fmt.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
enhancedPrompt, err := p.ContextIR(userInput)
if err != nil {
return err
}
video768p, err := p.Generate768p(enhancedPrompt)
if err != nil {
return err
}
err = p.Regenerate2K(video768p, enhancedPrompt, userInput)
if err != nil {
return err
}
fmt.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
fmt.Println("🎉 Full 2K workflow completed!")
return nil
}
func main() {
pipeline := NewPipeline(
"http://localhost:30011",
"https://api.minimaxi.com",
os.Getenv("MINIMAX_API_KEY"),
)
userInput := ContextIRRequest{
Text: "A red fox running through a pine forest, snow crunching underfoot",
}
if err := pipeline.RunFullPipeline(userInput); err != nil {
fmt.Fprintf(os.Stderr, "❌ Workflow failed: %v\n", err)
os.Exit(1)
}
}
10.2 Go Multimodal Preprocessor
package main
import (
"encoding/json"
"fmt"
"strings"
)
// MultiModalPreprocessor simulates core H3-Context-IR functionality
type MultiModalPreprocessor struct {
maxImages int
maxVideos int
maxAudios int
maxFiles int
}
// MediaAsset represents a media input asset
type MediaAsset struct {
Type string `json:"type"`
Path string `json:"path"`
Duration float64 `json:"duration"`
}
// StructuredPrompt represents the serialized intermediate representation
type StructuredPrompt struct {
OriginalText string `json:"original_text"`
EnhancedText string `json:"enhanced_text"`
ContextAssets []MediaAsset `json:"context_assets"`
TaskType string `json:"task_type"`
TokenBudget int `json:"token_budget"`
}
// NewPreprocessor creates a new preprocessor with H3 constraints
func NewPreprocessor() *MultiModalPreprocessor {
return &MultiModalPreprocessor{
maxImages: 9,
maxVideos: 3,
maxAudios: 3,
maxFiles: 12,
}
}
// ValidateInputs validates multimodal inputs against H3 constraints
func (p *MultiModalPreprocessor) ValidateInputs(
text string, images []MediaAsset, videos []MediaAsset, audios []MediaAsset,
) error {
if len(images) > p.maxImages {
return fmt.Errorf("image count %d exceeds max %d", len(images), p.maxImages)
}
if len(videos) > p.maxVideos {
return fmt.Errorf("video count %d exceeds max %d", len(videos), p.maxVideos)
}
if len(audios) > p.maxAudios {
return fmt.Errorf("audio count %d exceeds max %d", len(audios), p.maxAudios)
}
totalFiles := len(images) + len(videos) + len(audios)
if totalFiles > p.maxFiles {
return fmt.Errorf("total files %d exceeds max %d", totalFiles, p.maxFiles)
}
if len(audios) > 0 && len(images) == 0 && len(videos) == 0 {
return fmt.Errorf("audio must accompany image or video input")
}
for i, v := range videos {
if v.Duration < 2 || v.Duration > 15 {
return fmt.Errorf("video %d duration %.1f outside [2,15] range", i, v.Duration)
}
}
totalDuration := 0.0
for _, v := range videos {
totalDuration += v.Duration
}
if totalDuration > 15 {
return fmt.Errorf("total video duration %.1fs exceeds 15s limit", totalDuration)
}
return nil
}
// DetermineTaskType determines the generation task type
func (p *MultiModalPreprocessor) DetermineTaskType(
images []MediaAsset, videos []MediaAsset, audios []MediaAsset,
) string {
if len(videos) > 0 || len(audios) > 0 || len(images) > 2 {
return "ref2va"
}
return "fl2va"
}
// Process handles multimodal input, outputting structured representation
func (p *MultiModalPreprocessor) Process(
text string, images []MediaAsset, videos []MediaAsset, audios []MediaAsset,
) (*StructuredPrompt, error) {
if err := p.ValidateInputs(text, images, videos, audios); err != nil {
return nil, fmt.Errorf("validation failed: %w", err)
}
taskType := p.DetermineTaskType(images, videos, audios)
var assets []MediaAsset
assets = append(assets, images...)
assets = append(assets, videos...)
assets = append(assets, audios...)
enhanced := p.buildEnhancedPrompt(text, assets, taskType)
return &StructuredPrompt{
OriginalText: text,
EnhancedText: enhanced,
ContextAssets: assets,
TaskType: taskType,
TokenBudget: 4096,
}, nil
}
func (p *MultiModalPreprocessor) buildEnhancedPrompt(
text string, assets []MediaAsset, taskType string,
) string {
var parts []string
parts = append(parts, fmt.Sprintf("[Task: %s]", taskType))
parts = append(parts, text)
for i, a := range assets {
switch a.Type {
case "image":
parts = append(parts, fmt.Sprintf("[Ref image %d: %s]", i+1, a.Path))
case "video":
parts = append(parts, fmt.Sprintf("[Ref video %d: %s, %.1fs]", i+1, a.Path, a.Duration))
case "audio":
parts = append(parts, fmt.Sprintf("[Ref audio %d: %s, %.1fs]", i+1, a.Path, a.Duration))
}
}
return strings.Join(parts, "\n")
}
func main() {
proc := NewPreprocessor()
images := []MediaAsset{
{Type: "image", Path: "character.jpg"},
{Type: "image", Path: "scene_bg.jpg"},
}
videos := []MediaAsset{
{Type: "video", Path: "motion_ref.mp4", Duration: 5.0},
}
audios := []MediaAsset{
{Type: "audio", Path: "voice_ref.wav", Duration: 3.0},
}
result, err := proc.Process(
"Subject walks toward camera, motion ref video 1, voice timbre ref audio 1",
images, videos, audios,
)
if err != nil {
fmt.Printf("❌ Processing failed: %v\n", err)
return
}
output, _ := json.MarshalIndent(result, "", " ")
fmt.Println("✅ Structured Prompt:")
fmt.Println(string(output))
}
XI. Python Deployment Utility: Multi-GPU Inference Service
"""
MiniMax H3 Multi-GPU Inference Service Wrapper
Supports FL2VA and Ref2VA variants
"""
import subprocess
import time
import requests
import json
import base64
from typing import Optional, List, Dict, Any
from dataclasses import dataclass
@dataclass
class H3Config:
"""H3 deployment configuration"""
model_path: str = "MiniMaxAI/MiniMax-H3"
num_gpus: int = 4
ulysses_degree: int = 4
host: str = "0.0.0.0"
port: int = 30011
model_variant: str = "fl2va"
performance_mode: str = "speed"
class H3Server:
"""H3 inference service manager"""
def __init__(self, config: H3Config):
self.config = config
self.process = None
self.base_url = f"http://{config.host}:{config.port}"
def start(self):
"""Start SGLang inference service"""
cmd = [
"python", "-m", "sglang.launch_server",
"--model-path", self.config.model_path,
"--num-gpus", str(self.config.num_gpus),
"--ulysses-degree", str(self.config.ulysses_degree),
"--performance-mode", self.config.performance_mode,
"--host", self.config.host,
"--port", str(self.config.port),
"--model-variant", self.config.model_variant,
]
print(f"🚀 Starting H3 service ({self.config.model_variant})...")
self.process = subprocess.Popen(cmd)
self._wait_for_ready()
def _wait_for_ready(self, timeout: int = 300):
"""Wait for service readiness"""
start = time.time()
while time.time() - start < timeout:
try:
resp = requests.get(f"{self.base_url}/health", timeout=5)
if resp.status_code == 200:
print("✅ Service ready")
return
except requests.ConnectionError:
time.sleep(2)
raise TimeoutError("Service startup timed out")
def stop(self):
"""Stop the service"""
if self.process:
self.process.terminate()
self.process.wait()
print("🛑 Service stopped")
def generate(
self,
prompt: str,
num_frames: int = 124,
height: int = 768,
width: int = 1344,
num_steps: int = 30,
seed: int = 42,
) -> bytes:
"""Call the generation API"""
payload = {
"prompt": prompt,
"num_frames": num_frames,
"height": height,
"width": width,
"num_inference_steps": num_steps,
"seed": seed,
}
resp = requests.post(
f"{self.base_url}/v1/videos",
json=payload,
timeout=600,
)
resp.raise_for_status()
return resp.content
class Full2KWorkflow:
"""Complete 2K workflow orchestrator"""
def __init__(self, server: H3Server, api_base: str, api_key: str):
self.server = server
self.api_base = api_base
self.api_key = api_key
def context_ir(self, text: str, **kwargs) -> str:
"""Call H3-Context-IR API"""
payload = {"text": text, **kwargs}
resp = requests.post(
f"{self.api_base}/v1/h3-context-ir",
json=payload,
headers={"Authorization": f"Bearer {self.api_key}"},
timeout=120,
)
resp.raise_for_status()
return resp.json()["enhanced_prompt"]
def generate_768p(self, prompt: str) -> bytes:
"""Generate 768p via local H3-Base"""
return self.server.generate(prompt=prompt)
def regenerate_2k(self, video_data: bytes, prompt: str, **refs) -> bytes:
"""Call H3-Regenerate-2K API"""
b64 = base64.b64encode(video_data).decode()
payload = {
"base_video": f"data:video/mp4;base64,{b64}",
"prompt": prompt,
"resolution": "2k",
**refs,
}
resp = requests.post(
f"{self.api_base}/v1/h3-regenerate-2k",
json=payload,
headers={"Authorization": f"Bearer {self.api_key}"},
timeout=900,
)
resp.raise_for_status()
return resp.content
def run(self, text: str, **refs) -> bytes:
"""Execute the full 2K workflow"""
# Step 1: Context-IR
enhanced = self.context_ir(text, **refs)
print(f"📋 Context-IR enhancement complete")
# Step 2: 768p generation
video_768p = self.generate_768p(enhanced)
print(f"🎬 768p generation complete ({len(video_768p)} bytes)")
# Step 3: 2K regeneration
video_2k = self.regenerate_2k(video_768p, enhanced, **refs)
print(f"✨ 2K regeneration complete ({len(video_2k)} bytes)")
return video_2k
if __name__ == "__main__":
config = H3Config(model_variant="fl2va", num_gpus=4)
server = H3Server(config)
server.start()
workflow = Full2KWorkflow(
server=server,
api_base="https://api.minimaxi.com",
api_key="your-api-key",
)
video_2k = workflow.run(text="A red fox running through a pine forest")
with open("output_2k.mp4", "wb") as f:
f.write(video_2k)
print("✅ 2K video saved")
XII. Performance Benchmarks and Competitive Landscape
12.1 Artificial Analysis Leaderboard Results
| Capability | H3 Ranking | Notes |
|---|---|---|
| Audio-video editing | 🥇 Global #1 | Elo 1130 |
| Text-to-video | 🥈 Global #2 | — |
| Image-to-video | 🥉 Global #3 | — |
| Overall | Top tier globally | On par with Seedance 2.5 |
12.2 Key Strengths
- Complex Text Rendering: Text truly participates in visual composition — conforming to perspective, accepting scene lighting, following camera movement and subject motion
- Director-Level Instruction Following: Precisely executes complex lighting changes (candlelight → hard light → rainbow refraction → red-blue split → golden backlight)
- Generalized Editing: Dub video characters while simultaneously matching voice timbre and lip movement
- Cost Advantage: Per-second price at 2K resolution is less than 1/3 of mainstream models; at 768p, less than half
XIII. Technical Summary and Outlook
13.1 H3’s Three Technical Pillars
┌────────────────────────────────────────────────────────────────┐
│ MiniMax H3 Technical System Summary │
├────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Pillar 1: Contextual Omni Representation │ │
│ │ · Language as generalization bridge, unifying tasks │ │
│ │ into open descriptive form │ │
│ │ · 100K tokens → 4K tokens, 25:1 compression │ │
│ │ · Cross-modal relationship modeling │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Pillar 2: H3-Omni-Transformer + H3-VAE │ │
│ │ · 33.1B Dense Single-Stream Transformer │ │
│ │ · 13B AdaLN branch precomputable & cacheable │ │
│ │ · MM-RoPE 3D multimodal positional encoding │ │
│ │ · H3-VAE 4× sequence compression → native 2K support │ │
│ │ · 30% end-to-end training throughput improvement │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Pillar 3: In-Context Regeneration │ │
│ │ · Not traditional SR — base model regenerates │ │
│ │ · Reuses base model capability + original context │ │
│ │ · Recovers small text/fine textures that traditional │ │
│ │ super-resolution cannot restore │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────┘
13.2 Significance of Open-Sourcing
MiniMax H3’s open-sourcing is regarded as the “DeepSeek moment” for video generation:
- Breaking closed-source monopoly: Previously, all flagship video models were closed-source; H3 is the first to open commercially viable weights
- Hardware ecosystem: 16 chip manufacturers completed day-one adaptation, accelerating domestic chip support
- Community customization: Developers can fine-tune for their business needs and build domain-specific versions
13.3 Future Directions
According to official announcements, the next-generation H series will:
- Integrate capabilities from M-series models for enhanced multimodal understanding
- Explore scaling to larger model sizes
- Continue pushing resolution and visual fidelity higher
References
- Official technical blog: https://www.minimax.io/blog/minimax-h3
- HuggingFace model repository: https://huggingface.co/MiniMaxAI/MiniMax-H3
- vLLM-Omni deployment docs: https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3
- ComfyUI integration: https://comfyui-wiki.com/en/news/2026-08-03-minimax-h3-open-weights-comfyui
- diffusers integration PR: https://github.com/huggingface/diffusers/pull/14355
- MiniMax Platform: https://platform.minimax.io