MiniMax H3 Omni-Modal Deep Dive: H3-Omni-Transformer Three-Module Architecture and 2K Video Regeneration

MiniMax H3 Omni-Modal Deep Dive: H3-Omni-Transformer Three-Module Architecture and 2K Video Regeneration

On August 3, 2026, MiniMax officially open-sourced MiniMax H3, a next-generation general-purpose omni-modal generation model. It is the first commercially viable flagship video model with open weights, ranking #1 globally in video editing on the Artificial Analysis leaderboard. This article dissects the H3 technical system across architecture design, core technologies, information flow, and code implementation.


I. From “Expert Models” to “General-Purpose System”: H3’s Design Philosophy

1.1 Industry Pain Points: Task Islands and Modality Barriers

Before H3, the AI video generation landscape was dominated by “task silos”:

  • Image generation was fragmented into separate expert models for T2I, editing, subject reference, motion reference, and style reference
  • Audio generation treated voice, sound effects, and music as largely independent domains
  • Video generation was further splintered into text-to-video, image-to-video, first-and-last-frame, subject reference, motion reference, voice reference, video editing — with clear boundaries between image, video, and audio

These barriers constrained creative freedom in practice and capped the model’s ability to generalize on the training side. H3’s first design principle is cross-task, cross-modality unification and generalization.

1.2 Core Design Decisions

H3’s pretraining paradigm is built around the following pillars:

DimensionDesign Choice
Data & TasksText-to-image, text-to-video (with jointly generated audio & native multi-shot modeling), text-to-audio (voice/SFX/music jointly modeled), generalized reference & editing
Reference & Editing DefinitionBuilt entirely from real, natural data; relationships expressed through natural language rather than a fixed task set
ArchitectureFuse diverse data types and tasks as early as possible — the right mixing ratio is key
Training StrategyFuse diverse data types and tasks at the earliest training stages

In one sentence: Enable H3 to possess broad multimodal context understanding and generation capabilities from the pretraining stage onward.


II. Three-Module System Architecture: End-to-End Information Flow

2.1 System Overview

┌─────────────────────────────────────────────────────────────────────┐
│                   MiniMax H3 Complete System Architecture            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────┐    ┌──────────────────┐    ┌─────────────────┐   │
│  │ H3-Context-IR│    │    H3-Base       │    │H3-Regenerate-2K│   │
│  │ (Closed/API) │───▶│  (Open 33.1B)    │───▶│ (Closed/API)   │   │
│  │              │    │                  │    │                 │   │
│  │ · Instruction│    │ · H3-Encoder     │    │ · In-context    │   │
│  │   parsing    │    │   (Qwen3-VL-32B) │    │   regeneration  │   │
│  │ · Cross-modal│    │ · H3-VisualVAE   │    │ · Original      │   │
│  │   association│    │ · H3-AudioVAE    │    │   context for   │   │
│  │ · Temporal   │    │ · Omni-Transform │    │   detail recovery│  │
│  │   understanding  │ · 768p output    │    │ · 2K resolution │   │
│  │ · Logic      │    │                  │    │   output        │   │
│  │   reasoning  │    │                  │    │                 │   │
│  │ · Serialization│  │                  │    │                 │   │
│  └──────────────┘    └──────────────────┘    └─────────────────┘   │
│         │                     │                       │             │
│   User multimodal       768p Audio-Video         2K Audio-Video    │
│   input (text/img/      (short edge 768px)       (up to 2560×1440) │
│    video/audio)                                                   │
│                                                                     │
│  100K tokens ──compress──▶ ~4K tokens ──generate──▶ 768p ──regen──▶ 2K │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

2.2 Three-Module Collaboration Pipeline

Phase 1: H3-Context-IR (Context Intermediate Representation)

H3-Context-IR is a managed preprocessing and orchestration system responsible for:

  1. Instruction Parsing: Understanding the creative intent in user text input
  2. Cross-Modal Association: Parsing relationships among text, images, audio, and reference videos
  3. Temporal Understanding: Processing timeline information from video materials
  4. Complex Logic Reasoning: Inferring implicit semantic relationships between materials
  5. Serialization & Compression: Converting understanding into structured representations that H3-Base can directly consume

Most source materials require ~100K tokens of inference, distilled down to an average of ~4K tokens. This 25:1 compression ratio is a key driver of system efficiency.

⚠️ Note: H3-Context-IR relies on multi-stage workflows and multiple hosted models/services. It is not included in this open-source release. An API and prompt-writing guide are provided.

Phase 2: H3-Base (Core Generation Model)

H3-Base is the open-sourced 33.1B-parameter core generation model:

  • Receives structured representations from H3-Context-IR
  • Encodes different modality information into a unified sequence via multi-encoders
  • Jointly generates video and stereo audio through H3-Omni-Transformer
  • Outputs audio-video results at short edge 768 pixels

Phase 3: H3-Regenerate-2K (2K Regeneration)

The innovation of H3-Regenerate-2K lies in not using a conventional super-resolution module. Instead:

  1. The 768p video from H3-Base serves as base_video
  2. Combined with the enhanced prompt and original multimodal context
  3. The base model regenerates a 2K version in an in-context manner
  4. Recovers small text, fine textures, and other details that low resolution cannot restore

⚠️ This module is also not yet open-sourced and is available via API.


III. H3-Omni-Transformer Architecture Deep Dive

3.1 Architecture Overview

H3-Omni-Transformer is the core component of H3-Base — a 33.1B-parameter Dense Single-Stream Transformer designed specifically for task generalization.

┌───────────────────────────────────────────────────────────────────────┐
│              H3-Omni-Transformer Internal Architecture                │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│  Input Side:                                                          │
│  ┌──────────┐  ┌───────────────┐  ┌──────────────┐  ┌────────────┐  │
│  │ H3-Encoder│  │H3-VisualVAE   │  │H3-AudioVAE   │  │  Text      │  │
│  │(Qwen3-VL  │  │(32x spatial   │  │(40Hz temporal│  │  Token     │  │
│  │ -32B)     │  │ compression   │  │ frequency    │  │            │  │
│  │           │  │ 4x temporal)  │  │ stereo enc.) │  │            │  │
│  │ · Text    │  │ · Image/video │  │ · Audio      │  │            │  │
│  │   encoding│  │   spatial     │  │   waveform   │  │            │  │
│  │ · Visual  │  │   compression │  │   → latent   │  │            │  │
│  │   encoding│  │ · 4x sequence │  │              │  │            │  │
│  │ · Layer-50│  │   length gain │  │              │  │            │  │
│  │  hidden   │  │               │  │              │  │            │  │
│  │  states   │  │               │  │              │  │            │  │
│  └─────┬─────┘  └──────┬────────┘  └──────┬───────┘  └─────┬──────┘  │
│        │               │                   │                 │         │
│        ▼               ▼                   ▼                 ▼         │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │          Unified Packed Token Sequence                          │  │
│  │                                                                 │  │
│  │  [Text Tokens | Visual Tokens | Audio Tokens | Modality Tags]  │  │
│  │                                                                 │  │
│  │  · Each token carries an AdaLN modality tag                     │  │
│  │  · MM-RoPE (3D Multimodal Rotary Position Embeddings)           │  │
│  └─────────────────────────┬───────────────────────────────────────┘  │
│                            │                                          │
│                            ▼                                          │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │            52-Layer Dense Transformer Blocks                    │  │
│  │                                                                 │  │
│  │  ┌─────────────┐  ┌──────────────┐  ┌────────────────────┐    │  │
│  │  │  Multi-Head │  │   FFN /      │  │  AdaLN Modulation  │    │  │
│  │  │  Attention  │  │   SwiGLU     │  │  Branch (~13B)     │    │  │
│  │  │             │  │              │  │  · Precomputable   │    │  │
│  │  │  · Full     │  │  · Activation│  │  · Cacheable at    │    │  │
│  │  │    attention│  │   function   │  │    inference time  │    │  │
│  │  │  · Cross-   │  │              │  │                    │    │  │
│  │  │    modal    │  │              │  │                    │    │  │
│  │  │    interact │  │              │  │                    │    │  │
│  │  └─────────────┘  └──────────────┘  └────────────────────┘    │  │
│  │                                                                 │  │
│  │  Effective inference params: 33.1B - 13B ≈ 20B                  │  │
│  └─────────────────────────┬───────────────────────────────────────┘  │
│                            │                                          │
│                            ▼                                          │
│  ┌──────────────────┐          ┌──────────────────┐                   │
│  │  Video Output    │          │  Audio Output     │                   │
│  │  Head            │          │  Head             │                   │
│  │                  │          │                   │                   │
│  │  · Video latent  │          │  · Audio latent   │                   │
│  │  · Decoded via   │          │  · Decoded via    │                   │
│  │    H3-VisualVAE  │          │    H3-AudioVAE    │                   │
│  └──────────────────┘          └──────────────────┘                   │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

3.2 Key Architectural Decisions

Why abandon the Hailuo-02 architecture?

MiniMax explicitly stated in their blog: despite the significant architectural advantages Hailuo-02 once provided, “it would introduce unnecessary complexity for a model built around task generalization.” H3 chose a simpler path — a Dense Single-Stream Transformer — letting architecture serve the task rather than forcing tasks to adapt to the architecture.

Heterogeneous Compute Separation for Understanding and Generation

After introducing multimodal context, the variance in sequence length increased , and the compute workloads for understanding and generation became markedly more heterogeneous. H3 adopted a training architecture that separates these workloads:

  • Fine-tuning hardware utilization for each workload independently
  • Jointly balancing per-sample heterogeneous compute against cross-sample load balancing
  • End-to-end training throughput improvement of nearly 30%

3.3 Parameter Composition and AdaLN Optimization

Of H3-Omni-Transformer’s 33.1B parameters, approximately 13B reside in the AdaLN (Adaptive Layer Normalization) modulation branch. This design enables a critical optimization:

# AdaLN Modulation Branch Inference Optimization
# ~13B of 33.1B total params belong to the AdaLN branch

class AdaLNOptimization:
    """
    The AdaLN modulation branch can be precomputed and cached
    during inference, avoiding recomputation at each forward pass.
    """
    total_params = 33.1e9       # Total parameter count
    adaln_params = 13e9         # AdaLN modulation branch params
    effective_inference = total_params - adaln_params  # ≈ 20B
    
    def precompute_adaln_curves(self, timestep_embed):
        """
        Precompute AdaLN curve tables stored in cache.
        At inference time, direct table lookup skips 13B param computation.
        """
        # Timestep embedding → AdaLN modulation parameters
        gamma, beta = self.adaln_mlp(timestep_embed)
        # Cache curve tables
        self.curve_cache = {gamma: gamma, beta: beta}
        return self.curve_cache

The practical effect is visible in model file sizes: the pruned INT8 checkpoint is approximately 40% smaller than standard INT8 (19.5 GB vs 31.7 GB).

3.4 MM-RoPE: 3D Multimodal Rotary Position Embeddings

H3 employs MM-RoPE (Multimodal Rotary Position Embeddings) to handle positional information across multimodal sequences. Unlike standard RoPE, MM-RoPE must accommodate:

  • Text dimension: Linear sequence positions
  • Visual dimension: Spatial (H, W) + temporal (T) three-dimensional positions
  • Audio dimension: Temporal frequency positions
import torch
import math

class MMRoPE3D(torch.nn.Module):
    """
    3D Multimodal Rotary Position Embeddings (MM-RoPE)
    Assigns different positional encoding dimensions per modality:
    - Text tokens: 1D position
    - Visual tokens: 3D (T, H, W) position
    - Audio tokens: 1D temporal position
    """
    
    def __init__(self, dim: int, base: int = 10000):
        super().__init__()
        self.dim = dim
        self.base = base
        self.dim_per_modality = dim // 3
        
        inv_freq = 1.0 / (base ** (
            torch.arange(0, self.dim_per_modality, 2).float() / self.dim_per_modality
        ))
        self.register_buffer("inv_freq", inv_freq)
    
    def _compute_rope(self, positions: torch.Tensor) -> torch.Tensor:
        """Compute rotary position embeddings"""
        freqs = torch.einsum("i,j->ij", positions.float(), self.inv_freq)
        emb = torch.cat([freqs, freqs], dim=-1)
        return torch.stack([emb.cos(), emb.sin()], dim=-2)
    
    def forward(
        self,
        text_positions: torch.Tensor,      # (N_text,)
        visual_positions: torch.Tensor,     # (N_vis, 3) -> (T, H, W)
        audio_positions: torch.Tensor,      # (N_audio,)
    ) -> torch.Tensor:
        """
        Compute positional encodings separately for each modality and concatenate
        """
        # Text: 1D positional encoding
        text_rope = self._compute_rope(text_positions)
        
        # Visual: 3D positional encoding (temporal + height + width)
        vis_t = self._compute_rope(visual_positions[:, 0])
        vis_h = self._compute_rope(visual_positions[:, 1])
        vis_w = self._compute_rope(visual_positions[:, 2])
        visual_rope = torch.cat([vis_t, vis_h, vis_w], dim=-1)
        
        # Audio: 1D temporal positional encoding
        audio_rope = self._compute_rope(audio_positions)
        
        # Concatenate all modalities
        return torch.cat([text_rope, visual_rope, audio_rope], dim=0)

IV. Omni-Modal Context Encoding: Contextual Omni Representation

4.1 Core Concept

Traditional video generation models caption only the target video itself. H3 redefines “caption” as Contextual Omni Representation:

  • Not just describing the target video, but the relationships between context materials
  • Describing relationships between context and target video
  • Jointly describing video and audio, including audio-visual relationships across multiple shots

Language as the generalization bridge: Unifying “tasks” into an open, descriptive form is the root of H3’s broad instruction-following ability.

4.2 Multimodal Encoding Pipeline

┌──────────────────────────────────────────────────────────────┐
│              Multimodal Encoding Pipeline                     │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────┐  ┌──────────┐  ┌───────────┐  ┌───────────┐  │
│  │  Text   │  │  Image   │  │   Video   │  │   Audio   │  │
│  │         │  │          │  │           │  │           │  │
│  │ Tokenize│  │ Resize   │  │ Sample    │  │ Waveform  │  │
│  │    ↓    │  │  Patch   │  │  Frames   │  │     ↓     │  │
│  │         │  │     ↓    │  │     ↓     │  │   STFT    │  │
│  │ H3-     │  │ H3-      │  │ H3-Visual │  │   Audio   │  │
│  │ Encoder │  │ Encoder  │  │   VAE     │  │    VAE    │  │
│  │(Qwen3-  │  │(Qwen3-VL │  │ (32x spat │  │ (40Hz    │  │
│  │ VL-32B) │  │  Layer50)│  │  4x temp) │  │  temporal)│  │
│  │     ↓   │  │     ↓    │  │     ↓     │  │     ↓     │  │
│  │ Hidden  │  │ Visual   │  │ Compressed│  │  Audio    │  │
│  │ States  │  │ Tokens   │  │  Tokens   │  │  Latents  │  │
│  └────┬────┘  └────┬─────┘  └─────┬─────┘  └─────┬─────┘  │
│       │            │              │               │         │
│       ▼            ▼              ▼               ▼         │
│  ┌──────────────────────────────────────────────────────┐   │
│  │     Unified Sequence Assembly (w/ AdaLN Modality Tags)│  │
│  │                                                      │   │
│  │  [text_emb | vis_emb | vid_emb | aud_emb | tags]    │   │
│  │                                                      │   │
│  │  Each token tagged with modality origin for          │   │
│  │  Transformer-differentiated processing               │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└──────────────────────────────────────────────────────────────┘

4.3 Text Encoder: Qwen3-VL-32B

H3 uses Qwen3-VL-32B as its text encoder, extracting layer-50 hidden states as the conditioning signal:

class TextEncoderWrapper:
    """
    Text encoder wrapper based on Qwen3-VL-32B.
    Extracts layer-50 hidden states as the conditioning signal.
    """
    
    def __init__(self, model_path: str = "MiniMaxAI/MiniMax-H3"):
        from transformers import AutoModel, AutoProcessor
        self.processor = AutoProcessor.from_pretrained(model_path)
        self.encoder = AutoModel.from_pretrained(
            model_path, 
            subfolder="text_encoder",
            torch_dtype=torch.bfloat16
        )
        self.extract_layer = 50  # Layer-50 hidden states
    
    def encode_text(self, text: str) -> torch.Tensor:
        """Encode text, returning layer-50 hidden states"""
        inputs = self.processor(text=text, return_tensors="pt")
        outputs = self.encoder(
            **inputs,
            output_hidden_states=True
        )
        hidden = outputs.hidden_states[self.extract_layer]
        return hidden  # shape: (batch, seq_len, hidden_dim)

V. H3-VAE: A Major Breakthrough in Tokenizer Technology

5.1 Compression Efficiency

H3-VAE is the core technology enabling native 2K output, achieving a comprehensive tokenizer overhaul:

ComponentCompressionEffect
H3-VisualVAE spatial32× sampling factor32×32 = 1024× spatial compression
H3-VisualVAE temporal4× sampling factor4× temporal compression
Combined sequence gainEffective sequence length improved 4×
H3-AudioVAE40Hz temporal frequencyStereo compressed to unified latent space
class H3VisualVAE:
    """
    H3-VisualVAE: Video Variational Autoencoder
    32× spatial compression, 4× temporal compression
    """
    
    def __init__(self, model_path: str):
        self.spatial_compression = 32   # Spatial sampling factor
        self.temporal_compression = 4   # Temporal sampling factor
    
    def encode(self, video_tensor: torch.Tensor) -> torch.Tensor:
        """
        Encode pixel-space video to latent-space representation
        Input:  (B, C, T, H, W) — pixel space
        Output: (B, C', T//4, H//32, W//32) — latent space
        """
        B, C, T, H, W = video_tensor.shape
        latent_T = T // self.temporal_compression
        latent_H = H // self.spatial_compression
        latent_W = W // self.spatial_compression
        
        compression_ratio = (
            self.temporal_compression * 
            self.spatial_compression ** 2
        )
        print(f"Pixel → Latent compression: {compression_ratio}× (volumetric)")
        print(f"Sequence length gain: 4× (temporal dimension)")
        
        return self.encoder(video_tensor)

Key significance: The 4× sequence length compression directly reduces training and inference costs, making native 2K resolution generation feasible — without this compression ratio, the token sequence for 2K video would be prohibitively long.


VI. In-Context Regeneration: A Revolutionary 2K Approach

6.1 Why Not Traditional Super-Resolution?

Traditional super-resolution modules (Real-ESRGAN, SwinIR, etc.) face fundamental limitations:

  • Can only “guess” missing high-frequency information from low-resolution pixels
  • Cannot recover semantic details from original multimodal context (small text, brand logos)
  • Information disconnect between generative model and post-processing module

6.2 In-Context Regeneration Approach

┌──────────────────────────────────────────────────────────────┐
│           In-Context Regeneration Pipeline                    │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Input Combination:                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌─────────────────┐   │
│  │  768p Video  │  │  Original    │  │  Enhanced        │   │
│  │  (base_video)│  │  Context     │  │  Prompt          │   │
│  │              │  │  (text/img/  │  │  (from Context-  │   │
│  │              │  │   video/     │  │   IR)            │   │
│  │              │  │   audio)     │  │                  │   │
│  └──────┬───────┘  └──────┬───────┘  └────────┬────────┘   │
│         │                 │                    │             │
│         └─────────────────┼────────────────────┘             │
│                           │                                  │
│                           ▼                                  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              H3-Base (Running Again)                  │   │
│  │                                                      │   │
│  │  768p result as condition + original multimodal      │   │
│  │  context as reference → regenerate at 2K resolution  │   │
│  │                                                      │   │
│  │  Advantage 1: Maximally reuses base model capability │   │
│  │  Advantage 2: Original context supplements semantics │   │
│  │  Advantage 3: Recovers small text/fine textures that │   │
│  │    traditional SR cannot restore                     │   │
│  └──────────────────────────────────────────────────────┘   │
│                           │                                  │
│                           ▼                                  │
│                    ┌──────────────┐                          │
│                    │  2K Output   │                          │
│                    │ 2560×1440    │                          │
│                    │ 24FPS        │                          │
│                    │ 32kHz Stereo │                          │
│                    └──────────────┘                          │
│                                                              │
└──────────────────────────────────────────────────────────────┘

The elegance of this approach: 2K generation is not “upscaling” — it is “regenerating.” The model leverages the rich information in the original context (text descriptions, fine details from reference images, audio features) to construct 2K resolution video from scratch — not guessing pixels, but reconstructing based on understanding.


VII. Dual Checkpoints and Input Specifications

7.1 FL2VA vs Ref2VA

FeatureH3-Base-FL2VAH3-Base-Ref2VA
PurposeFirst/last-frame modeOmni-reference mode
Image input0-2 (first/last frame)Up to 9 (reference images)
Video inputNoneUp to 3 clips (2-15s each, total ≤ 15s)
Audio inputNoneUp to 3 clips (must accompany image/video)
Max files2 images12 files total
Use casesT2V, first-frame, last-frame, first+last-frameIdentity preservation, scene retention, motion transfer, lip-sync, voice timbre

7.2 Output Specifications

ParameterValue
Output duration4-15 seconds
Output frame rate24 FPS
Output audio32 kHz native stereo
Default resolutionShort edge 768px
2K resolutionShort edge 1440px (via Regenerate-2K)
Aspect ratios21:9, 16:9, 4:3, 1:1, 3:4, 9:16
Languages11 stably supported (ar, zh, en, fr, de, it, ja, ko, pt, ru, es)

VIII. Open-Source Ecosystem and Deployment

8.1 Model Repositories

  • HuggingFace: MiniMaxAI/MiniMax-H3
  • License: MiniMax H3 Community License
  • ComfyUI Native Support: Comfy-Org/ComfyUI #15224 merged

8.2 Quantization Options

FormatModel SizeNotes
BF1661.7 GBFull precision inference
INT831.7 GBStandard quantization
Pruned INT819.5 GBPrecomputed AdaLN curves, 40% smaller
NVFP4 AWQ (text encoder)14.6 GBRuns on any GPU

8.3 Multi-Framework Deployment

16 ecosystem partners completed day-one adaptation, including Huawei Ascend, Moore Threads, AMD, Intel, and frameworks like Hugging Face, ComfyUI, SGLang, vLLM.


IX. Code Practice: Python Complete Pipeline Examples

9.1 Text-to-Video Pipeline with diffusers

"""
MiniMax H3 Text-to-Video Pipeline (diffusers)
Generate 768p video with native stereo audio from text description
"""
import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video

def text_to_video_audio(
    prompt: str,
    output_path: str = "output.mp4",
    num_frames: int = 124,
    height: int = 768,
    width: int = 1344,
    num_inference_steps: int = 30,
    seed: int = 42,
):
    """
    Text-to-Video-Audio Pipeline
    
    Args:
        prompt: Text description
        output_path: Output video path
        num_frames: Frame count (snapped to 17n+5)
        height: Height (short edge 768px)
        width: Width
        num_inference_steps: Denoising steps
        seed: Random seed
    """
    # 1. Load model
    pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3")
    pipe.load_components(dtype=torch.bfloat16)
    pipe.to("cuda")
    
    # 2. Run inference
    state = pipe(
        prompt=prompt,
        num_frames=num_frames,  # snapped to 17n+5 at 24fps
        height=height,
        width=width,
        num_inference_steps=num_inference_steps,
        generator=torch.Generator("cpu").manual_seed(seed),
    )
    
    # 3. Encode output (video + audio muxed into one MP4)
    encode_video(
        state.get("videos")[0],
        fps=24,
        audio=state.get("audio")[0],
        audio_sample_rate=state.get("sampling_rate"),
        output_path=output_path,
    )
    
    print(f"✅ Video saved to: {output_path}")
    print(f"   Resolution: {width}x{height}")
    print(f"   Frames: {num_frames} ({num_frames/24:.1f}s)")
    print(f"   Audio: 32kHz stereo")
    
    return output_path

if __name__ == "__main__":
    text_to_video_audio(
        prompt="A red fox trotting through a snowy pine forest, snow crunching underfoot",
        output_path="fox.mp4",
        num_frames=124,
        height=768,
        width=1344,
    )

9.2 Omni-Reference Pipeline (Ref2VA)

"""
MiniMax H3 Omni-Reference Pipeline
Joint generation using reference images, video, and audio
"""
import torch
from diffusers.modular_pipelines import MiniMaxH3Ref2VABlocks
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Reference
from diffusers.utils import encode_video

def ref_to_video_audio(
    prompt: str,
    references: list,
    output_path: str = "ref_output.mp4",
    num_frames: int = 124,
    num_inference_steps: int = 30,
):
    """
    Omni-Reference Pipeline
    Supports up to 9 images + 3 videos + 3 audios, max 12 files total
    """
    pipe = MiniMaxH3Ref2VABlocks().init_pipeline("MiniMaxAI/MiniMax-H3")
    pipe.load_components(dtype=torch.bfloat16)
    pipe.to("cuda")
    
    state = pipe(
        prompt=prompt,
        references=references,
        num_frames=num_frames,
        num_inference_steps=num_inference_steps,
    )
    
    encode_video(
        state.get("videos")[0],
        fps=24,
        audio=state.get("audio")[0],
        audio_sample_rate=state.get("sampling_rate"),
        output_path=output_path,
    )
    print(f"✅ Reference-generated video saved: {output_path}")

if __name__ == "__main__":
    ref_to_video_audio(
        prompt="The subject walks toward camera, matching the reference video's shot rhythm",
        references=[
            MiniMaxH3Reference(video="motion_ref.mp4"),
            MiniMaxH3Reference(image="subject.jpg"),
            MiniMaxH3Reference(audio="voice.wav"),
        ],
        num_frames=124,
    )

9.3 SGLang Multi-GPU Deployment

#!/bin/bash
# MiniMax H3 SGLang Deployment (4 GPU parallel)

# FL2VA variant
python -m sglang.launch_server \
  --model-path MiniMaxAI/MiniMax-H3 \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --performance-mode speed \
  --host 0.0.0.0 \
  --port 30011 \
  --model-variant fl2va

# Ref2VA variant
python -m sglang.launch_server \
  --model-path MiniMaxAI/MiniMax-H3 \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --performance-mode speed \
  --host 0.0.0.0 \
  --port 30011 \
  --model-variant ref2va

X. Code Practice: Go Implementation of Full 2K Workflow

10.1 Complete 2K Workflow Client

package main

import (
	"bytes"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"os"
	"time"
)

// MiniMaxH3Pipeline Full 2K workflow pipeline
type MiniMaxH3Pipeline struct {
	SGLangEndpoint string
	APIBaseURL     string
	APIKey         string
}

// ContextIRRequest H3-Context-IR request
type ContextIRRequest struct {
	Text   string   `json:"text"`
	Images []string `json:"images,omitempty"`
	Videos []string `json:"videos,omitempty"`
	Audios []string `json:"audios,omitempty"`
}

// GenerationRequest H3-Base generation request
type GenerationRequest struct {
	Prompt    string `json:"prompt"`
	NumFrames int    `json:"num_frames"`
	Height    int    `json:"height"`
	Width     int    `json:"width"`
	NumSteps  int    `json:"num_inference_steps"`
	Seed      int    `json:"seed"`
}

// RegenerateRequest H3-Regenerate-2K request
type RegenerateRequest struct {
	BaseVideo  string   `json:"base_video"`
	Prompt     string   `json:"prompt"`
	Resolution string   `json:"resolution"`
	Images     []string `json:"images,omitempty"`
	Videos     []string `json:"videos,omitempty"`
	Audios     []string `json:"audios,omitempty"`
}

// NewPipeline creates a new H3 pipeline
func NewPipeline(sglangEndpoint, apiBaseURL, apiKey string) *MiniMaxH3Pipeline {
	return &MiniMaxH3Pipeline{
		SGLangEndpoint: sglangEndpoint,
		APIBaseURL:     apiBaseURL,
		APIKey:         apiKey,
	}
}

// Step1: Context-IR preprocessing
func (p *MiniMaxH3Pipeline) ContextIR(req ContextIRRequest) (string, error) {
	fmt.Println("📋 Step 1/3: H3-Context-IR multimodal understanding...")
	
	body, _ := json.Marshal(req)
	httpReq, _ := http.NewRequest("POST",
		p.APIBaseURL+"/v1/h3-context-ir", bytes.NewReader(body))
	httpReq.Header.Set("Authorization", "Bearer "+p.APIKey)
	httpReq.Header.Set("Content-Type", "application/json")
	
	client := &http.Client{Timeout: 120 * time.Second}
	resp, err := client.Do(httpReq)
	if err != nil {
		return "", fmt.Errorf("Context-IR request failed: %w", err)
	}
	defer resp.Body.Close()
	
	var result struct {
		EnhancedPrompt string `json:"enhanced_prompt"`
	}
	json.NewDecoder(resp.Body).Decode(&result)
	
	fmt.Printf("  ✅ Context-IR complete, prompt enhanced\n")
	fmt.Printf("  📝 Enhanced prompt length: %d chars\n", len(result.EnhancedPrompt))
	return result.EnhancedPrompt, nil
}

// Step2: H3-Base local 768p generation
func (p *MiniMaxH3Pipeline) Generate768p(prompt string) ([]byte, error) {
	fmt.Println("🎬 Step 2/3: H3-Base local 768p audio-video generation...")
	
	req := GenerationRequest{
		Prompt:    prompt,
		NumFrames: 124,
		Height:    768,
		Width:     1344,
		NumSteps:  30,
		Seed:      42,
	}
	
	body, _ := json.Marshal(req)
	httpReq, _ := http.NewRequest("POST",
		p.SGLangEndpoint+"/v1/videos", bytes.NewReader(body))
	httpReq.Header.Set("Content-Type", "application/json")
	
	client := &http.Client{Timeout: 600 * time.Second}
	resp, err := client.Do(httpReq)
	if err != nil {
		return nil, fmt.Errorf("H3-Base generation failed: %w", err)
	}
	defer resp.Body.Close()
	
	videoData, _ := io.ReadAll(resp.Body)
	fmt.Printf("  ✅ 768p video generated (%d bytes)\n", len(videoData))
	return videoData, nil
}

// Step3: Regenerate-2K
func (p *MiniMaxH3Pipeline) Regenerate2K(
	baseVideo []byte, prompt string, refs ContextIRRequest,
) error {
	fmt.Println("✨ Step 3/3: H3-Regenerate-2K regeneration to 2K...")
	
	b64 := base64.StdEncoding.EncodeToString(baseVideo)
	dataURL := "data:video/mp4;base64," + b64
	
	req := RegenerateRequest{
		BaseVideo:  dataURL,
		Prompt:     prompt,
		Resolution: "2k",
		Images:     refs.Images,
		Videos:     refs.Videos,
		Audios:     refs.Audios,
	}
	
	body, _ := json.Marshal(req)
	httpReq, _ := http.NewRequest("POST",
		p.APIBaseURL+"/v1/h3-regenerate-2k", bytes.NewReader(body))
	httpReq.Header.Set("Authorization", "Bearer "+p.APIKey)
	httpReq.Header.Set("Content-Type", "application/json")
	
	client := &http.Client{Timeout: 900 * time.Second}
	resp, err := client.Do(httpReq)
	if err != nil {
		return fmt.Errorf("Regenerate-2K failed: %w", err)
	}
	defer resp.Body.Close()
	
	outFile, _ := os.Create("output_2k.mp4")
	defer outFile.Close()
	io.Copy(outFile, resp.Body)
	
	fmt.Println("  ✅ 2K video saved: output_2k.mp4")
	return nil
}

// RunFullPipeline executes the complete 2K workflow
func (p *MiniMaxH3Pipeline) RunFullPipeline(userInput ContextIRRequest) error {
	fmt.Println("🚀 MiniMax H3 Full 2K Workflow initiated")
	fmt.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
	
	enhancedPrompt, err := p.ContextIR(userInput)
	if err != nil {
		return err
	}
	
	video768p, err := p.Generate768p(enhancedPrompt)
	if err != nil {
		return err
	}
	
	err = p.Regenerate2K(video768p, enhancedPrompt, userInput)
	if err != nil {
		return err
	}
	
	fmt.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
	fmt.Println("🎉 Full 2K workflow completed!")
	return nil
}

func main() {
	pipeline := NewPipeline(
		"http://localhost:30011",
		"https://api.minimaxi.com",
		os.Getenv("MINIMAX_API_KEY"),
	)
	
	userInput := ContextIRRequest{
		Text: "A red fox running through a pine forest, snow crunching underfoot",
	}
	
	if err := pipeline.RunFullPipeline(userInput); err != nil {
		fmt.Fprintf(os.Stderr, "❌ Workflow failed: %v\n", err)
		os.Exit(1)
	}
}

10.2 Go Multimodal Preprocessor

package main

import (
	"encoding/json"
	"fmt"
	"strings"
)

// MultiModalPreprocessor simulates core H3-Context-IR functionality
type MultiModalPreprocessor struct {
	maxImages int
	maxVideos int
	maxAudios int
	maxFiles  int
}

// MediaAsset represents a media input asset
type MediaAsset struct {
	Type     string  `json:"type"`
	Path     string  `json:"path"`
	Duration float64 `json:"duration"`
}

// StructuredPrompt represents the serialized intermediate representation
type StructuredPrompt struct {
	OriginalText  string       `json:"original_text"`
	EnhancedText  string       `json:"enhanced_text"`
	ContextAssets []MediaAsset `json:"context_assets"`
	TaskType      string       `json:"task_type"`
	TokenBudget   int          `json:"token_budget"`
}

// NewPreprocessor creates a new preprocessor with H3 constraints
func NewPreprocessor() *MultiModalPreprocessor {
	return &MultiModalPreprocessor{
		maxImages: 9,
		maxVideos: 3,
		maxAudios: 3,
		maxFiles:  12,
	}
}

// ValidateInputs validates multimodal inputs against H3 constraints
func (p *MultiModalPreprocessor) ValidateInputs(
	text string, images []MediaAsset, videos []MediaAsset, audios []MediaAsset,
) error {
	if len(images) > p.maxImages {
		return fmt.Errorf("image count %d exceeds max %d", len(images), p.maxImages)
	}
	if len(videos) > p.maxVideos {
		return fmt.Errorf("video count %d exceeds max %d", len(videos), p.maxVideos)
	}
	if len(audios) > p.maxAudios {
		return fmt.Errorf("audio count %d exceeds max %d", len(audios), p.maxAudios)
	}
	
	totalFiles := len(images) + len(videos) + len(audios)
	if totalFiles > p.maxFiles {
		return fmt.Errorf("total files %d exceeds max %d", totalFiles, p.maxFiles)
	}
	
	if len(audios) > 0 && len(images) == 0 && len(videos) == 0 {
		return fmt.Errorf("audio must accompany image or video input")
	}
	
	for i, v := range videos {
		if v.Duration < 2 || v.Duration > 15 {
			return fmt.Errorf("video %d duration %.1f outside [2,15] range", i, v.Duration)
		}
	}
	totalDuration := 0.0
	for _, v := range videos {
		totalDuration += v.Duration
	}
	if totalDuration > 15 {
		return fmt.Errorf("total video duration %.1fs exceeds 15s limit", totalDuration)
	}
	
	return nil
}

// DetermineTaskType determines the generation task type
func (p *MultiModalPreprocessor) DetermineTaskType(
	images []MediaAsset, videos []MediaAsset, audios []MediaAsset,
) string {
	if len(videos) > 0 || len(audios) > 0 || len(images) > 2 {
		return "ref2va"
	}
	return "fl2va"
}

// Process handles multimodal input, outputting structured representation
func (p *MultiModalPreprocessor) Process(
	text string, images []MediaAsset, videos []MediaAsset, audios []MediaAsset,
) (*StructuredPrompt, error) {
	if err := p.ValidateInputs(text, images, videos, audios); err != nil {
		return nil, fmt.Errorf("validation failed: %w", err)
	}
	
	taskType := p.DetermineTaskType(images, videos, audios)
	
	var assets []MediaAsset
	assets = append(assets, images...)
	assets = append(assets, videos...)
	assets = append(assets, audios...)
	
	enhanced := p.buildEnhancedPrompt(text, assets, taskType)
	
	return &StructuredPrompt{
		OriginalText:  text,
		EnhancedText:  enhanced,
		ContextAssets: assets,
		TaskType:      taskType,
		TokenBudget:   4096,
	}, nil
}

func (p *MultiModalPreprocessor) buildEnhancedPrompt(
	text string, assets []MediaAsset, taskType string,
) string {
	var parts []string
	parts = append(parts, fmt.Sprintf("[Task: %s]", taskType))
	parts = append(parts, text)
	
	for i, a := range assets {
		switch a.Type {
		case "image":
			parts = append(parts, fmt.Sprintf("[Ref image %d: %s]", i+1, a.Path))
		case "video":
			parts = append(parts, fmt.Sprintf("[Ref video %d: %s, %.1fs]", i+1, a.Path, a.Duration))
		case "audio":
			parts = append(parts, fmt.Sprintf("[Ref audio %d: %s, %.1fs]", i+1, a.Path, a.Duration))
		}
	}
	
	return strings.Join(parts, "\n")
}

func main() {
	proc := NewPreprocessor()
	
	images := []MediaAsset{
		{Type: "image", Path: "character.jpg"},
		{Type: "image", Path: "scene_bg.jpg"},
	}
	videos := []MediaAsset{
		{Type: "video", Path: "motion_ref.mp4", Duration: 5.0},
	}
	audios := []MediaAsset{
		{Type: "audio", Path: "voice_ref.wav", Duration: 3.0},
	}
	
	result, err := proc.Process(
		"Subject walks toward camera, motion ref video 1, voice timbre ref audio 1",
		images, videos, audios,
	)
	if err != nil {
		fmt.Printf("❌ Processing failed: %v\n", err)
		return
	}
	
	output, _ := json.MarshalIndent(result, "", "  ")
	fmt.Println("✅ Structured Prompt:")
	fmt.Println(string(output))
}

XI. Python Deployment Utility: Multi-GPU Inference Service

"""
MiniMax H3 Multi-GPU Inference Service Wrapper
Supports FL2VA and Ref2VA variants
"""
import subprocess
import time
import requests
import json
import base64
from typing import Optional, List, Dict, Any
from dataclasses import dataclass

@dataclass
class H3Config:
    """H3 deployment configuration"""
    model_path: str = "MiniMaxAI/MiniMax-H3"
    num_gpus: int = 4
    ulysses_degree: int = 4
    host: str = "0.0.0.0"
    port: int = 30011
    model_variant: str = "fl2va"
    performance_mode: str = "speed"

class H3Server:
    """H3 inference service manager"""
    
    def __init__(self, config: H3Config):
        self.config = config
        self.process = None
        self.base_url = f"http://{config.host}:{config.port}"
    
    def start(self):
        """Start SGLang inference service"""
        cmd = [
            "python", "-m", "sglang.launch_server",
            "--model-path", self.config.model_path,
            "--num-gpus", str(self.config.num_gpus),
            "--ulysses-degree", str(self.config.ulysses_degree),
            "--performance-mode", self.config.performance_mode,
            "--host", self.config.host,
            "--port", str(self.config.port),
            "--model-variant", self.config.model_variant,
        ]
        print(f"🚀 Starting H3 service ({self.config.model_variant})...")
        self.process = subprocess.Popen(cmd)
        self._wait_for_ready()
    
    def _wait_for_ready(self, timeout: int = 300):
        """Wait for service readiness"""
        start = time.time()
        while time.time() - start < timeout:
            try:
                resp = requests.get(f"{self.base_url}/health", timeout=5)
                if resp.status_code == 200:
                    print("✅ Service ready")
                    return
            except requests.ConnectionError:
                time.sleep(2)
        raise TimeoutError("Service startup timed out")
    
    def stop(self):
        """Stop the service"""
        if self.process:
            self.process.terminate()
            self.process.wait()
            print("🛑 Service stopped")
    
    def generate(
        self,
        prompt: str,
        num_frames: int = 124,
        height: int = 768,
        width: int = 1344,
        num_steps: int = 30,
        seed: int = 42,
    ) -> bytes:
        """Call the generation API"""
        payload = {
            "prompt": prompt,
            "num_frames": num_frames,
            "height": height,
            "width": width,
            "num_inference_steps": num_steps,
            "seed": seed,
        }
        resp = requests.post(
            f"{self.base_url}/v1/videos",
            json=payload,
            timeout=600,
        )
        resp.raise_for_status()
        return resp.content


class Full2KWorkflow:
    """Complete 2K workflow orchestrator"""
    
    def __init__(self, server: H3Server, api_base: str, api_key: str):
        self.server = server
        self.api_base = api_base
        self.api_key = api_key
    
    def context_ir(self, text: str, **kwargs) -> str:
        """Call H3-Context-IR API"""
        payload = {"text": text, **kwargs}
        resp = requests.post(
            f"{self.api_base}/v1/h3-context-ir",
            json=payload,
            headers={"Authorization": f"Bearer {self.api_key}"},
            timeout=120,
        )
        resp.raise_for_status()
        return resp.json()["enhanced_prompt"]
    
    def generate_768p(self, prompt: str) -> bytes:
        """Generate 768p via local H3-Base"""
        return self.server.generate(prompt=prompt)
    
    def regenerate_2k(self, video_data: bytes, prompt: str, **refs) -> bytes:
        """Call H3-Regenerate-2K API"""
        b64 = base64.b64encode(video_data).decode()
        payload = {
            "base_video": f"data:video/mp4;base64,{b64}",
            "prompt": prompt,
            "resolution": "2k",
            **refs,
        }
        resp = requests.post(
            f"{self.api_base}/v1/h3-regenerate-2k",
            json=payload,
            headers={"Authorization": f"Bearer {self.api_key}"},
            timeout=900,
        )
        resp.raise_for_status()
        return resp.content
    
    def run(self, text: str, **refs) -> bytes:
        """Execute the full 2K workflow"""
        # Step 1: Context-IR
        enhanced = self.context_ir(text, **refs)
        print(f"📋 Context-IR enhancement complete")
        
        # Step 2: 768p generation
        video_768p = self.generate_768p(enhanced)
        print(f"🎬 768p generation complete ({len(video_768p)} bytes)")
        
        # Step 3: 2K regeneration
        video_2k = self.regenerate_2k(video_768p, enhanced, **refs)
        print(f"✨ 2K regeneration complete ({len(video_2k)} bytes)")
        
        return video_2k


if __name__ == "__main__":
    config = H3Config(model_variant="fl2va", num_gpus=4)
    server = H3Server(config)
    server.start()
    
    workflow = Full2KWorkflow(
        server=server,
        api_base="https://api.minimaxi.com",
        api_key="your-api-key",
    )
    
    video_2k = workflow.run(text="A red fox running through a pine forest")
    
    with open("output_2k.mp4", "wb") as f:
        f.write(video_2k)
    print("✅ 2K video saved")

XII. Performance Benchmarks and Competitive Landscape

12.1 Artificial Analysis Leaderboard Results

CapabilityH3 RankingNotes
Audio-video editing🥇 Global #1Elo 1130
Text-to-video🥈 Global #2
Image-to-video🥉 Global #3
OverallTop tier globallyOn par with Seedance 2.5

12.2 Key Strengths

  1. Complex Text Rendering: Text truly participates in visual composition — conforming to perspective, accepting scene lighting, following camera movement and subject motion
  2. Director-Level Instruction Following: Precisely executes complex lighting changes (candlelight → hard light → rainbow refraction → red-blue split → golden backlight)
  3. Generalized Editing: Dub video characters while simultaneously matching voice timbre and lip movement
  4. Cost Advantage: Per-second price at 2K resolution is less than 1/3 of mainstream models; at 768p, less than half

XIII. Technical Summary and Outlook

13.1 H3’s Three Technical Pillars

┌────────────────────────────────────────────────────────────────┐
│               MiniMax H3 Technical System Summary              │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │  Pillar 1: Contextual Omni Representation                │ │
│  │  · Language as generalization bridge, unifying tasks     │ │
│  │    into open descriptive form                            │ │
│  │  · 100K tokens → 4K tokens, 25:1 compression            │ │
│  │  · Cross-modal relationship modeling                     │ │
│  └──────────────────────────────────────────────────────────┘ │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │  Pillar 2: H3-Omni-Transformer + H3-VAE                 │ │
│  │  · 33.1B Dense Single-Stream Transformer                 │ │
│  │  · 13B AdaLN branch precomputable & cacheable            │ │
│  │  · MM-RoPE 3D multimodal positional encoding             │ │
│  │  · H3-VAE 4× sequence compression → native 2K support    │ │
│  │  · 30% end-to-end training throughput improvement        │ │
│  └──────────────────────────────────────────────────────────┘ │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │  Pillar 3: In-Context Regeneration                       │ │
│  │  · Not traditional SR — base model regenerates           │ │
│  │  · Reuses base model capability + original context       │ │
│  │  · Recovers small text/fine textures that traditional    │ │
│  │    super-resolution cannot restore                       │ │
│  └──────────────────────────────────────────────────────────┘ │
│                                                                │
└────────────────────────────────────────────────────────────────┘

13.2 Significance of Open-Sourcing

MiniMax H3’s open-sourcing is regarded as the “DeepSeek moment” for video generation:

  • Breaking closed-source monopoly: Previously, all flagship video models were closed-source; H3 is the first to open commercially viable weights
  • Hardware ecosystem: 16 chip manufacturers completed day-one adaptation, accelerating domestic chip support
  • Community customization: Developers can fine-tune for their business needs and build domain-specific versions

13.3 Future Directions

According to official announcements, the next-generation H series will:

  1. Integrate capabilities from M-series models for enhanced multimodal understanding
  2. Explore scaling to larger model sizes
  3. Continue pushing resolution and visual fidelity higher

References