Getting AI Off the Screen: Agnes Harness Goes Open Source, Welding 'Agent Execution' and 'Physical Devices' into One System
Getting AI truly off the screen and into reality needs three things: a brain reliable enough, an execution system that can keep it working continuously, and a body that can connect physical devices. Over the past two years, model capabilities have advanced by leaps and bounds and agent execution frameworks have appeared endlessly, but most people remain stuck on one unassuming problem—how exactly should the wildly diverse devices of the real world be understood, invoked, and coordinated by a single agent?
Recently, Agnes AI gave its answer: officially open-sourcing Agnes Harness (AGH)—an agent execution infrastructure built for real tasks—while also opening up the MHS (Model Hardware Standard) device interaction protocol and related components, and previewing the capability upgrades and upcoming API of the next-generation model Agnes 3.0 Pro.
This doesn’t feel like an ordinary framework release. It feels like Agnes AI deliberately signaling something: the next battlefield for agents isn’t just the dialog box on screen—it’s the physical world off screen. In this article, I want to take you through this open-source stack completely: first understand what an “Agent Harness” is, then see how AGH organizes one continuous real task, then dig into what this “protocol devices can understand” (MHS) actually does, then demonstrate the engineering details of connecting agents and physical devices with code, and finally discuss Agnes 3.0 Pro and the strategic significance of this “from model to infrastructure” path.
Part One: First Get It—Why Does an Agent Need a “Harness”
Before diving into AGH, we have to answer a more basic question: What is an Agent Harness? And why can’t a modern agent survive without it?
If you compare a large model to a “brain,” then an agent that can actually work and handle long multi-step tasks needs far more than a model that can think. It also needs: a loop for continuous execution (the model thinks a step, takes an action, sees the result, thinks the next step), channels to call external tools (read files, run commands, call APIs, connect MCP), cross-step memory and session management (don’t lose the earlier context halfway through a task), context compression and organization, and a human-confirmation mechanism for critical operations.
Everything that packages up “everything supporting the model besides its thinking” is what a Harness is. To put it in a memorable phrase: a Harness is everything you build around a model to turn it from a “text generator” into an agent that can act in the real world. In agent engineering, the difficulty of building a Harness has often already exceeded that of choosing the model itself—models iterate every year, but what truly decides whether an agent can finish a task steadily, continuously, and safely is precisely that unassuming yet everywhere-critical “execution foundation.”
From a broader view, two routes have already emerged in the agent ecosystem: one is “managed Harness” (like AWS’s AgentCore, packaging orchestration loop, tool execution, memory, and sandbox into an out-of-the-box managed service); the other is “open-source Harness” (like the AGH Agnes just open-sourced, handing the source and protocol entirely to developers, so you can install it locally and extend on demand). The value of the open-source route lies in portability, pluggability, and auditability—you can see exactly what each layer is doing, and wire your own tools and hardware into the same system. What makes AGH different is that it brings two formerly fragmented connections—“software tools” and “physical devices”—into one execution environment.
Part Two: AGH in Daily Use—How an “Agent Loop” Organizes a Real Task
Let’s set aside the abstract architecture terms and look directly at how AGH, as a “usable Harness,” works in daily life.
When you open AGH’s web workbench, you see a dialog interface running in a local environment. You can specify a working directory, send a task instruction, and watch the agent read files, execute commands, and keep modifying the existing project based on results. All of this is driven forward by AGH’s core mechanism—the Agent Loop.
The Agent Loop works as a deceptively simple yet powerful cycle:
┌──────────────────────────────────────────────────────────────┐
│ AGH's Agent Loop │
│ │
│ ┌──────────┐ judge next step ┌──────────────┐ │
│ │ model │ ──────────────────────► │ choose act/tool│ │
│ │ (brain) │ └──────┬───────┘ │
│ └────▲─────┘ │ │
│ │ ▼ │
│ │ ┌──────────────┐ │
│ hand back│ │ Harness executes│ │
│ result │ │ call tool/cmd │ │
│ │ └──────┬───────┘ │
│ │ │ exec feedback │
│ └─────────── until task done ──────────┘ or human join│
└──────────────────────────────────────────────────────────────┘
In this loop, the model is responsible for “thinking,” and the Harness for “doing”: the model judges what to do next, the Harness calls the corresponding tool and executes the command, then hands the execution result back to the model, which judges the next step… and so on, until the task is done or hits a critical step requiring human involvement.
This loop itself isn’t unusual; many agent frameworks have it. But AGH has two design points worth calling out separately.
First, the traceability of “trajectory.” When a task gets complex (e.g., an agent modifies a dozen files, executes many commands, and still fails the test), it’s hard to judge from conversation logs alone “which step went wrong”—was the model’s judgment wrong, or did some tool fail? AGH’s “trajectory” page gives the answer: developers can replay the whole task execution process along a timeline, see what tools the model called and how long each step took, and expand the records to inspect tool parameters and execution status. This “auditable process” capability is a prerequisite for agents moving from toys to production.
Second, the “plugin + Skills” extension mechanism. AGH opens up plugin extensions: you can plug in your own business tools through plugins, connect existing services via MCP, and even develop dedicated interfaces for different roles. Even better are “Skills”—already-validated task methods can be preserved through Skills and reused in other projects. You can understand this as “solidifying one successful working experience into a reusable skill pack,” greatly lowering the cost of repeated setup.
Let me use a small piece of Python conceptual code to make “Agent Loop + tool feedback” concrete in your mind:
# agent_loop.py
# Conceptual demo of AGH's Agent Loop: model decides → tool executes → feedback returns
class ToolRegistry:
"""A minimal tool registry: name → callable"""
def __init__(self):
self._tools = {}
def register(self, name, fn):
self._tools[name] = fn
def call(self, name, **kwargs):
if name not in self._tools:
return {"ok": False, "error": f"unknown tool: {name}"}
try:
return {"ok": True, "result": self._tools[name](**kwargs)}
except Exception as e:
return {"ok": False, "error": str(e)}
def run_agent_loop(model_decision, registry: ToolRegistry, initial_state):
"""Keep decide→execute→feedback until the model says done or cap reached"""
state = initial_state
for step in range(10): # cap of 10 steps to avoid infinite loop
intent = model_decision(state) # model judges next step
if intent["done"]:
print(f"[DONE] step={step}")
return state
feedback = registry.call(intent["tool"], **intent.get("args", {}))
state = {**state, "last_feedback": feedback} # feedback back to model
print(f"[STEP {step}] tool={intent['tool']} ok={feedback['ok']}")
print("[STOP] reached step cap")
return state
# Simulate: model's three-step decision + two tools
def decide(state):
plan = {0: {"tool": "read_file", "args": {"path": "src/main.py"}},
1: {"tool": "run_cmd", "args": {"cmd": "go build ./..."}},
2: {"done": True}}
return plan.get(state.get("step", 0), {"done": True})
reg = ToolRegistry()
reg.register("read_file", lambda path: f"read {path} (24 lines)")
reg.register("run_cmd", lambda cmd: f"exec {cmd} -> exit 0")
result = run_agent_loop(decide, reg, {"step": 0})
# Sample output:
# [STEP 0] tool=read_file ok=True
# [STEP 1] tool=run_cmd ok=True
# [DONE] step=2
What this minimal demo wants to show is: the essence of a Harness is turning the loop of “model thinks, machine does, result comes back” into reliable, auditable, interruptible infrastructure. AGH’s real implementation is far more complex than this snippet, but the skeleton is the same—and this is the foundation for any agent capable of “working continuously.”
Part Three: The Main Event MHS—Letting an Agent Understand a Device It’s Never Seen
If AGH solves “how agents execute tasks continuously,” then MHS (Model Hardware Standard) solves the harder problem: how agents connect to the wildly diverse devices of the physical world.
Real-world devices are all over the map: mobile cars, robot arms, sensors, airlocks, aircraft… each has its own interfaces and control methods. If an agent had to write special adapter code for every new device it connects to, the cost of connecting agents to the physical world would be too high to ever land. What MHS wants to do is something plain yet profound: through a unified interaction style, let devices expose their “function and operation information” to agents, so that an agent facing a device it’s never seen can understand its capabilities and invoke the appropriate operations based on the task.
You can think of MHS as the “USB protocol of the agent world”: USB lets all kinds of peripherals be plug-and-play with a computer, without writing low-level device driver understanding for each device. MHS wants to let all kinds of hardware be “plug-and-play” understood and invoked by agents—if a device can “speak MHS’s unified language,” an agent can read it and control it.
MHS isn’t isolated; Agnes designed it together with another protocol, MOS (Model Operating Standard). The two play distinct roles:
┌────────────────────────────────────────────────────────────────┐
│ Agnes device protocol system │
├───────────────────────┬────────────────────────────────────────┤
│ MHS (Model Hardware │ handles the "action layer": device │
│ Standard) 1.0 │ state, tool calls, action requests │
├───────────────────────┼────────────────────────────────────────┤
│ MOS (Model Operating │ handles the "perception layer": camera, │
│ Standard) 1.0 │ sensors, maps and observation transport │
├───────────────────────┼────────────────────────────────────────┤
│ AgnesHub │ device connection component │
├───────────────────────┼────────────────────────────────────────┤
│ mhs-check │ conformance test suite for protocol │
└───────────────────────┴────────────────────────────────────────┘
In this system, the division of roles is clear: AGH organizes the whole task, MHS connects specific devices. The model judges what to do next based on the goal, AGH coordinates tool calls and execution feedback, then completes the operation through the device interface; when a critical step is involved, human confirmation can be introduced as the task demands.
Agnes used two examples to show this system’s actual capability. The first is very intuitive: have an office mobile car find a specified book and read its content aloud via voice. In the demo, the car completed finding the book and reading it per the task, and the agent’s execution scope genuinely extended from “operating software” to “operating a physical car.” The second example has even more imagination—Mars World, a virtual Mars base. Agnes designed a rescue task there: an astronaut loses contact outside the base, and the agent must use the base’s devices to find the astronaut and organize a rescue. In this virtual mode, the agent dispatches aircraft to search target areas, arranges robots to perform the rescue, coordinates the base’s airlocks and other facilities, and the whole process must advance many operation steps by synthesizing maps, device states, and task feedback. The official repo already provides a Mars World example with 11 virtual devices, plus local simulated devices like mobile robots, robot arms, and sensors—developers don’t need real hardware to experience the full device-access and task-invocation flow.
Part Four: Code View—Implementing an “MHS-style Unified Device Facade” in Go
The best way to understand MHS is to design a similar abstraction yourself. I’ll write a minimal “MHS-style device facade” in Go, showing how to let different devices expose a unified capability interface to agents.
// mhs_device.go
// A minimal MHS-style device abstraction: exposing hardware to agents via a unified interface
package main
import "fmt"
// DeviceCapability unified description of "what a device can do" exposed to agents
type DeviceCapability struct {
DeviceID string
Name string
Actions []string // standard actions the device supports (e.g. "move_to", "read_sensor", "grip")
}
// AgentDevice unified device interface: implement it and an agent can understand and invoke
type AgentDevice interface {
Capability() DeviceCapability
Invoke(action string, params map[string]string) (string, error)
}
// ---- an implementation of a mobile car ----
type Car struct{}
func (Car) Capability() DeviceCapability {
return DeviceCapability{DeviceID: "car-01", Name: "mobile-car",
Actions: []string{"move_to", "stop", "pick_up"}}
}
func (Car) Invoke(action string, params map[string]string) (string, error) {
switch action {
case "move_to":
return fmt.Sprintf("car moved to %v", params["pos"]), nil
case "pick_up":
return fmt.Sprintf("car picked up %v", params["target"]), nil
}
return "", fmt.Errorf("unsupported action %s", action)
}
// ---- an implementation of a robot arm ----
type Arm struct{}
func (Arm) Capability() DeviceCapability {
return DeviceCapability{DeviceID: "arm-01", Name: "robot-arm",
Actions: []string{"grip", "release", "rotate"}}
}
func (Arm) Invoke(action string, params map[string]string) (string, error) {
switch action {
case "grip":
return "arm gripped object", nil
case "rotate":
return fmt.Sprintf("arm rotated to %v deg", params["angle"]), nil
}
return "", fmt.Errorf("unsupported action %s", action)
}
// ---- a "plug-and-play" facade: face an unseen device, read and invoke it ----
func main() {
car := Car{}
arm := Arm{}
devices := []AgentDevice{car, arm}
for _, d := range devices {
cap := d.Capability()
fmt.Printf("[device] %s (%s) can: %v\n", cap.Name, cap.DeviceID, cap.Actions)
if cap.Name == "mobile-car" {
out, err := d.Invoke("move_to", map[string]string{"pos": "(2,3)"})
fmt.Printf(" -> %s err=%v\n", out, err)
}
}
// Sample output:
// [device] mobile-car (car-01) can: [move_to stop pick_up]
// -> car moved to (2,3) err=<nil>
// [device] robot-arm (arm-01) can: [grip release rotate]
}
The essence of this code is the AgentDevice interface: as long as a device implements this unified “expose capability + invoke by action” interface, an agent doesn’t need to know its underlying driver details to understand and operate it. That’s what MHS wants to do—reduce the cost of “connecting a new device” from “writing a special adapter for it” to “letting it say what it can do once.” As more and more devices learn this “unified language,” the barrier to connecting agents to the physical world will drop sharply.
Part Five: Agnes 3.0 Pro and the Full Chain “From Model to Physical Infrastructure”
AGH and MHS are the “execution foundation” and “connection protocol,” but a closed-loop agent system still lacks the “brain” at the top—the model. Agnes simultaneously previewed the next-generation Agnes 3.0 Pro capability upgrades, with its API coming soon. Behind this is Agnes’s clear strategy: integrate “model → execution → connecting the physical world” into one technical system.
This strategy can be understood this way: in the past, most AI companies only did “models,” agent frameworks were made by others, and physical device connectivity was made by yet another group—the three “limbs” limped along via various patchwork and adaptation. What Agnes wants is to provide a full chain “from brain to body”: Agnes-series models as the “brain,” AGH as the “execution system,” and MHS/MOS as the “nervous system connecting the physical world.” Putting all three into one technical system is how AI can possibly “live” and “work” stably in real, complex, cross-platform tasks.
Let me string the whole chain together in one diagram:
┌──────────────────────────────────────────────────────────────┐
│ Agnes: the full chain from model to physical infra│
│ │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Agnes 3.0 │ │ AGH │ │
│ │ Pro (brain) │ ─────► │ (exec system)│ │
│ │ reason/act │ │ Agent Loop │ │
│ └─────────────┘ └──────┬──────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ MHS/MOS (neural) │ │
│ │ unified device │ │
│ │ interaction proto │ │
│ └─────────┬─────────┘ │
│ ▼ │
│ ┌────────┬────────┼────────┬────────┐ │
│ ▼ ▼ ▼ ▼ ▼ │
│ mobile car arm sensor airlock aircraft │
│ (physical/virtual devices, plug-and-play unified) │
└──────────────────────────────────────────────────────────────┘
The significance of this chain is opening the last mile between “software agents” and “the physical world.” In the past, when we said “agents step off the screen,” it mostly still meant “operating a browser on screen, calling APIs.” What Agnes wants to deliver is letting the agent’s other foot truly step into the physical world—to operate cars, arms, and sensors. For developers exploring robotics inspection, scientific experiments, and on-site operations, this open-source stack offers a landable starting point “from real device connection to complex task coordination.”
Part Six: Open Source vs. Managed—AGH’s Path and the Imagination It Leaves the Industry
Finally, I want to place AGH in the larger agent-infrastructure landscape and talk about its positioning and significance.
The agent execution infrastructure layer is already being watched by many players: AWS’s AgentCore takes the “managed” route, packaging orchestration, tools, and sandbox into an out-of-the-box service; Microsoft’s Agent Framework also offers a programming model for Harness; and Agnes chose to fully open-source AGH—source, MHS, MOS, AgnesHub, dev libraries, and conformance test suite all open. Choosing open source means Agnes is betting its “standard for connecting the physical world” on open co-building: it hopes MHS becomes not just one company’s device protocol but a unified standard the whole ecosystem embraces.
The subtext here runs deep. Whether a protocol named “Model Hardware Standard” can become an industry-wide “de facto standard” depends largely on whether developers buy in and device vendors are willing to integrate. Open-sourcing the standard and attaching a conformance test suite (mhs-check) is exactly the move to lower everyone’s barrier to entry and dispel “vendor lock-in” fears—if you want to integrate, integrate against the public protocol and test suite; if you fear lock-in, the protocol is plainly open for you to see. This “trade openness for ecosystem” play is often more attractive than closed managed services in domains (like hardware access) that are naturally afraid of “binding.”
For developers, AGH’s value can be reduced to three sentences:
- If you’re building software agents, AGH gives you an out-of-the-box, audit-traced execution foundation—install it locally, connect services via MCP, and use Skills to accumulate reusable working experience.
- If you’re building physical/embodied agents, MHS/MOS gives you an abstraction layer where “you don’t write an adapter for every device”; with the 11-virtual-device example and local simulated devices, you can get the access-and-invocation logic working even without real hardware.
- If you want to avoid binding, open source + conformance test suite means this system is transparent, auditable, and portable—you can see what each layer does and extend as needed.
Conclusion: AI’s Next Battlefield Is Moving from “Screen” to “World”
From a task like “let a car find a book and read it” to a simulation “dispatching aircraft to rescue an astronaut in a virtual Mars base,” Agnes paints a clear direction with AGH and MHS: the capability boundary of agents is extending—from “dialog boxes on screen” and “APIs in software”—bit by bit—to “commanding the physical world.” And to do that, stronger models alone aren’t enough; you also need “execution infrastructure” that can organize agents to work continuously and let agents understand and invoke thousands of different devices.
By fully open-sourcing the Harness and execution protocol, Agnes is publicly betting on a judgment: the next wave of agent landing belongs to those willing to weld “brain, execution system, and physical connection” into one system and use openness to win the ecosystem. Whether MHS can really become the “USB protocol” of the agent world, and whether Agnes 3.0 Pro can hold up the “brain” on top of this system, still needs time to verify.
But one thing is increasingly clear: when an agent is no longer just a “program that chats,” but a complete system that can walk into the real world and operate real devices, whoever first fills the “last mile from model to physical world” is more likely to eat the next wave of dividends as agents move from screen to world. This door, Agnes has already opened for everyone.