Skip to main content
HUD uses the Model Context Protocol (MCP) to provide a standard way for AI agents to interact with any software environment through tool calls.

​
Why MCP?

Traditional agent frameworks couple agents tightly to specific environments. MCP decouples them:

Without MCP

  • Agent code hardcoded for each environment
  • No standardization across tools
  • Difficult to swap agents or environments

With MCP

  • Any agent works with any environment
  • Standard protocol for all interactions
  • Easy to swap components

​
How It Works

MCP standardizes agent-environment communication through JSON-RPC messages. Agents call tools exposed by environments and receive structured responses.

​
Core Concepts

​
Tools

Tools are functions exposed by the environment: 
{
  "name": "move",
  "description": "Move in a direction",
  "inputSchema": {
    "type": "object",
    "properties": {
      "direction": {
        "type": "string",
        "enum": ["up", "down", "left", "right"]
      }
    }
  }
}

​
Tool Calls & Results

Agents call tools and receive results: 
# Agent makes a tool call
result = await client.call_tool("move", {"direction": "up"})

# Environment returns result
{
  "content": [{
    "type": "text",
    "text": "Moved up. New board state: ..."
  }]
}

​
Lifecycle Management

MCP defines a rigorous lifecycle for connections:
  1. Initialization: Client and server negotiate capabilities and protocol version with client.initialize()
  2. Operation: Normal tool calling and message exchange
  3. Shutdown: Clean termination of the connection
The protocol ensures both sides understand each other’s capabilities before proceeding.

​
HUD’s MCP Extensions

HUD adds conventions on top of MCP:
  1. Setup Tools: Initialize environment state (setup_board, navigate_to_url)
  2. Evaluate Tools: Score agent performance (evaluate_max_tile, contains_text)
  3. Lifecycle Management: Clean initialization and shutdown with client.initialize() and proper cleanup
See the Tools Reference for implementation details.

​
Transport Options

HUD environments are designed for 100% reproducibility through Docker:
Run environments locally for development and debugging using stdio transport:
mcp_config = {
  "hud-text-2048": {
    "command": "docker",
    "args": ["run", "--rm", "-i", "hudpython/hud-text-2048:v1.2"]
  }
}
  • Transport: stdio - JSON-RPC over stdin/stdout
  • Pros: Full control, easy debugging, no network latency
  • Use case: Development, testing, single-agent runs
The -i flag enables Docker’s interactive mode, allowing stdio communication between the client and server process.
Both approaches use the exact same Docker image, ensuring identical behavior whether running locally or remotely

​
Next Steps

Environment Spec

See the full HUD environment specification