AppMaker MCP

An MCP server that lets agents see and control native GUI applications inside an isolated, nested X11 display.

The agent gets its own private desktop. It launches apps there, takes screenshots, moves the pointer, types, drags, scrolls — without touching your real session, and without the apps ever knowing they aren't on a real screen.

Index

• Why
• Features
• Quick start
• Tools
• How it works
• Limitations
• Contributing
• License

Why

LLM agents are great at reasoning about UIs, but they need a sandbox to act in. AppMaker MCP gives them one: a nested X11 display where they can spawn arbitrary apps, observe the pixels, and drive input — all isolated from your real desktop, all reversible by killing one process.

Features

• Isolated display. Each session runs inside its own Xephyr window. Tearing it down kills every app spawned in it.
• Visible by default. You can watch the agent work in real time. A software cursor makes pointer movement easy to follow.
• Real input, paced. Mouse movement is interpolated over time and clicks honor configurable hold/pre-press delays, so apps see input the way a human would produce it.
• Screenshots with cursor markers. Optional synthetic cursor overlay shows where the pointer is and whether a button is currently held.
• Window awareness. List, focus, and kill top-level windows by ID or title.
• Opaque interface. Tools speak in displays, coordinates, and window titles — no leaking xdotool or Xephyr flags into agent prompts.

Quick start

Prerequisites

Linux with the following packages available on PATH:

• Xephyr (usually packaged as xserver-xephyr)
• fluxbox
• xdotool
• scrot or ImageMagick's import

On Debian/Ubuntu:

sudo apt install xserver-xephyr fluxbox xdotool scrot

Install

cargo install app-maker-mcp

Build

git clone https://github.com/mii-nipah/app-maker-mcp
cd app-maker-mcp
cargo build --release

The binary lands at target/release/app-maker-mcp. It speaks MCP over stdio.

Wire it up to your MCP client

Example configuration snippet:

{
  "mcpServers": {
    "app-maker": {
      "command": "app-maker-mcp"
    }
  }
}

First session

From the agent's side, a typical flow looks like this:

1. appmaker-start — spin up a nested display (returns something like :99).
2. appmaker-run with command: "xterm" — launch an app inside it.
3. appmaker-screenshot with show_cursor: true — see what's on screen.
4. appmaker-send-click, appmaker-send-keyboard, ... — interact.
5. appmaker-stop — tear everything down.

Tools

Every motion-bearing tool exposes timing knobs (move duration, pre-click delay, hold duration, ...) so the agent can match an app's input expectations.

How it works

┌──────────────────┐    stdio (MCP)    ┌──────────────────────┐
│   MCP client     │ ◄───────────────► │   app-maker-mcp      │
│ (agent / IDE)    │                   │  (Rust, this crate)  │
└──────────────────┘                   └──────────┬───────────┘
                                                  │ spawns + supervises
                                                  ▼
                              ┌──────────────────────────────────┐
                              │  Xephyr :N  (nested X11 server)  │
                              │  ├── fluxbox (window manager)    │
                              │  └── user apps (DISPLAY=:N)      │
                              └──────────────────────────────────┘
                                                  ▲
                                  xdotool / scrot │ input + screenshots

AppSessionManager owns the Xephyr + fluxbox lifecycle and tracks every child process spawned through appmaker-run, so appmaker-stop can reliably reap the whole tree. Pointer movement is interpolated in small steps to produce realistic motion, and the screenshot path optionally rasterizes a synthetic cursor (idle vs. pressed style) onto PNG output.

Limitations

• Linux + X11 only. Pure Wayland-only apps won't run. Most toolkits can be coaxed into X11 mode (QT_QPA_PLATFORM=xcb, GDK_BACKEND=x11, --ozone-platform=x11).
• Visible nested window. Xephyr opens a window on your real display. That's by design — humans need to be able to watch — but it means there is no headless mode yet.
• One session per server process. The current AppSessionManager manages a single nested display at a time.

Contributing

Issues and PRs welcome. A few guidelines:

• Keep tool surfaces opaque. Agents shouldn't need to learn about xdotool flags to use a tool — translate at the boundary.
• Prefer adding timing knobs over hard-coding behavior; different apps need different pacing.
• Run cargo fmt and cargo clippy before sending a PR.

For larger changes, open an issue first to discuss the design.

License

BSD-3-Clause. See LICENSE.