Orbit documentation Open the editor →

Drive Orbit with an AI agent (MCP)

Orbit speaks MCP (the Model Context Protocol), so an external AI agent — Claude Code, Copilot, anything that speaks MCP — can drive the editor: scout the world, author whole scenes, seek to any moment and see the frame, iterate, validate, and export a playable .mctemplate / .mcworld / .mcaddon. Where Quarry's MCP makes the agent a builder, Orbit's makes it a cinematographer.

Everything the agent does is one undoable op labelled "MCP: …" in the History panel, and Orbit only ever reads the world — the project document is the only thing an agent can change.

How it works

Orbit runs entirely in a browser tab, and browsers can't accept incoming connections. So a tiny Node process bridges your agent (over stdio) to the tab (over a local WebSocket):

agent  ⇄ (MCP/stdio) ⇄  mcp/server.mjs  ⇄ (ws://127.0.0.1:7332) ⇄  Orbit tab

Setup (Local)

  1. Register the server with your agent (one time). From an Orbit checkout:

    claude mcp add orbit -- node /path/to/orbit/mcp/server.mjs

    The server defaults to port 7332 and the ORBIT_MCP_* env prefix, so it can run alongside Quarry's bridge (7331 / QUARRY_MCP_*) — one agent session can build a set in Quarry and film it in Orbit.

  2. Connect the tab. In Orbit, open the menu → AI agent (MCP)… → Local. The server prints a pairing token to its console on first run; paste it into the panel and click Connect. The status dot turns green when paired.

  3. Talk to your agent. Ask it to film something — it has the tools below.

Security. Local mode binds to 127.0.0.1, checks the browser Origin, and requires the pairing token. It's user-opted-in convenience, not a hardened sandbox — but every edit is undoable and the world itself is read-only, so an agent can't do anything you can't undo.

Setup (Cloud relay)

When the agent and the Orbit tab are on different machines — or you're using the deployed site at orbit.refrag.com, where a loopback WebSocket isn't reachable — pair through a small Cloudflare relay Worker instead. It's a dumb two-party router: the tab and the Node MCP server each connect out to a room and the relay forwards frames between them.

agent ⇄ (stdio) ⇄ orbit-mcp.mjs ⇄ (wss) ⇄ [ relay Worker ] ⇄ (wss) ⇄ Orbit tab

Deploy the relay (one time, to your own Cloudflare account — needs the Workers plan for Durable Objects):

npx wrangler login
npm run relay:deploy        # deploys party/server.ts → <name>.<acct>.workers.dev

The relay runs in one of two modes:

Identity mode (recommended) — sign in with GitHub

Tie pairing to a GitHub sign-in: your room becomes mcp-<your-github-id> and your agent gets a stable personal token you configure once (no per-session room/token). Set it up:

  1. Register a GitHub OAuth App (any user can sign in; it's identity-only):
    • Homepage: https://orbit.refrag.com
    • Authorization callback URL: https://orbit-mcp-relay.<account>.workers.dev/auth/callback
  2. Put the Client ID in wrangler.toml (GITHUB_CLIENT_ID), then set the secrets:
    npx wrangler secret put GITHUB_CLIENT_SECRET   # from the OAuth app
npx wrangler secret put SESSION_SECRET         # any long random string
npm run relay:deploy

Then in Orbit → AI agent (MCP)… → Cloud relay → Sign in with GitHub. The panel shows a ready-to-paste snippet with your personal token baked in — run it once and your agent is configured for good. Connect pairs this tab. Rotate the token anytime with the panel's Rotate token button (revokes the old one).

Ad-hoc mode — no sign-in

If you don't set SESSION_SECRET, the relay pairs by a generated room + token instead (trust-on-first-use: the first peer to present a token locks the room). The panel's Cloud relay tab generates both for you and shows the snippet — copy, paste, Connect. Keep the token private; it's the only thing gating the room.

Test the relay locally without deploying: npm run relay:dev (Worker on :8787), then npm run mcp:relay-verify (ad-hoc) or, with a SESSION_SECRET in .dev.vars, SESSION_SECRET=… npm run mcp:relay-auth-verify (identity).

The filming loop

The server teaches the agent this loop (and get_cinematography_guide returns the full shot grammar + the exact Scene JSON shape):

  1. Orient — project_info, then world_overview / surface_scan to scout the location and learn the terrain line a crane shot must clear.
  2. Author in bulk — design the whole scene, then land it with one upsert_scene (validated, one undoable op). Fine-grained set_* tools are for iterating on a single thing afterward.
  3. See the work — contact_sheet renders N frames across the scene into one labelled grid image (one picture ≈ reviewing the whole cutscene); seek + screenshot inspect a single moment. Because preview and the exported runtime share identical easing/evaluation math, what the agent sees is what ships.
  4. Pace — markers for story beats (cue:true fires /scriptevent orbit:marker <name>), the BPM grid when reference audio is bound.
  5. Ship — validate_project, resolve warnings, then export.

Tool catalogue

Orient — project_info, get_scene, list_profiles, get_cinematography_guide, world_info, world_overview, surface_scan, locate_blocks.

Author — upsert_scene (the workhorse), delete_scene, set_project_settings, rename_project, add_camera_track, set_camera_keyframe, set_cut, set_env_keyframe, set_effect, set_marker, set_actor, set_actor_keyframe, delete_item, retime, undo, redo, get_history.

See — seek, screenshot, contact_sheet, set_view, get_view, look_at, play_preview.

Ship — validate_project, export, save_project, load_project.

Try it

Once connected, ask your agent something like:

"Scout this world, film a 12-second establishing orbit of the village with a title card, show me a contact sheet, then export a .mcworld."

It will orient with world_overview, author the shot with upsert_scene, review it with contact_sheet, and hand you back a playable file — with every step visible and undoable in the History panel.