Run Northstar locally on a Mac

Sometimes you want to run computer-use agents locally, for example on a MacBook. We publish a Mac driver you build on top of, and an MLX build of Northstar on Hugging Face. Together they give you fully local execution. The two building blocks:

	What it is	License
MacControlKit	A Swift package (and maccontrol CLI) that screenshots and controls a Mac: mouse, keyboard, app launching, permissions. It has no model and no agent loop.	Apache 2.0
Northstar CUA Fast 1.6, MLX 4-bit	A 4-bit MLX build of Northstar 1.6 (~3 GB) that runs on-device via mlx-vlm.	Apache 2.0

MacControlKit controls the screen; the MLX model decides what to do. You write the loop that connects them, the same computer-use loop as the cloud API, run locally.

MLX runs on Apple Silicon only. MacControlKit requires macOS 15.2 or later.

Install both

# The local model runtime (Python)
pip install -U "mlx-vlm>0.6.4"   # 0.6.4 and earlier mis-load this model's weights

# The control layer (Swift): clone and build the CLI
git clone https://github.com/tzafon/MacControlKit.git
cd MacControlKit && swift build

Grant the built binary Screen Recording and Accessibility permissions once:

.build/debug/maccontrol request-permissions
.build/debug/maccontrol permissions   # confirm both are granted

A note on coordinates

Both pieces use a top-left-origin normalized grid, but not the same range. maccontrol and MacControlKit’s NormalizedPoint use 0–999 (the same as the cloud API). The MLX model’s card documents its grid as 0–1000. The difference is rounding: dividing a model coordinate by 1000 instead of 999 shifts a click by under a pixel on any real display. Divide by 1000 and pass the result to maccontrol.

The loop

The model speaks the same XML tool-call format (training_xml / computer_use) as the hosted tzafon.northstar-cua-fast-1.6. The loop is: screenshot, ask the model for one action, parse the XML, run it with maccontrol, repeat.

local_agent.py

import re
import subprocess

from mlx_vlm import generate, load
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

MACCONTROL = "/path/to/MacControlKit/.build/debug/maccontrol"
MODEL_PATH = "Tzafon/Northstar-CUA-Fast-1.6-MLX-4bit"

COMPUTER_USE_TOOL = {
    "type": "function",
    "function": {
        "name": "computer_use",
        "description": "Perform one action on the screen.",
        "parameters": {
            "type": "object",
            "properties": {
                "action": {"type": "string", "enum": ["left_click", "type", "key"]},
                "coordinate": {"type": "array", "items": {"type": "integer"}},
                "text": {"type": "string"},
            },
            "required": ["action"],
        },
    },
}

model, processor = load(MODEL_PATH)
config = load_config(MODEL_PATH)


def screenshot() -> str:
    path = "/tmp/screen.jpg"
    subprocess.run([MACCONTROL, "screenshot", path], check=True)
    return path


def next_action(image_path: str, instruction: str) -> str:
    prompt = apply_chat_template(
        processor, config, instruction, num_images=1, tools=[COMPUTER_USE_TOOL]
    )
    return generate(
        model, processor, prompt, [image_path],
        temp=0.0,       # deterministic decoding; the model card requires it
        verbose=False,
    )


def to_px(coord: int) -> float:
    return coord / 1000  # model's 0-1000 grid -> maccontrol's 0-999 normalized space


def run_action(xml: str) -> None:
    action = re.search(r"<parameter=action>\s*(\w+)\s*</parameter>", xml).group(1)
    coord = re.search(r"<parameter=coordinate>\s*\[(\d+),\s*(\d+)\]\s*</parameter>", xml)

    if action == "left_click" and coord:
        x, y = to_px(int(coord.group(1))), to_px(int(coord.group(2)))
        subprocess.run([MACCONTROL, "click", str(x), str(y)], check=True)
    elif action == "type":
        text = re.search(r"<parameter=text>\s*(.*?)\s*</parameter>", xml, re.S).group(1)
        subprocess.run([MACCONTROL, "type", text], check=True)
    elif action == "key":
        key = re.search(r"<parameter=text>\s*(.*?)\s*</parameter>", xml, re.S).group(1)
        subprocess.run([MACCONTROL, "key", key], check=True)


instruction = "Open TextEdit and type 'hello from Northstar'"
for _ in range(10):
    xml = next_action(screenshot(), instruction)
    print(xml)
    if "<tool_call>" not in xml:
        break  # model answered in plain text instead of calling the tool; treat as done
    run_action(xml)

• Pin mlx-vlm>0.6.4. Versions ≤0.6.4 double-apply this architecture’s RMSNorm weight shift on already-converted weights and produce garbage output.
• Use deterministic decoding (temp=0.0). The model was trained to click precisely at temperature 0; sampling makes clicks drift.
• Don’t force JSON output. The model was trained on the XML tool-call format above. Constraining generation to a JSON grammar pushes it off that distribution and degrades click accuracy. Parse the XML as shown, not JSON.

Building a fully native Swift app

Since MacControlKit is a Swift package, a fully native app (no Python subprocess) is possible by loading the same MLX weights directly in Swift via mlx-swift-lm (MLXVLM) instead of mlx-vlm. Both packages can sit in the same Package.swift alongside MacControlKit. mlx-swift-lm’s own VLM documentation covers loading a specific Hugging Face checkpoint and running vision-language inference. The loop is identical to the Python version above: screenshot via MacComputer.screenshot(), generate one action, perform(_:) it, repeat.

Local vs. cloud

	Local (this page)	Cloud (Computers / Tasks)
Setup	Install two packages, grant permissions	One API key
Network	None required after model download	Every action is a network call
Cost	Free, your own compute	Metered, see pricing
Scale	One Mac, one session	Any number of parallel browser/desktop sessions
Platforms	macOS (Apple Silicon) only	Linux browser and desktop, from anywhere
Loop	You write it (as above)	Managed for you via Tasks, or build your own via Responses
Observability	Whatever you build	Live view, screencast, replay built in

Use local when you want zero network dependency, zero per-run cost, and control stays entirely on one machine. Use the cloud API when you need scale, uptime beyond your own Mac, or a managed agent loop.

Next steps

• MacControlKit on GitHub: full API reference and DocC guides
• Northstar CUA Fast 1.6 MLX 4-bit on Hugging Face: model card and files
• Coordinates: the cloud API’s coordinate system, for comparison
• Computer-use loop: the same act/observe pattern, against a cloud computer