--- title: Run Northstar locally on a Mac | Lightcone description: Run computer-use agents on-device with MacControlKit and the Northstar MLX model. --- Sometimes you want to run computer-use agents locally, for example on a MacBook. We publish a Mac driver you build on top of, and an MLX build of Northstar on Hugging Face. Together they give you fully local execution. The two building blocks: | | What it is | License | | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | **[MacControlKit](https://github.com/tzafon/MacControlKit)** | A Swift package (and `maccontrol` CLI) that screenshots and controls a Mac: mouse, keyboard, app launching, permissions. It has no model and no agent loop. | Apache 2.0 | | **[Northstar CUA Fast 1.6, MLX 4-bit](https://huggingface.co/Tzafon/Northstar-CUA-Fast-1.6-MLX-4bit)** | A 4-bit MLX build of Northstar 1.6 (\~3 GB) that runs on-device via [mlx-vlm](https://github.com/Blaizzy/mlx-vlm). | Apache 2.0 | MacControlKit controls the screen; the MLX model decides what to do. You write the loop that connects them, the same [computer-use loop](/guides/cua-protocol/index.md) as the cloud API, run locally. MLX runs on Apple Silicon only. MacControlKit requires macOS 15.2 or later. ## Install both Terminal window ``` # The local model runtime (Python) pip install -U "mlx-vlm>0.6.4" # 0.6.4 and earlier mis-load this model's weights # The control layer (Swift): clone and build the CLI git clone https://github.com/tzafon/MacControlKit.git cd MacControlKit && swift build ``` Grant the built binary Screen Recording and Accessibility permissions once: Terminal window ``` .build/debug/maccontrol request-permissions .build/debug/maccontrol permissions # confirm both are granted ``` ## A note on coordinates Both pieces use a top-left-origin normalized grid, but not the same range. `maccontrol` and `MacControlKit`’s `NormalizedPoint` use **0–999** (the same as the [cloud API](/guides/coordinates/index.md)). The MLX model’s card documents its grid as **0–1000**. The difference is rounding: dividing a model coordinate by 1000 instead of 999 shifts a click by under a pixel on any real display. Divide by 1000 and pass the result to `maccontrol`. ## The loop The model speaks the same XML tool-call format (`training_xml` / `computer_use`) as the hosted `tzafon.northstar-cua-fast-1.6`. The loop is: screenshot, ask the model for one action, parse the XML, run it with `maccontrol`, repeat. local\_agent.py ``` import re import subprocess from mlx_vlm import generate, load from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config MACCONTROL = "/path/to/MacControlKit/.build/debug/maccontrol" MODEL_PATH = "Tzafon/Northstar-CUA-Fast-1.6-MLX-4bit" COMPUTER_USE_TOOL = { "type": "function", "function": { "name": "computer_use", "description": "Perform one action on the screen.", "parameters": { "type": "object", "properties": { "action": {"type": "string", "enum": ["left_click", "type", "key"]}, "coordinate": {"type": "array", "items": {"type": "integer"}}, "text": {"type": "string"}, }, "required": ["action"], }, }, } model, processor = load(MODEL_PATH) config = load_config(MODEL_PATH) def screenshot() -> str: path = "/tmp/screen.jpg" subprocess.run([MACCONTROL, "screenshot", path], check=True) return path def next_action(image_path: str, instruction: str) -> str: prompt = apply_chat_template( processor, config, instruction, num_images=1, tools=[COMPUTER_USE_TOOL] ) return generate( model, processor, prompt, [image_path], temp=0.0, # deterministic decoding; the model card requires it verbose=False, ) def to_px(coord: int) -> float: return coord / 1000 # model's 0-1000 grid -> maccontrol's 0-999 normalized space def run_action(xml: str) -> None: action = re.search(r"\s*(\w+)\s*", xml).group(1) coord = re.search(r"\s*\[(\d+),\s*(\d+)\]\s*", xml) if action == "left_click" and coord: x, y = to_px(int(coord.group(1))), to_px(int(coord.group(2))) subprocess.run([MACCONTROL, "click", str(x), str(y)], check=True) elif action == "type": text = re.search(r"\s*(.*?)\s*", xml, re.S).group(1) subprocess.run([MACCONTROL, "type", text], check=True) elif action == "key": key = re.search(r"\s*(.*?)\s*", xml, re.S).group(1) subprocess.run([MACCONTROL, "key", key], check=True) instruction = "Open TextEdit and type 'hello from Northstar'" for _ in range(10): xml = next_action(screenshot(), instruction) print(xml) if "" not in xml: break # model answered in plain text instead of calling the tool; treat as done run_action(xml) ``` - **Pin `mlx-vlm>0.6.4`.** Versions ≤0.6.4 double-apply this architecture’s RMSNorm weight shift on already-converted weights and produce garbage output. - **Use deterministic decoding** (`temp=0.0`). The model was trained to click precisely at temperature 0; sampling makes clicks drift. - **Don’t force JSON output.** The model was trained on the XML tool-call format above. Constraining generation to a JSON grammar pushes it off that distribution and degrades click accuracy. Parse the XML as shown, not JSON. ## Building a fully native Swift app Since MacControlKit is a Swift package, a fully native app (no Python subprocess) is possible by loading the same MLX weights directly in Swift via [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) (`MLXVLM`) instead of `mlx-vlm`. Both packages can sit in the same `Package.swift` alongside `MacControlKit`. mlx-swift-lm’s own [VLM documentation](https://swiftpackageindex.com/ml-explore/mlx-swift-lm/main/documentation/mlxvlm) covers loading a specific Hugging Face checkpoint and running vision-language inference. The loop is identical to the Python version above: screenshot via `MacComputer.screenshot()`, generate one action, `perform(_:)` it, repeat. ## Local vs. cloud | | Local (this page) | Cloud ([Computers](/guides/computers/index.md) / [Tasks](/guides/tasks/index.md)) | | ------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Setup | Install two packages, grant permissions | One API key | | Network | None required after model download | Every action is a network call | | Cost | Free, your own compute | Metered, see [pricing](/guides/pricing/index.md) | | Scale | One Mac, one session | Any number of parallel browser/desktop sessions | | Platforms | macOS (Apple Silicon) only | Linux browser and desktop, from anywhere | | Loop | You write it (as above) | Managed for you via [Tasks](/guides/tasks/index.md), or build your own via [Responses](/guides/responses-api/index.md) | | Observability | Whatever you build | [Live view, screencast, replay](/guides/observability/index.md) built in | Use local when you want zero network dependency, zero per-run cost, and control stays entirely on one machine. Use the cloud API when you need scale, uptime beyond your own Mac, or a managed agent loop. ## Next steps - [MacControlKit on GitHub](https://github.com/tzafon/MacControlKit): full API reference and DocC guides - [Northstar CUA Fast 1.6 MLX 4-bit on Hugging Face](https://huggingface.co/Tzafon/Northstar-CUA-Fast-1.6-MLX-4bit): model card and files - [Coordinates](/guides/coordinates/index.md): the cloud API’s coordinate system, for comparison - [Computer-use loop](/guides/cua-protocol/index.md): the same act/observe pattern, against a cloud computer