Verified structured extraction from screens

Legacy enterprise apps render data as pixels, not payloads: search-result rows, form fields, status banners. To build anything reliable on top, you need to turn those pixels into structured data you can trust. The pattern is simple and strict: ask the model for one thing: a JSON answer, no tools, with a confidence score. Then verify it against what you already know before acting on it.

This is the read-back layer used by Wrap a legacy app in a REST API for its search and verify steps.

The ask_screen helper

Send one screenshot plus one question through the Responses API, deliberately without the computer_use tool. With no tools available, the model can’t wander off clicking; its only possible output is a text answer.

extract.py

import asyncio
import json
import re

from tzafon import AsyncLightcone

MODEL = "tzafon.northstar-cua-fast-1.6"   # pin a version in production
# Illustrative gate. Calibrate against a labeled sample of your own screens;
# the right value depends on the model, the UI, and your tolerance for review.
MIN_CONFIDENCE = 0.6


async def ask_screen(client: AsyncLightcone, image_url: str, question: str) -> str:
    """One screenshot, one question, text-only answer. No tools on purpose."""
    resp = await asyncio.wait_for(
        client.responses.create(
            model=MODEL,
            input=[{
                "role": "user",
                "content": [
                    {"type": "input_image", "image_url": image_url, "detail": "auto"},
                    {"type": "input_text", "text": question},
                ],
            }],
        ),
        timeout=45,
    )
    return resp.output_text


def json_from_model_text(text: str) -> dict:
    """Models love ```json fences. Strip them before parsing."""
    stripped = text.strip()
    if stripped.startswith("```"):
        stripped = re.sub(r"^```(?:json)?\s*", "", stripped, flags=re.I)
        stripped = re.sub(r"\s*```$", "", stripped)
    try:
        data = json.loads(stripped)
    except json.JSONDecodeError:
        match = re.search(r"\{.*\}", stripped, flags=re.S)   # salvage embedded JSON
        if not match:
            raise ValueError(f"model did not return JSON: {text!r}")
        data = json.loads(match.group(0))
    if not isinstance(data, dict):
        raise ValueError(f"model JSON is not an object: {data!r}")
    return data

Fence-stripping and regex-salvaging JSON out of prose is a fallback. When the endpoint supports it, pass a JSON schema via response_format so the model is constrained to valid, typed output and you skip the salvage step entirely. See Chat Completions. The parser above is the belt-and-suspenders path for models or calls where you can’t set a schema.

Prompt for an exact JSON shape

Vague prompts produce prose. Dictate the exact shape, allow nulls, demand a confidence field, and give one worked example of the UI’s quirks (legacy apps often display “Last First Date”, not “First Last”):

extract.py (prompt)

FIRST_ROW_PROMPT = (
    "You are looking at a record search screen. Read ONLY the first visible "
    "row inside the results list in the upper-right panel. Ignore the search "
    "input fields, filters, and labels outside that row. Return ONLY one JSON "
    "object with this exact shape:\n"
    '{"visible": boolean, "row_text": string|null, '
    '"first_name": string|null, "last_name": string|null, '
    '"birth_date": "YYYY-MM-DD"|null, "confidence": number}\n'
    "Result rows display as LAST FIRST DATE. For example, a row reading "
    '"Smith John 2000-Jan-01" means {"first_name":"John","last_name":"Smith",'
    '"birth_date":"2000-01-01"}. If no result row is visible, set '
    "visible=false and all text fields to null. Do not infer a row from the "
    "search filters."
)

The "Do not infer..." line earns its keep: without it, models will helpfully “extract” the name you just typed into the search box and report it as a result row.

Verify before acting

The extraction is an input to a decision, never the decision itself. Before clicking a search result, require that the extracted row exactly matches what you searched for, at sufficient confidence. Otherwise fail the step for human review rather than act on the wrong record:

extract.py (verify)

def _norm(value) -> str:
    return re.sub(r"[^a-z0-9]+", "", str(value or "").casefold())


async def select_verified_result(client, cid, first_name, last_name, birth_date):
    """Click the first search result only if it is visibly the exact record."""
    shot = await client.computers.screenshot(cid)
    row = json_from_model_text(
        await ask_screen(client, shot.result.screenshot_url, FIRST_ROW_PROMPT)
    )
    try:
        confidence = float(row.get("confidence") or 0)
    except (TypeError, ValueError):
        confidence = 0.0

    matches = (
        row.get("visible")
        and _norm(row.get("first_name")) == _norm(first_name)
        and _norm(row.get("last_name")) == _norm(last_name)
        and row.get("birth_date") == birth_date
    )
    if confidence < MIN_CONFIDENCE or not matches:
        return False, f"first result not verified as exact match; extracted={row!r}"

    await client.computers.click(cid, x=880, y=176)   # measured first-row position
    return True, ""

Note what this deliberately does not do: guess which row index holds the right record. If the exact match isn’t the first visible row, the step fails with the extracted row attached to the error: cheap to debug, impossible to act on the wrong record. Tighten your search query rather than loosen the check.

Never let a fuzzy match through because “the confidence was high”. Confidence gates readability (was the row legible?); the exact-match comparison gates identity (is it the right record?). You need both.

Two-phase trick: explore with tools, extract without

Sometimes the data is not on one screen. It is behind a tab, a scroll, or a detail panel. Resist the urge to run one big tool-enabled agent that both navigates and answers: with tools available, models often report from memory mid-navigation, and you can’t tell which screenshot the answer came from.

Instead, split the phases:

1. Explore with tools. Run a bounded computer-use loop whose only goal is navigation: “Open the detail view for the selected record. Stop when the detail panel is visible.” (See the bounded-recovery loop in Wrap a legacy app in a REST API; same shape, same max_steps budget.)
2. Extract without tools. Take a fresh screenshot and run ask_screen with a strict JSON prompt, exactly as above.

extract.py (two-phase)

await run_bounded_loop(          # phase 1: tools on, answer off
    client, cid,
    "Open the detail view for the currently selected record. "
    "Stop when the detail panel is fully visible. Do not modify anything.",
    max_steps=6,
)
shot = await client.computers.screenshot(cid)
detail = json_from_model_text(   # phase 2: tools off, JSON only
    await ask_screen(client, shot.result.screenshot_url, DETAIL_PROMPT)
)

The boundary gives you an auditable artifact: the exact screenshot the JSON was extracted from. Log both together, which makes every wrong extraction diagnosable in seconds. See Observability for wiring that into request traces.

What you learned

1. ask_screen: one screenshot, one question, via the Responses API with no tools, so the only possible output is a text answer.
2. Strict JSON prompts with an exact shape, nulls, a confidence field, and a worked example of the UI’s display quirks, plus fence-stripping before json.loads.
3. Verify before acting: exact-match the extraction against the query and require a minimum confidence (calibrate the threshold on your own screens); fail closed instead of guessing.
4. Two-phase explore/extract: navigate with a bounded tool loop, then extract tool-free from a known screenshot.

Next steps

• Wrap a legacy app in a REST API: where these verified reads gate every success response
• Warm sessions and self-healing: keeping the desktop these screenshots come from alive
• Responses API: the full request/response reference
• Production: deployment patterns for services built on these recipes
• Observability: logging screenshots alongside extractions