---
title: Verified structured extraction from screens | Lightcone
description: Read data off a legacy UI as strict JSON with a confidence score, and verify every extraction before acting on it.
---

Legacy enterprise apps render data as pixels, not payloads: search-result rows, form fields, status banners. To build anything reliable on top, you need to turn those pixels into structured data you can trust. The pattern is simple and strict: ask the model for one thing: a JSON answer, no tools, with a confidence score. Then verify it against what you already know before acting on it.

This is the read-back layer used by [Wrap a legacy app in a REST API](/cookbook/wrap-a-legacy-app-in-an-api/index.md) for its search and verify steps.

## The `ask_screen` helper

Send one screenshot plus one question through the [Responses API](/guides/responses-api/index.md), deliberately **without** the `computer_use` tool. With no tools available, the model can’t wander off clicking; its only possible output is a text answer.

extract.py

````
import asyncio
import json
import re


from tzafon import AsyncLightcone


MODEL = "tzafon.northstar-cua-fast-1.6"   # pin a version in production
# Illustrative gate. Calibrate against a labeled sample of your own screens;
# the right value depends on the model, the UI, and your tolerance for review.
MIN_CONFIDENCE = 0.6




async def ask_screen(client: AsyncLightcone, image_url: str, question: str) -> str:
    """One screenshot, one question, text-only answer. No tools on purpose."""
    resp = await asyncio.wait_for(
        client.responses.create(
            model=MODEL,
            input=[{
                "role": "user",
                "content": [
                    {"type": "input_image", "image_url": image_url, "detail": "auto"},
                    {"type": "input_text", "text": question},
                ],
            }],
        ),
        timeout=45,
    )
    return resp.output_text




def json_from_model_text(text: str) -> dict:
    """Models love ```json fences. Strip them before parsing."""
    stripped = text.strip()
    if stripped.startswith("```"):
        stripped = re.sub(r"^```(?:json)?\s*", "", stripped, flags=re.I)
        stripped = re.sub(r"\s*```$", "", stripped)
    try:
        data = json.loads(stripped)
    except json.JSONDecodeError:
        match = re.search(r"\{.*\}", stripped, flags=re.S)   # salvage embedded JSON
        if not match:
            raise ValueError(f"model did not return JSON: {text!r}")
        data = json.loads(match.group(0))
    if not isinstance(data, dict):
        raise ValueError(f"model JSON is not an object: {data!r}")
    return data
````

Fence-stripping and regex-salvaging JSON out of prose is a fallback. When the endpoint supports it, pass a JSON schema via `response_format` so the model is constrained to valid, typed output and you skip the salvage step entirely. See [Chat Completions](/guides/chat-completions#key-parameters/index.md). The parser above is the belt-and-suspenders path for models or calls where you can’t set a schema.

## Prompt for an exact JSON shape

Vague prompts produce prose. Dictate the exact shape, allow nulls, demand a confidence field, and give one worked example of the UI’s quirks (legacy apps often display “Last First Date”, not “First Last”):

extract.py (prompt)

```
FIRST_ROW_PROMPT = (
    "You are looking at a record search screen. Read ONLY the first visible "
    "row inside the results list in the upper-right panel. Ignore the search "
    "input fields, filters, and labels outside that row. Return ONLY one JSON "
    "object with this exact shape:\n"
    '{"visible": boolean, "row_text": string|null, '
    '"first_name": string|null, "last_name": string|null, '
    '"birth_date": "YYYY-MM-DD"|null, "confidence": number}\n'
    "Result rows display as LAST FIRST DATE. For example, a row reading "
    '"Smith John 2000-Jan-01" means {"first_name":"John","last_name":"Smith",'
    '"birth_date":"2000-01-01"}. If no result row is visible, set '
    "visible=false and all text fields to null. Do not infer a row from the "
    "search filters."
)
```

The `"Do not infer..."` line earns its keep: without it, models will helpfully “extract” the name you just typed into the search box and report it as a result row.

## Verify before acting

The extraction is an *input* to a decision, never the decision itself. Before clicking a search result, require that the extracted row exactly matches what you searched for, at sufficient confidence. Otherwise fail the step for human review rather than act on the wrong record:

extract.py (verify)

```
def _norm(value) -> str:
    return re.sub(r"[^a-z0-9]+", "", str(value or "").casefold())




async def select_verified_result(client, cid, first_name, last_name, birth_date):
    """Click the first search result only if it is visibly the exact record."""
    shot = await client.computers.screenshot(cid)
    row = json_from_model_text(
        await ask_screen(client, shot.result.screenshot_url, FIRST_ROW_PROMPT)
    )
    try:
        confidence = float(row.get("confidence") or 0)
    except (TypeError, ValueError):
        confidence = 0.0


    matches = (
        row.get("visible")
        and _norm(row.get("first_name")) == _norm(first_name)
        and _norm(row.get("last_name")) == _norm(last_name)
        and row.get("birth_date") == birth_date
    )
    if confidence < MIN_CONFIDENCE or not matches:
        return False, f"first result not verified as exact match; extracted={row!r}"


    await client.computers.click(cid, x=880, y=176)   # measured first-row position
    return True, ""
```

Note what this deliberately does *not* do: guess which row index holds the right record. If the exact match isn’t the **first visible** row, the step fails with the extracted row attached to the error: cheap to debug, impossible to act on the wrong record. Tighten your search query rather than loosen the check.

Never let a fuzzy match through because “the confidence was high”. Confidence gates *readability* (was the row legible?); the exact-match comparison gates *identity* (is it the right record?). You need both.

## Two-phase trick: explore with tools, extract without

Sometimes the data is not on one screen. It is behind a tab, a scroll, or a detail panel. Resist the urge to run one big tool-enabled agent that both navigates *and* answers: with tools available, models often report from memory mid-navigation, and you can’t tell which screenshot the answer came from.

Instead, split the phases:

1. **Explore with tools.** Run a bounded computer-use loop whose only goal is navigation: “Open the detail view for the selected record. Stop when the detail panel is visible.” (See the bounded-recovery loop in [Wrap a legacy app in a REST API](/cookbook/wrap-a-legacy-app-in-an-api/index.md); same shape, same `max_steps` budget.)
2. **Extract without tools.** Take a fresh screenshot and run `ask_screen` with a strict JSON prompt, exactly as above.

extract.py (two-phase)

```
await run_bounded_loop(          # phase 1: tools on, answer off
    client, cid,
    "Open the detail view for the currently selected record. "
    "Stop when the detail panel is fully visible. Do not modify anything.",
    max_steps=6,
)
shot = await client.computers.screenshot(cid)
detail = json_from_model_text(   # phase 2: tools off, JSON only
    await ask_screen(client, shot.result.screenshot_url, DETAIL_PROMPT)
)
```

The boundary gives you an auditable artifact: the exact screenshot the JSON was extracted from. Log both together, which makes every wrong extraction diagnosable in seconds. See [Observability](/guides/observability/index.md) for wiring that into request traces.

## What you learned

1. **`ask_screen`**: one screenshot, one question, via the Responses API with **no tools**, so the only possible output is a text answer.
2. **Strict JSON prompts** with an exact shape, nulls, a confidence field, and a worked example of the UI’s display quirks, plus fence-stripping before `json.loads`.
3. **Verify before acting**: exact-match the extraction against the query and require a minimum confidence (calibrate the threshold on your own screens); fail closed instead of guessing.
4. **Two-phase explore/extract**: navigate with a bounded tool loop, then extract tool-free from a known screenshot.

## Next steps

- [**Wrap a legacy app in a REST API**](/cookbook/wrap-a-legacy-app-in-an-api/index.md): where these verified reads gate every success response
- [**Warm sessions and self-healing**](/cookbook/warm-sessions-and-self-healing/index.md): keeping the desktop these screenshots come from alive
- [**Responses API**](/guides/responses-api/index.md): the full request/response reference
- [**Production**](/guides/production/index.md): deployment patterns for services built on these recipes
- [**Observability**](/guides/observability/index.md): logging screenshots alongside extractions