Warm sessions and self-healing

Logging into a legacy enterprise app behind SSO can take minutes: username, password, TOTP, then a slow app launch. You do not want to pay that on every request, or every deploy. The pattern that works in production is a single warm session: one authenticated desktop that outlives your service process, gets adopted back on restart, is actively probed for liveness, and heals itself when it breaks.

This is the session layer underneath Wrap a legacy app in a REST API.

Adopt or create

Your service restarts far more often than the desktop needs to. On startup, look for a live desktop you already authenticated, matching your context_id, and adopt it. Only create a fresh one (from a saved environment_id snapshot of the logged-in state) when nothing live exists.

session.py

from tzafon import AsyncLightcone

CONTEXT_ID = "legacy-app-worker"       # tag every desktop you create with this
SNAPSHOT_ENV_ID = "env_..."            # saved environment with the app installed


async def adopt_or_create(client: AsyncLightcone) -> str:
    """Return the id of a live authenticated desktop, reusing one if possible."""
    # Fail closed: let a listing error propagate. If we can't confirm whether a
    # warm session already exists, creating one risks a duplicate login that
    # silently kills the first (see the note below). The caller should retry.
    computers = await client.computers.list()

    matches = [
        c for c in computers
        if c.kind == "desktop"
        and str(c.status).upper() in ("RUNNING", "READY")
        and c.context_id == CONTEXT_ID
    ]
    if matches:
        # Freshest first, so we don't adopt one about to hit its idle timeout.
        matches.sort(key=lambda c: str(c.created_at), reverse=True)
        print(f"adopted warm session {matches[0].id}")
        return str(matches[0].id)

    created = await client.computers.create(
        kind="desktop",
        context_id=CONTEXT_ID,
        environment_id=SNAPSHOT_ENV_ID,
        persistent=True,
        idle_timeout_enabled=False,     # the keepalive loop owns liveness
    )
    print(f"created warm session {created.id}")
    return str(created.id)

Track whether you adopted or created: on shutdown, an adopted session should be left running so the next process start re-adopts it. Deleting it would throw away exactly the warm state you built this machinery to keep.

Many SSO-fronted apps enforce a single concurrent login: a second desktop authenticating as the same user silently kills the first. Adopt-or-create reuses a live session within one process, but it does not stop two processes from racing: both can list, both see nothing, both create. For any multi-instance or multi-replica deployment, guard creation with a distributed lock or lease (a fencing token) so exactly one worker ever creates the session.

An active liveness probe

The trap: a screenshot of a frozen app looks identical to a healthy one. The remote-app window is painted on screen, but the session behind it is dead. The only trustworthy probe is active: perform a cheap real input and verify the screen responds.

session.py (probe)

async def is_live(client, cid) -> bool:
    """Type into a harmless field and check the screen actually changed."""
    PROBE = "zzprobe"
    await client.computers.click(cid, x=176, y=98)      # a known search field
    await client.computers.hotkey(cid, keys=["ctrl", "a"])
    await client.computers.type(cid, text=PROBE)
    await asyncio.sleep(1.0)

    img = await screenshot_image(client, cid)           # PIL image helper
    field = img.crop((120, 90, 320, 110))
    responded = region_contains_text(field)             # cheap dark-pixel count

    # Clean up the probe text either way.
    await client.computers.hotkey(cid, keys=["ctrl", "a"])
    await client.computers.hotkey(cid, keys=["Delete"])
    return responded

region_contains_text needs no OCR. Counting dark pixels in the field’s cropped region is enough to prove the app processed the keystrokes. A frozen session swallows them, and the field stays blank.

The active probe has a second benefit: many remote-desktop stacks count only input as activity. A screenshot alone does not reset their inactivity timers, so a probe that types doubles as the in-app keepalive.

Escalating recovery

When the probe fails, repair in order of increasing cost, and stop at the first level that restores health:

1. Re-login in place (~1–2 min). The desktop is fine, only the app session expired. Re-drive the SSO login on the same machine, keeping everything else warm.
2. Recreate from snapshot (~2–5 min). The desktop itself is gone or wedged. Boot a fresh one from SNAPSHOT_ENV_ID and authenticate.
3. Mark unhealthy. Both failed. This usually means something no amount of automation fixes (an expired SSO grant, a rotated credential). Serve 503s and page a human rather than retry-looping forever.

session.py (recovery)

class Session:
    def __init__(self, client):
        self.client = client
        self.cid: str | None = None
        self.healthy = False

    async def recover(self) -> bool:
        # Level 1: the VM is alive but the app logged out; re-auth in place.
        if self.cid and await computer_exists(self.client, self.cid):
            if await drive_login(self.client, self.cid):   # your SSO login FSM
                self.healthy = await is_live(self.client, self.cid)
                if self.healthy:
                    return True
            # Still live but unrecoverable: leave it for inspection rather
            # than creating a duplicate login for the same account.
            self.healthy = False
            return False

        # Level 2: the VM is gone; recreate from the snapshot and re-auth.
        self.cid = await adopt_or_create(self.client)
        if await drive_login(self.client, self.cid):
            self.healthy = await is_live(self.client, self.cid)
            if self.healthy:
                return True

        # Level 3: give up loudly. Manual re-auth is likely required.
        self.healthy = False
        return False

drive_login is a small screen-classification loop: screenshot, classify which login screen you are on (username, password, TOTP, app loading), act, and repeat, with a per-screen attempt budget so a wrong password fails fast instead of looping. See Logins and sessions for the full login-driving pattern.

If the desktop is live but recovery can’t restore the app, don’t create a second desktop. You’d double-book the SSO account and often kill the session you’re trying to save. Leave the machine running for manual inspection and mark the session unhealthy.

The keepalive loop

A background task keeps the platform-level session alive, runs the active probe, and recovers proactively, so the session is warm before the next request arrives instead of 503ing it:

session.py (keepalive)

async def keepalive_loop(session: Session, lock: asyncio.Lock, interval=180):
    while True:
        await asyncio.sleep(interval)
        try:
            if session.cid is None:
                continue
            await session.client.computers.keepalive(session.cid)
            async with lock:                    # never probe mid-request
                session.healthy = await is_live(session.client, session.cid)
                if not session.healthy:
                    print("keepalive saw a dead session; auto-recovering")
                    await session.recover()
        except Exception as exc:
            print(f"keepalive iteration failed: {exc}")

Two details matter:

• Share the request lock. The probe clicks and types; running it during a live request would corrupt the workflow. The same lock that serializes API requests must serialize the probe.
• Probe at startup too. A session recreated from a stale snapshot can boot looking healthy: app chrome painted, session behind it dead. Run the active probe (and recovery if it fails) once at startup, so you pay the restart cost before the first request, not during it.

What you learned

1. Adopt-or-create: list live desktops, adopt the freshest one matching your context_id, and only create from a saved environment_id when none exists, leaving adopted sessions running on shutdown.
2. Active liveness probes: type a real input and verify the screen responds; screenshots alone can’t distinguish healthy from frozen.
3. Escalating recovery: re-login in place, then recreate from snapshot, then mark unhealthy for a human, never spawning duplicate logins for one account.
4. A keepalive loop that shares the request lock and heals proactively.

Next steps

• Wrap a legacy app in a REST API: the request-handling layer that leases this warm session
• Verified structured extraction: reading data off the screen once the session is healthy
• Logins and sessions: driving SSO logins and persisting authenticated state
• Production: deployment and capacity patterns
• Observability: knowing what your warm session did and when it broke