Pond

Attach — operator attaches a TUI to a live agent

On-demand, end-to-end-encrypted attach to a running agent: an operator runs pond run attach <run> and the real Codex TUI drives a live, headless agent through pond’s relay. Pond is otherwise outbound-poll-only; this is its one full-duplex transport, and it’s a zero-knowledge relay — the payload is sealed (Noise/ChaCha20) inside glyph on the worker and in the operator’s client, so pond only ever moves ciphertext.

Composes with glyph (the lean Codex fork): the agent is codex-app-server, and the e2e + tunnel halves are glyph’s attach-bridge (serve/connect) and attach-e2e crates. The RFC is neotype-perf/REMOTE_ATTACH.md in that repo.

Code: app/routers/attach.py (rendezvous + trigger), app/attach_bridge.py (operator ws⇄tcp), swarm/src/ws_client.py + swarm/src/core.py (worker auto-dial), cli/commands/run.py::attach + cli/attach_bridge.py (the CLI).

Topology

Both ends dial inward (no inbound to a NAT’d worker); the e2e terminates in glyph, so the bridges and rendezvous see only ciphertext.

 codex-tui --remote unix://OP            codex-app-server (unix socket)
     │                                        ▲
 attach-bridge connect (e2e initiator)    attach-bridge serve (e2e responder)
     │ 127.0.0.1                               │ 127.0.0.1
 operator bridge ──ws──▶  /v1/attach/{sid}  ◀──ws── worker-host bridge
   (cli/CLI)            (app/routers/attach.py)      (swarm worker)

How a run becomes attachable

  1. Harness — the codex-attachable preset (app/harness_presets.py) runs codex-app-server --listen unix://{socket} (long-lived; the TUI drives turns). The worker fills {socket} with a short per-job path (build_job_spec); a deep $CODEX_HOME path overflows the unix-socket SUN_LEN (~108) limit.
  2. Request — POST /v1/runs/{id}/attach mints a one-shot rendezvous session, records it on the run’s live dispatch job (dispatch_jobs.attach_session_id), and returns the serving worker’s enrollment-pinned X25519 pubkey for the operator to pin as the Noise responder key.
  3. Trigger — GET /v1/orch/poll surfaces attachRequests for the orch’s running jobs; the orchestrator builds the rendezvous ws url and relays it to the owning worker on its next heartbeat (swarm/src/orchestrator.py, backend.py).
  4. Worker dials out — core.beat() runs one _run_attach_bridge per session: glyph attach-bridge serve (signed with the worker’s pinned X25519 key) against the agent’s socket, piped to the rendezvous via the stdlib ws_client (swarm is stdlib-only — no websockets).
  5. Operator — pond run attach opens the rendezvous, runs attach-bridge connect pinning the worker key, and launches codex-tui --remote. Quit to detach; the run keeps running.

Status

Verified end-to-end on a live profile:none pool: operator request → poll → orchestrator relay → worker auto-dial → the operator reached the real codex-app-server (HTTP/1.1 101 Switching Protocols). Tests: rendezvous (tests/test_attach_rendezvous.py), the trigger, the {socket} substitution (swarm/tests/test_sandbox.py), and the reproducible transport smoke (scripts/attach_e2e.py). P1 (interactive) — no prompt, the agent is idle until attached and the TUI drives turns — and P2 (headless + steerable) — the stage carries a prompt, the worker runs the first turn headless, and an operator attaches to observe/steer the same thread — both work (see the checklist below).

Remaining work (pick-up list)

Functionality is proven; these are real gaps a contributor can take independently.

  • Attachable-stage lifecycle. (done) A stage is attachable when its command template carries the {socket} placeholder — the same signal the worker uses, so the two agree with no extra schema. The orchestrated executor (_run_tasks_orchestrated) now skips the wall-clock deadline for an attachable stage: it runs detach-driven, ending on cancel rather than being marked failed at timeout_sec. The idle half is also done: the worker tracks attach activity per job and gracefully ends an agent that’s never attached past a connect window, or attached-then-detached past an idle window (re-attach resets it) — POND_ATTACH_CONNECT_SEC / POND_ATTACH_IDLE_SEC. Tripping the job’s cancel event SIGTERMs the app-server (exits 0 → done), and the executor no longer downgrades an attachable stage’s empty stdout to failed.
  • Reap the agent on job end/timeout. (done) NoneProvider now launches the agent with start_new_session=True, and the base SandboxHandle terminate/kill signal the whole process group (guarded so we never killpg the worker’s own group). worker_run_job also reaps in a finally on every exit path (normal / cancel / timeout / exception), so the long-lived codex-app-server and any children can’t be orphaned.
  • Multi-worker pubkey lookup. (done) dispatch_jobs.claimed_by_worker now records the claiming worker’s name — the orchestrator reports it on the running transition (swarm backend.py wraps state.claim; OrchJobStateIn carries an optional worker). _worker_pubkey_for_run resolves that specific (orch_id, worker)’s pinned key, unambiguous on a multi-worker pool, with the single-worker heuristic kept as a fallback for older orchestrators. Verified live on a 2-worker pool (the claiming worker’s key is returned, not the other’s).
  • Pin the operator key worker-side. (done) The operator mints an ephemeral X25519 keypair per attach and sends its public half with the request (POST /attach operatorPubkey); it rides the orchestrator-poll relay (dispatch_jobs.attach_operator_pubkey → orch set_attach/heartbeat → worker) and the worker passes it to attach-bridge serve --peer. glyph’s serve gained --peer: in Noise IK the responder learns the initiator’s static key during the handshake, so it authorizes it post-handshake and drops a mismatch. Verified live: a matching operator key reaches 101; a different key is rejected.
  • Multi-replica rendezvous. (done) Set POND_ATTACH_REDIS_URL and the ws handler validates the session against the DB (the cross-replica source of truth) and relays the byte stream over Redis pub/sub keyed by attach_session_id — so the operator and worker ws connections can land on different replicas. Each peer subscribes before announcing presence and waits for the other (pub/sub has no buffering); duplicate roles are rejected via a Redis SET NX. Still zero-knowledge — only ciphertext crosses Redis. Unset = the in-process pairing (single replica), unchanged. Verified live across two replicas.
  • P2 — headless + steerable. (done) No codex-app-server auto-run mode was needed — the worker drives the first turn. An attachable stage that carries an explicit prompt is P2 (the prompt resolver’s built-in fallback assets do NOT count — else P1 would be unreachable): the executor ships the stage’s own prompt to the worker (POND_INITIAL_PROMPT), which — once the app-server binds its socket — runs initialize → thread/start → turn/start over a stdlib WS-over-unix client (swarm/src/app_server_client.py), publishes the thread_id on the job’s stdout, and keeps the turn-owning connection open. POST /attach returns that agentThreadId; pond run attach launches codex-tui --remote --resume <thread_id> so the operator lands on (and steers) the SAME thread — the app-server fans turn events to every subscribed connection. The only glyph change was exposing codex-tui’s existing resume_session_id as --resume. No prompt ⇒ P1 (idle). Verified live through the e2e tunnel: the operator resumed the worker’s headless session and the agent recalled context from it.
  • Confined (glyph-brokered) attach. Verified on profile:none; the Docker/confined path ({socket} inside the container, the broker network) is untested (nested-container limits block it locally). Wire + verify on real infra.

Threat model notes

  • Zero-knowledge relay: the rendezvous and both ws⇄tcp bridges carry only Noise-sealed frames; the control plane can’t read or forge session traffic.
  • Mutual authentication. The worker is authenticated by its enrollment-pinned X25519 key (the operator pins it as connect --peer). The operator is authenticated by the ephemeral key it registers in the attach request, which the worker pins as serve --peer — so only that operator can complete the handshake (no longer TOFU).
  • The agent container gains no new inbound: glyph serve binds localhost on the worker host; the worker host makes the outbound rendezvous connection.