Skip to content

feat(tui): ingest image attachments through Runtime Host (#4171) - #4248

Open
abhinav-phi wants to merge 2 commits into
apache:mainfrom
abhinav-phi:feat/tui-image-attachments
Open

feat(tui): ingest image attachments through Runtime Host (#4171)#4248
abhinav-phi wants to merge 2 commits into
apache:mainfrom
abhinav-phi:feat/tui-image-attachments

Conversation

@abhinav-phi

Copy link
Copy Markdown
Contributor

Summary

Closes #4171. Adds a self-contained TUI image-attachment ingestion path: the user attaches an image with /attach <path>, the bytes are ingested through the existing Runtime Host artifact.ingest authority, and the draft carries only an atomic [image N] marker. At submit time, each marker serializes to the exact committed Session AttachmentRef alongside the visible label — on every path: idle send, mid-turn steering, Alt+Enter queued successors, failed-dispatch retry, and retraction.

Before this change, the TUI's submit path carried text only, while Runtime Host accepts only canonical session_file attachments already committed to the Session ArtifactStore. The Host protocol, admission validation (validateTurnAttachments), and model-facing consumption (native image parts for vision models; name + MIME + Read reference otherwise) already existed — this PR builds the missing client ingestion surface and keeps its ownership boundaries explicit.

How it works

  1. Attach (explicit action, never blocks Send): /attach <path> (TUI slash catalog + guidance copy) reads the file client-side with a byte cap (MAX_ATTACHMENT_BYTES), refuses non-images and over-count drafts (MAX_ATTACHMENT_COUNT), and calls the new driver ingestAttachment — which sequences artifact.ingest begin → chunk… → commit over the same connection used for turns, aborting the staged upload on any failure. The [image N] marker is inserted atomically at the cursor immediately; the committed ref binds when the ingest lands.
  2. Bind (marker → reference): tui-attachments.ts holds a draft-scoped registry mapping ordinals to committed refs. bindForSubmit scans the outgoing text: every distinct [image N] token binds its staged ref (first-appearance order, deduped); tokens with no staged ordinal are inert literal text and bind nothing. Binding is token-identity based, so editing marker text can orphan, duplicate, or renumber markers — it can never make one image's bytes travel under another image's marker.
  3. Submit: MakaSubmitMessageOptions / MakaPreparePromptOptions carry attachments, and the driver puts them on content.attachments for turn.message.submit and turn.start (steering and follow-up queue entries carry the full MessageContent, so queued images ride the existing Host machinery unchanged).
  4. Retraction / interrupt: MakaRetractedMessages gains per-entry grouping (entries: { text, attachments }). Alt+↑ and the interrupt path re-stage every retracted ref and renumber markers into one collision-proof draft sequence (planRestoredImageAttachments + admitRestoredPlan), so the recalled text keeps a marker where each image was and every marker rebinds to exactly the image it rode with.
  5. Recovery: stored user messages, steering_message echoes, and the transient row render committed attachments as 📎 name · mime · size chips — names only, never a local path; resumed sessions show them through the same hydration path.

Cleanup behavior (defined in this PR)

Case Behavior
Marker deleted from the draft The staged ref is abandoned at the next submit/dispatch and its user-upload Artifact is deleted best-effort via artifact.delete.
Failed ingest The staged upload is aborted (artifact.ingest abort); the marker is removed from the draft; nothing is staged and there is nothing to clean.
Failed dispatch (submit rejected) Bound refs return to staging under the same ordinals, so recalling the draft from history and retrying rebinds identically.
Abandoned draft Editor clear (Ctrl+C), session switch//new, and graceful exit all abandon staged refs and delete their committed Artifacts best-effort (refs carry their owning Session, so cleanup is correct even after switching).
Hard crash The committed Artifact remains in the Session's artifact list — the Host artifact authority stays the only durable store; the user can delete it there. In-flight (never-committed) uploads are reclaimed by the Host's 5-minute upload TTL.

Ownership boundaries

  • The Runtime Host remains the only durable attachment authority: bytes enter exclusively through artifact.ingest, deletion through artifact.delete, and the TUI never touches an ArtifactStore. Local and remote Hosts take the identical protocol path.
  • The draft registry is client-local view state, not a second store; no Surface-owned durable authority is introduced.
  • No temporary or local filesystem path ever enters the editor text, the model-facing content, or the transcript.
  • Vision/non-vision handling is untouched: the runtime retains the existing native-image path for vision models and sends name + MIME + Read reference otherwise; no OCR inference or model switching.

Non-goals honored

No second attachment store; no OCR/vision inference; no temp paths in prompts or transcripts.

Test plan

New packages/cli/src/__tests__/tui-image-attachments.test.ts (20 tests): marker/registry unit semantics (binding order, dedupe, stranger tokens, restage, restore renumbering, capped file reads, path/MIME resolution) and end-to-end TUI flows — upload + exact-ref submission, Send-not-blocked by in-flight ingest, failed ingest, marker deletion → Artifact deletion, failed submit → retry rebind, queue → retraction → same-ref resubmission, session-switch recovery rendering without path leakage, no-authority refusal, non-image refusal.

Extended runtime-host-session-driver.test.ts: artifact.ingest begin/chunk/commit chunking and payload SHA-256, abort-on-failure, artifact.delete by exact reference (tolerating not-found), attachments on the submit wire, and per-entry retraction grouping.

packages/cli: core + cli typecheck clean, Biome clean; the affected suites pass (73/73 driver tests; 20/20 attachment tests; pi-tui-runner and pi-transcript suites pass with the same pre-existing Windows-only environmental flakes — resize/SIGTERM child-process probes — verified identical on a clean baseline).

Refs #4171
Refs #4079
Refs #4080

@github-actions github-actions Bot added the effort/XL Over 1000 readable lines label Aug 30, 2026
@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for taking on TUI image attachments — the user need is real, and reusing the Runtime Host Artifact authority is the right durable boundary.

Before we patch the individual lifecycle findings, could we revisit the shape of the first slice? Desktop already keeps selected files or approval tokens local to the draft and performs Artifact ingestion inside the message-submit boundary. This PR instead ingests during /attach, then adds a TUI-specific marker, deletion, retraction, retry, and orphan lifecycle around the already-created Artifact.

That is not a second durable store, but it is a parallel draft lifecycle, and it appears to be the source of most of the 1,600-line change and the current concurrency/recovery findings. My suggestion would be to stage only a local image descriptor, ingest through the existing Host boundary at submit, and let the committed Message’s AttachmentRef become the durable identity. The TUI can render staged images separately from editable message text, as Desktop does.

Please do push back if inline marker placement is itself a demonstrated product requirement rather than an implementation choice; that would materially change the smallest complete design.

AI-assisted review disclosure: Codex assisted with comparing the TUI and Desktop attachment lifecycles; Astro-Han verified the production boundaries and owns this position.

Add a self-contained TUI image-attachment ingestion path that mirrors the
Desktop composer's shape: a /attach pick stages only a client-local
descriptor, the message-submit boundary owns Artifact ingestion, and the
committed Message's AttachmentRef is the durable identity.

- new tui-attachments.ts: draft-scoped staging (local descriptors and
  retained refs) plus the client-side capped file reader
- /attach <path> (TUI surface) and /detach <n>: stage and remove client-
  locally with MIME/count guards; staged images render as a separate
  composer strip beside the pending queues - name + media type only,
  never a local path, and never inside the editable draft text
- submit boundary: staged descriptors are read (byte-capped) and ingested
  through the Runtime Host artifact.ingest authority (begin/chunk/commit,
  abort on failure) in staging order, then ride content.attachments on
  turn.message.submit / turn.start; Send stays enabled the whole time
- failed dispatch or refusal keeps the already-committed refs staged as
  retained items, so a retry reuses the exact Artifacts instead of
  orphaning a second copy; a mid-batch ingest failure deletes only the
  freshly committed refs and leaves staging untouched
- Alt+Up / interrupt retraction: retracted queue entries return their
  attachments and are restaged as retained refs - resubmission reuses
  the same Artifacts with no re-ingest
- cleanup semantics: local descriptors drop with the draft (editor clear,
  session switch, /new, graceful exit) with nothing to clean; retained
  refs are deleted best-effort via artifact.delete
- transcript rendering: stored user messages, steering echoes, and the
  transient row render committed attachments as name + MIME + size chips
  - never a local path
- tests: staging unit semantics and TUI flows (stage-then-submit, exact-
  ref dispatch, ingest failure, dispatch retry, /detach, retraction
  reuse, session-switch cleanup + recovery rendering, authority and
  non-image refusals, count cap) plus driver-level artifact.ingest and
  artifact.delete wire coverage

Refs apache#4171
Refs apache#4079
Refs apache#4080
@abhinav-phi
abhinav-phi force-pushed the feat/tui-image-attachments branch from 623eff3 to 152426a Compare August 30, 2026 20:50
@abhinav-phi

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @Astro-Han — the diagnosis is correct, and I've reworked the PR along exactly the shape you suggested (pushed as 152426a, rebased onto current main).

On your question first: inline marker placement was an implementation choice, not a demonstrated product requirement — I have no evidence users need [image N] inside the editable text, so I did not preserve it. The issue's "atomic marker / visible label" language is now satisfied by the separate staged strip, and the "marker serializes to the exact committed Session Artifact reference" criterion maps to the staged item → exact AttachmentRef serialization inside the submit boundary. Happy to see that reflected in the issue text.

What changed:

  • /attach <path> now stages only a client-local descriptor (path, name, MIME, size) with MIME/count guards; new /detach <n> removes one. Nothing is read or ingested at stage time, so abandoning a draft needs no cleanup round trip at all.
  • Ingestion happens inside the message-submit boundary, like the Desktop composer: descriptors are read (byte-capped) and ingested through artifact.ingest in staging order, then ride content.attachments. Send stays enabled the whole time.
  • Each committed ref replaces its descriptor in staging, so a failed dispatch or skill refusal retries with the same Artifact; a mid-batch ingest failure deletes only that attempt's fresh refs. This removed the orphan/retry/identity machinery the first slice needed.
  • Alt+↑ / interrupt retraction restages retracted entries' committed refs as retained items — resubmission reuses the exact Artifacts with no re-ingest.
  • Staged images render as a separate Staged: chip strip beside the pending queues (never inside editable text); committed attachments render as 📎 name · mime · size chips in the transcript. No path ever serializes.
  • Cleanup: local descriptors drop with the draft (Ctrl+C, /new, session switch, exit); only retained refs delete best-effort via artifact.delete.
  • The marker/ordinal/renumbering machinery is deleted; the source diff is now ~630 lines of TUI source + tests (plus driver-level artifact.ingest/artifact.delete wire coverage and the already-reviewed contract additions for attachments on submit/prepare and retracted-entry grouping).

CI note: the failing Check renderer architecture step reproduces on a clean checkout of the merge base with this PR's changes stashed, so it appears pre-existing on main rather than caused by this PR — but I'll keep an eye on the next run.

All TUI flows from the issue are covered by tests in the new shape: stage-then-submit with exact refs, dispatch failure → retry reuses same Artifacts, ingest failure → nothing dispatched and nothing orphaned, /detach, retraction reuse, session-switch cleanup + recovery rendering, and the authority/non-image/count refusals.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reworking this around the existing Runtime Host Artifact authority; the new shape is materially simpler than the previous design. I found one submit-ownership issue and several recovery/lifecycle gaps below. The exact-head required test check is also currently red in the changed CLI autocomplete surface, so I could not treat the current gate as passing. This was an AI-assisted review using Codex; I independently traced the production runner/driver paths and exact-head CI. These are suggestions from an outside read, so please push back if I have missed an enforced TUI invariant.

// same Artifact instead of orphaning a second copy of the bytes.
const attachments: AttachmentRef[] = [];
const freshlyIngested: AttachmentRef[] = [];
for (const item of imageAttachments.list()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving ingestion to the submit boundary. [P1] (category ② — supported submit/session concurrency) This copies the current items but does not atomically claim them: while the loop awaits file reads and Host ingestion, /attach, /detach, a second Send, /session, /new, and close can still mutate or reset the shared staging object. The first attempt can therefore send an item the user detached, clear an item staged for the next message, duplicate one image across two messages, or ingest under one Session and submit under another because ingest and submit each resolve the live Session separately. Could Send synchronously take an attempt-owned batch plus Session generation, with later draft edits going to a fresh batch and only that exact batch being retired? Gated-ingest regressions for second Send, attach/detach, and Session switch would pin the boundary down. Please push back if the input layer serializes all of these operations through an invariant I missed.

} catch (error) {
// Nothing dispatched, so this attempt's Artifacts would be orphaned:
// delete them and leave staging untouched for the retry.
for (const ref of freshlyIngested) deleteAttachmentBestEffort(ref);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for preserving retry state on an ingest failure. [P2] (category ② — partial batch failure) The earlier successful descriptors have already been replaced with retained refs at line 1227, then this branch deletes those Artifacts without restoring the descriptors. The strip still has the same item count, but the next retry submits refs that now point to deleted Artifacts and fails with Attachment Artifact was not found. Could the batch keep descriptors untouched until every ingest succeeds, or restore them during rollback, and extend this test through an actual second submit rather than only checking the strip count? Please push back if delete is guaranteed to leave these refs reusable.

return;
}
// The Message is the attachments' durable owner now.
imageAttachments.clear();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for keeping the transient row when admission is uncertain. [P2] (category ② — disconnect/outcome-unknown recovery) The driver deliberately returns undefined when a dispatched submit has an unknown outcome, but this branch treats every non-blocked result as durable ownership and clears the attachments. If the Message was not admitted, reconnect later retires the transient row while the user has neither staged refs to retry nor an owner that cleans up the ingested Artifacts. Could attachment ownership stay with the attempt until transcript/query reconciliation proves admission or cancellation, and keep staging on undefined? An attached outcome-unknown → reconnect regression would cover both branches. Please push back if another reconciliation path restores these refs.

return;
}
const removed = imageAttachments.list()[index - 1]!;
imageAttachments.remove(removed.stagingKey);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding explicit draft removal. [P2] (category ① — normal /detach path) This is not always client-local: retracted queued Messages re-stage already committed retained refs. Removing one here drops the last draft/message owner but never calls the existing Artifact delete authority, so repeated retract → detach leaves persistent user-upload Artifacts behind. Could this delete best-effort when the removed item is retained, while local descriptors remain a pure list removal? Please push back if retained refs have a separate garbage-collection owner.

// reference as a retained item, so resubmitting reuses the exact Artifact
// instead of re-ingesting the same bytes into a second copy.
for (const entry of retracted.entries ?? []) {
for (const attachment of entry.attachments) imageAttachments.stageRetained(attachment);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for preserving attachments during Alt+Up retraction. [P2] (category ① — normal multi-message retraction) The Host returns per-message entries, but this flattens every attachment into one draft and refillEditorFromQueues joins the texts. Two valid queued messages can each carry the allowed 8 images; retracting both creates a 16-image draft whose next submit is deterministically rejected by the per-message MAX_ATTACHMENT_COUNT. Could the retraction preserve entry grouping, or define a bounded restaging behavior that always remains submit-able, with a two-message regression? Please push back if the Host guarantees that only one attachment-bearing entry can be retracted here.

/attach now sorts ahead of /compact, so the initial slash-autocomplete
cursor lands on it; the resize-visibility assertions must wait for the
cursor on /attach. Counter, wrap, and resize behavior are unchanged.
@abhinav-phi

Copy link
Copy Markdown
Contributor Author

Follow-up on 197968a (the rework commit plus one test fix):

  • The rebase onto current main resolved the tui-primary-guidance.ts conflict by adopting the new tui-copy-catalog.ts resource structure; the PR is mergeable again.
  • The earlier test job failure is fixed: /attach sorts ahead of /compact, so the initial slash-autocomplete cursor no longer lands on /compact — the resize-visibility assertions in pi-tui-runner.test.ts now target /attach (counter, wrap, and resize behavior unchanged).
  • The Check renderer architecture step that failed the first run passed on this run (same base code either way — it looks environment/order-sensitive rather than related to this PR).
  • CI is green on this commit: test ✓ (11m48s), windows_recovery ✓.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XL Over 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(tui): ingest image attachments through Runtime Host

2 participants