Skip to content

fix(buffers): make byte limits Unicode-correct - #1175

Open
GautamSharma99 wants to merge 1 commit into
anthropics:mainfrom
GautamSharma99:fix/1165-utf8-byte-accounting
Open

fix(buffers): make byte limits Unicode-correct#1175
GautamSharma99 wants to merge 1 commit into
anthropics:mainfrom
GautamSharma99:fix/1165-utf8-byte-accounting

Conversation

@GautamSharma99

Copy link
Copy Markdown

Summary

Make every limit documented as bytes enforce the actual UTF-8 encoded size instead of Python Unicode character counts.

  • track pending stdout/stderr line size incrementally in UTF-8 bytes
  • validate completed stdout messages using their encoded byte length
  • batch imported JSONL entries using encoded line bytes
  • add multibyte boundary regressions for stdout, stderr, and session import

Problem

len(str) counts Unicode code points, not bytes. That made max_buffer_size and MAX_PENDING_BYTES substantially more permissive for multibyte content. For example, emoji generally require four UTF-8 bytes per Python character, so a nominal 1 MiB byte limit could allow several MiB of encoded data.

The mismatch affected three related protections:

  1. _LineFramer tracked pending stdout and stderr fragments in characters.
  2. Complete stdout lines were guarded using character length.
  3. Session import batches accumulated JSONL line characters while documenting a byte threshold.

Implementation

The transport still frames decoded text, preserving the existing stream and parsing behavior. _LineFramer now maintains pending_bytes by adding the UTF-8 encoded size of each decoded chunk and recomputing the encoded size of the residual tail after newline framing. Complete stdout lines use the same UTF-8 measurement before parsing.

Session import now adds len(line.encode("utf-8")) to its byte accumulator. Batch-size and entry-count semantics remain otherwise unchanged.

This is exact for the decoded UTF-8 text delivered by TextReceiveStream and avoids changing transport decoding or line-framing contracts.

Tests

Added regressions proving that:

  • a multibyte stdout message split across chunks is accepted exactly at its encoded byte size and rejected one byte below it
  • a newline-free multibyte stderr fragment flushes when encoded bytes exceed max_buffer_size, even when its character count does not
  • Unicode JSONL imports flush at MAX_PENDING_BYTES based on encoded bytes and preserve entry ordering

Validation

  • uv run --extra dev pytest -q — 1294 passed, 5 skipped
  • uv run --extra dev ruff check src tests — passed
  • uv run --extra dev ruff format --check src tests — passed
  • uv run --extra dev mypy src — passed

Fixes #1165

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

max_buffer_size and import byte limits count characters instead of UTF-8 bytes

1 participant