fix(buffers): make byte limits Unicode-correct - #1175
Open
GautamSharma99 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make every limit documented as bytes enforce the actual UTF-8 encoded size instead of Python Unicode character counts.
Problem
len(str)counts Unicode code points, not bytes. That mademax_buffer_sizeandMAX_PENDING_BYTESsubstantially more permissive for multibyte content. For example, emoji generally require four UTF-8 bytes per Python character, so a nominal 1 MiB byte limit could allow several MiB of encoded data.The mismatch affected three related protections:
_LineFramertracked pending stdout and stderr fragments in characters.Implementation
The transport still frames decoded text, preserving the existing stream and parsing behavior.
_LineFramernow maintainspending_bytesby adding the UTF-8 encoded size of each decoded chunk and recomputing the encoded size of the residual tail after newline framing. Complete stdout lines use the same UTF-8 measurement before parsing.Session import now adds
len(line.encode("utf-8"))to its byte accumulator. Batch-size and entry-count semantics remain otherwise unchanged.This is exact for the decoded UTF-8 text delivered by
TextReceiveStreamand avoids changing transport decoding or line-framing contracts.Tests
Added regressions proving that:
max_buffer_size, even when its character count does notMAX_PENDING_BYTESbased on encoded bytes and preserve entry orderingValidation
uv run --extra dev pytest -q— 1294 passed, 5 skippeduv run --extra dev ruff check src tests— passeduv run --extra dev ruff format --check src tests— passeduv run --extra dev mypy src— passedFixes #1165