2026.7.26

Hybrid passthrough for Scylla backup UploadPartCopy part 2 (#137)

Fixes InvalidPart failures on large Scylla dedup copies (e.g. main.companies *-big-Data.db) where part 2 was blocked by ranged_copy_nonzero_start and fell back to frame-by-frame streaming (~190 Hetzner GETs per 1.5GB part).

• Extend range-aware passthrough to nonzero bytes= starts (part 2 after part 1's deferred mid-frame tail)
• Add streaming_head split: consume deferred tail + re-encrypt only the misaligned frame prefix, then server-side UploadPartCopy for aligned bulk
• Remove ranged_copy_nonzero_start gate
• Add route diagnostics: hybrid_mode, passthrough segment counts, UPLOAD_PART_COPY_PART2_STREAMING_FALLBACK warning

Deploy

Bump image.tag to 2026.7.26 in argocd-system. After rollout, part 2 logs should show UPLOAD_PART_COPY_ROUTE route=passthrough hybrid_mode=head_and_passthrough, not UPLOAD_PART_COPY_STREAMING.

2026.7.25

Graceful drain on scale-in (#136)

Chart-only release. s3proxy pods now drain instead of dying mid-upload when KEDA scales the fleet in:

• preStop sleep 10s — HAProxy stops routing new connections before the pod stops listening
• terminationGracePeriodSeconds 30 → 600 (now a values knob) — uvicorn finishes in-flight requests before exit

Fixes the remaining InvalidPart failures on Scylla backups: pods killed with 30s grace mid-UploadPart/UploadPartCopy orphaned multipart state, failing CompleteMultipartUpload (2026-07-19 23:49 UTC run error).

2026.7.24

Fixes

• Pin uvicorn's event loop to asyncio (#135). 2026.7.22 (#134) removed the app's own uvloop.install() call, but uvicorn's default loop="auto" silently installs uvloop whenever the package is importable, so the libuv uv__io_poll abort (fd double-close under backend connection churn, upstream bug fixed-but-unreleased in MagicStack/uvloop#740) kept killing pods mid-upload. The uvicorn config now passes loop="asyncio" unless S3PROXY_UVLOOP=1 explicitly opts back in.
• The Starting log line now reports the actually-running loop class (event_loop=asyncio.SelectorEventLoop) so deployments can be verified against the runtime.

Includes #134 (2026.7.22): drop S3PROXY_MEMORY_DEBUG tracemalloc mode, gate uvloop.install() behind S3PROXY_UVLOOP.

Images: ghcr.io/serversidehannes/s3proxy-python:2026.7.24, ghcr.io/serversidehannes/s3proxy-dashboard:2026.7.24
Chart: oci://ghcr.io/serversidehannes/charts/s3proxy-python:2026.7.24

2026.7.21

Fixes

• Retry transient source-read and segment-copy failures in UploadPartCopy (#133): a backend dropping a long-lived connection mid-body (observed against Hetzner: ClientPayloadError: received 8331264 of 8388636 bytes after ~4 min) or failing a segment copy after streaming a 200 (InternalError: The server did not respond in time) no longer fails the whole client part. Frame GETs and single-object reads retry on a fresh connection; raw copy-source streams resume with a ranged GET at the exact byte offset already delivered; passthrough segment copies retry idempotently. New env knobs: S3PROXY_SOURCE_READ_ATTEMPTS (default 4), S3PROXY_SOURCE_READ_BACKOFF (default 0.5s).

2026.7.20

Fixes

• Listings now report plaintext size/etag for multipart objects (#131). ListObjects previously fell back to the backend (ciphertext) size for multipart-uploaded objects because their plaintext size lives in the .meta sidecar, not user metadata. Sync clients comparing sizes (scylla-manager's rclone) saw every multipart object as changed and re-uploaded it on every pass, doubling backup transfer and defeating dedup. LIST now resolves the sidecar, returns the same synthetic etag as HEAD/CompleteMultipartUpload, and caches resolved attributes in a per-pod LRU keyed by backend etag (primed at complete), so repeated listings skip the per-object round-trips.

2026.7.19

fix: deferred copy tail double-counted in completed object plaintext size (#126)

The deferred copy tail was counted both in the client part that produced it and in the part that eventually stored it (next part or complete-time flush). This inflated total_plaintext_size by the tail size, so HEAD reported size+tail and rclone failed every >copy-cutoff copy with "corrupted on transfer: sizes differ".

• Stop adding the deferred tail to the producing part's plaintext total.
• Count the tail's plaintext on the part that flushes it at complete.
• Add a PART_PLAINTEXT_MISMATCH drift warning in _build_s3_parts.
• Regression tests for the 2-part defer and complete-time flush paths.

2026.7.18

Fix InvalidPart on 500+ part uploads (ClickHouse, Postgres barman, Scylla compressed streams) and noisy GetBucketVersioning 400s.

Problem: Sparse internal-part allocation maps client part N → internal (N-1)*20+1. Client part 501 needs internal 10001, exceeding S3's 10k part limit. Affected ClickHouse shadow tars, Postgres barman base backups (5MB parts), and Scylla compressed stream uploads.

Fix (#125):

• Dense single-internal mode — client part N → internal N while every client part uses exactly one internal part (typical 5MB uploads)
• Scylla SST path preserved — first multi-internal client part clears dense mode and reverts to sparse ranges (out-of-order safe)
• Legacy Redis state — infer dense_single_internal=false from existing sparse parts so in-flight uploads don't switch modes mid-stream
• GetBucketVersioning — answer locally with Suspended (Hetzner backend rejects the probe with 400)
• Diagnostic logs: allocation_mode, HIGH_CLIENT_PART_ALLOCATED, LEGACY_DENSE_MODE_INFERRED

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.18

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.18

Post-deploy check:

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'allocation_mode=dense|HIGH_CLIENT_PART_ALLOCATED|INTERNAL_PART_ALLOCATION_REJECTED'

Expect allocation_mode=dense and internal_start=501 on part 501. Zero new INTERNAL_PART_ALLOCATION_REJECTED on fresh upload IDs.

Note: Uploads already past part 500 under sparse numbering cannot be salvaged — clients must abort and retry with a fresh upload ID.

2026.7.17

Fix EntityTooSmall on Scylla manifest dedup copies after hybrid passthrough (2026.7.16).

Problem: After 2026.7.16, part 1 hybrid passthrough worked (~572 segments) but the ~1.08 MB mid-frame tail was uploaded as internal part 573. Part 2 then added parts 574+, so S3 rejected CompleteMultipartUpload with EntityTooSmall (part 573 < 5 MB and not last). Dedup bytes counted, then the same bytes retried as Failed.

Fix (#124):

• Defer hybrid tail when part 1 is large (≥ 1 GiB) and more client parts follow — store sub-5 MB plaintext suffix in upload state instead of uploading it on part 1
• Part 2 prepends deferred tail into streaming re-encrypt (_PlaintextReader prefix)
• Complete flushes any leftover deferred tail
• Debug logs: HYBRID_TAIL_DEFER_DECISION, UPLOAD_PART_COPY_PASSTHROUGH_TAIL_DEFERRED, UPLOAD_PART_COPY_CONSUME_DEFERRED_TAIL, COMPLETE_MULTIPART_SUCCESS

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.17

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.17

Post-deploy check:

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'HYBRID_TAIL_DEFER_DECISION defer=true|PASSTHROUGH_TAIL_DEFERRED|CONSUME_DEFERRED_TAIL|COMPLETE_MULTIPART_SUCCESS|ENTITY_TOO_SMALL'

Expect defer=true → TAIL_DEFERRED → CONSUME_DEFERRED_TAIL → COMPLETE_MULTIPART_SUCCESS on fresh .sm_ manifest upload IDs. Zero new ENTITY_TOO_SMALL on new uploads.

2026.7.16

Fix Scylla manifest UploadPartCopy timeouts, zombie copies, and two latent CI integration bugs.

Problem: Live-cluster analysis showed failing manifest copies are UploadPartCopy calls that send zero response bytes until the copy completes. rclone (scylla-manager-agent) hangs up after exactly 300s idle — a 4.7GB part took ~400s streaming, so every attempt was doomed. The proxy kept abandoned copies running as zombies (1,953 streaming copies started vs 232 completed in 9h, plus 18k MEMORY_BACKPRESSURE events from the retry storm).

Fix (#123):

• Keepalive streaming: UploadPartCopy now commits to 200 OK and trickles whitespace (default every 5s, S3PROXY_COPY_KEEPALIVE_INTERVAL) while the copy runs — the AWS S3 long-copy pattern — then emits CopyPartResult or an <Error> document as the body
• Client disconnect cancels work — no more zombie copies burning the memory budget after the client gives up
• Parallel passthrough segments (bounded, default 8, S3PROXY_PASSTHROUGH_SEGMENT_CONCURRENCY) instead of ~570 sequential backend calls
• MD5 pass overlapped with segment copies via TaskGroup instead of serially re-reading the whole 4.7GB source afterwards
• Hybrid passthrough + tail when Scylla manifest range ends mid internal frame
• CompleteMultipartUpload now forwards stored backend part etags (fixes InvalidPart on single-segment passthrough complete)
• Error handler drains small unread request bodies on keep-alive connections (fixes Abort after rejected UploadPart)
• Chart default memoryLimitMb raised to 128; backpressure timeout 120s; Redis WATCH retries hardened

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.16

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.16

Post-deploy check:

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'UPLOAD_PART_COPY_KEEPALIVE|UPLOAD_PART_COPY_PASSTHROUGH_COMPLETE|UPLOAD_PART_COPY_CLIENT_GONE|FAILED_AFTER_200'

Expect PASSTHROUGH_COMPLETE with elapsed_sec well under 300s and CLIENT_GONE to disappear.

2026.7.15

Fix Scylla backup manifest UploadPartCopy still streaming after 2026.7.14.

Problem: PR #121 only normalized ranges matching sidecar total_plaintext_size. Scylla Manager sends bytes=0-(~4768MB-1) on .sm_* manifests while source big-Data.db metadata totals ~6GB. Copies still routed to UPLOAD_PART_COPY_STREAMING, queued behind the 2-slot pipeline, and timed out at ~300s → HAProxy 400.

Fix (#122):

• Range-aware passthrough for large copies from offset 0 when the range aligns to internal ciphertext segments
• Server-side upload_part_copy per segment (no decrypt/re-encrypt) instead of streaming
• UPLOAD_PART_COPY_ROUTE log with route, range sizes, and passthrough_blocked_reason for diagnosis

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.15

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.15

Post-deploy: Scylla manifest uploads (*.sm_*) should log UPLOAD_PART_COPY_ROUTE route=passthrough and UPLOAD_PART_COPY_PASSTHROUGH. Upgrade from 2026.7.14 if backups still show 400 @ ~300s on manifest copies.