Skip to content

feat(lfs): add FastCDC media transport - #461

Open
marshawcoco wants to merge 2 commits into
libra-tools:mainfrom
marshawcoco:feat/lfs-fastcdc
Open

feat(lfs): add FastCDC media transport#461
marshawcoco wants to merge 2 commits into
libra-tools:mainfrom
marshawcoco:feat/lfs-fastcdc

Conversation

@marshawcoco

@marshawcoco marshawcoco commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • connect Libra's existing frozen fastcdc-v1 chunker, manifest, and local chunk store to normal LFS push/download flows
  • negotiate the repository-scoped Mega media extension while preserving standard LFS fallback before a chunked transfer is selected
  • upload only missing chunks, resume interrupted transfers, reuse verified cached chunks, and repair corrupt cache entries
  • fail closed on authentication or integrity errors after selecting a finalized manifest
  • derive repository-scoped LFS URLs without dropping paths, query strings, ports, or IPv6 hosts
  • atomically publish hydrated files after whole-object verification
  • flush ordinary LFS downloads and checksum-error pointers before returning, fixing an observed Tokio write-completion race
  • document the opt-in configuration, authentication flow, behavior, and known production gaps

This is the client half of a coordinated Mega server change. Mega companion PR: gitmono-dev/mega#2178

Verification

  • cargo test --features fastcdc --test compat_fastcdc_feature_gate_guard: 2 passed
  • cargo test --features fastcdc --test media_fastcdc_test: 6 passed, 1 interop case ignored
  • cargo test --features fastcdc --lib utils::media::: 12 passed
  • default and FastCDC checks passed; the FastCDC release build passed on Windows/MSVC
  • the ignored Libra-to-Mega HTTP case passed against the companion Mega production routes and test fixture
  • cargo +nightly fmt --all --check, git diff --check, and a Playwright 1.62.1 Chromium install dry-run passed after repairing the self-hosted Web E2E workflow

The PR also removes Playwright's per-job --with-deps invocation from the self-hosted Web E2E job. The runner must provide its Linux browser dependencies; the old command attempted interactive sudo and had timed out for five minutes across multiple unrelated PRs before any E2E test ran.

cargo test --features fastcdc --lib completed with 4,196 passed and 98 failures in non-FastCDC Windows-sensitive areas. Those failures were not individually proven to be baseline failures because the clean-baseline comparison was itself blocked by unguarded Unix-only test code. The two --all runs and strict all-targets/all-features Clippy were also non-zero on Windows. This PR does not claim full CI is green.

Deployment boundary

The feature is disabled by default and can be disabled per repository with lfs.fastcdc=false. The HTTP fixture uses loopback, an injected test client/token, SQLite, and Local object storage. It does not validate production credential storage, a complete Mono deployment, S3/GCS backends, shared repository ACLs, quotas, or garbage collection.

Signed-off-by: Henry Guo <marshawcoco@gmail.com>
Signed-off-by: Henry Guo <marshawcoco@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant