Skip to content

feat(lfs): support FastCDC media transport - #2178

Merged
benjamin-747 merged 3 commits into
gitmono-dev:mainfrom
marshawcoco:feat/lfs-fastcdc
Aug 31, 2026
Merged

feat(lfs): support FastCDC media transport#2178
benjamin-747 merged 3 commits into
gitmono-dev:mainfrom
marshawcoco:feat/lfs-fastcdc

Conversation

@marshawcoco

@marshawcoco marshawcoco commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in fastcdc LFS media extension under the repository-scoped /info/lfs/libra/media/v1 API
  • validate chunk SHA-256, complete-object SHA-256, manifest identity, and the frozen fastcdc-v1 boundaries before publication
  • persist the interoperable full LFS fallback and database metadata before publishing a finalized manifest
  • isolate pending manifests and chunks by authenticated Mono user and repository, with no public raw-hash chunk endpoint
  • stream complete fallback objects through bounded multipart/local atomic writes instead of whole-file buffering
  • harden standard LFS object validation and propagate storage/database failures instead of panicking
  • document authentication, limits, fallback behavior, deployment boundaries, and the Libra/Mega interop fixture

This is the server half of a coordinated Libra client change. Libra companion PR: libra-tools/libra#461

Verification

  • cargo test -p io-orbit --lib: 11 passed
  • cargo test -p ceres --features fastcdc --lib lfs::: 20 passed
  • cargo test -p mono --features fastcdc --lib lfs_router::: 19 passed, 1 interop helper ignored
  • relevant io-orbit/Ceres/Mono --no-deps -D warnings Clippy commands passed after the validation fixes in this commit
  • default Mono build, FastCDC debug build, and FastCDC release build passed on Windows/MSVC
  • the fresh two-process Libra-to-Mega HTTP fixture passed on both sides; shutdown returned 204 and left no helper process
  • the review regression test verifies that object absence returns false while backend head failures propagate as storage errors

The Windows workspace suite is not fully green: three Ceres path-separator assertions and the orion-scheduler Unix-signal compile errors reproduce on the clean upstream baseline. This PR does not claim full workspace CI is green.

Deployment boundary

The feature is disabled by default. The fixture uses loopback HTTP, isolated SQLite, Local object storage, and test tokens. Production Mono startup, the real libra auth login credential path, S3/GCS compatibility, shared-repository ACLs, quotas, orphan GC, and incomplete multipart cleanup were not validated and remain rollout requirements.

CI follow-up — 2026-08-31

Follow-up head: 39fa85e7622ce6312ffb03679fde23a2af76f42b (signed and pushed). The earlier verification results above are historical; they are not a fresh full-suite or HTTP interop result for this head.

  • CI on 44bc65dd passed Full Test and Services Test. Clippy reported eight result_large_err findings introduced by this PR; three files now use smaller error types locally.
  • Local strict Mono library-and-tests Clippy with FastCDC passed on Rust 1.97.1. The remote job uses Rust 1.98 and must be rerun before claiming the CI failure is cleared.
  • Follow-up Mono route tests were stopped during compilation/codegen when this host had less than 1 GiB of system commit headroom. No tests executed; no runtime pass is claimed for the new error-response tests. The Clippy run type-checked the new tests.

The new Base CI, Services, and CodeQL runs require maintainer approval (action_required; the Base run has no jobs yet). Please approve these runs so Rust 1.98 Clippy and the new runtime regressions can execute. The production and cloud-storage validation boundaries above are unchanged.

Signed-off-by: Henry Guo <marshawcoco@gmail.com>
@marshawcoco
marshawcoco marked this pull request as ready for review August 30, 2026 14:45
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-31T02:36:23.785397Z 39fa85e New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76b6b234fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ceres/src/lfs/media/mod.rs
Signed-off-by: Henry Guo <marshawcoco@gmail.com>
@marshawcoco

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 44bc65dd4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: Henry Guo <marshawcoco@gmail.com>
@benjamin-747
benjamin-747 self-requested a review August 31, 2026 03:16
@benjamin-747
benjamin-747 added this pull request to the merge queue Aug 31, 2026
Merged via the queue into gitmono-dev:main with commit bb3ef17 Aug 31, 2026
8 checks passed
@marshawcoco
marshawcoco deleted the feat/lfs-fastcdc branch August 31, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants