Skip to content

[deckhouse-cli] Add plugin mirroring to d8 mirror pull and push - #445

Draft
Glitchy-Sheep wants to merge 12 commits into
mainfrom
feat/mirror-plugins
Draft

[deckhouse-cli] Add plugin mirroring to d8 mirror pull and push#445
Glitchy-Sheep wants to merge 12 commits into
mainfrom
feat/mirror-plugins

Conversation

@Glitchy-Sheep

@Glitchy-Sheep Glitchy-Sheep commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

d8 mirror now carries CLI plugins into air-gapped installs.

Pull picks the plugin versions the mirrored modules need, packs each plugin into its own tar, and push publishes them so d8 inside the closed network can install them.

How it works

  • Plugins live at deckhouse-cli/plugins/<name>, next to the installer, outside the edition path.
  • Each plugin version carries a contract: a base64 JSON annotation with its requirements (modules, other plugins, platform version).
  • The plugins phase runs last in pull. It sees which modules and platform versions actually made it into the bundle.
  • Selection is "nothing extra": a plugin enters the bundle only for a module that needs it, as a dependency of a selected plugin, or via --include-plugin.
  • Dependencies resolve recursively: built-in d8 commands count as present, already picked versions are reused.
  • A candidate with a dependency cycle is rejected, and its half-resolved dependencies roll back.
  • Content problems (no compatible version, broken contract) skip the plugin with a reason in the summary. Network problems fail the whole pull.
  • Multi-platform plugin images stay whole: the OCI index survives the bundle unchanged.

Before / After

Before: d8 mirror carried the platform, modules, packages and security databases; plugins could not reach closed networks at all.

After: the bundle also holds plugin-<name>.tar for every plugin the modules need; push uploads them to the same paths and adds a name tag to the catalog.

Tests

  • Resolver unit suite (31 scenarios): selection gates, dependencies, explicit includes, error policy.
  • TestPullE2E_* (9 tests): the whole PullService.Pull on a fake registry, plus a pull-to-push roundtrip into a target registry.
  • The e2e suite checks that module versions reach the plugin resolver and every picked version lands in the summary with its reason.
  • Command level: a real Puller.Execute with the registry stub produces plugin tars next to module tars; dry-run writes nothing.
  • Multi-platform round-trip against a real in-memory HTTP registry.

Notes

  • --include-plugin accepts ranges and exact pins (name@=vX.Y.Z); only pins reach pre-releases. With --proxy-registry, only exact pins are allowed.

@Glitchy-Sheep
Add plugins registry service under deckhouse-cli/plugins
d2e3f70 
- `PluginsService` lists plugin names from the catalog path, `Plugin(name)` scopes to one plugin repo with its version tags.
- `ContractAnnotation` reads the base64 contract from the manifest: index annotation first, first child as fallback.
- The catalog hangs off the bare registry root, outside the edition segment - same asymmetry as the installer, pinned by tests.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Store multi-platform OCI indexes in image layouts
ed3d1c8 
- `AddIndex` writes a whole image index into the layout: platform children, their descriptors, and index annotations stay as published.
- Same (tag, digest) idempotency guard as `AddImage`, so retried pulls do not duplicate descriptors.
- Needed for CLI plugins: their images are multi-platform and flattening to one platform would lose the rest.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Push multi-platform OCI indexes from layouts
ceca572 
- `pushManifest` picks the push call by descriptor media type: `PushIndex` for a nested index, `PushImage` for a single image.
- Before, push failed on any layout with an index: `index.Image()` rejects index media types.
- CLI plugin images are such indexes: platform children and the contract annotation must reach the target registry whole.
- Round-trip test against an in-memory registry: index digest, both platform children, and the contract annotation survive the push.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Add plugin catalog access for mirror
7f6d98b 
- `Catalog` reads the plugins registry: names from the directory-as-tags index, stable semver versions newest first, contracts decoded from the manifest annotation.
- Results are memoized per pull: one tags listing per plugin, one manifest fetch per version.
- `ErrInvalidContract` marks broken published contracts, so the resolver can skip that version; transport errors stay plain and fail the pull.
- Resolver input/output types: typed module versions in, selected plugin versions with provenance out.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Resolve plugin versions against bundled module versions
1bc3ad8 
- For each bundled version of a mirrored module, the newest plugin version the bundle satisfies is picked; picks are deduplicated.
- Mandatory plugin dependencies resolve recursively: shared versions are reused, disjoint constraints get two versions, cycles and depth are guarded.
- A plugin with no compatible version is skipped with the reason recorded; an unmet `--include-plugin` fails the pull.
- Module and Deckhouse constraints go through `requirements.NormalizedForConstraint` (now exported), so mirror and install compare versions the same way.
- Exact pins bypass the stable-version list, so pre-releases stay reachable.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Add plugins phase to d8 mirror pull
fb00666 
- `PullPlugins` resolves plugin versions for the bundle, pulls them, and packs one `plugin-<name>.tar` per plugin.
- Multi-platform indexes are pulled whole: children fetched by digest, so platform binaries and the contract annotation stay as published.
- Tars carry the `deckhouse-cli/plugins/<name>` prefix - the path mirror push uploads verbatim and registry-packages-proxy serves.
- A registry without a plugins catalog skips auto-selection quietly; dry-run prints the plan and fills stats without downloads.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Wire plugins phase into mirror pull pipeline
2ceed48 
- The plugins phase runs last: it resolves against the module and platform versions the earlier phases actually selected.
- New `--include-plugin name[@constraint]` flag adds plugins on top of the automatic selection; with `--proxy-registry` an exact pin is required.
- `PullSummary.Plugins` carries per-version provenance, skips, and advisories for the summary renderer.
- Dependencies on built-in d8 commands (`delivery-kit`, `package`) are never mirrored.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Publish plugin name index on mirror push
959123c 
- `createPluginsIndex` tags the deckhouse-cli/plugins path with one tag per plugin, so `ListTags` discovery works on the target registry.
- The same directory-as-tags convention modules and packages use; matches what registry-bundle serves for bundle-backed registries.
- `PushSummary.Plugins` counts pushed plugin repositories; the push summary prints a Plugins line.
- Plugin layouts are not touched by `--modules-path-suffix`: their registry path is fixed, pinned by a test.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Render plugins in mirror pull summary
615f589 
- The Plugins line shows the count with a provenance breakdown: how many serve modules, how many are dependencies, how many were explicit.
- Verbose mode draws the tree: plugins grouped by the module they serve, dependencies nested under their dependents.
- Skipped plugins with their reasons are always visible - losing a plugin in an air-gapped bundle should not hide behind a flag.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Document plugin mirroring in d8 mirror
886116c 
- Plugin Mirroring section in the mirror README: selection rules, registry layout, what mirror checks vs what install checks.
- Air-gapped flow in docs/plugins.md: pull -> push -> `d8 plugins install` through the proxy.
- Proxy-registry doc: auto-selection needs a catalog, so only exact `--include-plugin` pins work there.
- The push layout doc-tree gains the deckhouse-cli/plugins branch.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep Glitchy-Sheep self-assigned this Aug 14, 2026
@Glitchy-Sheep Glitchy-Sheep added the enhancement New feature or request label Aug 14, 2026
@Glitchy-Sheep Glitchy-Sheep changed the title [deckhouse-cli] Add plugins mirroring [deckhouse-cli] Add plugin mirroring to d8 mirror pull and push Aug 14, 2026
@Glitchy-Sheep
Add e2e coverage for module-driven plugin mirroring
9737b66 
- The e2e suite runs the whole `PullService.Pull`: module versions from release channels reach the plugin resolver and pick compatible plugin versions.
- Checks summary reasons for picked and skipped plugins, the `deckhouse` constraint, exact pins and the `--only-extra-images` skip.
- A pull-to-push roundtrip carries plugin tars into a target registry and checks the plugin name index.
- `--modules-path-suffix` moves modules only - the roundtrip checks that plugins stay put.
- Test stubs now carry a pullable module and a plugin catalog: the plugins phase is no longer a no-op in orchestration and command-level tests.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Drop the unused version parameter from the stub moduleImage
474d565 
- unparam: every call passed stubModuleVersion, so the helper now reads the constant itself.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ldmonster ldmonster Awaiting requested review from ldmonster ldmonster will be requested when the pull request is marked ready for review ldmonster is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@Glitchy-Sheep Glitchy-Sheep

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Glitchy-Sheep