diff --git a/.github/workflows/test-kubectl-plugin.yml b/.github/workflows/test-kubectl-plugin.yml new file mode 100644 index 000000000..4417e18e1 --- /dev/null +++ b/.github/workflows/test-kubectl-plugin.yml @@ -0,0 +1,75 @@ +# Build, vet and unit-test the kubectl-documentdb plugin. +# +# The plugin lives in its own Go module (documentdb-kubectl-plugin/) which is +# outside the path filters of test-unit.yml, so its tests were never executed +# in CI. This workflow gives the module the same build/vet/test treatment the +# operator module gets. +# +# The plugin imports the operator's api/preview package through a `replace` +# directive, so changes under operator/src/api/ can break it; those paths are +# included in the filter. + +name: Unit Test - kubectl plugin + +on: + push: + branches: [main] + paths: + - 'documentdb-kubectl-plugin/**' + - 'operator/src/api/**' + - '.github/workflows/test-kubectl-plugin.yml' + pull_request: + branches: [main] + paths: + - 'documentdb-kubectl-plugin/**' + - 'operator/src/api/**' + - '.github/workflows/test-kubectl-plugin.yml' + +permissions: + contents: read + +jobs: + kubectl-plugin-test: + name: Build and Test kubectl Plugin + runs-on: ubuntu-22.04 + timeout-minutes: 10 + defaults: + run: + working-directory: documentdb-kubectl-plugin + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Go + uses: actions/setup-go@v5 + with: + go-version-file: documentdb-kubectl-plugin/go.mod + cache-dependency-path: documentdb-kubectl-plugin/go.sum + + - name: Download dependencies + run: go mod download + + - name: Check formatting + run: | + unformatted=$(gofmt -l .) + if [ -n "$unformatted" ]; then + echo "The following files are not gofmt'd:" + echo "$unformatted" + exit 1 + fi + + - name: Build + run: go build ./... + + - name: Run go vet + run: go vet ./... + + - name: Run unit tests + run: go test -v -race -count=1 -timeout 10m ./... + + - name: Test Summary + if: always() + run: | + echo "## kubectl Plugin Test Results" >> "$GITHUB_STEP_SUMMARY" + echo "✅ kubectl plugin build, vet and unit tests completed" >> "$GITHUB_STEP_SUMMARY" diff --git a/CHANGELOG.md b/CHANGELOG.md index 424044906..d624bbfbf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] ### Major Features +- **Backup and restore commands in `kubectl documentdb`**: The kubectl plugin now covers the backup lifecycle end to end. `kubectl documentdb backup create` starts an on-demand backup (with `--retention-days` and an optional `--wait` that exits non-zero on a failed or skipped backup), `kubectl documentdb backup list` shows running/completed/failed/skipped backups with their owning schedule, expiry, and captured schema version, and `kubectl documentdb backup schedule create|list` manages `ScheduledBackup` resources with client-side cron validation. `kubectl documentdb restore` creates a new DocumentDB that bootstraps from an existing backup by cloning the source cluster's spec, dropping `spec.clusterReplication`, and setting `spec.bootstrap.recovery.backup`; use `--dry-run` to review the manifest first. See the [kubectl plugin guide](docs/operator-public-documentation/preview/kubectl-plugin.md). Resolves [#139](https://github.com/documentdb/documentdb-kubernetes-operator/issues/139). - **Fail-fast ImageVolume capability check**: The operator now depends on the Kubernetes [ImageVolume](https://kubernetes.io/docs/concepts/storage/volumes/#image) feature to mount the DocumentDB extension into PostgreSQL pods. Instead of gating on a Kubernetes version number, the validating webhook performs a capability probe (a server-side dry-run) when a `DocumentDB` is created and **rejects the resource with an actionable error if ImageVolume is unavailable**, so you find out immediately instead of waiting for pods that never become ready. ImageVolume is GA (on by default) in Kubernetes **1.35+**; on **1.33/1.34** it is beta and must be enabled via the `ImageVolume` feature gate on a containerd/CRI-O runtime. The Helm chart's `kubeVersion` floor is relaxed to `>= 1.33.0-0` accordingly. See [Before you start](docs/operator-public-documentation/preview/getting-started/before-you-start.md). ## [0.3.0] - 2026-07-15 diff --git a/docs/operator-public-documentation/preview/kubectl-plugin.md b/docs/operator-public-documentation/preview/kubectl-plugin.md index 48bc8c3a3..351e282a6 100644 --- a/docs/operator-public-documentation/preview/kubectl-plugin.md +++ b/docs/operator-public-documentation/preview/kubectl-plugin.md @@ -1,6 +1,6 @@ # kubectl-documentdb Plugin -The `kubectl documentdb` plugin provides operational tooling for Azure Cosmos DB for MongoDB (DocumentDB) deployments managed by this operator. It targets day-two operations such as status inspection, event triage, and primary promotion workflows. +The `kubectl documentdb` plugin provides operational tooling for Azure Cosmos DB for MongoDB (DocumentDB) deployments managed by this operator. It targets day-two operations such as status inspection, event triage, backup and restore, and primary promotion workflows. ## Installation @@ -26,10 +26,15 @@ Copy `bin/kubectl-documentdb` onto your `PATH` (renaming is not required). Verif | `kubectl documentdb status` | Collects cluster-wide health information for a DocumentDB CR across all member clusters. | | `kubectl documentdb events` | Streams Kubernetes events scoped to a DocumentDB CR, optionally following new events. | | `kubectl documentdb promote` | Switches the primary cluster in a fleet by patching `spec.clusterReplication.primary` and waiting for convergence. | +| `kubectl documentdb backup create` | Starts an on-demand backup by creating a `Backup` resource. | +| `kubectl documentdb backup list` | Lists running, completed, failed, and skipped backups. | +| `kubectl documentdb backup schedule create` | Creates a recurring backup schedule by creating a `ScheduledBackup` resource. | +| `kubectl documentdb backup schedule list` | Lists backup schedules with their last and next run times. | +| `kubectl documentdb restore` | Creates a new DocumentDB cluster that bootstraps from an existing backup. | Run `kubectl documentdb --help` to review all flags. Key options include: -- `--documentdb`: (required) name of the `DocumentDB` custom resource. +- `--documentdb`: name of the `DocumentDB` custom resource. Required by `status`, `events`, `promote`, `backup create`, and `backup schedule create`; optional as a filter on `backup list` and `backup schedule list`. - `--namespace/-n`: namespace containing the resource. Defaults to `documentdb-preview-ns` for all commands. - `--context`: kubeconfig context to use for hub-level operations (defaults to the current context). - `--show-connections`: include connection strings in `status` output. @@ -37,6 +42,72 @@ Run `kubectl documentdb --help` to review all flags. Key options inclu - `--since`: limit historical events to a relative duration (for example `--since=1h`). - `--target-cluster`: target cluster name for `promote` (required). - `--hub-context` and `--cluster-context`: override hub and target kubeconfig contexts when promoting. +- `--retention-days`: per-backup retention override for `backup create` and `backup schedule create`. Defaults to the cluster's `spec.backup.retentionDays`. +- `--status`: phase filter for `backup list` (`all`, `running`, `completed`, `failed`, `skipped`). +- `--wait`, `--wait-timeout`, and `--poll-interval`: block until a backup or restore reaches a terminal state. + +## Backup and Restore + +`backup` and `restore` operate on the `Backup`, `ScheduledBackup`, and `DocumentDB` custom resources in a single cluster, so they use `--context` (not `--hub-context`). + +### Taking a backup + +```bash +# Start a backup and return immediately +kubectl documentdb backup create --documentdb sample + +# Start a backup, keep it for 7 days, and block until it finishes +kubectl documentdb backup create --documentdb sample --retention-days 7 --wait +``` + +The backup name defaults to `-`; override it with `--name`. The command verifies the DocumentDB exists before creating the `Backup`, so a typo fails immediately instead of leaving a resource the operator can only reject later. + +Backups are taken from the primary cluster only. In a multi-region deployment a `Backup` created against a standby is marked `skipped` by the operator, and `--wait` reports that as an error. + +### Listing backups + +```bash +# Every backup in the namespace +kubectl documentdb backup list + +# Only the backups of one cluster that are still running +kubectl documentdb backup list --documentdb sample --status running + +# Only the backups produced by a given schedule +kubectl documentdb backup list --scheduled-backup nightly +``` + +The table reports phase, the owning schedule (if any), start/stop/expiry times, and the DocumentDB schema version captured at backup time. Newest backups are listed first. + +### Scheduling backups + +```bash +# Back up every day at 02:00, keeping each backup for 14 days +kubectl documentdb backup schedule create --documentdb sample --schedule "0 2 * * *" --retention-days 14 + +kubectl documentdb backup schedule list --documentdb sample +``` + +The schedule name defaults to `-schedule`. Cron expressions are validated locally with the same parser the operator uses, so an invalid expression is rejected before the resource is created. + +### Restoring + +```bash +# Preview the manifest that would be created +kubectl documentdb restore --from-backup sample-20260101-020000 --name sample-restored --dry-run + +# Create the restored cluster and wait for it to become healthy +kubectl documentdb restore --from-backup sample-20260101-020000 --name sample-restored --wait +``` + +`restore` builds a **new** DocumentDB resource; it never overwrites an existing cluster. The new spec is cloned from the DocumentDB the backup was taken from, so storage, resources, and version settings carry over, with two deliberate changes: + +- `spec.bootstrap.recovery.backup.name` is set to the backup being restored. +- `spec.clusterReplication` is dropped, because the restored cluster starts standalone. + +Use `--source-documentdb` when the original cluster no longer exists and you want to use another cluster's spec as the template. By default only `completed` backups can be restored; `--allow-incomplete-backup` overrides that guard. + +A restore must target a binary at or above the schema version recorded on the backup. The command prints the backup's schema version so you can check this before the operator rejects the restore. ## Kubeconfig Expectations @@ -49,12 +120,16 @@ The plugin never modifies kubeconfig files; it only reads them through `client-g - **Status** prints a table containing cluster role, phase, pod readiness, service endpoints, and any retrieval errors per member cluster. Pass `--show-connections` to include the hub-reported primary connection string. - **Events** prints the latest matching events immediately and switches to watch mode while `--follow` remains true. - **Promote** patches the DocumentDB resource in the fleet hub, then (unless `--skip-wait` is used) polls both the hub and the target cluster until the reconciliation reports the desired primary cluster. +- **Backup** prints the created resource name and, with `--wait`, exits non-zero when the backup ends in `failed` or `skipped` so it can be used in scripts. +- **Restore** prints the rendered manifest with `--dry-run`, otherwise creates the DocumentDB and reports the backup's schema version. ## Troubleshooting - Ensure the operator has already synchronized status for the target resource; otherwise `status` may report unknown phases. - If you see context lookup errors, verify the context name exists via `kubectl config get-contexts` and matches the cluster list entry. - Promotion waits until `status.status` reports a healthy phase on both hub and target contexts. Use `--poll-interval` and `--wait-timeout` to tune. +- A backup stuck before `running` usually means no default `VolumeSnapshotClass` exists. Check `kubectl documentdb events --documentdb ` for the operator's warning. +- `restore` reports `source DocumentDB ... not found` when the original cluster has been deleted. Pass `--source-documentdb` to point at another cluster whose spec should be used as the template. ## Contributing diff --git a/documentdb-kubectl-plugin/cmd/backup.go b/documentdb-kubectl-plugin/cmd/backup.go new file mode 100644 index 000000000..d6055d7cd --- /dev/null +++ b/documentdb-kubectl-plugin/cmd/backup.go @@ -0,0 +1,678 @@ +package cmd + +import ( + "context" + "errors" + "fmt" + "sort" + "strings" + "text/tabwriter" + "time" + + cnpgv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" + "github.com/robfig/cron" + "github.com/spf13/cobra" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/dynamic" + + "github.com/documentdb/documentdb-operator/api/preview" +) + +const ( + backupKind = "Backup" + scheduledBackupKind = "ScheduledBackup" + + // scheduledBackupLabel is the label the operator stamps on Backup resources + // it creates from a ScheduledBackup. See ScheduledBackup.CreateBackup. + scheduledBackupLabel = "scheduledbackup" + + // backupNameTimestampLayout matches the suffix the operator uses when it + // generates Backup names for a ScheduledBackup, so manually created and + // scheduled backups sort and read consistently. + backupNameTimestampLayout = "20060102-150405" +) + +// nowFunc is overridable in tests so generated resource names are deterministic. +var nowFunc = time.Now + +// backupStatusFilter enumerates the accepted values of `backup list --status`. +type backupStatusFilter string + +const ( + backupStatusAll backupStatusFilter = "all" + backupStatusRunning backupStatusFilter = "running" + backupStatusCompleted backupStatusFilter = "completed" + backupStatusFailed backupStatusFilter = "failed" + backupStatusSkipped backupStatusFilter = "skipped" +) + +func newBackupCommand() *cobra.Command { + cmd := &cobra.Command{ + Use: "backup", + Short: "Create and inspect DocumentDB backups", + RunE: func(cmd *cobra.Command, args []string) error { + return cmd.Help() + }, + } + + cmd.AddCommand(newBackupCreateCommand()) + cmd.AddCommand(newBackupListCommand()) + cmd.AddCommand(newBackupScheduleCommand()) + + return cmd +} + +// --------------------------------------------------------------------------- +// backup create +// --------------------------------------------------------------------------- + +type backupCreateOptions struct { + documentDBName string + backupName string + namespace string + kubeContext string + retentionDays int + wait bool + waitTimeout time.Duration + pollInterval time.Duration +} + +func newBackupCreateCommand() *cobra.Command { + opts := &backupCreateOptions{namespace: defaultDocumentDBNamespace} + + cmd := &cobra.Command{ + Use: "create", + Short: "Start an on-demand backup of a DocumentDB cluster", + RunE: func(cmd *cobra.Command, args []string) error { + if err := opts.complete(); err != nil { + return err + } + return opts.run(cmd.Context(), cmd) + }, + } + + cmd.Flags().StringVar(&opts.documentDBName, "documentdb", opts.documentDBName, "Name of the DocumentDB resource to back up") + cmd.Flags().StringVar(&opts.backupName, "name", opts.backupName, "Name of the Backup resource to create (defaults to -)") + cmd.Flags().StringVarP(&opts.namespace, "namespace", "n", opts.namespace, "Namespace containing the DocumentDB resource") + cmd.Flags().StringVar(&opts.kubeContext, "context", opts.kubeContext, "Kubeconfig context to use (defaults to current context)") + cmd.Flags().IntVar(&opts.retentionDays, "retention-days", 0, "Days to retain this backup (defaults to the cluster's backup retention policy)") + cmd.Flags().BoolVar(&opts.wait, "wait", false, "Wait for the backup to reach a terminal phase before returning") + cmd.Flags().DurationVar(&opts.waitTimeout, "wait-timeout", 30*time.Minute, "Maximum time to wait when --wait is set") + cmd.Flags().DurationVar(&opts.pollInterval, "poll-interval", 10*time.Second, "Polling interval when --wait is set") + + _ = cmd.MarkFlagRequired("documentdb") + + return cmd +} + +func (o *backupCreateOptions) complete() error { + o.documentDBName = strings.TrimSpace(o.documentDBName) + if o.documentDBName == "" { + return errors.New("--documentdb is required") + } + + o.namespace = strings.TrimSpace(o.namespace) + if o.namespace == "" { + o.namespace = defaultDocumentDBNamespace + } + + o.kubeContext = strings.TrimSpace(o.kubeContext) + + o.backupName = strings.TrimSpace(o.backupName) + if o.backupName == "" { + o.backupName = fmt.Sprintf("%s-%s", o.documentDBName, nowFunc().UTC().Format(backupNameTimestampLayout)) + } + + if o.retentionDays < 0 { + return fmt.Errorf("--retention-days must be greater than zero, got %d", o.retentionDays) + } + + if o.waitTimeout <= 0 { + o.waitTimeout = 30 * time.Minute + } + if o.pollInterval <= 0 { + o.pollInterval = 10 * time.Second + } + + return nil +} + +func (o *backupCreateOptions) run(ctx context.Context, cmd *cobra.Command) error { + config, contextName, err := loadConfigFunc(o.kubeContext) + if err != nil { + return fmt.Errorf("failed to load kubeconfig: %w", err) + } + if contextName == "" { + contextName = "(current)" + } + + dyn, err := dynamicClientForConfig(config) + if err != nil { + return fmt.Errorf("failed to create dynamic client: %w", err) + } + + // Fail fast with a clear message instead of leaving an orphaned Backup that + // the operator can only reject once it reconciles. + if _, err := dyn.Resource(documentDBGVR()).Namespace(o.namespace).Get(ctx, o.documentDBName, metav1.GetOptions{}); err != nil { + if apierrors.IsNotFound(err) { + return fmt.Errorf("DocumentDB %q not found in namespace %q", o.documentDBName, o.namespace) + } + return fmt.Errorf("failed to get DocumentDB %q in namespace %q: %w", o.documentDBName, o.namespace, err) + } + + backup := &preview.Backup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: documentDBGVRGroup + "/" + documentDBGVRVersion, + Kind: backupKind, + }, + ObjectMeta: metav1.ObjectMeta{ + Name: o.backupName, + Namespace: o.namespace, + }, + Spec: preview.BackupSpec{ + Cluster: cnpgv1.LocalObjectReference{Name: o.documentDBName}, + }, + } + if o.retentionDays > 0 { + retention := o.retentionDays + backup.Spec.RetentionDays = &retention + } + + obj, err := toUnstructured(backup) + if err != nil { + return err + } + + if _, err := dyn.Resource(backupGVR()).Namespace(o.namespace).Create(ctx, obj, metav1.CreateOptions{}); err != nil { + if apierrors.IsAlreadyExists(err) { + return fmt.Errorf("Backup %q already exists in namespace %q; pass --name to choose a different name", o.backupName, o.namespace) + } + return fmt.Errorf("failed to create Backup %q in namespace %q: %w", o.backupName, o.namespace, err) + } + + fmt.Fprintf(cmd.OutOrStdout(), "Backup %s/%s requested for DocumentDB %q (context %s).\n", o.namespace, o.backupName, o.documentDBName, contextName) + + if !o.wait { + fmt.Fprintf(cmd.OutOrStdout(), "Track progress with: kubectl documentdb backup list --documentdb %s -n %s\n", o.documentDBName, o.namespace) + return nil + } + + fmt.Fprintf(cmd.OutOrStdout(), "Waiting up to %s for the backup to finish...\n", o.waitTimeout) + return o.waitForBackup(ctx, cmd, dyn) +} + +func (o *backupCreateOptions) waitForBackup(ctx context.Context, cmd *cobra.Command, dyn dynamic.Interface) error { + ctx, cancel := context.WithTimeout(ctx, o.waitTimeout) + defer cancel() + + ticker := time.NewTicker(o.pollInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return fmt.Errorf("timed out after %s waiting for Backup %s/%s to finish", o.waitTimeout, o.namespace, o.backupName) + case <-ticker.C: + obj, err := dyn.Resource(backupGVR()).Namespace(o.namespace).Get(ctx, o.backupName, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + // The operator garbage-collects expired backups; treat a + // disappearing resource as a hard failure rather than + // spinning until the timeout. + return fmt.Errorf("Backup %s/%s no longer exists", o.namespace, o.backupName) + } + return fmt.Errorf("failed to get Backup %s/%s: %w", o.namespace, o.backupName, err) + } + + backup, err := toBackup(obj) + if err != nil { + return err + } + if !backup.Status.IsDone() { + continue + } + + switch backup.Status.Phase { + case cnpgv1.BackupPhaseCompleted: + fmt.Fprintf(cmd.OutOrStdout(), "Backup %s/%s completed.\n", o.namespace, o.backupName) + return nil + case preview.BackupPhaseSkipped: + return fmt.Errorf("Backup %s/%s was skipped: %s", o.namespace, o.backupName, safeValue(backup.Status.Message)) + default: + return fmt.Errorf("Backup %s/%s failed: %s", o.namespace, o.backupName, safeValue(backup.Status.Message)) + } + } + } +} + +// --------------------------------------------------------------------------- +// backup list +// --------------------------------------------------------------------------- + +type backupListOptions struct { + documentDBName string + scheduledBackup string + namespace string + kubeContext string + status string + statusFilter backupStatusFilter +} + +func newBackupListCommand() *cobra.Command { + opts := &backupListOptions{namespace: defaultDocumentDBNamespace, status: string(backupStatusAll)} + + cmd := &cobra.Command{ + Use: "list", + Short: "List completed, running, and failed backups", + RunE: func(cmd *cobra.Command, args []string) error { + if err := opts.complete(); err != nil { + return err + } + return opts.run(cmd.Context(), cmd) + }, + } + + cmd.Flags().StringVar(&opts.documentDBName, "documentdb", opts.documentDBName, "Only list backups of this DocumentDB resource") + cmd.Flags().StringVar(&opts.scheduledBackup, "scheduled-backup", opts.scheduledBackup, "Only list backups created by this ScheduledBackup") + cmd.Flags().StringVarP(&opts.namespace, "namespace", "n", opts.namespace, "Namespace to list backups from") + cmd.Flags().StringVar(&opts.kubeContext, "context", opts.kubeContext, "Kubeconfig context to use (defaults to current context)") + cmd.Flags().StringVar(&opts.status, "status", opts.status, "Filter by phase: all, running, completed, failed, or skipped") + + return cmd +} + +func (o *backupListOptions) complete() error { + o.documentDBName = strings.TrimSpace(o.documentDBName) + o.scheduledBackup = strings.TrimSpace(o.scheduledBackup) + o.kubeContext = strings.TrimSpace(o.kubeContext) + + o.namespace = strings.TrimSpace(o.namespace) + if o.namespace == "" { + o.namespace = defaultDocumentDBNamespace + } + + status := strings.ToLower(strings.TrimSpace(o.status)) + if status == "" { + status = string(backupStatusAll) + } + switch backupStatusFilter(status) { + case backupStatusAll, backupStatusRunning, backupStatusCompleted, backupStatusFailed, backupStatusSkipped: + o.statusFilter = backupStatusFilter(status) + default: + return fmt.Errorf("invalid --status %q: must be one of all, running, completed, failed, skipped", o.status) + } + + return nil +} + +func (o *backupListOptions) run(ctx context.Context, cmd *cobra.Command) error { + config, contextName, err := loadConfigFunc(o.kubeContext) + if err != nil { + return fmt.Errorf("failed to load kubeconfig: %w", err) + } + if contextName == "" { + contextName = "(current)" + } + + dyn, err := dynamicClientForConfig(config) + if err != nil { + return fmt.Errorf("failed to create dynamic client: %w", err) + } + + listOptions := metav1.ListOptions{} + if o.scheduledBackup != "" { + listOptions.LabelSelector = fmt.Sprintf("%s=%s", scheduledBackupLabel, o.scheduledBackup) + } + + list, err := dyn.Resource(backupGVR()).Namespace(o.namespace).List(ctx, listOptions) + if err != nil { + return fmt.Errorf("failed to list backups in namespace %q: %w", o.namespace, err) + } + + backups := make([]preview.Backup, 0, len(list.Items)) + for idx := range list.Items { + backup, err := toBackup(&list.Items[idx]) + if err != nil { + return err + } + if o.documentDBName != "" && backup.Spec.Cluster.Name != o.documentDBName { + continue + } + if !matchesBackupStatus(backup, o.statusFilter) { + continue + } + backups = append(backups, *backup) + } + + // Newest first so the most relevant backups are at the top of the table. + sort.SliceStable(backups, func(i, j int) bool { + return backups[i].CreationTimestamp.Time.After(backups[j].CreationTimestamp.Time) + }) + + fmt.Fprintf(cmd.OutOrStdout(), "Backups in namespace %s (context %s)\n\n", o.namespace, contextName) + + if len(backups) == 0 { + fmt.Fprintln(cmd.OutOrStdout(), "No backups found.") + return nil + } + + tw := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0) + fmt.Fprintln(tw, "NAME\tDOCUMENTDB\tPHASE\tSCHEDULE\tSTARTED\tCOMPLETED\tEXPIRES\tSCHEMA\tMESSAGE") + for idx := range backups { + backup := &backups[idx] + fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", + backup.Name, + safeValue(backup.Spec.Cluster.Name), + safeValue(string(backup.Status.Phase)), + safeValue(backup.Labels[scheduledBackupLabel]), + formatTime(backup.Status.StartedAt), + formatTime(backup.Status.StoppedAt), + formatTime(backup.Status.ExpiredAt), + safeValue(backup.Status.SchemaVersion), + safeValue(truncateString(backup.Status.Message, 60)), + ) + } + return tw.Flush() +} + +func matchesBackupStatus(backup *preview.Backup, filter backupStatusFilter) bool { + switch filter { + case backupStatusAll: + return true + case backupStatusRunning: + return !backup.Status.IsDone() + case backupStatusCompleted: + return backup.Status.Phase == cnpgv1.BackupPhaseCompleted + case backupStatusSkipped: + return backup.Status.Phase == preview.BackupPhaseSkipped + case backupStatusFailed: + return backup.Status.IsDone() && + backup.Status.Phase != cnpgv1.BackupPhaseCompleted && + backup.Status.Phase != preview.BackupPhaseSkipped + default: + return true + } +} + +// --------------------------------------------------------------------------- +// backup schedule +// --------------------------------------------------------------------------- + +func newBackupScheduleCommand() *cobra.Command { + cmd := &cobra.Command{ + Use: "schedule", + Short: "Manage recurring DocumentDB backup schedules", + RunE: func(cmd *cobra.Command, args []string) error { + return cmd.Help() + }, + } + + cmd.AddCommand(newBackupScheduleCreateCommand()) + cmd.AddCommand(newBackupScheduleListCommand()) + + return cmd +} + +type backupScheduleCreateOptions struct { + documentDBName string + scheduleName string + schedule string + namespace string + kubeContext string + retentionDays int +} + +func newBackupScheduleCreateCommand() *cobra.Command { + opts := &backupScheduleCreateOptions{namespace: defaultDocumentDBNamespace} + + cmd := &cobra.Command{ + Use: "create", + Short: "Create a recurring backup schedule for a DocumentDB cluster", + Example: ` # Back up every day at 02:00 + kubectl documentdb backup schedule create --documentdb sample --schedule "0 2 * * *"`, + RunE: func(cmd *cobra.Command, args []string) error { + if err := opts.complete(); err != nil { + return err + } + return opts.run(cmd.Context(), cmd) + }, + } + + cmd.Flags().StringVar(&opts.documentDBName, "documentdb", opts.documentDBName, "Name of the DocumentDB resource to back up") + cmd.Flags().StringVar(&opts.scheduleName, "name", opts.scheduleName, "Name of the ScheduledBackup resource (defaults to -schedule)") + cmd.Flags().StringVar(&opts.schedule, "schedule", opts.schedule, "Cron expression describing when backups run (required)") + cmd.Flags().StringVarP(&opts.namespace, "namespace", "n", opts.namespace, "Namespace containing the DocumentDB resource") + cmd.Flags().StringVar(&opts.kubeContext, "context", opts.kubeContext, "Kubeconfig context to use (defaults to current context)") + cmd.Flags().IntVar(&opts.retentionDays, "retention-days", 0, "Days to retain each backup (defaults to the cluster's backup retention policy)") + + _ = cmd.MarkFlagRequired("documentdb") + _ = cmd.MarkFlagRequired("schedule") + + return cmd +} + +func (o *backupScheduleCreateOptions) complete() error { + o.documentDBName = strings.TrimSpace(o.documentDBName) + if o.documentDBName == "" { + return errors.New("--documentdb is required") + } + + o.schedule = strings.TrimSpace(o.schedule) + if o.schedule == "" { + return errors.New("--schedule is required") + } + // Validate with the same parser the operator uses so an invalid expression + // is rejected here instead of silently stalling reconciliation. + if _, err := cron.ParseStandard(o.schedule); err != nil { + return fmt.Errorf("invalid --schedule %q: %w", o.schedule, err) + } + + o.namespace = strings.TrimSpace(o.namespace) + if o.namespace == "" { + o.namespace = defaultDocumentDBNamespace + } + + o.kubeContext = strings.TrimSpace(o.kubeContext) + + o.scheduleName = strings.TrimSpace(o.scheduleName) + if o.scheduleName == "" { + o.scheduleName = o.documentDBName + "-schedule" + } + + if o.retentionDays < 0 { + return fmt.Errorf("--retention-days must be greater than zero, got %d", o.retentionDays) + } + + return nil +} + +func (o *backupScheduleCreateOptions) run(ctx context.Context, cmd *cobra.Command) error { + config, contextName, err := loadConfigFunc(o.kubeContext) + if err != nil { + return fmt.Errorf("failed to load kubeconfig: %w", err) + } + if contextName == "" { + contextName = "(current)" + } + + dyn, err := dynamicClientForConfig(config) + if err != nil { + return fmt.Errorf("failed to create dynamic client: %w", err) + } + + if _, err := dyn.Resource(documentDBGVR()).Namespace(o.namespace).Get(ctx, o.documentDBName, metav1.GetOptions{}); err != nil { + if apierrors.IsNotFound(err) { + return fmt.Errorf("DocumentDB %q not found in namespace %q", o.documentDBName, o.namespace) + } + return fmt.Errorf("failed to get DocumentDB %q in namespace %q: %w", o.documentDBName, o.namespace, err) + } + + scheduledBackup := &preview.ScheduledBackup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: documentDBGVRGroup + "/" + documentDBGVRVersion, + Kind: scheduledBackupKind, + }, + ObjectMeta: metav1.ObjectMeta{ + Name: o.scheduleName, + Namespace: o.namespace, + }, + Spec: preview.ScheduledBackupSpec{ + Cluster: cnpgv1.LocalObjectReference{Name: o.documentDBName}, + Schedule: o.schedule, + }, + } + if o.retentionDays > 0 { + retention := o.retentionDays + scheduledBackup.Spec.RetentionDays = &retention + } + + obj, err := toUnstructured(scheduledBackup) + if err != nil { + return err + } + + if _, err := dyn.Resource(scheduledBackupGVR()).Namespace(o.namespace).Create(ctx, obj, metav1.CreateOptions{}); err != nil { + if apierrors.IsAlreadyExists(err) { + return fmt.Errorf("ScheduledBackup %q already exists in namespace %q; pass --name to choose a different name", o.scheduleName, o.namespace) + } + return fmt.Errorf("failed to create ScheduledBackup %q in namespace %q: %w", o.scheduleName, o.namespace, err) + } + + fmt.Fprintf(cmd.OutOrStdout(), "ScheduledBackup %s/%s created for DocumentDB %q with schedule %q (context %s).\n", + o.namespace, o.scheduleName, o.documentDBName, o.schedule, contextName) + return nil +} + +type backupScheduleListOptions struct { + documentDBName string + namespace string + kubeContext string +} + +func newBackupScheduleListCommand() *cobra.Command { + opts := &backupScheduleListOptions{namespace: defaultDocumentDBNamespace} + + cmd := &cobra.Command{ + Use: "list", + Short: "List backup schedules", + RunE: func(cmd *cobra.Command, args []string) error { + if err := opts.complete(); err != nil { + return err + } + return opts.run(cmd.Context(), cmd) + }, + } + + cmd.Flags().StringVar(&opts.documentDBName, "documentdb", opts.documentDBName, "Only list schedules targeting this DocumentDB resource") + cmd.Flags().StringVarP(&opts.namespace, "namespace", "n", opts.namespace, "Namespace to list schedules from") + cmd.Flags().StringVar(&opts.kubeContext, "context", opts.kubeContext, "Kubeconfig context to use (defaults to current context)") + + return cmd +} + +func (o *backupScheduleListOptions) complete() error { + o.documentDBName = strings.TrimSpace(o.documentDBName) + o.kubeContext = strings.TrimSpace(o.kubeContext) + + o.namespace = strings.TrimSpace(o.namespace) + if o.namespace == "" { + o.namespace = defaultDocumentDBNamespace + } + + return nil +} + +func (o *backupScheduleListOptions) run(ctx context.Context, cmd *cobra.Command) error { + config, contextName, err := loadConfigFunc(o.kubeContext) + if err != nil { + return fmt.Errorf("failed to load kubeconfig: %w", err) + } + if contextName == "" { + contextName = "(current)" + } + + dyn, err := dynamicClientForConfig(config) + if err != nil { + return fmt.Errorf("failed to create dynamic client: %w", err) + } + + list, err := dyn.Resource(scheduledBackupGVR()).Namespace(o.namespace).List(ctx, metav1.ListOptions{}) + if err != nil { + return fmt.Errorf("failed to list scheduled backups in namespace %q: %w", o.namespace, err) + } + + schedules := make([]preview.ScheduledBackup, 0, len(list.Items)) + for idx := range list.Items { + var scheduledBackup preview.ScheduledBackup + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(list.Items[idx].Object, &scheduledBackup); err != nil { + return fmt.Errorf("failed to convert ScheduledBackup %q: %w", list.Items[idx].GetName(), err) + } + if o.documentDBName != "" && scheduledBackup.Spec.Cluster.Name != o.documentDBName { + continue + } + schedules = append(schedules, scheduledBackup) + } + + sort.SliceStable(schedules, func(i, j int) bool { + return schedules[i].Name < schedules[j].Name + }) + + fmt.Fprintf(cmd.OutOrStdout(), "Backup schedules in namespace %s (context %s)\n\n", o.namespace, contextName) + + if len(schedules) == 0 { + fmt.Fprintln(cmd.OutOrStdout(), "No backup schedules found.") + return nil + } + + tw := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0) + fmt.Fprintln(tw, "NAME\tDOCUMENTDB\tSCHEDULE\tRETENTION DAYS\tLAST SCHEDULED\tNEXT SCHEDULED") + for idx := range schedules { + scheduledBackup := &schedules[idx] + retention := "-" + if scheduledBackup.Spec.RetentionDays != nil { + retention = fmt.Sprintf("%d", *scheduledBackup.Spec.RetentionDays) + } + fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\n", + scheduledBackup.Name, + safeValue(scheduledBackup.Spec.Cluster.Name), + safeValue(scheduledBackup.Spec.Schedule), + retention, + formatTime(scheduledBackup.Status.LastScheduledTime), + formatTime(scheduledBackup.Status.NextScheduledTime), + ) + } + return tw.Flush() +} + +// --------------------------------------------------------------------------- +// shared helpers +// --------------------------------------------------------------------------- + +func toBackup(obj *unstructured.Unstructured) (*preview.Backup, error) { + var backup preview.Backup + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, &backup); err != nil { + return nil, fmt.Errorf("failed to convert Backup %q: %w", obj.GetName(), err) + } + return &backup, nil +} + +func toUnstructured(obj runtime.Object) (*unstructured.Unstructured, error) { + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(obj) + if err != nil { + return nil, fmt.Errorf("failed to convert %T to unstructured: %w", obj, err) + } + // The status subresource is server-owned; sending an empty one on create is + // noise at best and rejected at worst. + delete(content, "status") + return &unstructured.Unstructured{Object: content}, nil +} + +func formatTime(t *metav1.Time) string { + if t == nil || t.IsZero() { + return "-" + } + return t.UTC().Format(time.RFC3339) +} diff --git a/documentdb-kubectl-plugin/cmd/backup_test.go b/documentdb-kubectl-plugin/cmd/backup_test.go new file mode 100644 index 000000000..d7e460401 --- /dev/null +++ b/documentdb-kubectl-plugin/cmd/backup_test.go @@ -0,0 +1,567 @@ +package cmd + +import ( + "bytes" + "context" + "strings" + "testing" + "time" + + cnpgv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" + "github.com/spf13/cobra" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/rest" + + "github.com/documentdb/documentdb-operator/api/preview" +) + +// installFakeCluster points the command factories at an in-memory cluster +// seeded with objs and returns the client so tests can assert on writes. +func installFakeCluster(t *testing.T, objs ...*unstructured.Unstructured) dynamic.Interface { + t.Helper() + + prevLoad := loadConfigFunc + prevDynamic := dynamicClientForConfig + t.Cleanup(func() { + loadConfigFunc = prevLoad + dynamicClientForConfig = prevDynamic + }) + + client := newFakeDynamicClient(objs...) + loadConfigFunc = func(string) (*rest.Config, string, error) { + return &rest.Config{Host: "test"}, "test-context", nil + } + dynamicClientForConfig = func(*rest.Config) (dynamic.Interface, error) { + return client, nil + } + return client +} + +func newBackupObject(t *testing.T, name, namespace, cluster string, created time.Time, labels map[string]string, status preview.BackupStatus) *unstructured.Unstructured { + t.Helper() + + backup := &preview.Backup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: documentDBGVRGroup + "/" + documentDBGVRVersion, + Kind: backupKind, + }, + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + Labels: labels, + CreationTimestamp: metav1.NewTime(created), + }, + Spec: preview.BackupSpec{ + Cluster: cnpgv1.LocalObjectReference{Name: cluster}, + }, + Status: status, + } + + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(backup) + if err != nil { + t.Fatalf("failed to build Backup object: %v", err) + } + return &unstructured.Unstructured{Object: content} +} + +func newScheduledBackupObject(t *testing.T, name, namespace, cluster, schedule string, retentionDays *int) *unstructured.Unstructured { + t.Helper() + + scheduledBackup := &preview.ScheduledBackup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: documentDBGVRGroup + "/" + documentDBGVRVersion, + Kind: scheduledBackupKind, + }, + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}, + Spec: preview.ScheduledBackupSpec{ + Cluster: cnpgv1.LocalObjectReference{Name: cluster}, + Schedule: schedule, + RetentionDays: retentionDays, + }, + } + + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(scheduledBackup) + if err != nil { + t.Fatalf("failed to build ScheduledBackup object: %v", err) + } + return &unstructured.Unstructured{Object: content} +} + +func newTestCommand() (*cobra.Command, *bytes.Buffer) { + cmd := &cobra.Command{} + var stdout bytes.Buffer + cmd.SetOut(&stdout) + cmd.SetErr(&stdout) + return cmd, &stdout +} + +// --------------------------------------------------------------------------- +// backup create +// --------------------------------------------------------------------------- + +func TestBackupCreateOptionsCompleteDefaults(t *testing.T) { + prevNow := nowFunc + defer func() { nowFunc = prevNow }() + nowFunc = func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) } + + o := &backupCreateOptions{documentDBName: " sample ", namespace: " "} + if err := o.complete(); err != nil { + t.Fatalf("complete returned error: %v", err) + } + if o.documentDBName != "sample" { + t.Fatalf("expected documentDBName trimmed, got %q", o.documentDBName) + } + if o.namespace != defaultDocumentDBNamespace { + t.Fatalf("expected default namespace, got %q", o.namespace) + } + if o.backupName != "sample-20260102-030405" { + t.Fatalf("expected generated backup name, got %q", o.backupName) + } + if o.waitTimeout <= 0 || o.pollInterval <= 0 { + t.Fatalf("expected positive wait settings, got %v/%v", o.waitTimeout, o.pollInterval) + } +} + +func TestBackupCreateOptionsCompleteValidates(t *testing.T) { + t.Parallel() + + cases := map[string]backupCreateOptions{ + "missing documentdb": {}, + "negative retention": {documentDBName: "sample", retentionDays: -1}, + } + for name, opts := range cases { + opts := opts + if err := opts.complete(); err == nil { + t.Fatalf("expected error for case %q", name) + } + } +} + +func TestBackupCreateRunCreatesBackup(t *testing.T) { + namespace := defaultDocumentDBNamespace + client := installFakeCluster(t, newDocument("sample", namespace, "cluster-a", "Ready")) + + cmd, out := newTestCommand() + opts := &backupCreateOptions{ + documentDBName: "sample", + backupName: "sample-backup", + namespace: namespace, + retentionDays: 7, + } + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + + obj, err := client.Resource(backupGVR()).Namespace(namespace).Get(context.Background(), "sample-backup", metav1.GetOptions{}) + if err != nil { + t.Fatalf("expected Backup to be created: %v", err) + } + + backup, err := toBackup(obj) + if err != nil { + t.Fatalf("failed to convert created Backup: %v", err) + } + if backup.Spec.Cluster.Name != "sample" { + t.Fatalf("expected spec.cluster.name 'sample', got %q", backup.Spec.Cluster.Name) + } + if backup.Spec.RetentionDays == nil || *backup.Spec.RetentionDays != 7 { + t.Fatalf("expected retentionDays 7, got %v", backup.Spec.RetentionDays) + } + if _, found := obj.Object["status"]; found { + t.Fatal("expected the server-owned status to be stripped before create") + } + if !strings.Contains(out.String(), "sample-backup") { + t.Fatalf("expected output to name the backup, got %q", out.String()) + } +} + +func TestBackupCreateRunRequiresExistingDocumentDB(t *testing.T) { + installFakeCluster(t) + + cmd, _ := newTestCommand() + opts := &backupCreateOptions{documentDBName: "missing", backupName: "b", namespace: defaultDocumentDBNamespace} + + err := opts.run(context.Background(), cmd) + if err == nil { + t.Fatal("expected an error when the DocumentDB does not exist") + } + if !strings.Contains(err.Error(), "not found") { + t.Fatalf("expected a not-found error, got %v", err) + } +} + +func TestBackupCreateRunRejectsDuplicateName(t *testing.T) { + namespace := defaultDocumentDBNamespace + existing := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, preview.BackupStatus{}) + installFakeCluster(t, newDocument("sample", namespace, "cluster-a", "Ready"), existing) + + cmd, _ := newTestCommand() + opts := &backupCreateOptions{documentDBName: "sample", backupName: "sample-backup", namespace: namespace} + + err := opts.run(context.Background(), cmd) + if err == nil || !strings.Contains(err.Error(), "already exists") { + t.Fatalf("expected an already-exists error, got %v", err) + } +} + +func TestBackupCreateWaitReportsCompletion(t *testing.T) { + namespace := defaultDocumentDBNamespace + client := installFakeCluster(t, newDocument("sample", namespace, "cluster-a", "Ready")) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + go setBackupPhase(ctx, client, namespace, "sample-backup", cnpgv1.BackupPhaseCompleted, "") + + cmd, out := newTestCommand() + opts := &backupCreateOptions{ + documentDBName: "sample", + backupName: "sample-backup", + namespace: namespace, + wait: true, + waitTimeout: 5 * time.Second, + pollInterval: 5 * time.Millisecond, + } + if err := opts.run(ctx, cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + if !strings.Contains(out.String(), "completed") { + t.Fatalf("expected completion message, got %q", out.String()) + } +} + +func TestBackupCreateWaitReportsFailure(t *testing.T) { + namespace := defaultDocumentDBNamespace + client := installFakeCluster(t, newDocument("sample", namespace, "cluster-a", "Ready")) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + go setBackupPhase(ctx, client, namespace, "sample-backup", cnpgv1.BackupPhaseFailed, "snapshot class missing") + + cmd, _ := newTestCommand() + opts := &backupCreateOptions{ + documentDBName: "sample", + backupName: "sample-backup", + namespace: namespace, + wait: true, + waitTimeout: 5 * time.Second, + pollInterval: 5 * time.Millisecond, + } + + err := opts.run(ctx, cmd) + if err == nil || !strings.Contains(err.Error(), "snapshot class missing") { + t.Fatalf("expected the failure message to surface, got %v", err) + } +} + +// setBackupPhase waits for the Backup to appear and then writes a terminal phase, +// standing in for the operator's status updates. +func setBackupPhase(ctx context.Context, client dynamic.Interface, namespace, name, phase, message string) { + for { + select { + case <-ctx.Done(): + return + default: + } + + obj, err := client.Resource(backupGVR()).Namespace(namespace).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + time.Sleep(time.Millisecond) + continue + } + status := map[string]any{"phase": phase} + if message != "" { + status["message"] = message + } + if err := unstructured.SetNestedMap(obj.Object, status, "status"); err != nil { + return + } + if _, err := client.Resource(backupGVR()).Namespace(namespace).Update(ctx, obj, metav1.UpdateOptions{}); err != nil { + return + } + return + } +} + +// --------------------------------------------------------------------------- +// backup list +// --------------------------------------------------------------------------- + +func TestBackupListOptionsCompleteRejectsUnknownStatus(t *testing.T) { + t.Parallel() + + o := &backupListOptions{status: "bogus"} + if err := o.complete(); err == nil { + t.Fatal("expected an error for an unknown --status value") + } + + o = &backupListOptions{} + if err := o.complete(); err != nil { + t.Fatalf("complete returned error: %v", err) + } + if o.statusFilter != backupStatusAll { + t.Fatalf("expected default status filter 'all', got %q", o.statusFilter) + } + if o.namespace != defaultDocumentDBNamespace { + t.Fatalf("expected default namespace, got %q", o.namespace) + } +} + +func TestBackupListRendersAndFilters(t *testing.T) { + namespace := defaultDocumentDBNamespace + base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + + completed := newBackupObject(t, "sample-completed", namespace, "sample", base, + map[string]string{scheduledBackupLabel: "nightly"}, + preview.BackupStatus{ + Phase: cnpgv1.BackupPhaseCompleted, + StartedAt: ptrTime(base), + StoppedAt: ptrTime(base.Add(time.Minute)), + SchemaVersion: "0.113-0", + }) + running := newBackupObject(t, "sample-running", namespace, "sample", base.Add(time.Hour), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseRunning, StartedAt: ptrTime(base.Add(time.Hour))}) + failed := newBackupObject(t, "sample-failed", namespace, "sample", base.Add(2*time.Hour), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseFailed, Message: "boom"}) + other := newBackupObject(t, "other-completed", namespace, "other", base, nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseCompleted}) + + installFakeCluster(t, completed, running, failed, other) + + t.Run("all backups for one documentdb", func(t *testing.T) { + cmd, out := newTestCommand() + opts := &backupListOptions{documentDBName: "sample", namespace: namespace, statusFilter: backupStatusAll} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + output := out.String() + for _, want := range []string{"sample-completed", "sample-running", "sample-failed", "nightly", "0.113-0", "boom"} { + if !strings.Contains(output, want) { + t.Fatalf("expected output to contain %q, got:\n%s", want, output) + } + } + if strings.Contains(output, "other-completed") { + t.Fatalf("expected backups of other clusters to be filtered out, got:\n%s", output) + } + // Newest first. + if strings.Index(output, "sample-failed") > strings.Index(output, "sample-completed") { + t.Fatalf("expected newest backups first, got:\n%s", output) + } + }) + + t.Run("running only", func(t *testing.T) { + cmd, out := newTestCommand() + opts := &backupListOptions{documentDBName: "sample", namespace: namespace, statusFilter: backupStatusRunning} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + output := out.String() + if !strings.Contains(output, "sample-running") { + t.Fatalf("expected running backup, got:\n%s", output) + } + if strings.Contains(output, "sample-completed") || strings.Contains(output, "sample-failed") { + t.Fatalf("expected terminal backups to be filtered out, got:\n%s", output) + } + }) + + t.Run("failed only", func(t *testing.T) { + cmd, out := newTestCommand() + opts := &backupListOptions{namespace: namespace, statusFilter: backupStatusFailed} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + output := out.String() + if !strings.Contains(output, "sample-failed") { + t.Fatalf("expected failed backup, got:\n%s", output) + } + if strings.Contains(output, "sample-running") || strings.Contains(output, "sample-completed") { + t.Fatalf("expected only failed backups, got:\n%s", output) + } + }) + + t.Run("by scheduled backup label", func(t *testing.T) { + cmd, out := newTestCommand() + opts := &backupListOptions{namespace: namespace, scheduledBackup: "nightly", statusFilter: backupStatusAll} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + output := out.String() + if !strings.Contains(output, "sample-completed") { + t.Fatalf("expected the scheduled backup, got:\n%s", output) + } + if strings.Contains(output, "sample-running") { + t.Fatalf("expected non-scheduled backups to be filtered out, got:\n%s", output) + } + }) + + t.Run("no matches", func(t *testing.T) { + cmd, out := newTestCommand() + opts := &backupListOptions{documentDBName: "nope", namespace: namespace, statusFilter: backupStatusAll} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + if !strings.Contains(out.String(), "No backups found") { + t.Fatalf("expected an empty-result message, got:\n%s", out.String()) + } + }) +} + +func TestMatchesBackupStatusTreatsSkippedSeparately(t *testing.T) { + t.Parallel() + + skipped := &preview.Backup{Status: preview.BackupStatus{Phase: preview.BackupPhaseSkipped}} + if !matchesBackupStatus(skipped, backupStatusSkipped) { + t.Fatal("expected a skipped backup to match the skipped filter") + } + if matchesBackupStatus(skipped, backupStatusFailed) { + t.Fatal("expected a skipped backup not to be reported as failed") + } + if matchesBackupStatus(skipped, backupStatusRunning) { + t.Fatal("expected a skipped backup not to be reported as running") + } +} + +// --------------------------------------------------------------------------- +// backup schedule +// --------------------------------------------------------------------------- + +func TestBackupScheduleCreateOptionsComplete(t *testing.T) { + t.Parallel() + + o := &backupScheduleCreateOptions{documentDBName: " sample ", schedule: " 0 2 * * * "} + if err := o.complete(); err != nil { + t.Fatalf("complete returned error: %v", err) + } + if o.scheduleName != "sample-schedule" { + t.Fatalf("expected generated schedule name, got %q", o.scheduleName) + } + if o.namespace != defaultDocumentDBNamespace { + t.Fatalf("expected default namespace, got %q", o.namespace) + } +} + +func TestBackupScheduleCreateOptionsRejectsInvalidCron(t *testing.T) { + t.Parallel() + + o := &backupScheduleCreateOptions{documentDBName: "sample", schedule: "every tuesday"} + err := o.complete() + if err == nil || !strings.Contains(err.Error(), "invalid --schedule") { + t.Fatalf("expected an invalid schedule error, got %v", err) + } +} + +func TestBackupScheduleCreateRun(t *testing.T) { + namespace := defaultDocumentDBNamespace + client := installFakeCluster(t, newDocument("sample", namespace, "cluster-a", "Ready")) + + cmd, out := newTestCommand() + opts := &backupScheduleCreateOptions{ + documentDBName: "sample", + scheduleName: "nightly", + schedule: "0 2 * * *", + namespace: namespace, + retentionDays: 14, + } + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + + obj, err := client.Resource(scheduledBackupGVR()).Namespace(namespace).Get(context.Background(), "nightly", metav1.GetOptions{}) + if err != nil { + t.Fatalf("expected ScheduledBackup to be created: %v", err) + } + + var scheduledBackup preview.ScheduledBackup + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, &scheduledBackup); err != nil { + t.Fatalf("failed to convert ScheduledBackup: %v", err) + } + if scheduledBackup.Spec.Schedule != "0 2 * * *" { + t.Fatalf("unexpected schedule %q", scheduledBackup.Spec.Schedule) + } + if scheduledBackup.Spec.Cluster.Name != "sample" { + t.Fatalf("unexpected cluster %q", scheduledBackup.Spec.Cluster.Name) + } + if scheduledBackup.Spec.RetentionDays == nil || *scheduledBackup.Spec.RetentionDays != 14 { + t.Fatalf("expected retentionDays 14, got %v", scheduledBackup.Spec.RetentionDays) + } + if !strings.Contains(out.String(), "nightly") { + t.Fatalf("expected output to name the schedule, got %q", out.String()) + } +} + +func TestBackupScheduleCreateRunRequiresExistingDocumentDB(t *testing.T) { + installFakeCluster(t) + + cmd, _ := newTestCommand() + opts := &backupScheduleCreateOptions{ + documentDBName: "missing", + scheduleName: "nightly", + schedule: "0 2 * * *", + namespace: defaultDocumentDBNamespace, + } + + err := opts.run(context.Background(), cmd) + if err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("expected a not-found error, got %v", err) + } +} + +func TestBackupScheduleListRenders(t *testing.T) { + namespace := defaultDocumentDBNamespace + retention := 14 + installFakeCluster(t, + newScheduledBackupObject(t, "nightly", namespace, "sample", "0 2 * * *", &retention), + newScheduledBackupObject(t, "weekly", namespace, "other", "0 3 * * 0", nil), + ) + + cmd, out := newTestCommand() + opts := &backupScheduleListOptions{documentDBName: "sample", namespace: namespace} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + + output := out.String() + if !strings.Contains(output, "nightly") || !strings.Contains(output, "0 2 * * *") || !strings.Contains(output, "14") { + t.Fatalf("expected the matching schedule in the output, got:\n%s", output) + } + if strings.Contains(output, "weekly") { + t.Fatalf("expected schedules for other clusters to be filtered out, got:\n%s", output) + } +} + +func TestBackupScheduleListEmpty(t *testing.T) { + installFakeCluster(t) + + cmd, out := newTestCommand() + opts := &backupScheduleListOptions{namespace: defaultDocumentDBNamespace} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + if !strings.Contains(out.String(), "No backup schedules found") { + t.Fatalf("expected an empty-result message, got:\n%s", out.String()) + } +} + +func TestFormatTime(t *testing.T) { + t.Parallel() + + if got := formatTime(nil); got != "-" { + t.Fatalf("expected '-' for a nil time, got %q", got) + } + if got := formatTime(&metav1.Time{}); got != "-" { + t.Fatalf("expected '-' for a zero time, got %q", got) + } + stamp := metav1.NewTime(time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) + if got := formatTime(&stamp); got != "2026-01-02T03:04:05Z" { + t.Fatalf("unexpected formatted time %q", got) + } +} + +func ptrTime(t time.Time) *metav1.Time { + stamp := metav1.NewTime(t) + return &stamp +} diff --git a/documentdb-kubectl-plugin/cmd/fake_dynamic_test.go b/documentdb-kubectl-plugin/cmd/fake_dynamic_test.go index 7885984ab..b812034c0 100644 --- a/documentdb-kubectl-plugin/cmd/fake_dynamic_test.go +++ b/documentdb-kubectl-plugin/cmd/fake_dynamic_test.go @@ -4,11 +4,14 @@ import ( "context" "encoding/json" "fmt" + "sort" + "strings" "sync" apierrors "k8s.io/apimachinery/pkg/api/errors" - "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" @@ -16,14 +19,23 @@ import ( "k8s.io/client-go/dynamic" ) -func documentDBGVR() schema.GroupVersionResource { - return schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: documentDBGVRResource} -} - func documentDBGK() schema.GroupVersionKind { return schema.GroupVersionKind{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Kind: "DocumentDB"} } +// kindForResource maps the resources the plugin touches to their kinds so the +// fake client can keep objects of different types apart. +func kindForResource(resource string) string { + switch resource { + case backupGVRResource: + return backupKind + case scheduledBackupGVRResource: + return scheduledBackupKind + default: + return documentDBKind + } +} + func newDocumentScheme() *runtime.Scheme { scheme := runtime.NewScheme() gk := documentDBGK() @@ -47,7 +59,7 @@ func newFakeDynamicClient(objs ...*unstructured.Unstructured) dynamic.Interface if obj == nil { continue } - key := namespacedName(obj.GetNamespace(), obj.GetName()) + key := objectKey(obj.GetKind(), obj.GetNamespace(), obj.GetName()) c.objects[key] = obj.DeepCopy() } return c @@ -133,7 +145,28 @@ type fakeResource struct { } func (r *fakeResource) Create(ctx context.Context, obj *unstructured.Unstructured, opts metav1.CreateOptions, subresources ...string) (*unstructured.Unstructured, error) { - return nil, fmt.Errorf("not implemented") + if len(subresources) != 0 { + return nil, fmt.Errorf("not implemented") + } + if obj == nil { + return nil, fmt.Errorf("nil object") + } + name := obj.GetName() + if name == "" { + return nil, fmt.Errorf("missing name") + } + + stored := obj.DeepCopy() + stored.SetNamespace(r.namespace) + key := r.key(name) + + r.client.mu.Lock() + defer r.client.mu.Unlock() + if _, exists := r.client.objects[key]; exists { + return nil, apierrors.NewAlreadyExists(schema.GroupResource{Group: r.gvr.Group, Resource: r.gvr.Resource}, name) + } + r.client.objects[key] = stored + return stored.DeepCopy(), nil } func (r *fakeResource) Update(ctx context.Context, obj *unstructured.Unstructured, opts metav1.UpdateOptions, subresources ...string) (*unstructured.Unstructured, error) { @@ -147,7 +180,7 @@ func (r *fakeResource) Update(ctx context.Context, obj *unstructured.Unstructure if name == "" { return nil, fmt.Errorf("missing name") } - key := namespacedName(r.namespace, name) + key := r.key(name) r.client.mu.Lock() defer r.client.mu.Unlock() @@ -171,7 +204,7 @@ func (r *fakeResource) Get(ctx context.Context, name string, opts metav1.GetOpti if len(subresources) != 0 { return nil, fmt.Errorf("not implemented") } - key := namespacedName(r.namespace, name) + key := r.key(name) r.client.mu.RLock() defer r.client.mu.RUnlock() @@ -183,7 +216,37 @@ func (r *fakeResource) Get(ctx context.Context, name string, opts metav1.GetOpti } func (r *fakeResource) List(ctx context.Context, opts metav1.ListOptions) (*unstructured.UnstructuredList, error) { - return nil, fmt.Errorf("not implemented") + selector, err := labels.Parse(opts.LabelSelector) + if err != nil { + return nil, fmt.Errorf("parse label selector: %w", err) + } + + kind := kindForResource(r.gvr.Resource) + prefix := objectKey(kind, r.namespace, "") + + r.client.mu.RLock() + defer r.client.mu.RUnlock() + + list := &unstructured.UnstructuredList{} + list.SetAPIVersion(r.gvr.GroupVersion().String()) + list.SetKind(kind + "List") + + names := make([]string, 0, len(r.client.objects)) + for key := range r.client.objects { + if strings.HasPrefix(key, prefix) { + names = append(names, key) + } + } + sort.Strings(names) + + for _, key := range names { + obj := r.client.objects[key] + if !selector.Matches(labels.Set(obj.GetLabels())) { + continue + } + list.Items = append(list.Items, *obj.DeepCopy()) + } + return list, nil } func (r *fakeResource) Watch(ctx context.Context, opts metav1.ListOptions) (watch.Interface, error) { @@ -203,7 +266,7 @@ func (r *fakeResource) Patch(ctx context.Context, name string, pt types.PatchTyp return nil, fmt.Errorf("unmarshal patch: %w", err) } - key := namespacedName(r.namespace, name) + key := r.key(name) r.client.mu.Lock() defer r.client.mu.Unlock() @@ -245,6 +308,16 @@ func namespacedName(namespace, name string) string { return namespace + "/" + name } +// key scopes a stored object to the resource type being addressed so that a +// DocumentDB, a Backup, and a ScheduledBackup can share a name. +func (r *fakeResource) key(name string) string { + return objectKey(kindForResource(r.gvr.Resource), r.namespace, name) +} + +func objectKey(kind, namespace, name string) string { + return kind + "/" + namespacedName(namespace, name) +} + func mergeMaps(dst map[string]any, patch map[string]any) { for k, v := range patch { patchMap, ok := v.(map[string]any) diff --git a/documentdb-kubectl-plugin/cmd/gvr.go b/documentdb-kubectl-plugin/cmd/gvr.go new file mode 100644 index 000000000..6cc61883f --- /dev/null +++ b/documentdb-kubectl-plugin/cmd/gvr.go @@ -0,0 +1,20 @@ +package cmd + +import "k8s.io/apimachinery/pkg/runtime/schema" + +const ( + backupGVRResource = "backups" + scheduledBackupGVRResource = "scheduledbackups" +) + +func documentDBGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: documentDBGVRResource} +} + +func backupGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: backupGVRResource} +} + +func scheduledBackupGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: scheduledBackupGVRResource} +} diff --git a/documentdb-kubectl-plugin/cmd/promote.go b/documentdb-kubectl-plugin/cmd/promote.go index cdb7d6a50..a3d8deb3e 100644 --- a/documentdb-kubectl-plugin/cmd/promote.go +++ b/documentdb-kubectl-plugin/cmd/promote.go @@ -12,7 +12,6 @@ import ( apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/dynamic" "k8s.io/client-go/rest" @@ -151,7 +150,7 @@ func (o *promoteOptions) run(ctx context.Context, cmd *cobra.Command) error { } func (o *promoteOptions) patchDocumentDB(ctx context.Context, dyn dynamic.Interface) error { - gvr := schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: documentDBGVRResource} + gvr := documentDBGVR() clusterReplicationPatch := map[string]any{ "primary": o.targetCluster, @@ -211,7 +210,7 @@ func (o *promoteOptions) waitForPromotion(ctx context.Context, dynHub, dynTarget ticker := time.NewTicker(o.pollInterval) defer ticker.Stop() - gvr := schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: documentDBGVRResource} + gvr := documentDBGVR() for { select { diff --git a/documentdb-kubectl-plugin/cmd/restore.go b/documentdb-kubectl-plugin/cmd/restore.go new file mode 100644 index 000000000..bbf82f8e9 --- /dev/null +++ b/documentdb-kubectl-plugin/cmd/restore.go @@ -0,0 +1,269 @@ +package cmd + +import ( + "context" + "errors" + "fmt" + "strings" + "time" + + cnpgv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" + "github.com/spf13/cobra" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/client-go/dynamic" + "sigs.k8s.io/yaml" +) + +const documentDBKind = "DocumentDB" + +type restoreOptions struct { + backupName string + targetName string + sourceName string + namespace string + kubeContext string + dryRun bool + wait bool + waitTimeout time.Duration + pollInterval time.Duration + skipPhaseCheck bool +} + +func newRestoreCommand() *cobra.Command { + opts := &restoreOptions{namespace: defaultDocumentDBNamespace} + + cmd := &cobra.Command{ + Use: "restore", + Short: "Restore a backup into a new DocumentDB cluster", + Long: `Restore creates a new DocumentDB resource that bootstraps from an existing Backup. + +The new resource reuses the spec of the DocumentDB the backup was taken from, so +storage, resources, and version settings are carried over. Cluster replication +settings are dropped because the restored cluster starts as a standalone primary.`, + Example: ` # Restore backup 'sample-20260101-020000' into a new cluster named 'sample-restored' + kubectl documentdb restore --from-backup sample-20260101-020000 --name sample-restored + + # Preview the DocumentDB manifest without creating it + kubectl documentdb restore --from-backup sample-20260101-020000 --name sample-restored --dry-run`, + RunE: func(cmd *cobra.Command, args []string) error { + if err := opts.complete(); err != nil { + return err + } + return opts.run(cmd.Context(), cmd) + }, + } + + cmd.Flags().StringVar(&opts.backupName, "from-backup", opts.backupName, "Name of the Backup to restore from (required)") + cmd.Flags().StringVar(&opts.targetName, "name", opts.targetName, "Name of the DocumentDB resource to create (required)") + cmd.Flags().StringVar(&opts.sourceName, "source-documentdb", opts.sourceName, "DocumentDB whose spec is used as the template (defaults to the backup's source cluster)") + cmd.Flags().StringVarP(&opts.namespace, "namespace", "n", opts.namespace, "Namespace containing the Backup and the new DocumentDB resource") + cmd.Flags().StringVar(&opts.kubeContext, "context", opts.kubeContext, "Kubeconfig context to use (defaults to current context)") + cmd.Flags().BoolVar(&opts.dryRun, "dry-run", false, "Print the DocumentDB manifest that would be created and exit") + cmd.Flags().BoolVar(&opts.skipPhaseCheck, "allow-incomplete-backup", false, "Restore even if the backup has not completed (the restore will likely fail)") + cmd.Flags().BoolVar(&opts.wait, "wait", false, "Wait for the restored cluster to report a healthy status") + cmd.Flags().DurationVar(&opts.waitTimeout, "wait-timeout", 30*time.Minute, "Maximum time to wait when --wait is set") + cmd.Flags().DurationVar(&opts.pollInterval, "poll-interval", 15*time.Second, "Polling interval when --wait is set") + + _ = cmd.MarkFlagRequired("from-backup") + _ = cmd.MarkFlagRequired("name") + + return cmd +} + +func (o *restoreOptions) complete() error { + o.backupName = strings.TrimSpace(o.backupName) + if o.backupName == "" { + return errors.New("--from-backup is required") + } + + o.targetName = strings.TrimSpace(o.targetName) + if o.targetName == "" { + return errors.New("--name is required") + } + + o.sourceName = strings.TrimSpace(o.sourceName) + o.kubeContext = strings.TrimSpace(o.kubeContext) + + o.namespace = strings.TrimSpace(o.namespace) + if o.namespace == "" { + o.namespace = defaultDocumentDBNamespace + } + + if o.targetName == o.sourceName { + return fmt.Errorf("--name %q must differ from --source-documentdb; restoring in place is not supported", o.targetName) + } + + if o.waitTimeout <= 0 { + o.waitTimeout = 30 * time.Minute + } + if o.pollInterval <= 0 { + o.pollInterval = 15 * time.Second + } + + return nil +} + +func (o *restoreOptions) run(ctx context.Context, cmd *cobra.Command) error { + config, contextName, err := loadConfigFunc(o.kubeContext) + if err != nil { + return fmt.Errorf("failed to load kubeconfig: %w", err) + } + if contextName == "" { + contextName = "(current)" + } + + dyn, err := dynamicClientForConfig(config) + if err != nil { + return fmt.Errorf("failed to create dynamic client: %w", err) + } + + backupObj, err := dyn.Resource(backupGVR()).Namespace(o.namespace).Get(ctx, o.backupName, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + return fmt.Errorf("Backup %q not found in namespace %q", o.backupName, o.namespace) + } + return fmt.Errorf("failed to get Backup %q in namespace %q: %w", o.backupName, o.namespace, err) + } + + backup, err := toBackup(backupObj) + if err != nil { + return err + } + + if !o.skipPhaseCheck && backup.Status.Phase != cnpgv1.BackupPhaseCompleted { + return fmt.Errorf("Backup %s/%s is in phase %q, not %q; pass --allow-incomplete-backup to restore anyway", + o.namespace, o.backupName, safeValue(string(backup.Status.Phase)), cnpgv1.BackupPhaseCompleted) + } + + sourceName := o.sourceName + if sourceName == "" { + sourceName = backup.Spec.Cluster.Name + } + if sourceName == "" { + return fmt.Errorf("Backup %s/%s does not record a source cluster; pass --source-documentdb", o.namespace, o.backupName) + } + if sourceName == o.targetName { + return fmt.Errorf("--name %q must differ from the backup's source DocumentDB; restoring in place is not supported", o.targetName) + } + + sourceObj, err := dyn.Resource(documentDBGVR()).Namespace(o.namespace).Get(ctx, sourceName, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + return fmt.Errorf("source DocumentDB %q not found in namespace %q; pass --source-documentdb to use another cluster's spec as the template", + sourceName, o.namespace) + } + return fmt.Errorf("failed to get source DocumentDB %q in namespace %q: %w", sourceName, o.namespace, err) + } + + target, err := buildRestoreTarget(sourceObj, o.targetName, o.namespace, o.backupName) + if err != nil { + return err + } + + if o.dryRun { + manifest, err := yaml.Marshal(target.Object) + if err != nil { + return fmt.Errorf("failed to render DocumentDB manifest: %w", err) + } + fmt.Fprint(cmd.OutOrStdout(), string(manifest)) + return nil + } + + if _, err := dyn.Resource(documentDBGVR()).Namespace(o.namespace).Create(ctx, target, metav1.CreateOptions{}); err != nil { + if apierrors.IsAlreadyExists(err) { + return fmt.Errorf("DocumentDB %q already exists in namespace %q; pick a different --name", o.targetName, o.namespace) + } + return fmt.Errorf("failed to create DocumentDB %q in namespace %q: %w", o.targetName, o.namespace, err) + } + + fmt.Fprintf(cmd.OutOrStdout(), "DocumentDB %s/%s created from Backup %q (template %q, context %s).\n", + o.namespace, o.targetName, o.backupName, sourceName, contextName) + if backup.Status.SchemaVersion != "" { + fmt.Fprintf(cmd.OutOrStdout(), "Backup schema version: %s. The restored cluster must run a binary at or above this version.\n", backup.Status.SchemaVersion) + } + + if !o.wait { + fmt.Fprintf(cmd.OutOrStdout(), "Track progress with: kubectl documentdb status --documentdb %s -n %s\n", o.targetName, o.namespace) + return nil + } + + fmt.Fprintf(cmd.OutOrStdout(), "Waiting up to %s for the restored cluster to become healthy...\n", o.waitTimeout) + if err := o.waitForRestore(ctx, dyn); err != nil { + return err + } + + fmt.Fprintf(cmd.OutOrStdout(), "DocumentDB %s/%s is healthy.\n", o.namespace, o.targetName) + return nil +} + +// buildRestoreTarget clones the source DocumentDB spec and points it at the +// backup to recover from. Working on the unstructured spec (rather than the +// typed one) keeps fields the plugin does not know about intact. +func buildRestoreTarget(source *unstructured.Unstructured, targetName, namespace, backupName string) (*unstructured.Unstructured, error) { + sourceSpec, found, err := unstructured.NestedMap(source.Object, "spec") + if err != nil { + return nil, fmt.Errorf("failed to read spec of DocumentDB %q: %w", source.GetName(), err) + } + if !found { + return nil, fmt.Errorf("DocumentDB %q has no spec to use as a restore template", source.GetName()) + } + + // The restored cluster comes up standalone; carrying over the source's + // replication topology would point it at clusters it is not a member of. + delete(sourceSpec, "clusterReplication") + + recovery := map[string]any{ + "backup": map[string]any{"name": backupName}, + } + if err := unstructured.SetNestedMap(sourceSpec, recovery, "bootstrap", "recovery"); err != nil { + return nil, fmt.Errorf("failed to set bootstrap.recovery: %w", err) + } + + target := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": documentDBGVRGroup + "/" + documentDBGVRVersion, + "kind": documentDBKind, + "metadata": map[string]any{ + "name": targetName, + "namespace": namespace, + }, + "spec": sourceSpec, + }} + + return target, nil +} + +func (o *restoreOptions) waitForRestore(ctx context.Context, dyn dynamic.Interface) error { + ctx, cancel := context.WithTimeout(ctx, o.waitTimeout) + defer cancel() + + ticker := time.NewTicker(o.pollInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return fmt.Errorf("timed out after %s waiting for DocumentDB %s/%s to become healthy", o.waitTimeout, o.namespace, o.targetName) + case <-ticker.C: + obj, err := dyn.Resource(documentDBGVR()).Namespace(o.namespace).Get(ctx, o.targetName, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + continue + } + return fmt.Errorf("failed to get DocumentDB %s/%s: %w", o.namespace, o.targetName, err) + } + + // A missing status means the operator has not observed the resource + // yet, which isHealthyPhase optimistically reports as healthy. Wait + // for a reported phase before declaring the restore done. + phase, found, err := unstructured.NestedString(obj.Object, "status", "status") + if err != nil || !found || strings.TrimSpace(phase) == "" { + continue + } + if healthy, _ := isDocumentHealthy(obj); healthy { + return nil + } + } + } +} diff --git a/documentdb-kubectl-plugin/cmd/restore_test.go b/documentdb-kubectl-plugin/cmd/restore_test.go new file mode 100644 index 000000000..cb969adb4 --- /dev/null +++ b/documentdb-kubectl-plugin/cmd/restore_test.go @@ -0,0 +1,289 @@ +package cmd + +import ( + "context" + "strings" + "testing" + "time" + + cnpgv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/dynamic" + + "github.com/documentdb/documentdb-operator/api/preview" +) + +func newRestoreSourceDocument(name, namespace string) *unstructured.Unstructured { + doc := &unstructured.Unstructured{Object: map[string]any{ + "spec": map[string]any{ + "nodeCount": int64(1), + "instancesPerNode": int64(3), + "documentDBImage": "ghcr.io/documentdb/documentdb:0.113.0", + "resource": map[string]any{ + "pvcSize": "10Gi", + }, + // A field the plugin knows nothing about; it must survive the clone. + "someFutureField": "keep-me", + "clusterReplication": map[string]any{ + "primary": "cluster-a", + "clusterList": []any{map[string]any{"name": "cluster-a"}}, + }, + }, + "status": map[string]any{"status": "Ready"}, + }} + doc.SetGroupVersionKind(schema.GroupVersionKind{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Kind: documentDBKind}) + doc.SetName(name) + doc.SetNamespace(namespace) + return doc +} + +func TestRestoreOptionsCompleteValidates(t *testing.T) { + t.Parallel() + + cases := map[string]restoreOptions{ + "missing backup": {targetName: "restored"}, + "missing target": {backupName: "backup"}, + "restore in place": {backupName: "backup", targetName: "sample", sourceName: "sample"}, + } + for name, opts := range cases { + opts := opts + if err := opts.complete(); err == nil { + t.Fatalf("expected error for case %q", name) + } + } + + o := &restoreOptions{backupName: " backup ", targetName: " restored ", namespace: " "} + if err := o.complete(); err != nil { + t.Fatalf("complete returned error: %v", err) + } + if o.backupName != "backup" || o.targetName != "restored" { + t.Fatalf("expected trimmed names, got %q/%q", o.backupName, o.targetName) + } + if o.namespace != defaultDocumentDBNamespace { + t.Fatalf("expected default namespace, got %q", o.namespace) + } + if o.waitTimeout <= 0 || o.pollInterval <= 0 { + t.Fatalf("expected positive wait settings, got %v/%v", o.waitTimeout, o.pollInterval) + } +} + +func TestBuildRestoreTargetClonesSpec(t *testing.T) { + t.Parallel() + + source := newRestoreSourceDocument("sample", defaultDocumentDBNamespace) + + target, err := buildRestoreTarget(source, "sample-restored", defaultDocumentDBNamespace, "sample-backup") + if err != nil { + t.Fatalf("buildRestoreTarget returned error: %v", err) + } + + if target.GetName() != "sample-restored" || target.GetNamespace() != defaultDocumentDBNamespace { + t.Fatalf("unexpected target metadata %s/%s", target.GetNamespace(), target.GetName()) + } + if target.GetKind() != documentDBKind { + t.Fatalf("unexpected kind %q", target.GetKind()) + } + + backupName, found, err := unstructured.NestedString(target.Object, "spec", "bootstrap", "recovery", "backup", "name") + if err != nil || !found { + t.Fatalf("expected spec.bootstrap.recovery.backup.name to be set (found=%v, err=%v)", found, err) + } + if backupName != "sample-backup" { + t.Fatalf("expected recovery backup 'sample-backup', got %q", backupName) + } + + if _, found, _ := unstructured.NestedMap(target.Object, "spec", "clusterReplication"); found { + t.Fatal("expected clusterReplication to be dropped from the restored spec") + } + + if _, found, _ := unstructured.NestedMap(target.Object, "status"); found { + t.Fatal("expected the source status not to be copied") + } + + preserved, _, _ := unstructured.NestedString(target.Object, "spec", "someFutureField") + if preserved != "keep-me" { + t.Fatalf("expected unknown spec fields to be preserved, got %q", preserved) + } + image, _, _ := unstructured.NestedString(target.Object, "spec", "documentDBImage") + if image != "ghcr.io/documentdb/documentdb:0.113.0" { + t.Fatalf("expected the source image to be carried over, got %q", image) + } + + // The source must not be mutated in the process. + if _, found, _ := unstructured.NestedMap(source.Object, "spec", "clusterReplication"); !found { + t.Fatal("expected the source document to be left untouched") + } +} + +func TestRestoreRunRejectsIncompleteBackup(t *testing.T) { + namespace := defaultDocumentDBNamespace + backup := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseRunning}) + installFakeCluster(t, newRestoreSourceDocument("sample", namespace), backup) + + cmd, _ := newTestCommand() + opts := &restoreOptions{backupName: "sample-backup", targetName: "sample-restored", namespace: namespace} + + err := opts.run(context.Background(), cmd) + if err == nil || !strings.Contains(err.Error(), "allow-incomplete-backup") { + t.Fatalf("expected a phase-guard error, got %v", err) + } +} + +func TestRestoreRunRejectsMissingBackup(t *testing.T) { + installFakeCluster(t) + + cmd, _ := newTestCommand() + opts := &restoreOptions{backupName: "nope", targetName: "restored", namespace: defaultDocumentDBNamespace} + + err := opts.run(context.Background(), cmd) + if err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("expected a not-found error, got %v", err) + } +} + +func TestRestoreRunRejectsMissingSourceDocumentDB(t *testing.T) { + namespace := defaultDocumentDBNamespace + backup := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseCompleted}) + installFakeCluster(t, backup) + + cmd, _ := newTestCommand() + opts := &restoreOptions{backupName: "sample-backup", targetName: "sample-restored", namespace: namespace} + + err := opts.run(context.Background(), cmd) + if err == nil || !strings.Contains(err.Error(), "--source-documentdb") { + t.Fatalf("expected guidance about --source-documentdb, got %v", err) + } +} + +func TestRestoreRunCreatesDocumentDB(t *testing.T) { + namespace := defaultDocumentDBNamespace + backup := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseCompleted, SchemaVersion: "0.113-0"}) + client := installFakeCluster(t, newRestoreSourceDocument("sample", namespace), backup) + + cmd, out := newTestCommand() + opts := &restoreOptions{backupName: "sample-backup", targetName: "sample-restored", namespace: namespace} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + + created, err := client.Resource(documentDBGVR()).Namespace(namespace).Get(context.Background(), "sample-restored", metav1.GetOptions{}) + if err != nil { + t.Fatalf("expected the restored DocumentDB to be created: %v", err) + } + backupRef, _, _ := unstructured.NestedString(created.Object, "spec", "bootstrap", "recovery", "backup", "name") + if backupRef != "sample-backup" { + t.Fatalf("expected the restored cluster to bootstrap from the backup, got %q", backupRef) + } + + output := out.String() + if !strings.Contains(output, "sample-restored") { + t.Fatalf("expected output to name the new cluster, got:\n%s", output) + } + if !strings.Contains(output, "0.113-0") { + t.Fatalf("expected the backup schema version to be surfaced, got:\n%s", output) + } +} + +func TestRestoreRunDryRunDoesNotCreate(t *testing.T) { + namespace := defaultDocumentDBNamespace + backup := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseCompleted}) + client := installFakeCluster(t, newRestoreSourceDocument("sample", namespace), backup) + + cmd, out := newTestCommand() + opts := &restoreOptions{backupName: "sample-backup", targetName: "sample-restored", namespace: namespace, dryRun: true} + if err := opts.run(context.Background(), cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + + if _, err := client.Resource(documentDBGVR()).Namespace(namespace).Get(context.Background(), "sample-restored", metav1.GetOptions{}); err == nil { + t.Fatal("expected --dry-run not to create the DocumentDB") + } + + output := out.String() + for _, want := range []string{"kind: DocumentDB", "name: sample-restored", "bootstrap:", "sample-backup"} { + if !strings.Contains(output, want) { + t.Fatalf("expected the rendered manifest to contain %q, got:\n%s", want, output) + } + } + if strings.Contains(output, "clusterReplication") { + t.Fatalf("expected clusterReplication to be dropped from the manifest, got:\n%s", output) + } +} + +func TestRestoreRunRejectsExistingTarget(t *testing.T) { + namespace := defaultDocumentDBNamespace + backup := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseCompleted}) + installFakeCluster(t, + newRestoreSourceDocument("sample", namespace), + newRestoreSourceDocument("sample-restored", namespace), + backup, + ) + + cmd, _ := newTestCommand() + opts := &restoreOptions{backupName: "sample-backup", targetName: "sample-restored", namespace: namespace} + + err := opts.run(context.Background(), cmd) + if err == nil || !strings.Contains(err.Error(), "already exists") { + t.Fatalf("expected an already-exists error, got %v", err) + } +} + +func TestRestoreWaitReportsHealthyCluster(t *testing.T) { + namespace := defaultDocumentDBNamespace + backup := newBackupObject(t, "sample-backup", namespace, "sample", time.Now(), nil, + preview.BackupStatus{Phase: cnpgv1.BackupPhaseCompleted}) + client := installFakeCluster(t, newRestoreSourceDocument("sample", namespace), backup) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + go setDocumentPhase(ctx, client, namespace, "sample-restored", "Healthy") + + cmd, out := newTestCommand() + opts := &restoreOptions{ + backupName: "sample-backup", + targetName: "sample-restored", + namespace: namespace, + wait: true, + waitTimeout: 5 * time.Second, + pollInterval: 5 * time.Millisecond, + } + if err := opts.run(ctx, cmd); err != nil { + t.Fatalf("run returned error: %v", err) + } + if !strings.Contains(out.String(), "is healthy") { + t.Fatalf("expected a healthy message, got:\n%s", out.String()) + } +} + +// setDocumentPhase stands in for the operator writing status.status once the +// restored cluster settles. +func setDocumentPhase(ctx context.Context, client dynamic.Interface, namespace, name, phase string) { + for { + select { + case <-ctx.Done(): + return + default: + } + + obj, err := client.Resource(documentDBGVR()).Namespace(namespace).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + time.Sleep(time.Millisecond) + continue + } + if err := unstructured.SetNestedField(obj.Object, phase, "status", "status"); err != nil { + return + } + if _, err := client.Resource(documentDBGVR()).Namespace(namespace).Update(ctx, obj, metav1.UpdateOptions{}); err != nil { + return + } + return + } +} diff --git a/documentdb-kubectl-plugin/cmd/root.go b/documentdb-kubectl-plugin/cmd/root.go index 548d211e8..d607e1972 100644 --- a/documentdb-kubectl-plugin/cmd/root.go +++ b/documentdb-kubectl-plugin/cmd/root.go @@ -27,4 +27,6 @@ func init() { rootCmd.AddCommand(newPromoteCommand()) rootCmd.AddCommand(newStatusCommand()) rootCmd.AddCommand(newEventsCommand()) + rootCmd.AddCommand(newBackupCommand()) + rootCmd.AddCommand(newRestoreCommand()) } diff --git a/documentdb-kubectl-plugin/cmd/status.go b/documentdb-kubectl-plugin/cmd/status.go index b7545825c..481a73478 100644 --- a/documentdb-kubectl-plugin/cmd/status.go +++ b/documentdb-kubectl-plugin/cmd/status.go @@ -12,7 +12,6 @@ import ( apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" - "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" ) @@ -90,7 +89,7 @@ func (o *statusOptions) run(ctx context.Context, cmd *cobra.Command) error { return fmt.Errorf("failed to create hub dynamic client: %w", err) } - gvr := schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: documentDBGVRResource} + gvr := documentDBGVR() document, err := dynHub.Resource(gvr).Namespace(o.namespace).Get(ctx, o.documentDBName, metav1.GetOptions{}) if err != nil { @@ -191,7 +190,7 @@ func (o *statusOptions) populateClusterStatus(ctx context.Context, st *clusterSt return fmt.Errorf("dynamic client: %w", err) } - gvr := schema.GroupVersionResource{Group: documentDBGVRGroup, Version: documentDBGVRVersion, Resource: documentDBGVRResource} + gvr := documentDBGVR() document, err := dynClient.Resource(gvr).Namespace(o.namespace).Get(ctx, o.documentDBName, metav1.GetOptions{}) if err != nil { diff --git a/documentdb-kubectl-plugin/go.mod b/documentdb-kubectl-plugin/go.mod index 71b30febc..6ac8f2a4a 100644 --- a/documentdb-kubectl-plugin/go.mod +++ b/documentdb-kubectl-plugin/go.mod @@ -3,11 +3,14 @@ module github.com/documentdb/documentdb-operator/documentdb-kubectl-plugin go 1.26.6 require ( + github.com/cloudnative-pg/cloudnative-pg v1.29.2 github.com/documentdb/documentdb-operator v0.0.0 + github.com/robfig/cron v1.2.0 github.com/spf13/cobra v1.10.2 k8s.io/api v0.36.2 k8s.io/apimachinery v0.36.2 k8s.io/client-go v0.36.2 + sigs.k8s.io/yaml v1.6.0 ) replace github.com/documentdb/documentdb-operator => ../operator/src @@ -17,7 +20,6 @@ require ( github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cloudnative-pg/barman-cloud v0.5.1 // indirect - github.com/cloudnative-pg/cloudnative-pg v1.29.2 // indirect github.com/cloudnative-pg/cnpg-i v0.5.0 // indirect github.com/cloudnative-pg/machinery v0.5.0 // indirect github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect @@ -58,7 +60,6 @@ require ( github.com/prometheus/client_model v0.6.2 // indirect github.com/prometheus/common v0.67.5 // indirect github.com/prometheus/procfs v0.19.2 // indirect - github.com/robfig/cron v1.2.0 // indirect github.com/spf13/pflag v1.0.10 // indirect github.com/thoas/go-funk v0.9.3 // indirect github.com/x448/float16 v0.8.4 // indirect @@ -88,5 +89,4 @@ require ( sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect sigs.k8s.io/randfill v1.0.0 // indirect sigs.k8s.io/structured-merge-diff/v6 v6.4.0 // indirect - sigs.k8s.io/yaml v1.6.0 // indirect ) diff --git a/documentdb-kubectl-plugin/kubectl-plugin.md b/documentdb-kubectl-plugin/kubectl-plugin.md index 48bc8c3a3..351e282a6 100644 --- a/documentdb-kubectl-plugin/kubectl-plugin.md +++ b/documentdb-kubectl-plugin/kubectl-plugin.md @@ -1,6 +1,6 @@ # kubectl-documentdb Plugin -The `kubectl documentdb` plugin provides operational tooling for Azure Cosmos DB for MongoDB (DocumentDB) deployments managed by this operator. It targets day-two operations such as status inspection, event triage, and primary promotion workflows. +The `kubectl documentdb` plugin provides operational tooling for Azure Cosmos DB for MongoDB (DocumentDB) deployments managed by this operator. It targets day-two operations such as status inspection, event triage, backup and restore, and primary promotion workflows. ## Installation @@ -26,10 +26,15 @@ Copy `bin/kubectl-documentdb` onto your `PATH` (renaming is not required). Verif | `kubectl documentdb status` | Collects cluster-wide health information for a DocumentDB CR across all member clusters. | | `kubectl documentdb events` | Streams Kubernetes events scoped to a DocumentDB CR, optionally following new events. | | `kubectl documentdb promote` | Switches the primary cluster in a fleet by patching `spec.clusterReplication.primary` and waiting for convergence. | +| `kubectl documentdb backup create` | Starts an on-demand backup by creating a `Backup` resource. | +| `kubectl documentdb backup list` | Lists running, completed, failed, and skipped backups. | +| `kubectl documentdb backup schedule create` | Creates a recurring backup schedule by creating a `ScheduledBackup` resource. | +| `kubectl documentdb backup schedule list` | Lists backup schedules with their last and next run times. | +| `kubectl documentdb restore` | Creates a new DocumentDB cluster that bootstraps from an existing backup. | Run `kubectl documentdb --help` to review all flags. Key options include: -- `--documentdb`: (required) name of the `DocumentDB` custom resource. +- `--documentdb`: name of the `DocumentDB` custom resource. Required by `status`, `events`, `promote`, `backup create`, and `backup schedule create`; optional as a filter on `backup list` and `backup schedule list`. - `--namespace/-n`: namespace containing the resource. Defaults to `documentdb-preview-ns` for all commands. - `--context`: kubeconfig context to use for hub-level operations (defaults to the current context). - `--show-connections`: include connection strings in `status` output. @@ -37,6 +42,72 @@ Run `kubectl documentdb --help` to review all flags. Key options inclu - `--since`: limit historical events to a relative duration (for example `--since=1h`). - `--target-cluster`: target cluster name for `promote` (required). - `--hub-context` and `--cluster-context`: override hub and target kubeconfig contexts when promoting. +- `--retention-days`: per-backup retention override for `backup create` and `backup schedule create`. Defaults to the cluster's `spec.backup.retentionDays`. +- `--status`: phase filter for `backup list` (`all`, `running`, `completed`, `failed`, `skipped`). +- `--wait`, `--wait-timeout`, and `--poll-interval`: block until a backup or restore reaches a terminal state. + +## Backup and Restore + +`backup` and `restore` operate on the `Backup`, `ScheduledBackup`, and `DocumentDB` custom resources in a single cluster, so they use `--context` (not `--hub-context`). + +### Taking a backup + +```bash +# Start a backup and return immediately +kubectl documentdb backup create --documentdb sample + +# Start a backup, keep it for 7 days, and block until it finishes +kubectl documentdb backup create --documentdb sample --retention-days 7 --wait +``` + +The backup name defaults to `-`; override it with `--name`. The command verifies the DocumentDB exists before creating the `Backup`, so a typo fails immediately instead of leaving a resource the operator can only reject later. + +Backups are taken from the primary cluster only. In a multi-region deployment a `Backup` created against a standby is marked `skipped` by the operator, and `--wait` reports that as an error. + +### Listing backups + +```bash +# Every backup in the namespace +kubectl documentdb backup list + +# Only the backups of one cluster that are still running +kubectl documentdb backup list --documentdb sample --status running + +# Only the backups produced by a given schedule +kubectl documentdb backup list --scheduled-backup nightly +``` + +The table reports phase, the owning schedule (if any), start/stop/expiry times, and the DocumentDB schema version captured at backup time. Newest backups are listed first. + +### Scheduling backups + +```bash +# Back up every day at 02:00, keeping each backup for 14 days +kubectl documentdb backup schedule create --documentdb sample --schedule "0 2 * * *" --retention-days 14 + +kubectl documentdb backup schedule list --documentdb sample +``` + +The schedule name defaults to `-schedule`. Cron expressions are validated locally with the same parser the operator uses, so an invalid expression is rejected before the resource is created. + +### Restoring + +```bash +# Preview the manifest that would be created +kubectl documentdb restore --from-backup sample-20260101-020000 --name sample-restored --dry-run + +# Create the restored cluster and wait for it to become healthy +kubectl documentdb restore --from-backup sample-20260101-020000 --name sample-restored --wait +``` + +`restore` builds a **new** DocumentDB resource; it never overwrites an existing cluster. The new spec is cloned from the DocumentDB the backup was taken from, so storage, resources, and version settings carry over, with two deliberate changes: + +- `spec.bootstrap.recovery.backup.name` is set to the backup being restored. +- `spec.clusterReplication` is dropped, because the restored cluster starts standalone. + +Use `--source-documentdb` when the original cluster no longer exists and you want to use another cluster's spec as the template. By default only `completed` backups can be restored; `--allow-incomplete-backup` overrides that guard. + +A restore must target a binary at or above the schema version recorded on the backup. The command prints the backup's schema version so you can check this before the operator rejects the restore. ## Kubeconfig Expectations @@ -49,12 +120,16 @@ The plugin never modifies kubeconfig files; it only reads them through `client-g - **Status** prints a table containing cluster role, phase, pod readiness, service endpoints, and any retrieval errors per member cluster. Pass `--show-connections` to include the hub-reported primary connection string. - **Events** prints the latest matching events immediately and switches to watch mode while `--follow` remains true. - **Promote** patches the DocumentDB resource in the fleet hub, then (unless `--skip-wait` is used) polls both the hub and the target cluster until the reconciliation reports the desired primary cluster. +- **Backup** prints the created resource name and, with `--wait`, exits non-zero when the backup ends in `failed` or `skipped` so it can be used in scripts. +- **Restore** prints the rendered manifest with `--dry-run`, otherwise creates the DocumentDB and reports the backup's schema version. ## Troubleshooting - Ensure the operator has already synchronized status for the target resource; otherwise `status` may report unknown phases. - If you see context lookup errors, verify the context name exists via `kubectl config get-contexts` and matches the cluster list entry. - Promotion waits until `status.status` reports a healthy phase on both hub and target contexts. Use `--poll-interval` and `--wait-timeout` to tune. +- A backup stuck before `running` usually means no default `VolumeSnapshotClass` exists. Check `kubectl documentdb events --documentdb ` for the operator's warning. +- `restore` reports `source DocumentDB ... not found` when the original cluster has been deleted. Pass `--source-documentdb` to point at another cluster whose spec should be used as the template. ## Contributing