feat(search): search index migration for both backends#3146
Draft
dschmidt wants to merge 3 commits into
Draft
Conversation
Not up to standards ⛔🟢 Coverage
|
| Metric | Results |
|---|---|
| Coverage variation | Report missing for cbd19621 |
| Diff coverage | ✅ 52.59% diff coverage |
Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (cbd1962) Report Missing Report Missing Report Missing Head commit (3c98a31) 83519 19844 23.76% Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>
Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#3146) 521 274 52.59% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
1 Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
dschmidt
force-pushed
the
feat/search-index-migration
branch
2 times, most recently
from
July 16, 2026 16:03
2e076d0 to
7cfe40e
Compare
Migrate the search index in place instead of a full Tika re-crawl when the shared search.SchemaRevision is bumped. bleve (single instance): rebuild from the existing documents into a temp index and atomically swap, with crash recovery; auto-migrate on startup (opt-out via SEARCH_ENGINE_BLEVE_AUTO_MIGRATE). OpenSearch (scaled): versioned indices <base>-v<rev> (no alias), filled from the newest older index via a create-only scroll reindex - idempotent, never overwrites, safe to run live. 'opencloud search index migrate' fills the current-revision index; 'opencloud search index prune' drops older ones. Per-document fixups are gated on the source schema version. Trash preserved, geopoint siblings synthesized, all fields round-tripped.
dschmidt
force-pushed
the
feat/search-index-migration
branch
from
July 16, 2026 19:53
97936ad to
4f71ff8
Compare
Alternative that drops the in-place rebuild machinery (buildReplacement, swap, recoverMigration, verifyMigrated, the fixup hook) and just wipes the outdated bleve index so a full 'search index --force-rescan' repopulates it. ~272 lines lighter, but trades the fast no-rescan migration for a Tika re-crawl on every schema bump. Kept in history for reference; reverted in the next commit - the in-place migration is worth the ~300 lines for bleve too.
…tion" Keep the in-place bleve migration: the fast no-rescan rebuild is worth its ~300 lines even for bleve (avoids a Tika re-crawl on schema bumps). The reset variant stays in the previous commit for reference. This reverts commit 4788a98.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3098 (base = #3098 + the Mtime
*time.Timefix), so this shows only its own commits.In-process index migration for bleve (dir swap) and OpenSearch (alias flip), triggered by a shared
search.SchemaRevisionbump, with anopencloud search migrateCLI. Reindexes from the existing index instead of a full Tika rescan, preserves trash, paginates with search_after, logs start/progress/end.bleve migrates on startup by default (opt-out via
SEARCH_ENGINE_BLEVE_AUTO_MIGRATE=false; safe because a single instance holds the data-path lock). OpenSearch is CLI-only, meant as a pre-rollout hook: the migration builds the new index and flips the alias atomically while the old index stays live, so the new version starts against a ready index.The
Applyschema classifier stays as the runtime safety net: it refuses to start on an index that doesn't match the code schema (e.g. a forgotten revision bump), rather than serving wrong results.Unsure about the last commit (refresh/replicas off during reindex, ~19% on a 2-node cluster) being worth the extra code; happy to drop it.
Not here (follow-up): live-write catch-up during migration (seq_no delta) and the readiness gate.