fix(kb): wait for the knowledge base to be ACTIVE before its data source

[DerrickF]

The first knowledge base past IAM was created and then abandoned

ConflictException: The Knowledge Base is not in a valid status.
Wait for the knowledge base to reach a valid status and try again.

CreateKnowledgeBase returns while the knowledge base is still CREATING. This module's own header records 47–124 s to ACTIVE (n=7) — the knowledge was there, just not in the code path, which called CreateDataSource immediately.

Why not just widen the retry set

ConflictException is deliberately absent from RETRYABLE_ERROR_CODES so a genuine conflict fails fast, and _call's backoff tops out near 60 s anyway — short of the measured upper bound.

More fundamentally, retrying the dependent call is the wrong shape: it spends attempts on a precondition instead of waiting for it. So the wait is explicit, bounded at 300 s (~2.5x the observed worst case, well inside the worker's 15-minute timeout). A terminal status — FAILED, DELETING — fails on the first poll rather than waiting out the budget.

The orphan, and what contained it

The create succeeded, the data source did not, and attach_aws_ids never ran — so the knowledge base existed in AWS with no record pointing at it. Two pre-existing decisions kept that from being serious:

• The clientToken is derived from app_kb_id, so the next attempt adopts that knowledge base rather than creating a second one.
• Tags are written at create, so the reconciler can find it either way.

The failure message now says this, because an operator reading it needs to know a retry is safe rather than duplicating.

Why no test caught it

FakeBedrockAgent.create_knowledge_base returned status: "ACTIVE" — which the real API never does. Every provisioning test therefore skipped straight past the window where this breaks.

The fake now returns CREATING and answers get_knowledge_base from a configurable status sequence, so the wait is exercised rather than assumed. That change alone made 11 existing tests fail until the wait was added, which is the behaviour you want from a test double.

Five new tests: the data source is created only after ACTIVE; it is never attempted while CREATING; a terminal status fails on the first poll; the budget is read at call time rather than bound as a default argument; and the failure message tells the operator a retry is safe.

Mutation verified: deleting the wait fails the named tests.

Verification

2,321 tests pass across tests/shared, tests/lambdas and tests/property. ruff clean on both files.

Observed in dev before the fix: the knowledge base reached ACTIVE, type MANAGED, correctly tagged ManagedKbEnvironment=dev — so provisioning, the managed KB type, and the tag contract are all confirmed working. Only the data-source step was blocked.

After merge

Backend-only, so backend.yml rebuilds the kb-migration image and updates the four Lambdas — no platform deploy needed. Then retrying adopts the existing ACTIVE knowledge base, the wait returns immediately, and the data source is created.