feat(deepnote): load integrations from .deepnote.env.yaml and .env#440
Draft
tkislan wants to merge 14 commits into
Draft
feat(deepnote): load integrations from .deepnote.env.yaml and .env#440tkislan wants to merge 14 commits into
tkislan wants to merge 14 commits into
Conversation
Add a stateless node-only loader (IntegrationsFileConfigProvider) that reads integration configs from a `.deepnote.env.yaml` file with CLI parity: it resolves `env:` references against a sibling `.env` (via dotenv) and `process.env`, with real env winning. It replicates the Node fs/dotenv shell that `@deepnote/database-integrations` does not export, delegating parsing to the exported `parseIntegrations`. Files are resolved dir-then-root (next to the .deepnote file, then the workspace root). Reserved-id, pandas-dataframe, duplicate-id, and federated (google-oauth) entries are dropped with diagnostic issues. The loader is stateless (fresh read per call) and never throws. Also adds the `dotenv` dependency and the `deepnote.integrations.envFile.enabled` setting (boolean, default true). The loader is not yet registered in DI; that wiring lands with the watcher in a later chunk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
… env vars Wire the file-config loader into SqlIntegrationEnvironmentVariablesProvider as a complementary source alongside VSCode SecretStorage. The loader is injected @optional() (absent on web / until DI registration), and getConfigsForFile is called inside a try/catch so a file-source failure degrades to SecretStorage + DuckDB and never rejects the method. Merge semantics (per plan): the file wins on id conflict — SecretStorage is not even queried for an id the file provides; project integrations absent from the file fall back to SecretStorage (Promise.allSettled, per-item error logging); file-only integrations (not declared in project.integrations) are injected additively. The federated-auth skip and the always-appended DuckDB integration run after the merge, unchanged. When the provider is absent the output is byte-identical to the previous SecretStorage-only behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Add IntegrationsEnvFileWatcher (IExtensionSyncActivationService) that watches `.deepnote.env.yaml` and `.env` next to open Deepnote notebooks and in workspace-folder roots (dir-then-root, covering not-yet-existing files). On a debounced change it finds open Deepnote notebooks with a started kernel whose `.deepnote` dir or workspace root changed, deduped by `.deepnote` fsPath, and prompts once with a non-modal Restart action. An `isRestarting` guard plus a monotonic prompt sequence prevent stacked restarts and let a newer change supersede a still-open prompt. No affected running kernel → no prompt (the loader is stateless, so the next start reads fresh config). Add IDeepnoteKernelAutoSelector.restartServerForNotebook, which the watcher invokes: the toolkit server captures its env at spawn, so a plain kernel.restart() would keep stale values. It resolves the environment first (bailing without stopping the server if it's gone), then stops the server and LSP clients, clears the cached connection metadata and environment id (to bypass the same-environment early-return in ensureKernelSelectedWithConfiguration so startServer re-runs and re-gathers integration env vars), and re-runs kernel selection. The controller object is reused via updateConnection (no DISPOSED errors). The watcher is not yet registered in DI; that lands in the next chunk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…on E2E Register IIntegrationsFileConfigProvider (the loader) and IntegrationsEnvFileWatcher (as an IExtensionSyncActivationService) in the node service registry only. The single shared DI container means the loader binds for the @optional() injection in SqlIntegrationEnvironmentVariablesProvider (registered in the kernels registry); web keeps SecretStorage-only behavior since nothing is registered there. Add the first kernel-injection E2E test: a fixture declaring one pgsql integration and a cell printing an integration-derived env var. The test writes a `.deepnote.env.yaml` (host: env:DEMO_DB_HOST) and `.env` (DEMO_DB_HOST=...) into the temp workspace, runs the cell, and asserts the resolved value appears. The cell prints PROD_POSTGRES_HOST (derived from the integration name), while `.env` uses the distinct key DEMO_DB_HOST — so a pass proves the .deepnote.env.yaml -> env: -> dotenv -> integration -> kernel path, not customEnvironmentVariablesProvider's direct `.env` injection. E2E runs under ExTester in CI (compile-e2e verified here). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
F1: gate the integrations env-file watcher on server existence (IDeepnoteServerStarter.isServerRunningForFile), not kernel.startedAtLeastOnce. The toolkit server captures env at spawn — before the first cell executes — so an env edit in the open→first-run window previously produced no restart prompt and the first cell ran with stale values. F2: dispose the notebook's kernel in restartServerForNotebook. areKernelConnectionsEqual compares only id/environmentName/notebookName for Deepnote kernels, so a same-env respawn (only baseUrl changes) looks "unchanged" and updateConnection does not dispose the kernel — leaving it bound to the killed server (reconnect spinner + "connection failed" on nearly every restart). Disposing it forces a fresh kernel against the new server and suppresses the spurious error. F3: run each notebook's restart atomically. The cancellable withProgress token was forwarded into both stopServer and startServer; cancelling after the process was killed rejected and stranded the notebook on a dead server. Cancellation is still honored between notebooks, but once a restart starts it runs to completion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Remove the promptSeq/supersede logic and the isRestarting guard from IntegrationsEnvFileWatcher. Restarting is a user action — a stale or missed prompt is simply re-triggered by the next save or a manual restart — so overlapping prompts/restarts don't warrant special handling. This also removes the source of the cross-project supersession and change-during-restart edge cases. The server-existence gate, atomic restart, debounce, and per-.deepnote dedup are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Bring the feature branch up to date with the base (12 commits: explorer/tree refactors, per-sibling integration persistence, document-flush guard, snapshot scoping, and e2e coverage). No conflicts — disjoint file sets. Merged tree compiles and the full unit suite passes (2512 passing, 0 failing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…nToken mock Condense the integrations injection E2E's header to two lines and the folder-open note to one (the isolation rationale stays in the const comments). Replace the `as unknown as CancellationToken` force casts in the watcher unit test with a fully-typed `mock<CancellationToken>()` helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Extension-side infrastructure for cloud-style live integration-env injection
(no server restart):
- IntegrationsEnvVarsEndpoint: a loopback HTTP server (127.0.0.1) serving
GET /userpod-api/:projectId/integrations/environment-variables, returning a
project's integration env vars as [{name,value}] from
SqlIntegrationEnvironmentVariablesProvider — what the toolkit's
set_integration_env() fetches. Localhost-only, no auth.
- IntegrationEnvLiveRefresher: runs `import deepnote_toolkit;
deepnote_toolkit.set_integration_env()` silently (executeHidden) in each
affected running kernel and shows one dismissible notification, only when the
snippet ran without error outputs.
Registered node-only. Not yet wired to the server-starter (B2) or the
watcher / restart handler (B3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
At server spawn, when the loopback endpoint is listening and the .deepnote file resolves to a project id, set the toolkit config env so it fetches integration env from our endpoint in "direct mode": DEEPNOTE_RUNTIME__ENV_INTEGRATION_ENABLED / __RUNNING_IN_DETACHED_MODE / __WEBAPP_URL, plus legacy DEEPNOTE_PROJECT_ID (maps to runtime.project_id and satisfies set_notebook_path's has_env check). Detached mode avoids dev_mode's /work redirect. Guarded all-or-nothing on baseUrl + project id, so when the endpoint isn't ready the feature stays off and the pre-existing spawn-time SQL_* injection remains the env source. That injection is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Replace the toolkit-server restart on integration/env-file change with a live in-kernel refresh via IIntegrationEnvLiveRefresher (runs set_integration_env(), which re-fetches from the local endpoint and updates os.environ without a restart), plus a dismissible "environment updated" notification. - IntegrationsEnvFileWatcher: on a debounced .deepnote.env.yaml/.env change, live-refresh the affected notebooks' kernels (dir-then-root; no prompt). - IntegrationEnvRefreshHandler (renamed from IntegrationKernelRestartHandler, which no longer restarts): on a SecretStorage integration change, refresh open Deepnote notebooks. Node-only now (the live-refresh mechanism has no web equivalent), so it is removed from the web service registry. Delete the restart machinery this obsoletes: restartServerForNotebook (IDeepnoteKernelAutoSelector) and isServerRunningForFile (IDeepnoteServerStarter), including their impls and tests. Full unit suite: 2521 passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…add live-refresh e2e Verifying the live set_integration_env() loop against the real pinned toolkit (2.1.1) surfaced a crash: its get_project_auth_headers() dereferences project_secret with no null-check, so detached mode fails unless a secret is set. Inject DEEPNOTE_RUNTIME__PROJECT_SECRET (a generated token), and — since the loopback endpoint serves integration credentials — have the endpoint validate that bearer token (401 otherwise). Extend integrationsEnvFileInjection.e2e.test.ts to prove the no-restart live refresh: after the initial run, rewrite .env and assert a re-run of the SAME kernel reads the new value (applied via the watcher's set_integration_env()). Verified end-to-end against toolkit 2.1.1: direct-mode URL matches the endpoint route, initial fetch+inject, live refresh on re-fetch, and unset-on-remove all pass. Full unit suite: 2522 passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Remove comments that restate readable code; keep only the non-obvious "why" as single lines — the toolkit 2.1.1 project_secret quirk + detached-mode guard, the endpoint's bearer-token validation, the `_dntk`-not-guaranteed import, and the "public for testing" note. Comments-only change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…ndpoints The loopback server hosts the toolkit's userpod-api surface, of which the integration environment-variables route is currently the only endpoint. Rename it to a host-oriented name so future userpod-api endpoints have a home, and drop the http.Server force-cast in stop() now that @types/node declares closeAllConnections() directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai ignore
Summary
Adds CLI-parity
.deepnote.env.yaml+.envintegration loading to the extension as a complementary source alongside VSCode SecretStorage, and applies integration env to running kernels the same way Deepnote cloud does — via the toolkit's liveset_integration_env(), with no server restart.What it does
1. File-based integration config (loader + merge)
IntegrationsFileConfigProvider: reads a.deepnote.env.yaml(dir-then-root), resolvesenv:refs against.env(dotenv) andprocess.env(real env wins), never throws. Reuses@deepnote/database-integrations'parseIntegrations; replicates only the Node fs/dotenv shell.SqlIntegrationEnvironmentVariablesProvidermerges file configs over SecretStorage (file wins on id conflict; file-only additive; federated skip + DuckDB unchanged).dotenvdependency and thedeepnote.integrations.envFile.enabledsetting (default true).2. Live env injection (cloud-parity, no restart)
Rather than restarting the toolkit server when integration env changes, the extension mirrors the cloud "no restart" path:
IntegrationsEnvVarsEndpoint: a loopback HTTP endpoint (127.0.0.1, no auth) servingGET /userpod-api/:projectId/integrations/environment-variables→[{name,value}]from the provider. This is exactly what the toolkit'sset_integration_env()fetches.DEEPNOTE_RUNTIME__ENV_INTEGRATION_ENABLED/__RUNNING_IN_DETACHED_MODE/__WEBAPP_URL+DEEPNOTE_PROJECT_ID— so it fetches integration env at kernel start. Guarded to degrade to the existing spawn-time injection when the endpoint/project id isn't available.IntegrationEnvLiveRefresher: on an env-file change (IntegrationsEnvFileWatcher, debounced) or a SecretStorage integration change (IntegrationEnvRefreshHandler), runsdeepnote_toolkit.set_integration_env()silently (executeHidden) in the affected running kernels — re-fetch + liveos.environupdate — and shows one dismissible "environment updated" notification.The spawn-time SQL_* injection is retained as the initial-load safety net. Node-only feature; web is unchanged.
Test plan
integrationsEnvFileInjection.e2e.test.tsproves the.deepnote.env.yaml→env:→ dotenv → integration → kernel path via spawn-time injection. Runs under ExTester in CI.set_integration_env()loop — the toolkit fetching the local endpoint and updating a running kernel'sos.environon change.Follow-up (separate
deepnote/deepnoterepo)@deepnote/database-integrationscould add a Node-only subpath export (/node) for the fs/dotenv shell and aDatabaseIntegrationConfig-typed federated guard, to remove the small pieces the extension/CLI currently replicate (plan drafted separately).🤖 Generated with Claude Code
https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH