[Feature] Supporting dstack-inside-dstack for dev environments and tasks#4023
Open
peterschmidt85 wants to merge 3 commits into
Open
[Feature] Supporting dstack-inside-dstack for dev environments and tasks#4023peterschmidt85 wants to merge 3 commits into
peterschmidt85 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Tasks and dev environments may need to call dstack themselves, for example to
inspect runs, submit another run, or attach to it. This currently requires the
dstack server to accept direct connections from the run.
UX
Add
server: trueto task and dev-environment configurations.With
server: true, dstack exposes the server managing the run through/run/dstack/server.sockinside each job. It setsDSTACK_SERVER_URLto thecorresponding
http+unixURL andDSTACK_PROJECTto the current project,allowing the CLI and API clients to use that connection.
Authentication remains explicit. dstack does not inject
DSTACK_TOKEN; it canbe passed through the existing environment or secret mechanisms.
The CLI also accepts
DSTACK_SERVER_URL,DSTACK_PROJECT, andDSTACK_TOKENas a complete configuration without requiring
~/.dstack/config.yml.serveris not supported for services.Implementation
Before starting commands, the server opens an SSH connection to each job and
creates a remote Unix-socket forward. Connections to
/run/dstack/server.sockinside the job are forwarded to the HTTP port of theserver managing the run.
DSTACK_SERVER_URLpoints clients to this socket.This uses the existing job SSH path, so the server does not need a public URL
or gateway. Health checks use the same socket path end to end; if the SSH
connection or socket becomes stale, the server closes and recreates the
tunnel. The tunnel is removed when the job terminates.