[POC] DABs AIR native fields

[vinchenzo-db]

Proposal for AIR-on-DABs code_source parity (see DABs x AIR CLI Alignment). Extends the existing artifacts block so DABs can build the code tarball itself instead of only uploading a user build command's output:

  artifacts:
    code_source:
      type: tgz            # new ArtifactType (was whl|jar)
      include: [src/...]   # subpaths to pack, gitignore-honored
      git: {branch|commit} # snapshot a ref instead of the working tree
      files: [{source: ./dist/code.tgz}]

build and git/include are mutually exclusive: either the user's command produces the tarball (today's behavior) or DABs does. The produced file flows through the existing artifact upload path, so an ai_runtime_task's code_source_path pointing at it is uploaded and rewritten to the remote path with no new reference syntax.

• include: reuses the bundle sync walker (matches file-sync filtering)
• git: shells out to git archive Verified end-to-end on staging (both modes deploy; code_source_path rewritten).

Open for review: this is a proposal to react to, not a finished feature. jsonschema annotations for the new fields are a follow-up.

Co-authored-by: Isaac

Changes

Why

Tests

Show that the include field is respected, we notice the ./dist/code.tgz is properly uploaded.

 v.chen at https://dbc-04ac0685-8857 in ~/.worktrees/cli-artifacts (git:air-artifacts-code-source) [20:04:39]
$ rm -rf /tmp/demo-include && mkdir -p /tmp/demo-include/src && cd /tmp/demo-include
printf 'print("hello from packaged code")\n' > src/train.py
printf '#!/bin/bash\ncd "$(dirname "$0")"\npython src/train.py\n' > command.sh
cat > databricks.yml <<YAML
bundle: {name: demo-include}
targets: {dev: {mode: development, default: true, workspace: {host: $HOST}}}
artifacts:
  code_source:
    type: tgz
    include: [src]
    files: [{source: ./dist/code.tgz}]
resources:
  jobs:
    demo_include:
      name: demo-include
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: demo-include
            deployments: [{command_path: ./command.sh, compute: {accelerator_type: GPU_1xA10, accelerator_count: 1}}]
            code_source_path: ./dist/code.tgz
      environments: [{environment_key: default, spec: {environment_version: "5", dependencies: []}}]
YAML

$CLI bundle deploy -t dev                       # → "Building code_source..." → "Uploading dist/code.tgz..."
tar tzf dist/code.tgz                            # → src/train.py   (DABs built it from ./src)
JID=$($CLI bundle summary -t dev | grep -oE 'jobs/[0-9]+' | head -1 | cut -d/ -f2)
$CLI jobs get $JID | grep code_source_path       # → rewritten to /Workspace/.../artifacts/.internal/code.tgz
$CLI bundle destroy -t dev --auto-approve

Building code_source...
Uploading dist/code.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev/files...
Created jobs.demo_include
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
src/train.py
          "code_source_path": "/Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev/artifacts/.internal/code.tgz",
The following resources will be deleted:
  delete resources.jobs.demo_include

All files and directories at the following location will be deleted: /Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev

Destroy: 1 deleted

Show that git refs are respected:

# v.chen at https://dbc-04ac0685-8857 in /tmp/demo-include (git:) [20:04:57]
$ rm -rf /tmp/demo-git && mkdir -p /tmp/demo-git/src && cd /tmp/demo-git
git init -q && git config user.email t@t.co && git config user.name t
printf 'print("committed code")\n' > src/train.py
printf '#!/bin/bash\ncd "$(dirname "$0")"\npython src/train.py\n' > command.sh
git add -A && git commit -qm init
BR=$(git rev-parse --abbrev-ref HEAD)
cat > databricks.yml <<YAML
bundle: {name: demo-git}
targets: {dev: {mode: development, default: true, workspace: {host: $HOST}}}
artifacts:
  code_source:
    type: tgz
    git: {branch: $BR}          # or  commit: <sha>
    include: [src]
    files: [{source: ./dist/code.tgz}]
resources:
  jobs:
    demo_git:
      name: demo-git
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: demo-git
            deployments: [{command_path: ./command.sh, compute: {accelerator_type: GPU_1xA10, accelerator_count: 1}}]
            code_source_path: ./dist/code.tgz
      environments: [{environment_key: default, spec: {environment_version: "5", dependencies: []}}]
YAML

$CLI bundle deploy -t dev
tar tzf dist/code.tgz                            # → src/train.py   (from `git archive` of the ref)
$CLI bundle destroy -t dev --auto-approve
Databricks pre-commit Git Hook V2.5.0
Running secret scanning on changes staged for commit.
secret-scan hook completed in 126 ms
Unknown project name: None, skipping linting.
pre-commit-total hook completed in 153 ms
Databricks commit-msg Git Hook V2.5.0
Running secret scanning on commit message.
secret-scan hook completed in 40 ms
commit-msg-total hook completed in 50 ms
Building code_source...
Uploading dist/code.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/demo-git/dev/files...
Created jobs.demo_git
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
src/
src/train.py
The following resources will be deleted:
  delete resources.jobs.demo_git

All files and directories at the following location will be deleted: /Workspace/Users/v.chen@databricks.com/.bundle/demo-git/dev

Destroy: 1 deleted

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

Files: acceptance/bundle/validate/strict/output.txt
Suggested: @janniklasrose
Also eligible: @pietern, @denik, @shreyas-goenka, @andrewnester, @anton-107, @lennartkats-db

/bundle/ - needs approval

8 files changed
Suggested: @janniklasrose
Also eligible: @pietern, @denik, @shreyas-goenka, @andrewnester, @anton-107, @lennartkats-db

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: f2a8fd9

Run: 33203627766

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	1	1	274	1207	3:59
💚​	aws windows	1	1	276	1205	4:06
💚​	azure linux	1	1	273	1207	3:59
💚​	azure windows	1	1	275	1205	3:38
💚​	gcp linux	1	1	274	1207	4:18
💚​	gcp windows	1	1	276	1205	3:44

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
4:01	aws windows	TestAccept
3:39	gcp windows	TestAccept
3:32	azure windows	TestAccept