Skip to content

feature: adding smarter unit test execution - #5966

Open
cmgoffena13 wants to merge 15 commits into
SQLMesh:mainfrom
cmgoffena13:feature/smarter-unit-tests
Open

feature: adding smarter unit test execution#5966
cmgoffena13 wants to merge 15 commits into
SQLMesh:mainfrom
cmgoffena13:feature/smarter-unit-tests

Conversation

@cmgoffena13

@cmgoffena13 cmgoffena13 commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Description

  • Add in --test-changed-only flag to only run tests for changed models in the plan.
  • Default behavior remains running all tests for all models
  • Add --select-model flag to the test command for improved functionality

Addresses #5965

Test Plan

  • Plan runs all unit tests by default (including no-change plans)
  • Plan with model changes runs the full suite unless --test-changed-only is set
  • --test-changed-only runs only changed/restated models’ unit tests
  • Plan with no changes and --test-changed-only skips unit tests
  • --skip-tests still skips; incompatible with --test-changed-only
  • Docs cover run-all default and --test-changed-only

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
…ow too

Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
@cmgoffena13 cmgoffena13 linked an issue Aug 21, 2026 that may be closed by this pull request
Comment thread sqlmesh/cli/main.py Outdated
default=None,
)
@click.option(
"--all-tests",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the other flags are verb_*. should this be --include_all_tests, or something similar?

@mday-io

mday-io commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Nice idea! I do worry about the default behavior being only running impacted model's unit tests. An indirect model change could take place (i.e., things like a SQLGlot upgrade changing rendering/types, or an engine adapter change) and then tests would fail silently for a long time before it's caught.

Also, would be good to add a console output (x tests skipped) when only some tests were selected.

@cmgoffena13

cmgoffena13 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

@mday-io -- yeah, now that I have time to think about it, I think it would make more sense to flip the logic I have and maybe add a flag for smarter unit test execution, thoughts? That way we wouldn't change default behavior and its opt-in. I like the Skipped X tests as well. Just need to think of what the flag would be called. Maybe --only-relevant-tests -- not sure

@mday-io

mday-io commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

yeah, I think that's the safest option for now.

and, what about --test-changed-only which auto-passes the list of changed models in. and then also add --test-model TEXT which the user can specify which to run (i.e., orders_*, to parallel --select-model)

cmgoffena13 and others added 10 commits August 28, 2026 22:29
…est command

Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Smarter Unit Test Execution

2 participants