diff --git a/docs/concepts/overview.md b/docs/concepts/overview.md index 32ad4176ae..47ae8ffd24 100644 --- a/docs/concepts/overview.md +++ b/docs/concepts/overview.md @@ -52,7 +52,9 @@ Bad data is worse than no data. The best way to keep bad data out of your system ### [Tests](./tests.md) SQLMesh "tests" are similar to unit tests in software development, where the unit is a single model. SQLMesh tests validate model *code* — you specify the input data and expected output, then SQLMesh runs the test and compares the expected and actual output. -SQLMesh automatically runs tests when you apply a `plan`, or you can run them on demand with the [`test` command](../reference/cli.md#test). +SQLMesh automatically runs all unit tests when a `plan` is created. Use `--test-changed-only` to run tests only for models included in the plan (added, modified, or restated), `--skip-tests` to skip, run tests for specific models with [`sqlmesh test --select-model`](../reference/cli.md#test), or run the full suite on demand with the [`test` command](../reference/cli.md#test). + +Learn more in the [testing guide](../guides/testing.md). ### [Audits](./audits.md) In contrast to tests, SQLMesh "audits" validate the results of model code applied to your actual data. diff --git a/docs/concepts/tests.md b/docs/concepts/tests.md index c1714ea982..a293237687 100644 --- a/docs/concepts/tests.md +++ b/docs/concepts/tests.md @@ -1,6 +1,8 @@ # Testing -Testing allows you to protect your project from regression by continuously verifying that the output of each model matches your expectations. Unlike [audits](audits.md), tests are executed either on demand (for example, as part of a CI/CD job) or every time a new [plan](plans.md) is created. +Testing allows you to protect your project from regression by continuously verifying that the output of each model matches your expectations. Unlike [audits](audits.md), tests are executed either on demand (for example, as part of a CI/CD job or via [`sqlmesh test`](../reference/cli.md#test)) or when a new [plan](plans.md) is created. + +By default, `sqlmesh plan` runs all unit tests. Use `--test-changed-only` to run tests only for models included in the plan (added, modified, or restated), or `--skip-tests` to run none. With both `--select-model` and `--test-changed-only`, tests run only for selected models that changed. Similar to unit testing in software development, SQLMesh evaluates the model's logic against predefined inputs and then compares the output to expected outcomes provided as part of each test. diff --git a/docs/faq/faq.md b/docs/faq/faq.md index b4a0d7e4d4..accd41a85f 100644 --- a/docs/faq/faq.md +++ b/docs/faq/faq.md @@ -102,7 +102,7 @@ SQLMesh's default behavior is appropriate for most deployments, but you can override where SQLMesh creates physical tables and views with [schema configuration options](../guides/configuration.md#environment-schemas). ??? question "What's the difference between a `test` and an `audit`?" - A SQLMesh [`test`](../concepts/tests.md) is analogous to a "unit test" in software engineering. It tests *code* based on known inputs and outputs. In SQLMesh, the inputs and outputs are specified in a YAML file, and SQLMesh automatically runs them when `sqlmesh plan` is executed. + A SQLMesh [`test`](../concepts/tests.md) is analogous to a "unit test" in software engineering. It tests *code* based on known inputs and outputs. In SQLMesh, the inputs and outputs are specified in a YAML file, and SQLMesh runs all unit tests when `sqlmesh plan` is executed (use `--test-changed-only` to run only tests for models in the plan). Writing YAML is annoying and error-prone, so SQLMesh's [`create_test` command](../concepts/tests.md#automatic-test-generation) allows you to automatically generate YAML test files based on queries of existing data tables. diff --git a/docs/guides/models.md b/docs/guides/models.md index e3b4ab1cfa..074e581937 100644 --- a/docs/guides/models.md +++ b/docs/guides/models.md @@ -161,7 +161,7 @@ Reverting to a previous model version is a quick operation since no additional w SQLMesh automatically validates your models in order to ensure the quality and accuracy of your data. This is done via the following: -* Running unit tests by default when you execute the `plan` command. This ensures all changes to applied to any environment are logically validated. Refer to [testing](../concepts/tests.md) for more information. +* Running all unit tests when you execute the `plan` command (use `--test-changed-only` to run only tests for models in the plan). This ensures changes applied to any environment are logically validated. Refer to [testing](../concepts/tests.md) for more information. * Running audits whenever data is loaded to a table (either for backfill or loading on a cadence). This way you know all data present in any table has passed all defined audits. Refer to [auditing](../concepts/audits.md) for more information. SQLMesh also provides automatic validation via CI/CD by automatically creating a preview environment. diff --git a/docs/guides/testing.md b/docs/guides/testing.md index 1045759e25..f837e9f00a 100644 --- a/docs/guides/testing.md +++ b/docs/guides/testing.md @@ -12,11 +12,19 @@ OK ``` As the unit tests run, SQLMesh will identify any that fail. +By default, `sqlmesh plan` runs all unit tests. Use `--test-changed-only` to run tests only for models included in the plan (added, modified, or restated), or `--skip-tests` to run none. With both `--select-model` and `--test-changed-only`, tests run only for selected models that changed. + For more information about tests, refer to [testing](../concepts/tests.md). ### Test changes to a specific model -To run a specific model test, pass in the suite file name followed by `::` and the name of the test; for example: `sqlmesh test tests/test_suite.yaml::test_example_full_model`. +To run unit tests for a specific model, use `--select-model`: + +```bash +$ sqlmesh test --select-model sqlmesh_example.full_model +``` + +Alternatively, pass in the suite file name followed by `::` and the name of the test; for example: `sqlmesh test tests/test_suite.yaml::test_example_full_model`. ### Run a subset of tests diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 9b6e28fe14..e5a684a5b4 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -357,6 +357,8 @@ Options: Default: prod. --skip-tests Skip tests prior to generating the plan if they are defined. + --test-changed-only Run unit tests only for models included in + the plan instead of all tests. --skip-linter Skip linting prior to generating the plan if the linter is enabled. -r, --restate-model TEXT Restate data for specified models and models @@ -626,6 +628,8 @@ Options: -v, --verbose Verbose output. --preserve-fixtures Preserve the fixture tables in the testing database, useful for debugging. + --select-model TEXT Select specific models to run unit tests for. Can be + specified multiple times. --help Show this message and exit. ``` diff --git a/docs/reference/notebook.md b/docs/reference/notebook.md index 6cac4e1078..17ce26b4c6 100644 --- a/docs/reference/notebook.md +++ b/docs/reference/notebook.md @@ -96,7 +96,7 @@ options: #### plan ``` %plan [--start START] [--end END] [--execution-time EXECUTION_TIME] - [--create-from CREATE_FROM] [--skip-tests] + [--create-from CREATE_FROM] [--skip-tests] [--test-changed-only] [--restate-model [RESTATE_MODEL ...]] [--no-gaps] [--skip-backfill, --dry-run] [--forward-only] [--effective-from EFFECTIVE_FROM] [--no-prompts] [--auto-apply] @@ -120,6 +120,8 @@ options: The environment to create the target environment from if it doesn't exist. Default: prod. --skip-tests, -t Skip the unit tests defined for the model. + --test-changed-only Run unit tests only for models included in the plan + instead of all tests. --restate-model <[RESTATE_MODEL ...]>, -r <[RESTATE_MODEL ...]> Restate data for specified models (and models downstream from the one specified). For production @@ -131,7 +133,8 @@ options: comparing to existing snapshots for matching models in the target environment. --skip-backfill, --dry-run - Skip the backfill step and only create a virtual update for the plan. + Skip the backfill step and only create a virtual + update for the plan. --forward-only Create a plan for forward-only changes. --effective-from EFFECTIVE_FROM The effective date from which to apply forward-only @@ -430,7 +433,8 @@ options: #### run_test ``` -%run_test [--pattern [PATTERN ...]] [--verbose] [--preserve-fixtures] [tests ...] +%run_test [--pattern [PATTERN ...]] [--verbose] [--preserve-fixtures] + [--select-model [SELECT_MODEL ...]] [tests ...] Run unit test(s). @@ -443,6 +447,8 @@ options: --verbose, -v Verbose output. --preserve-fixtures Preserve the fixture tables in the testing database, useful for debugging. + --select-model <[SELECT_MODEL ...]> + Select specific models to run unit tests for. ``` #### audit diff --git a/sqlmesh/cli/main.py b/sqlmesh/cli/main.py index 5574a892cb..e868a58eac 100644 --- a/sqlmesh/cli/main.py +++ b/sqlmesh/cli/main.py @@ -423,6 +423,12 @@ def diff(ctx: click.Context, environment: t.Optional[str] = None) -> None: help="Skip tests prior to generating the plan if they are defined.", default=None, ) +@click.option( + "--test-changed-only", + is_flag=True, + help="Run unit tests only for models included in the plan instead of all tests.", + default=None, +) @click.option( "--skip-linter", is_flag=True, @@ -795,6 +801,12 @@ def create_test( default=False, help="Preserve the fixture tables in the testing database, useful for debugging.", ) +@click.option( + "--select-model", + type=str, + multiple=True, + help="Select specific models to run unit tests for.", +) @click.argument("tests", nargs=-1) @click.pass_obj @error_handler @@ -804,14 +816,19 @@ def test( k: t.List[str], verbose: int, preserve_fixtures: bool, + select_model: t.List[str], tests: t.List[str], ) -> None: """Run model unit tests.""" + model_names = ( + obj._new_selector().expand_model_selections(select_model) if select_model else None + ) result = obj.test( match_patterns=k, tests=tests, verbosity=Verbosity(verbose), preserve_fixtures=preserve_fixtures, + model_names=model_names, ) if not result.wasSuccessful(): exit(1) diff --git a/sqlmesh/core/console.py b/sqlmesh/core/console.py index f9a758b405..ca11b6e607 100644 --- a/sqlmesh/core/console.py +++ b/sqlmesh/core/console.py @@ -2232,6 +2232,9 @@ def log_test_results(self, result: ModelTextTestResult, target_dialect: str) -> message = ( f"Ran {result.testsRun} tests against {target_dialect} in {result.duration} seconds." ) + if result.tests_skipped: + message = f"{message}\nSkipped {result.tests_skipped} tests" + if result.wasSuccessful(): self._print("=" * divider_length) self._print( @@ -3160,6 +3163,8 @@ def log_test_results(self, result: ModelTextTestResult, target_dialect: str) -> message = ( f"Ran {result.testsRun} tests against {target_dialect} in {result.duration} seconds." ) + if result.tests_skipped: + message = f"{message}\nSkipped {result.tests_skipped} tests" if result.wasSuccessful(): success_color = {"color": "#008000"} @@ -3600,6 +3605,8 @@ def log_test_results(self, result: ModelTextTestResult, target_dialect: str) -> return message = f"Ran `{result.testsRun}` Tests Against `{target_dialect}`" + if result.tests_skipped: + message = f"{message}\n**Skipped `{result.tests_skipped}` Tests**" if result.wasSuccessful(): self._print(f"**Successfully {message}**\n\n") diff --git a/sqlmesh/core/context.py b/sqlmesh/core/context.py index c3abff1d94..004a57ee72 100644 --- a/sqlmesh/core/context.py +++ b/sqlmesh/core/context.py @@ -115,6 +115,7 @@ ModelTestMetadata, generate_test, run_tests, + filter_tests_by_model_names, filter_tests_by_patterns, ) from sqlmesh.core.user import User @@ -1347,6 +1348,7 @@ def plan( execution_time: t.Optional[TimeLike] = None, create_from: t.Optional[str] = None, skip_tests: t.Optional[bool] = None, + test_changed_only: t.Optional[bool] = None, restate_models: t.Optional[t.Iterable[str]] = None, no_gaps: t.Optional[bool] = None, skip_backfill: t.Optional[bool] = None, @@ -1384,6 +1386,7 @@ def plan( create_from: The environment to create the target environment from if it doesn't exist. If not specified, the "prod" environment will be used. skip_tests: Unit tests are run by default so this will skip them if enabled + test_changed_only: Run unit tests only for models included in the plan instead of all tests restate_models: A list of either internal or external models, or tags, that need to be restated for the given plan interval. If the target environment is a production environment, ALL snapshots that depended on these upstream tables will have their intervals deleted @@ -1430,6 +1433,7 @@ def plan( execution_time=execution_time, create_from=create_from, skip_tests=skip_tests, + test_changed_only=test_changed_only, restate_models=restate_models, no_gaps=no_gaps, skip_backfill=skip_backfill, @@ -1484,6 +1488,7 @@ def plan_builder( execution_time: t.Optional[TimeLike] = None, create_from: t.Optional[str] = None, skip_tests: t.Optional[bool] = None, + test_changed_only: t.Optional[bool] = None, restate_models: t.Optional[t.Iterable[str]] = None, no_gaps: t.Optional[bool] = None, skip_backfill: t.Optional[bool] = None, @@ -1518,6 +1523,7 @@ def plan_builder( create_from: The environment to create the target environment from if it doesn't exist. If not specified, the "prod" environment will be used. skip_tests: Unit tests are run by default so this will skip them if enabled + test_changed_only: Run unit tests only for models included in the plan instead of all tests restate_models: A list of either internal or external models, or tags, that need to be restated for the given plan interval. If the target environment is a production environment, ALL snapshots that depended on these upstream tables will have their intervals deleted @@ -1559,6 +1565,7 @@ def plan_builder( "execution_time": execution_time, "create_from": create_from, "skip_tests": skip_tests, + "test_changed_only": test_changed_only, "restate_models": list(restate_models) if restate_models is not None else None, "no_gaps": no_gaps, "skip_backfill": skip_backfill, @@ -1588,6 +1595,11 @@ def plan_builder( } skip_tests = explain or skip_tests or False + test_changed_only = test_changed_only or False + + if skip_tests and test_changed_only: + raise PlanError("Cannot combine --skip-tests with --test-changed-only.") + no_gaps = no_gaps or False skip_backfill = skip_backfill or False empty_backfill = empty_backfill or False @@ -1614,8 +1626,6 @@ def plan_builder( if not skip_linter: self.lint_models() - self._run_plan_tests(skip_tests=skip_tests) - environment_ttl = ( self.environment_ttl if environment not in self.pinned_environments else None ) @@ -1698,6 +1708,19 @@ def plan_builder( *[s.name for s in context_diff.added], } + plan_test_model_names: t.Set[str] = { + *modified_model_names, + *(expanded_restate_models or set()), + } + if select_models and test_changed_only: + plan_test_model_names &= selected_fqns + + self._run_plan_tests( + skip_tests=skip_tests, + test_changed_only=test_changed_only, + model_names=plan_test_model_names, + ) + if ( is_dev and not include_unmodified @@ -2314,6 +2337,7 @@ def test( verbosity: Verbosity = Verbosity.DEFAULT, preserve_fixtures: bool = False, stream: t.Optional[t.TextIO] = None, + model_names: t.Optional[t.Collection[str]] = None, ) -> ModelTextTestResult: """Discover and run model tests""" if verbosity >= Verbosity.VERBOSE: @@ -2321,7 +2345,15 @@ def test( pd.set_option("display.max_columns", None) - test_meta = self.select_tests(tests=tests, patterns=match_patterns) + baseline_meta = self.select_tests(tests=tests, patterns=match_patterns, model_names=None) + if model_names is not None: + test_meta = self.select_tests( + tests=tests, patterns=match_patterns, model_names=model_names + ) + tests_skipped = len(baseline_meta) - len(test_meta) + else: + test_meta = baseline_meta + tests_skipped = 0 result = run_tests( model_test_metadata=test_meta, @@ -2335,6 +2367,7 @@ def test( default_catalog=self.default_catalog, default_catalog_dialect=self.config.dialect or "", ) + result.tests_skipped = tests_skipped self.console.log_test_results( result, @@ -2781,15 +2814,28 @@ def _run_tests( result = self.test(stream=test_output_io, verbosity=verbosity) return result, test_output_io.getvalue() - def _run_plan_tests(self, skip_tests: bool = False) -> t.Optional[ModelTextTestResult]: - if not skip_tests: - result = self.test() - if not result.wasSuccessful(): - raise PlanError( - "Cannot generate plan due to failing test(s). Fix test(s) and run again." - ) - return result - return None + def _run_plan_tests( + self, + skip_tests: bool = False, + test_changed_only: bool = False, + model_names: t.Optional[t.Collection[str]] = None, + ) -> t.Optional[ModelTextTestResult]: + if skip_tests: + return None + + effective_names: t.Optional[t.Set[str]] = None + + if test_changed_only: + effective_names = set(model_names or []) + if not effective_names: + return None + + result = self.test(model_names=effective_names) + if not result.wasSuccessful(): + raise PlanError( + "Cannot generate plan due to failing test(s). Fix test(s) and run again." + ) + return result def _warn_if_virtual_catalog_rematerialization(self, plan: "Plan") -> None: """Warn when ClickHouse models appear as new snapshots solely because a virtual catalog @@ -3465,6 +3511,7 @@ def select_tests( self, tests: t.Optional[t.List[str]] = None, patterns: t.Optional[t.List[str]] = None, + model_names: t.Optional[t.Collection[str]] = None, ) -> t.List[ModelTestMetadata]: """Filter pre-loaded test metadata based on tests and patterns.""" @@ -3488,6 +3535,14 @@ def select_tests( if patterns: test_meta = filter_tests_by_patterns(test_meta, patterns) + if model_names is not None: + test_meta = filter_tests_by_model_names( + test_meta, + set(model_names), + default_catalog=self.default_catalog, + dialect=self.default_dialect, + ) + return test_meta diff --git a/sqlmesh/core/test/__init__.py b/sqlmesh/core/test/__init__.py index 6353370f45..279c97b744 100644 --- a/sqlmesh/core/test/__init__.py +++ b/sqlmesh/core/test/__init__.py @@ -3,6 +3,7 @@ from sqlmesh.core.test.definition import ModelTest as ModelTest, generate_test as generate_test from sqlmesh.core.test.discovery import ( ModelTestMetadata as ModelTestMetadata, + filter_tests_by_model_names as filter_tests_by_model_names, filter_tests_by_patterns as filter_tests_by_patterns, ) from sqlmesh.core.test.result import ModelTextTestResult as ModelTextTestResult diff --git a/sqlmesh/core/test/discovery.py b/sqlmesh/core/test/discovery.py index 9afe3dd7fc..bfc67fd1e3 100644 --- a/sqlmesh/core/test/discovery.py +++ b/sqlmesh/core/test/discovery.py @@ -9,6 +9,7 @@ from sqlmesh.utils import unique from sqlmesh.utils.pydantic import PydanticModel +from sqlmesh.core.dialect import normalize_model_name class ModelTestMetadata(PydanticModel): @@ -46,3 +47,35 @@ def filter_tests_by_patterns( if ("*" in pattern and fnmatch.fnmatchcase(test.fully_qualified_test_name, pattern)) or pattern in test.fully_qualified_test_name ) + + +def filter_tests_by_model_names( + tests: list[ModelTestMetadata], + model_names: set[str], + *, + default_catalog: t.Optional[str] = None, + dialect: t.Optional[str] = None, +) -> list[ModelTestMetadata]: + """Keep tests whose YAML ``model:`` resolves to one of the given model names. + + Args: + tests: Loaded test metadata. + model_names: Model FQNs / names to keep (typically from a plan change set). + default_catalog: Catalog used when normalizing short model names. + dialect: Dialect used when normalizing model names. + + Returns: + Tests that target a model in ``model_names``. + """ + + normalized_models = { + normalize_model_name(name, default_catalog=default_catalog, dialect=dialect) + for name in model_names + } + return [ + test + for test in tests + if test.model_name + and normalize_model_name(test.model_name, default_catalog=default_catalog, dialect=dialect) + in normalized_models + ] diff --git a/sqlmesh/core/test/result.py b/sqlmesh/core/test/result.py index eefa0be513..9ed671b1c6 100644 --- a/sqlmesh/core/test/result.py +++ b/sqlmesh/core/test/result.py @@ -15,6 +15,7 @@ class ModelTextTestResult(unittest.TextTestResult): successes: t.List[unittest.TestCase] + tests_skipped: int def __init__(self, *args: t.Any, **kwargs: t.Any): self.console = kwargs.pop("console", None) @@ -24,6 +25,7 @@ def __init__(self, *args: t.Any, **kwargs: t.Any): self.failure_tables: t.List[t.Tuple[t.Any, ...]] = [] self.original_errors: t.List[t.Tuple[unittest.TestCase, ErrorType]] = [] self.duration: t.Optional[float] = None + self.tests_skipped = 0 def addSubTest( self, @@ -123,6 +125,7 @@ def merge(self, other: ModelTextTestResult) -> None: self.addSkip(skipped_args[0], skipped_args[1]) self.testsRun += other.testsRun + self.tests_skipped += other.tests_skipped def get_fail_and_error_tests(self) -> t.List[ModelTest]: # If tests contain failed subtests (e.g testing CTE outputs) we don't want diff --git a/sqlmesh/magics.py b/sqlmesh/magics.py index ed6a1b62de..90b44cd62c 100644 --- a/sqlmesh/magics.py +++ b/sqlmesh/magics.py @@ -407,6 +407,11 @@ def test(self, context: Context, line: str, test_def_raw: t.Optional[str] = None action="store_true", help="Skip the unit tests defined for the model.", ) + @argument( + "--test-changed-only", + action="store_true", + help="Run unit tests only for models included in the plan instead of all tests.", + ) @argument( "--skip-linter", action="store_true", @@ -533,6 +538,7 @@ def plan(self, context: Context, line: str) -> None: execution_time=args.execution_time, create_from=args.create_from, skip_tests=args.skip_tests, + test_changed_only=args.test_changed_only, restate_models=args.restate_model, backfill_models=args.backfill_model, no_gaps=args.no_gaps, @@ -1078,18 +1084,31 @@ def create_test(self, context: Context, line: str) -> None: action="store_true", help="Preserve the fixture tables in the testing database, useful for debugging.", ) + @argument( + "--select-model", + type=str, + nargs="*", + help="Select specific models to run unit tests for.", + ) @line_magic @pass_sqlmesh_context def run_test(self, context: Context, line: str) -> None: """Run unit test(s).""" args = parse_argstring(self.run_test, line) + model_names = ( + context._new_selector().expand_model_selections(args.select_model) + if args.select_model + else None + ) + context.test( match_patterns=args.pattern, tests=args.tests, verbosity=Verbosity(args.verbose), preserve_fixtures=args.preserve_fixtures, stream=StringIO(), # consume the output instead of redirecting to stdout + model_names=model_names, ) @magic_arguments() diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py index c625cb084d..892864b6af 100644 --- a/tests/cli/test_cli.py +++ b/tests/cli/test_cli.py @@ -59,6 +59,31 @@ def create_example_project(temp_dir, template=ProjectTemplate.DEFAULT) -> None: ) +def add_incremental_model_test(temp_dir) -> None: + with open(temp_dir / "tests" / "test_incremental_model.yaml", "w", encoding="utf-8") as f: + f.write( + """ +test_example_incremental_model: + model: sqlmesh_example.incremental_model + vars: + start: 2020-01-01 + end: 2020-01-02 + inputs: + sqlmesh_example.seed_model: + rows: + - id: 1 + item_id: 1 + event_date: 2020-01-01 + outputs: + query: + rows: + - id: 1 + item_id: 1 + event_date: 2020-01-01 +""" + ) + + def update_incremental_model(temp_dir) -> None: with open(temp_dir / "models" / "incremental_model.sql", "w", encoding="utf-8") as f: f.write( @@ -193,6 +218,105 @@ def test_plan_skip_tests(runner, tmp_path): assert_backfill_success(result) +def test_plan_no_changes_runs_tests_by_default(runner, tmp_path): + create_example_project(tmp_path) + init_prod_and_backfill(runner, tmp_path) + add_incremental_model_test(tmp_path) + + result = runner.invoke( + cli, ["--log-file-dir", tmp_path, "--paths", tmp_path, "plan", "--no-prompts"], input="\n" + ) + assert result.exit_code == 0 + assert "Successfully Ran 2 tests against duckdb" in result.output + assert "No changes to plan" in result.output or "No changes" in result.output + + +def test_plan_test_changed_only_with_no_changes(runner, tmp_path): + create_example_project(tmp_path) + init_prod_and_backfill(runner, tmp_path) + + result = runner.invoke( + cli, + [ + "--log-file-dir", + tmp_path, + "--paths", + tmp_path, + "plan", + "--test-changed-only", + "--no-prompts", + ], + input="\n", + ) + assert result.exit_code == 0 + assert "Successfully Ran" not in result.output + + +def test_plan_test_changed_only_runs_only_changed_model_tests(runner, tmp_path): + create_example_project(tmp_path) + init_prod_and_backfill(runner, tmp_path) + add_incremental_model_test(tmp_path) + + full_model_path = tmp_path / "models" / "full_model.sql" + full_model_path.write_text( + full_model_path.read_text().replace("COUNT(DISTINCT id)", "COUNT(id)") + ) + + result = runner.invoke( + cli, + [ + "--log-file-dir", + tmp_path, + "--paths", + tmp_path, + "plan", + "--test-changed-only", + "--no-prompts", + "--auto-apply", + ], + ) + assert result.exit_code == 0 + assert "Successfully Ran 1 tests against duckdb" in result.output + assert "Skipped 1 tests" in result.output + + +def test_plan_select_model_test_changed_only_scopes_tests(runner, tmp_path): + create_example_project(tmp_path) + init_prod_and_backfill(runner, tmp_path) + add_incremental_model_test(tmp_path) + + full_model_path = tmp_path / "models" / "full_model.sql" + full_model_path.write_text( + full_model_path.read_text().replace("COUNT(DISTINCT id)", "COUNT(id)") + ) + incremental_model_path = tmp_path / "models" / "incremental_model.sql" + incremental_model_path.write_text( + incremental_model_path.read_text().replace( + " item_id,\n event_date,", + " item_id,\n 'b' as new_col,\n event_date,", + ) + ) + + result = runner.invoke( + cli, + [ + "--log-file-dir", + tmp_path, + "--paths", + tmp_path, + "plan", + "--select-model", + "sqlmesh_example.full_model", + "--test-changed-only", + "--no-prompts", + "--auto-apply", + ], + ) + assert result.exit_code == 0 + assert "Successfully Ran 1 tests against duckdb" in result.output + assert "Skipped 1 tests" in result.output + + def test_plan_skip_linter(runner, tmp_path): create_example_project(tmp_path) diff --git a/tests/core/test_context.py b/tests/core/test_context.py index e41382b078..eba19c9989 100644 --- a/tests/core/test_context.py +++ b/tests/core/test_context.py @@ -4,7 +4,7 @@ import re from datetime import date, timedelta, datetime from tempfile import TemporaryDirectory -from unittest.mock import PropertyMock, call, patch +from unittest.mock import ANY, PropertyMock, call, patch import time_machine import pytest @@ -3603,11 +3603,19 @@ def test_prompt_if_uncategorized_snapshot(mocker: MockerFixture, tmp_path: Path) assert context.config.plan.no_prompts == True +def test_plan_skip_tests_and_test_changed_only(sushi_context: Context) -> None: + with pytest.raises( + PlanError, + match="Cannot combine --skip-tests with --test-changed-only.", + ): + sushi_context.plan("dev", skip_tests=True, test_changed_only=True, no_prompts=True) + + def test_plan_explain_skips_tests(sushi_context: Context, mocker: MockerFixture) -> None: sushi_context.console = TerminalConsole() spy = mocker.spy(sushi_context, "_run_plan_tests") sushi_context.plan(environment="dev", explain=True, no_prompts=True, include_unmodified=True) - spy.assert_called_once_with(skip_tests=True) + spy.assert_called_once_with(skip_tests=True, test_changed_only=False, model_names=ANY) def test_dev_environment_virtual_update_with_environment_statements(tmp_path: Path) -> None: diff --git a/tests/core/test_test.py b/tests/core/test_test.py index d679f09393..6ac20cc2d8 100644 --- a/tests/core/test_test.py +++ b/tests/core/test_test.py @@ -3527,3 +3527,29 @@ def test_cte_failure(tmp_path: Path) -> None: assert "Ran 1 tests" in output assert "Failed tests (1)" in output + + +def test_filter_tests_by_model_names(): + from sqlmesh.core.test.discovery import ModelTestMetadata, filter_tests_by_model_names + + tests = [ + ModelTestMetadata(path=Path("a.yaml"), test_name="t1", body={"model": "sushi.a"}), + ModelTestMetadata(path=Path("b.yaml"), test_name="t2", body={"model": "sushi.b"}), + ModelTestMetadata(path=Path("c.yaml"), test_name="t3", body={"model": ""}), + ] + + filtered = filter_tests_by_model_names( + tests, + {'"memory"."sushi"."a"'}, + default_catalog="memory", + dialect="duckdb", + ) + assert [t.test_name for t in filtered] == ["t1"] + + filtered_short = filter_tests_by_model_names( + tests, + {"sushi.a"}, + default_catalog="memory", + dialect="duckdb", + ) + assert [t.test_name for t in filtered_short] == ["t1"] diff --git a/tests/integrations/github/cicd/test_github_controller.py b/tests/integrations/github/cicd/test_github_controller.py index 786341d361..dbb09f4336 100644 --- a/tests/integrations/github/cicd/test_github_controller.py +++ b/tests/integrations/github/cicd/test_github_controller.py @@ -3,7 +3,7 @@ import os import pathlib from unittest import mock -from unittest.mock import PropertyMock, call +from unittest.mock import ANY, PropertyMock, call import pytest import time_machine @@ -256,7 +256,9 @@ def test_pr_plan(github_client, make_controller): assert controller.pr_plan.skip_backfill assert not controller.pr_plan.no_gaps assert not controller._context.apply.called - assert controller._context._run_plan_tests.call_args == call(skip_tests=True) + assert controller._context._run_plan_tests.call_args == call( + skip_tests=True, test_changed_only=False, model_names=ANY + ) assert ( controller._pr_plan_builder._categorizer_config == controller._context.auto_categorize_changes @@ -278,7 +280,9 @@ def test_pr_plan_auto_categorization(github_client, make_controller): assert controller.pr_plan.skip_backfill assert not controller.pr_plan.no_gaps assert not controller._context.apply.called - assert controller._context._run_plan_tests.call_args == call(skip_tests=True) + assert controller._context._run_plan_tests.call_args == call( + skip_tests=True, test_changed_only=False, model_names=ANY + ) assert controller._pr_plan_builder._categorizer_config == custom_categorizer_config assert controller.pr_plan.start == default_start_absolute assert not controller.pr_plan.start_override_per_model @@ -365,7 +369,9 @@ def test_prod_plan(github_client, make_controller): assert not controller.prod_plan.skip_backfill assert controller.prod_plan.no_gaps assert not controller._context.apply.called - assert controller._context._run_plan_tests.call_args == call(skip_tests=True) + assert controller._context._run_plan_tests.call_args == call( + skip_tests=True, test_changed_only=False, model_names=ANY + ) assert ( controller._prod_plan_builder._categorizer_config == controller._context.auto_categorize_changes @@ -387,7 +393,9 @@ def test_prod_plan_auto_categorization(github_client, make_controller): assert not controller.prod_plan.skip_backfill assert controller.prod_plan.no_gaps assert not controller._context.apply.called - assert controller._context._run_plan_tests.call_args == call(skip_tests=True) + assert controller._context._run_plan_tests.call_args == call( + skip_tests=True, test_changed_only=False, model_names=ANY + ) assert controller._prod_plan_builder._categorizer_config == custom_categorizer_config # default PR start should be ignored for prod plans assert controller.prod_plan.start != default_pr_start @@ -404,7 +412,9 @@ def test_prod_plan_with_gaps(github_client, make_controller): assert controller._prod_plan_with_gaps_builder._auto_categorization_enabled assert not controller.prod_plan_with_gaps.no_gaps assert not controller._context.apply.called - assert controller._context._run_plan_tests.call_args == call(skip_tests=True) + assert controller._context._run_plan_tests.call_args == call( + skip_tests=True, test_changed_only=False, model_names=ANY + ) def test_run_tests(github_client, make_controller):