Document collect_schemas tuning options and partition counting - #39408
Document collect_schemas tuning options and partition counting#39408databergs wants to merge 1 commit into
Conversation
max_tables was only discoverable in conf.yaml.example. Adds the collect_schemas options for PostgreSQL, SQL Server, and MySQL to the Schemas page (the options and defaults differ by engine), documents that PostgreSQL counts each partition individually toward max_tables, and distinguishes this setting from the collect_column_statistics max_tables option.
Preview links (active after the
|
![datadog-prod-us1-3[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The PostgreSQL guidance says that a longer query timeout helps reduce database load. A longer timeout can increase resource use because the schema query can run for more time.
🤖 Datadog Autotest · Commit be01a89 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
|
||
| <div class="alert alert-warning">Each partition of a partitioned table counts as a separate table toward PostgreSQL's <code>max_tables</code> limit. For example, if <code>table_1</code> is partitioned into <code>partition_a.table_1</code> and <code>partition_b.table_1</code>, it counts as two tables, not one. Heavily partitioned databases can reach the default limit of 300 with far fewer logical tables than expected. If tables are missing from the Schemas page, raise <code>max_tables</code> to account for the total partition count. See <a href="/database_monitoring/setup_postgres/advanced_configuration/#handling-many-relations">Handling many relations</a> for more information.</div> | ||
|
|
||
| Raising `max_tables` increases the cost of each collection run. On instances with a large number of tables, also consider raising `max_query_duration` and `collection_interval` to reduce load on the database. |
There was a problem hiding this comment.
Users can increase database resource use while they try to reduce it on large PostgreSQL instances.
Assertion details
- Input: A user follows the new advice for a PostgreSQL instance with many tables and raises max_query_duration to reduce load.
- Expected:
Describe a longer timeout as an option that lets large schema collections finish. Describe a longer collection interval as the option that reduces collection frequency and load. - Actual:
The text says that a longer query timeout helps reduce database load. A longer timeout lets the query use resources for more time before cancellation.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
OliviaShoup
left a comment
OliviaShoup
left a comment
There was a problem hiding this comment.
hey thanks for this PR! it looks great. i left some comments/questions just to make sure about some info but we merge this
|
|
||
| | Option | Default | Description | | ||
| |---|---|---| | ||
| | `enabled` | `true` | Set to `false` to disable schema collection. | |
There was a problem hiding this comment.
integrations-core's postgres/datadog_checks/postgres/data/conf.yaml.example documents enabled as @param enabled - boolean - optional - default: false, same as SQL Server and MySQL. this row says the default is true
| | `enabled` | `true` | Set to `false` to disable schema collection. | | |
| | `enabled` | `false` | Set to `true` to enable schema collection. | |
| max_tables: 1000 | ||
| ``` | ||
|
|
||
| <div class="alert alert-warning">Each partition of a partitioned table counts as a separate table toward PostgreSQL's <code>max_tables</code> limit. For example, if <code>table_1</code> is partitioned into <code>partition_a.table_1</code> and <code>partition_b.table_1</code>, it counts as two tables, not one. Heavily partitioned databases can reach the default limit of 300 with far fewer logical tables than expected. If tables are missing from the Schemas page, raise <code>max_tables</code> to account for the total partition count. See <a href="/database_monitoring/setup_postgres/advanced_configuration/#handling-many-relations">Handling many relations</a> for more information.</div> |
There was a problem hiding this comment.
❓question❓
tracing postgres/datadog_checks/postgres/schemas.py: PG_TABLES_QUERY_V10_PLUS (the query max_tables limits) explicitly excludes partitions (WHERE c.relkind IN ('r','p','f') AND c.relispartition != 't'), and partition count is tracked separately as num_partitions metadata on the parent table rather than as additional rows. that suggests each partitioned table counts as one table toward max_tables, not one per partition. could someone from database-monitoring confirm before this merges? I may be missing a second code path, but this line hasn't changed since 2025-11-21
| replace_digits: true | ||
| ``` | ||
|
|
||
| Partitioning also affects schema collection. Each partition counts individually toward the `max_tables` limit for `collect_schemas`, so partitioned databases may need a higher limit for full coverage. See [Tuning schema collection][2] for details. |
There was a problem hiding this comment.
this repeats the partition-counting claim that doesn't match what I found in postgres/datadog_checks/postgres/schemas.py
| # dbname: '<DB_NAME>' | ||
| ``` | ||
|
|
||
| ### Tuning schema collection |
There was a problem hiding this comment.
you could use our tabs shortcode for the PostgreSQL/SQL Server/MySQL stuff. hugo/content/en/database_monitoring/agent_integration_overhead.md is a good example:
{{< tabs >}}
{{% tab "Postgres" %}}
...
{{% /tab %}}
{{% /tabs %}}
converting would match the docs, improve scannability, and make each engine's section self-contained (helpful for AI chunking, since right now nothing but bold text separates the three)
2 participants
What does this PR do? What is the motivation?
The
collect_schemastuning options (includingmax_tables) were only discoverable inconf.yaml.exampleinintegrations-core, not on the docs site. Additionally, on PostgreSQL, each partition of a partitioned table counts individually towardmax_tables, which can silently truncate the Schemas page for heavily partitioned databases with no visible cause.This PR:
max_tables.collect_schemasmax_tablesoption (default 300) from the separate, differently-defaultedmax_tablesoption undercollect_column_statistics(default 500) on the same page.Merge readiness
For Datadog employees:
<name>/<description>convention and include the forward slash (/). If you've already created your PR with an incorrect branch name, please rename your branch and open a fresh PR./reviewto run an automated check that catches common issues before a Documentation team member reviews your PR.AI assistance
Drafted and verified with Claude Code, based on a review of the current
collect_schemasdefaults inintegrations-corefor PostgreSQL, SQL Server, and MySQL.Additional notes
The
collect_schemasoptions and the partition-counting behavior formax_tablesdiffer across PostgreSQL, SQL Server, and MySQL (verified against the currentconf.yaml.exampleand check source inintegrations-core). Only PostgreSQL counts partitions individually towardmax_tables; SQL Server counts each logical table once (reporting partition count separately), and MySQL'scollect_schemashas nomax_tablesoption at all.