diff --git a/README.md b/README.md index 4d554d12..24d0c5a3 100644 --- a/README.md +++ b/README.md @@ -6,3 +6,17 @@ This repository contains the documentation hosted on [https://docs.pgdog.dev](ht Contributions are welcome. Please open a pull request / issue with requested changes. Once the PR is merged, you should see changes in production within a few minutes. + +## Regenerating `llms.txt` + +The LLM-friendly documentation index is generated from the Markdown sources with +[Sourcey](https://sourcey.com). After changing the documentation, regenerate it +with Node.js and Python installed: + +```bash +python3 scripts/generate_llms.py +``` + +The script pins the Sourcey version and updates `docs/llms.txt` only when the +generated output changes. Commit the updated file together with the documentation +changes. diff --git a/docs/llms.txt b/docs/llms.txt new file mode 100644 index 00000000..a86ba046 --- /dev/null +++ b/docs/llms.txt @@ -0,0 +1,68 @@ +# PgDog + +> PgDog is a connection pooler, load balancer and database sharder for PostgreSQL. Written in Rust, PgDog is fast, reliable and scales Postgres databases without requiring changes to your application. + +## Documentation + +- [Introduction to PgDog](/): PgDog is a connection pooler, load balancer and database sharder for PostgreSQL. Written in Rust, PgDog is fast, reliable and scales Postgres databases without requiring changes to your application. +- [Installation](/installation/): PgDog comes with its own Helm chart. You can install it directly from our chart repository: +- [Client drivers compatibility](/client-drivers/): PgDog is generally compatible with all PostgreSQL client drivers. Some of them are used heavily in production by our customers and users, and we know them to work well. Others, less so, but they are generally expected to be compatible. A lot of them are tested in our CI. +- [About this project](/about/): Contributions in all forms are welcome. If you find a bug and want to provide a fix, feel free to submit a pull request directly, or open an issue. If you'd like to see or implement a new feature, please create an issue first to discuss. +- [Roadmap](/roadmap/): PgDog is being actively developed. Along with performance improvements and bug fixes, new features are constantly being added. The goal for PgDog is to manage most aspects of at-scale Postgres production operations, and we're just getting started. +- [Migration overview](/migrating-to-pgdog/index/): PgDog attempts to make the migration from other connection poolers as smooth as possible. That being said, some changes to your infrastructure may be required to benefit from all our features. +- [Migrating from PgBouncer](/migrating-to-pgdog/from-pgbouncer/): PgBouncer is a popular PostgreSQL connection pooler. PgDog implements the majority of its features, which makes the migration process relatively straightforward. +- [Administration overview](/administration/index/): PgDog keeps track of clients, servers and connection pools. It provides real time statistics on its internal operations for system administrators to keep track of and integrate with monitoring tools like Datadog. +- [Client connections](/administration/clients/): PgDog provides real time statistics and information on all client connections. They can be accessed by connecting to the admin database and running the `SHOW CLIENTS` command: +- [Server connections](/administration/servers/): PgDog provides real time statistics and information on all connections to PostgreSQL databases. They can be accessed by connecting to the admin database and running the `SHOW SERVERS` command: +- [Connection pools](/administration/pools/): PgDog provides real time statistics and information on its connection pools. You can view them by connecting to the admin database and running the `SHOW POOLS` command: +- [Configuration](/administration/config/): PgDog provides real time access to its current configuration values. They can be accessed by connecting to the admin database and running the `SHOW CONFIG` command: +- [Replication](/administration/replication/): PgDog provides a real time view into PostgreSQL replication for the purposes of monitoring replication delay and performing query traffic failover. +- [Background tasks](/administration/tasks/): Long-running operations like resharding don't block the connection that started them. Instead, `RESHARD`, `COPY_DATA`, `REPLICATE`, and `SCHEMA_SYNC` each return a `task_id` and run in the background. `SHOW TASKS` is how you track those tasks: it reports their lifecycle status and progress, lets you find the `task_id` to pass to `STOP_TASK` or `CUTOVER`, and surfaces failures. +- [Architecture overview](/architecture/index/): PgDog is written in the Rust programming language. It is also asynchronous, powered by the Tokio runtime. This allows PgDog to serve hundreds of thousands of connections on one machine and to take advantage of multiple CPUs. +- [Comparison to other poolers](/architecture/comparison/): PgDog aims to be the de facto PostgreSQL proxy and pooler. Below is a feature comparison between PgDog and a few popular alternatives. +- [Benchmarks](/architecture/benchmarks/): PgDog does its best to minimize its impact on database performance. Great care is taken to make sure as few operations as possible are performed when passing data between clients and servers. +- [Configuration overview](/configuration/index/): PgDog uses the TOML configuration language for its two configuration files: +- [Features](/features/index/): PgDog provides foundational and unique features which make it a great choice for modern PostgreSQL deployments. +- [Authentication](/features/authentication/): PostgreSQL servers support many authentication mechanisms. PgDog supports a subset of those, with the aim to support all of them over time. Since PostgreSQL 14, `scram-sha-256` is widely used to encrypt passwords and PgDog supports this algorithm for both client and server connections. +- [Metrics](/features/metrics/): PgDog exposes real-time metrics and statistics about clients, servers, connection pools, and more. They are available via three media: +- [Mirroring](/features/mirroring/): Database mirroring streams traffic, byte for byte, from one database to another. This allows you to test how databases respond to real, production traffic. +- [Multitenancy](/features/multi-tenancy/): PgDog is a natural fit for multitenant databases. It allows to separate data using a key, like `tenant_id`, and automatically route queries with that key to the right database or schema. +- [Plugins overview](/features/plugins/index/): PgDog comes with a powerful plugin system that allows you to customize the query routing behavior. Plugins are written in Rust, compiled into shared libraries, and loaded at runtime. +- [TLS encryption](/features/tls/): PgDog supports TLS for both client and server connections. TLS encryption protects your connections from eavesdropping, especially if used across the public Internet, and is often required to pass security audits. +- [Connection pooler](/features/connection-pooler/index/): PgDog is first and foremost a connection pooler. It can proxy thousands (even hundreds of thousands) of application connections with only a handful of actual PostgreSQL connections. This feature is essential to large and busy databases. Without connection pooling, it would be very difficult to use Postgres in production. +- [Connection recovery](/features/connection-pooler/connection-recovery/): PostgreSQL database connections are expensive to create so PgDog does its best not to close them unless absolutely necessary. In case a client disconnects before fully processing a query response, PgDog will attempt to preserve the connection using several recovery methods. +- [Prepared statements](/features/connection-pooler/prepared-statements/): Prepared statements are SQL queries that are sent to the server in advance. They are parsed by the server, avoiding that cost at execution time. The client can request a statement to be executed by using its name and by passing optional parameters. +- [Session mode](/features/connection-pooler/session-mode/): In session mode, PgDog allocates one PostgreSQL server connection per client. This ensures that all PostgreSQL features work as expected not supported in transaction mode work as expected. +- [Transaction mode](/features/connection-pooler/transaction-mode/): Transaction mode allows PgDog to share just a few of PostgreSQL server connections with thousands of clients. This is required for at-scale production deployments where the number of clients is much higher than the number of available connections to the database. +- [Load balancer overview](/features/load-balancer/index/): PgDog understands the PostgreSQL wire protocol and uses the native PostgreSQL parser to understand queries. This allows it to split read queries from write queries and distribute traffic evenly between databases. +- [Health checks](/features/load-balancer/healthchecks/): All databases load balanced by PgDog are regularly checked with health checks. A health check is a small query that ensures the database is reachable and able to handle requests. +- [Manual routing](/features/load-balancer/manual-routing/): PgDog's load balancer uses the PostgreSQL parser to understand and route queries between the primary and replicas. If you want more control, you can provide the load balancer with hints, influencing its routing decisions. +- [Replication and failover](/features/load-balancer/replication-failover/): PgDog has built-in functionality for monitoring the state of Postgres replica databases. If configured, it can also automatically detect when a replica is promoted and redirect write queries to the new primary, or block replicas from serving traffic if they have fallen far behind in the replication stream. +- [Transactions](/features/load-balancer/transactions/): PgDog's load balancer is transaction-aware and will ensure that all statements inside a transaction are sent to the same PostgreSQL connection on just one database. +- [Sharding PostgreSQL](/features/sharding/index/): Sharding splits up a PostgreSQL database with all its tables and indices between multiple servers. Each machine runs its own, independent PostgreSQL server, while PgDog takes care of routing queries and moving data between databases. +- [Sharding basics](/features/sharding/basics/): Sharding a PostgreSQL database splits it, with all its tables and indices, between multiple machines. Each machine will run its own PostgreSQL server and is, on its own, an independent database. +- [Dry run mode](/features/sharding/dry-run/): In dry run mode, PgDog will parse every single query and record the routing decision in the admin database. If you're experimenting with sharding, this allows you to test the compatibility of your application without resharding data in production. +- [EXPLAIN command](/features/sharding/explain/): `EXPLAIN` is a command that produces the query plan Postgres will use to execute a query. This allows you to check that the query is optimal, before running it. +- [Manual query routing](/features/sharding/manual-routing/): In case the sharding key is not configured or cannot be extracted from the query, PgDog supports explicit sharding directions, provided by the client in a query comment or a `SET` statement. +- [Omnisharded tables](/features/sharding/omnishards/): Omnisharded tables are tables that contain the same data on all shards. This is useful for storing relatively static metadata used in joins or data that doesn't fit the sharding schema of the database, e.g., list of countries, global settings, list of blocked IPs, etc. +- [Direct-to-shard queries](/features/sharding/query-routing/): PgDog has a powerful parser that can extract sharding hints directly from SQL queries. Queries that refer to a column in one of the sharded tables are sent directly to the corresponding database in the configuration. +- [Sharded sequences](/features/sharding/sequences/): !!! note "Unique IDs" Sharded sequences require a bit more configuration to get working. If you're looking for an easy way to generate cross-shard unique 64-bit integers, consider Unique IDs. +- [Sharding functions](/features/sharding/sharding-functions/): The sharding functions determine how to route SQL queries to one or more shard numbers. They can use arbitrary input data to make this decision, and PgDog supports multiple sharding functions. Once a shard number is determined, PgDog will send the query to one or more databases configured in `pgdog.toml`. +- [Supported queries](/features/sharding/supported-queries/): Automatic routing in PgDog works by parsing queries and extracting the sharding key. SQL is a complex language and we are aiming to support as many queries as possible. As the development moves forward, this page will be updated with the latest features. +- [Unique IDs](/features/sharding/unique-ids/): To generate unique identifiers, regular PostgreSQL databases use sequences. For example, `BIGSERIAL` and `SERIAL` columns get their values by calling `SELECT nextval('users_id_seq')`. +- [Two-phase commit](/features/sharding/2pc/index/): Two-phase commit takes advantage of prepared transactions in Postgres to provide eventually consistent cross-shard writes. When enabled, transactions spanning multiple shards have a very high chance of being atomic. +- [Crash recovery](/features/sharding/2pc/crash-recovery/): !!! note "New feature" This feature is new and experimental. Please make sure to test it before deploying to production and let us know if you run into any issues. +- [Cross-shard queries](/features/sharding/cross-shard-queries/index/): If a client can't specify a sharding key, or doesn't specify one in the query, PgDog will send that query to all shards concurrently and combine the results automatically. To the client, this looks like the query was executed by a single database. +- [COPY](/features/sharding/cross-shard-queries/copy/): `COPY` is a special PostgreSQL command that can ingest a file directly into a specified database table. This allows for writing data faster than by using individual `INSERT` queries. +- [CREATE, ALTER, DROP](/features/sharding/cross-shard-queries/ddl/): `CREATE`, `ALTER` and `DROP`, also known as **D**ata **D**efinition **L**anguage (DDL), are, by design, cross-shard statements. When a client sends over a DDL command, PgDog will send it to all shards in parallel, ensuring the table, index, view and sequence definitions are identical across the database cluster. +- [Cross-shard INSERT](/features/sharding/cross-shard-queries/insert/): An INSERT statement that doesn't specify a sharding key, has multiple sharding keys, or targets an omnisharded table is a cross-shard statement and requires special handling, as described below. +- [Cross-shard SELECT](/features/sharding/cross-shard-queries/select/): A cross-shard SELECT query has either no sharding key or multiple sharding keys, which requires it to be executed by multiple database shards. PgDog can perform this in parallel, assembling the results from each shard automatically. This makes it a powerful scatter/gather engine, with data nodes powered by regular PostgreSQL. +- [Cross-shard UPDATE and DELETE](/features/sharding/cross-shard-queries/update/): `UPDATE` and `DELETE` statements that provide none or more than one sharding key in the `WHERE` clause are cross-shard and will be sent to all shards concurrently. +- [Logical replication overview](/features/sharding/internals/logical-replication/index/): One of PgDog's most interesting features is its ability to interpret the logical replication protocol used by Postgres to synchronize replicas. This allows PgDog to reroute data depending on which shard it should go to in a sharded cluster. Since logical replication is streaming data in real time, PgDog can move data between shards invisibly to the client and without database downtime. +- [PostgreSQL query protocol](/features/sharding/internals/query-protocol/): Postgres has two ways to send queries to the server: +- [Resharding Postgres](/features/sharding/resharding/index/): !!! note "Work in progress" This feature is in active development. Support for resharding with logical replication was started in #279 and received major improvements in #784. +- [Traffic cutover](/features/sharding/resharding/cutover/): !!! note "Experimental feature" This is a new and experimental feature. Please make sure to test it before deploying to production and report any issues you find. +- [New databases](/features/sharding/resharding/databases/): PgDog's strategy for resharding Postgres databases is to create a new, independent cluster of machines and move data over to it in real-time. Creating new databases is environment-specific, and PgDog doesn't currently automate this step[^1]. +- [Move data](/features/sharding/resharding/move/): Moving data from the source to the destination database is done using logical replication. This is an online operation, and doesn't require a maintenance window or pausing query traffic. +- [Replica identity](/features/sharding/resharding/replica-identity/): PostgreSQL replica identity controls what the WAL records include in `UPDATE` and `DELETE` entries to identify the affected row. PgDog validates each table's identity mode before resharding begins and handles each mode differently during data movement. +- [Schema sync](/features/sharding/resharding/schema/): PostgreSQL logical replication requires that tables on both the source and destination databases contain the same columns, with compatible data types. PgDog takes care of this, by using `pg_dump` under the hood, and re-creating table and index definitions, in an optimal order, on the new shards. diff --git a/docs/sourcey.config.ts b/docs/sourcey.config.ts new file mode 100644 index 00000000..a8546913 --- /dev/null +++ b/docs/sourcey.config.ts @@ -0,0 +1,118 @@ +import { defineConfig, markdown } from "sourcey"; + +export default defineConfig({ + name: "PgDog", + siteUrl: "https://docs.pgdog.dev", + prettyUrls: "slash", + repo: "https://github.com/pgdogdev/docs", + editBranch: "main", + navigation: { + tabs: [ + { + tab: "Documentation", + slug: "", + source: markdown({ + groups: [ + { + group: "Getting started", + pages: [ + "index", + "installation", + "client-drivers", + "about", + "roadmap", + "migrating-to-pgdog/index", + "migrating-to-pgdog/from-pgbouncer", + ], + }, + { + group: "Administration", + pages: [ + "administration/index", + "administration/clients", + "administration/servers", + "administration/pools", + "administration/config", + "administration/replication", + "administration/tasks", + ], + }, + { + group: "Architecture and configuration", + pages: [ + "architecture/index", + "architecture/comparison", + "architecture/benchmarks", + "configuration/index", + ], + }, + { + group: "Features", + pages: [ + "features/index", + "features/authentication", + "features/metrics", + "features/mirroring", + "features/multi-tenancy", + "features/plugins/index", + "features/tls", + ], + }, + { + group: "Connection pooling", + pages: [ + "features/connection-pooler/index", + "features/connection-pooler/connection-recovery", + "features/connection-pooler/prepared-statements", + "features/connection-pooler/session-mode", + "features/connection-pooler/transaction-mode", + ], + }, + { + group: "Load balancing", + pages: [ + "features/load-balancer/index", + "features/load-balancer/healthchecks", + "features/load-balancer/manual-routing", + "features/load-balancer/replication-failover", + "features/load-balancer/transactions", + ], + }, + { + group: "Sharding", + pages: [ + "features/sharding/index", + "features/sharding/basics", + "features/sharding/dry-run", + "features/sharding/explain", + "features/sharding/manual-routing", + "features/sharding/omnishards", + "features/sharding/query-routing", + "features/sharding/sequences", + "features/sharding/sharding-functions", + "features/sharding/supported-queries", + "features/sharding/unique-ids", + "features/sharding/2pc/index", + "features/sharding/2pc/crash-recovery", + "features/sharding/cross-shard-queries/index", + "features/sharding/cross-shard-queries/copy", + "features/sharding/cross-shard-queries/ddl", + "features/sharding/cross-shard-queries/insert", + "features/sharding/cross-shard-queries/select", + "features/sharding/cross-shard-queries/update", + "features/sharding/internals/logical-replication/index", + "features/sharding/internals/query-protocol", + "features/sharding/resharding/index", + "features/sharding/resharding/cutover", + "features/sharding/resharding/databases", + "features/sharding/resharding/move", + "features/sharding/resharding/replica-identity", + "features/sharding/resharding/schema", + ], + }, + ], + }), + }, + ], + }, +}); diff --git a/scripts/generate_llms.py b/scripts/generate_llms.py new file mode 100644 index 00000000..26b94cdd --- /dev/null +++ b/scripts/generate_llms.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Regenerate docs/llms.txt with the pinned Sourcey release.""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import tempfile +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +CONFIG = ROOT / "docs" / "sourcey.config.ts" +OUTPUT = ROOT / "docs" / "llms.txt" +SOURCEY_VERSION = "3.6.5" + + +def main() -> None: + npx = "npx.cmd" if os.name == "nt" else "npx" + + with tempfile.TemporaryDirectory(prefix="pgdog-sourcey-") as directory: + generated_dir = Path(directory) + subprocess.run( + [ + npx, + "--yes", + f"sourcey@{SOURCEY_VERSION}", + "build", + "--config", + str(CONFIG), + "--output", + str(generated_dir), + ], + cwd=ROOT, + check=True, + ) + + generated = generated_dir / "llms.txt" + if not generated.is_file(): + raise RuntimeError(f"Sourcey did not create {generated}") + + if OUTPUT.is_file() and OUTPUT.read_bytes() == generated.read_bytes(): + print(f"{OUTPUT.relative_to(ROOT)} is already up to date") + return + + shutil.copyfile(generated, OUTPUT) + print(f"Updated {OUTPUT.relative_to(ROOT)} with Sourcey {SOURCEY_VERSION}") + + +if __name__ == "__main__": + main()