GH-47686: [Docs][Python] Split the Python Parquet docs into separate items

[AlenkaF]

Rationale for this change

The current parquet section is one-page and covering lots of topics. As per #46601 the parquet section is most visited and would therefore need a bit of maintenance/content update.

What changes are included in this PR?

• Parquet documentation is split in 4 different subpages (index.rst, parquet.rst, parquet_datasets.rst and parquet_encryption.rst).
• Some additional content is added

Are these changes tested?

Via the docs preview

Are there any user-facing changes?

No

• GitHub Issue: [Docs][Python] Split the Python Parquet docs into separate items #47686

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 156850c

Submitted crossbow builds: ursacomputing/crossbow @ actions-937dad9db2

Task	Status
preview-docs

[AlenkaF]

I have used Claude for suggesting the split of pyarrow parquet page and to look through the code and the git log history for any features/changes that might be missing from the docs. From what was suggested I continued with the work on my own.

I have split the changes in different commits so the change will be easier to review (due to the docs split, the change is bigger than it really is).

commit 1: restructure of the content, no content changes @thisisnic, do you mind having a look?
commit 2: add content about filtering and ability to write bloom filters @mapleFU, do you mind having a look?
commit 3: add section on iter_batches @raulcd, do you mind having a look?
commit 4: add section on binary/list_type option on read, reading extension types @pitrou @paleolimbot, do you mind having a look?
commit 5: datasets note on single file partitions not being included
commit 6: update parquet writer's parameter list @raulcd, do you mind having a look?
commit 7: update encryption section adding info on key rotation and direct encryption @adamreeve, do you mind having a look?

Rendered docs are available here: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html

[Copilot]

Pull request overview

Splits the PyArrow Parquet user-guide documentation into multiple focused pages to improve navigability and enable incremental maintenance, aligning with the broader Python docs revision effort.

Changes:

• Replaces the single large Parquet guide page with a section index plus dedicated subpages (single-file IO, datasets, type handling, encryption).
• Updates the Python docs toctree to point at the new Parquet section entry point.
• Adds/relocates some additional Parquet-related content during the split (e.g., content-defined chunking).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
docs/source/python/parquet/parquet.rst	New “single files” Parquet IO page (read/write, metadata, filtering, etc.).
docs/source/python/parquet/parquet_type_handling.rst	New page for Parquet read/write type-handling options.
docs/source/python/parquet/parquet_datasets.rst	New page for partitioned datasets / multi-file Parquet usage.
docs/source/python/parquet/parquet_encryption.rst	New page for Parquet modular encryption docs.
docs/source/python/parquet/index.rst	New Parquet section landing page with toctrees to subpages.
docs/source/python/parquet.rst	Removes the previous monolithic Parquet documentation page.
docs/source/python/index.rst	Updates the Python guide toctree to reference the new Parquet section entry.

[paleolimbot]

Thank you!

I took a read through and this content seems well organized and I appreciate the additions for Geometry/Geography types!

[paleolimbot]

🎉 !

[adamreeve]

Thanks @AlenkaF, I've skimmed over most of this and looked more thoroughly at the encryption section changes.

[adamreeve]

These examples use Pandas where plain PyArrow could be easily used, which might give the impression to users that they need to use Pandas when reading or writing Parquet. I can see these are existing examples that have been moved though, so this isn't really related to this PR. Maybe as a follow up we should look at using pure PyArrow in more places, except where we are explicitly documenting Pandas integration?

[AlenkaF]

Yes, I totally agree. We need to make sure pandas is not used for creation of tables only (added that to the docs umbrella issue here).

Why I have kept the pandas df here was due to the content further on connected to maintaining pandas index column, see line 74 and 75.

[AlenkaF]

Thank you all for the reviews!
I still need to go through the Copilot review comments, but I have applied other review comment suggestions.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (3)

docs/source/python/parquet/parquet_type_handling.rst:120

• Sentence casing/grammar: “In addition, We provide …” should be “In addition, we provide …”.

In addition, We provide the ``coerce_timestamps`` option to allow you to select

docs/source/python/parquet/parquet_encryption.rst:71

• Grammar: “as following” should be “as follows”.

defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:

docs/source/python/parquet/parquet_datasets.rst:26

• Grammar: “These may present in a number of ways” is missing “be”.

Multiple Parquet files constitute a Parquet *dataset*. These may present in a
number of ways:

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (4)

docs/source/python/parquet/parquet_encryption.rst:108

• :download: target is now one directory deeper (docs/source/python/parquet/…), so ../../../python/... resolves under docs/ and breaks the download link. Update the relative path to reach repo root.

An :download:`example <../../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`

docs/source/python/parquet/parquet_type_handling.rst:73

• This code example uses pq.write_table / pq.read_table but doesn't import pyarrow.parquet as pq, making the snippet non-runnable as written.

   >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
   >>> pq.write_table(table, 'binary.parquet', store_schema=False)
   >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
   data: large_binary

docs/source/python/parquet/parquet_type_handling.rst:121

• Sentence casing/grammar: "In addition, We provide" should use lowercase "we".

the desired resolution:

docs/source/python/parquet/parquet_datasets.rst:26

• Grammar: "These may present in a number of ways" is missing a verb (should be "may be present" / "may be presented").

Multiple Parquet files constitute a Parquet *dataset*. These may be present in a
number of ways:

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

docs/source/python/parquet/parquet.rst:284

• The cross-reference uses :func:`~parquet.read_metadata``` which likely resolves to a top-level parquetmodule rather thanpyarrow.parquet`` and can break the generated link. Use a relative target under the currentmodule (or the fully-qualified name).

or can also be read directly using :func:`~parquet.read_metadata`:

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

docs/source/python/parquet/parquet.rst:269

• :class:~.ParquetFile`` resolves to pyarrow.ParquetFile under `.. currentmodule:: pyarrow`, but the actual class is `pyarrow.parquet.ParquetFile`. Use the fully-qualified reference so Sphinx can link it correctly.

The ``FileMetaData`` of a Parquet file can be accessed through
:class:`~.ParquetFile` as shown above:

docs/source/python/index.rst:57

• Switching the root toctree entry from parquet to parquet/index removes the generated python/parquet.html page (the old docs/source/python/parquet.rst is deleted), which will break existing inbound links/bookmarks. Consider keeping docs/source/python/parquet.rst as a shim landing page (old URL) that links/includes the new subpages, and keep the toctree entry as parquet.

   timestamps
   parquet/index
   dataset

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

docs/source/python/parquet/parquet.rst:103

• This section labels the output as "RSS" but prints pa.total_allocated_bytes() (Arrow-allocated bytes), and it also uses path without defining it. This makes the example misleading and not directly copy/pasteable.

Because Parquet data needs to be decoded from the Parquet format
and compression, it can't be directly mapped from disk.
Thus the ``memory_map`` option might perform better on some systems
but won't help much with resident memory consumption.

docs/source/python/parquet/index.rst:72

• PR description says the Parquet docs are split into 4 subpages, but the docs index includes an additional subpage (parquet_type_handling). Please update the PR description (or adjust the split) so the change summary matches what is actually being added.

Data Type Handling
------------------

.. toctree::
   :maxdepth: 2

   parquet_type_handling

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 58575d9

Submitted crossbow builds: ursacomputing/crossbow @ actions-c31e44a7c8

Task	Status
preview-docs

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 9773c76

Submitted crossbow builds: ursacomputing/crossbow @ actions-4a1d237853

Task	Status
preview-docs

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

docs/source/python/parquet/parquet_encryption.rst:108

• The :download: link path is no longer correct after moving this doc under docs/source/python/parquet/ (it now points under docs/python/... instead of repo-root python/examples/...). This will break the generated download link.

An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`

docs/source/python/parquet/parquet.rst:112

• pa.total_allocated_bytes() reports Arrow memory pool allocations, not OS RSS. Labeling this as "RSS" is misleading (and the sample output repeats that label). Consider renaming to something accurate (or use a real RSS measurement if that’s the intent).

   >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20))  # doctest: +SKIP
   RSS: 4299MB

   >>> pq_array = pq.read_table(path, memory_map=False)  # doctest: +SKIP
   >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20))  # doctest: +SKIP

docs/source/python/index.rst:56

• Switching the toctree entry from parquet to parquet/index (and deleting docs/source/python/parquet.rst) changes the generated page from python/parquet.html to python/parquet/index.html. That will break any existing inbound links/bookmarks to the old URL; consider keeping a thin docs/source/python/parquet.rst stub that links/includes the new index page to preserve the old docname.

   parquet/index