GH-39961: [C++][Python] Propagate CSV parse delimiter to write options

[egolearner]

Rationale for this change

resolve #39961

What changes are included in this PR?

When writing a dataset with CsvFileFormat configured with a custom ParseOptions delimiter (e.g. delimiter=">"), the output CSV still used the default "," delimiter. This was because:

• C++ CsvFileFormat::DefaultWriteOptions() always created WriteOptions::Defaults() with delimiter=',', ignoring the parse_options.delimiter stored on the format object.

• Python CsvFileFormat.make_write_options() unconditionally overwrote the C++ write options with a fresh WriteOptions(**kwargs), discarding any C++-side default that might have been set.

Fix the C++ side by propagating parse_options.delimiter into the write options in DefaultWriteOptions(). Fix the Python side by reading the delimiter from the C++ default when the caller does not explicitly specify one, preventing the overwrite from losing the propagated value.

Are these changes tested?

Yes

Are there any user-facing changes?

No

• GitHub Issue: [C++][Python] write_dataset with csv file format ignores delimiter #39961

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #39961 has been automatically assigned in GitHub to PR creator.

[kou]

Could you also add a C++ level test for this case?

[Copilot]

Pull request overview

Fixes pyarrow.dataset.write_dataset for CSV when CsvFileFormat is configured with a non-default ParseOptions.delimiter, ensuring the writer uses the same delimiter by default.

Changes:

• C++: propagate CsvFileFormat::parse_options.delimiter into DefaultWriteOptions().
• Python: preserve the C++-provided default delimiter in CsvFileFormat.make_write_options() unless explicitly overridden.
• Tests: add coverage verifying delimiter propagation and an end-to-end write_dataset roundtrip.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
python/pyarrow/tests/test_csv.py	Adds regression + end-to-end test for delimiter propagation from parse options into dataset writes.
python/pyarrow/_dataset.pyx	Keeps delimiter from C++ default write options when delimiter isn’t explicitly provided by the caller.
cpp/src/arrow/dataset/file_csv.cc	Sets CSV write-options delimiter from the format’s parse-options delimiter in DefaultWriteOptions().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[egolearner]

Could you also add a C++ level test for this case?

done

[Copilot]

Pull request overview

Fixes a cross-language mismatch where dataset.CsvFileFormat(parse_options=ParseOptions(delimiter=...)) was correctly stored for reading but not reflected in default write options, causing write_dataset to keep writing ,-delimited CSV output.

Changes:

• C++: propagate CsvFileFormat::parse_options.delimiter into CsvFileFormat::DefaultWriteOptions().
• Python: preserve the C++-provided default delimiter in CsvFileFormat.make_write_options() when the caller doesn’t explicitly pass delimiter=....
• Adds C++ and Python tests to validate delimiter propagation and end-to-end dataset write/read behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
python/pyarrow/tests/test_csv.py	Adds coverage for delimiter propagation in make_write_options() and an end-to-end write_dataset roundtrip.
python/pyarrow/_dataset.pyx	Ensures Python does not clobber the C++ default delimiter when creating WriteOptions.
cpp/src/arrow/dataset/file_csv_test.cc	Adds a unit test verifying C++ DefaultWriteOptions() carries over the parse delimiter.
cpp/src/arrow/dataset/file_csv.cc	Implements delimiter propagation from parse options to default write options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[egolearner]

Hi @kou Could you please take a look and help re-run the CI? The previous failure was due to a network issue.

[kou]

@AlenkaF @raulcd @rok Do you want to review the Python part?

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[egolearner]

Hi @AlenkaF PTAL

[kou]

Could you rebase on main to fix the CI failure?

[egolearner]

Could you rebase on main to fix the CI failure?

Done.

[AlenkaF]

LGTM, thanks!
I think the CI failure is not related. Will wait for @kou before merging.