Add compression to replicator

[lacklacklack]

Overview

Adds optional gzip compression of outbound request bodies in the replicator.
When enabled, _bulk_docs and _revs_diff request bodies are gzip-compressed
before sending. CouchDB already supports Content-Encoding: gzip on inbound
requests so no server-side changes are needed.

Compression is disabled by default to avoid breaking setups with older or
non-CouchDB targets.

Testing recommendations

Enable compression with a low threshold and run a replication, then check the stats:

# Start CouchDB
./dev/run --admin=admin:admin

# Lower the threshold for testing
curl -X PUT "http://admin:admin@127.0.0.1:15984/_node/_local/_config/replicator/compress_requests" -d '"true"'
curl -X PUT "http://admin:admin@127.0.0.1:15984/_node/_local/_config/replicator/compress_min_size" -d '"10"'

# Create DBs, add docs, replicate
curl -X PUT http://admin:admin@127.0.0.1:15984/source_db
curl -X PUT http://admin:admin@127.0.0.1:25984/target_db
curl -X PUT http://admin:admin@127.0.0.1:15984/source_db/doc1 \
  -H "Content-Type: application/json" \
  -d '{"data":"'$(python3 -c "print('x'*2000)'"}'
curl -X POST http://admin:admin@127.0.0.1:15984/_replicate \
  -H "Content-Type: application/json" \
  -d '{"source":"http://admin:admin@127.0.0.1:15984/source_db","target":"http://admin:admin@127.0.0.1:25984/target_db"}'

# Check stats — value should be > 0
curl -s http://admin:admin@127.0.0.1:15984/_node/_local/_stats/couch_replicator/requests_compressed | python3 -m json.tool

Automated tests: couch_replicator_compression_tests in src/couch_replicator/test/eunit/.

Related Issues or Pull Requests

• https://github.ibm.com/cloudant/dbcore/issues/694 — Enhance replicator to compress HTTP request bodies (this PR)
• https://github.ibm.com/cloudant/dbcore/issues/693 — Enhance replicator to decompress HTTP response bodies (follow-up)

Checklist

• This is my own work, I did not use AI, LLM's or similar technology
• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches

[janl]

Can you please use our PR template?

[nickva]

Nice work @lacklacklack!

A few suggestions / ideas as bullet points:

1. We're trying to do a bit of both: request and response sides; let's keep it simpler at first and focus on the request side (_bulk_docs and _revs_diff). Since we already have server side handling for gzip, let's handle gzip only for the request side. Then we can cleanly test the feature in CI without extra proxies or having to add server side compression sending here too.

2. Let's skip deflate and use gzip only. But add the possibility to add zstd in the future (not this PR though) so keep the configurable compression algorithm. The reason to skip deflate is simplicity (our server handles gzip only), and deflate is a bit of a mess according to https://en.wikipedia.org/wiki/HTTP_compression

Another problem found while deploying HTTP compression on large scale is due to the deflate encoding definition: while HTTP 1.1 defines the deflate encoding as data compressed with deflate (RFC 1951) inside a zlib formatted stream (RFC 1950), Microsoft server and client products historically implemented it as a "raw" deflated stream making its deployment unreliable. For this reason, some software, including the Apache HTTP Server, only implements gzip encoding.

3. As it stands, the most important thing to compress (_bulk_docs body) won't actually be compressed. The body there isn't an iolist or binary but {BodyFun, [prefix | Docs]}}. So that makes me think maybe a better place for this is not in httpc but in api_wrap

4. Do not set AcceptEncodings = config:get("replicator", "accept_encodings", "gzip, deflate, zstd") unless we can always handle these responses and decompress them. If the server then sends us zstd data and we're on OTP 27 we won't be able to handle it and fail the request. For this pr let's just skip setting that altogether

5. Do not enable gzip compression by default. Since that is not a negotiated setting, if the replicator was talking to an older CouchDB or other server not implementing gzip decompression we'd break a customers' setup as soon as they upgrade.

6. Don't forget to fill out the template like Jan suggested

[nickva]

Looks much better!

Added a few more comments with some minor tweaks

One new major bit I thought of is if we can make this a per job overridable. In theory we have https://docs.couchdb.org/en/stable/config/replicator.html#replicator/worker_processes to copy from (and a few others). But if that proves difficult we can punt it for later

[nickva]

Looks good but needs a few more tweaks.

We could also consider gzipping _bulk_get requests. There are not as big as _bulk_docs of course but we're bothering with _revs_diff and _bulk_get is on the order of _revs_diff so if it's good for the goose -- it's good for the gander, as they say.

[nickva]

Very nice improvements. Just added a few tiny style nits then it should be good to go

[nickva]

Can we use cluster_compare_dbs/2 instead from the test utils module?

[nickva]

Since we're testing large compression bodies. At at least one test with say 500 docs

[nickva]

As a style nit, typically we don't do a lot of try...after...cleanup in tests. It's better to use proper test setup/cleanup functions. For example this pattern:

            fun setup/0,
            fun teardown/1,
            [
                ?TDEF_FE(some_test)
            ]



setup() ->
    config:set("replicator", "request_compression", "none", false),
    ... other set...
    couch_replicator_test_helper:test_setup().

teardown(Ctx) ->
    config:delete("replicator", "request_compression", false),
    ... other config:delete
    couch_replicator_test_helper:test_teardown(Ctx).

Then each test only has to set non-default values and we skip an extra indent level and an an extra 5 lines with try...after....end block.

[nickva]

I think we forgot to mention _bulk_gets as well?

[nickva]

Style nit: this line is getting a bit too long. Maybe break out Len = byte_size(Compressed) to shorten it a bit