fix: prolong and track shared request queue locks to prevent duplicate processing

[vdusek]

The shared request-queue client (access='shared') locked a batch of up to 25 requests for a fixed 180s in list_and_lock_head but never prolonged, released, or tracked those locks. As the batch drained slowly, locks expired mid-flight and requests got processed twice — across clients (another consumer re-locked and processed an expired-lock request) and within one process (an expired request was re-listed and handed to a second worker). This defeated the class's documented "request locking to prevent duplicate processing" guarantee (double charges in PPE, duplicate dataset items).

Changes:

• Store lock_expires_at per cached head entry (it was returned by the API but discarded).
• Add a _requests_in_progress set; skip in-progress IDs both when re-listing the head and when fetching, so a request is never handed to a second consumer in the same process.
• In fetch_next_request, skip queued entries whose lock has already expired (another client may own them now) and prolong the chosen request's lock so the consumer gets a full window starting at hand-off; release the reservation on every early-return path.

Out of scope (not part of the reported failure, and not minimally feasible today):

• Deleting leftover head locks on shutdown/migration - Crawlee's RequestQueueClient exposes no close/finalize hook to attach it to; leftover locks still lapse naturally within the lock TTL.
• Handlers running longer than the (prolonged) lock window would need a background lock-refresh loop; the reported scenarios (handlers shorter than the lock) are fully covered by prolonging at hand-off.

✍️ Drafted by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 89.18919% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.85%. Comparing base (58d6da1) to head (f3be1be).

Files with missing lines	Patch %	Lines
...age_clients/_apify/_request_queue_shared_client.py	89.18%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1062      +/-   ##
==========================================
- Coverage   91.89%   91.85%   -0.05%     
==========================================
  Files          51       51              
  Lines        3232     3265      +33     
==========================================
+ Hits         2970     2999      +29     
- Misses        262      266       +4     

Flag	Coverage Δ
e2e	?
integration	57.09% <62.16%> (+<0.01%)	⬆️
unit	85.42% <89.18%> (+2.15%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vdusek]

Btw. this is a different case than #887

[Pijukatel]

What about the cost increase? Won't this lead to each request being locked at least twice? Maybe we could lock only if the remaining time is something very small?

I will run benchmarks with this change to quantify the cost increase. Maybe it is not serious

[Pijukatel]

Let's wait for them to finish:
Without change: https://github.com/apify/actor-benchmarks/actions/runs/29900513281
With change: https://github.com/apify/actor-benchmarks/actions/runs/29900596790

[Pijukatel]

Summary for the repeated 10 benchmark runs
Without change:

Overall benchmark of all runs, :aggregated_benchmark=Actor: parsel-crawler-benchmark, Valid results: 2013.6, Runtime: 59.6927 s, Costs: 0.1262127994999134 USD

With change:

Overall benchmark of all runs, :aggregated_benchmark=Actor: parsel-crawler-benchmark, Valid results: 2013.3, Runtime: 61.2107 s, Costs: 0.14725829787564015 USD,

[Pijukatel]

Around 16 % cost increase seems pretty high to me. Maybe some optimization could be used to mitigate this increase.

How is it done in JS SDK?