Skip to content

fix(logging): Preserve host application logging configuration#2010

Open
vdusek wants to merge 3 commits into
masterfrom
worktree-fix-first-run-embedding-surprises
Open

fix(logging): Preserve host application logging configuration#2010
vdusek wants to merge 3 commits into
masterfrom
worktree-fix-first-run-embedding-surprises

Conversation

@vdusek

@vdusek vdusek commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Problem

BasicCrawler.__init__ (with the default configure_logging=True) called configure_logger(root_logger, remove_old_handlers=True), wiping the host application's root logging handlers. Embedding Crawlee in a larger app silently removed the host's own logging configuration.

Changes

  • The root logger is now configured only when it has no handlers yet, mirroring logging.basicConfig semantics. A host that has already configured logging keeps its configuration.
  • The httpx logger silencing was moved out of BasicCrawler into HttpxHttpClient. It used to live unconditionally in the base crawler, even though the default HTTP client (ImpitHttpClient) doesn't use the httpx logger at all, and it reached into a third-party logger straight from the base class. The silencing now applies only when HttpxHttpClient is actually instantiated, which is where the noisy per-request httpx INFO logs originate.

Verification

Regression tests that fail without the fix:

  • host-installed root handlers survive crawler construction,
  • Crawlee still configures an unconfigured root logger,
  • HttpxHttpClient silences the httpx logger to WARNING by default.

Behavior change

This intentionally changes observable behavior, since the old behavior was the bug: host apps keep their logging config.

For HttpxHttpClient users, the httpx logger is now silenced whenever the client is instantiated. Previously the silencing happened only during BasicCrawler construction, with configure_logging=True and an unconfigured root logger. As a result it now also applies with configure_logging=False and to standalone send_request usage.

@vdusek vdusek self-assigned this Jul 2, 2026
@vdusek vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Jul 2, 2026
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.35%. Comparing base (7780e78) to head (3567e12).
⚠️ Report is 46 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2010      +/-   ##
==========================================
- Coverage   93.35%   93.35%   -0.01%     
==========================================
  Files         179      179              
  Lines       12482    12497      +15     
==========================================
+ Hits        11652    11666      +14     
- Misses        830      831       +1     
Flag Coverage Δ
unit 93.35% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek vdusek changed the title fix: Preserve host logging, keep named request queues across runs, fix adaptive-crawler extra fix: Preserve host application logging configuration Jul 3, 2026
@vdusek vdusek marked this pull request as ready for review July 10, 2026 16:28
@vdusek vdusek changed the title fix: Preserve host application logging configuration fix(logging): Preserve host application logging configuration Jul 10, 2026
@vdusek vdusek requested review from Mantisus and Pijukatel July 10, 2026 16:37

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants