docs: add Great Expectations validation recipe - #180
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughAdds an optional Great Expectations recipe that validates raw order data, cleans it with ChangesGreat Expectations recipe
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to This change adds an optional Great Expectations validation recipe without altering FreshData’s core dependencies or existing behavior. The documented checks pass, and no actionable merge-blocking risk remains. Sequence Diagram(s)sequenceDiagram
participant main
participant GreatExpectationsCheckpoint
participant freshdata_clean
main->>GreatExpectationsCheckpoint: Validate raw DataFrame
GreatExpectationsCheckpoint-->>main: Validation failure
main->>freshdata_clean: Clean raw DataFrame
freshdata_clean-->>main: Return cleaned DataFrame and report
main->>GreatExpectationsCheckpoint: Validate cleaned DataFrame
GreatExpectationsCheckpoint-->>main: Validation success
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
CI note: every check except |
|
Strix is installed on this repository, but we couldn't run this PR security review because this workspace's trial has ended. Add a card to resume code reviews here. So far, Strix has reviewed 36 pull requests, surfaced 3 security issues (1 critical/high) and blocked 1 risky merge across this workspace. |
JohnnyWilson16
left a comment
There was a problem hiding this comment.
Looks great! The repair-then-validate example is clean, well-structured, and keeps Great Expectations nicely decoupled as an optional dependency.
Really appreciate your continuous contributions to FreshData @WilliamK112 — thanks for helping improve the project! 🚀
Description
Add a runnable Great Expectations interoperability recipe that demonstrates a repair-then-validate workflow without adding Great Expectations to FreshData's core dependencies.
The example builds one in-memory GX Core checkpoint, runs it against raw currency and boolean strings (expected to fail), cleans the same DataFrame with
fd.clean(), and reruns the checkpoint (expected to pass). It also prints the FreshData repair report so the relationship between repair and validation is explicit.The recipe is linked from both examples indexes, and the user-facing addition is recorded under
Unreleasedin the changelog.Closes #8
Type of Change
Validation
GX_ANALYTICS_ENABLED=false PYTHONPATH=src python examples/10_great_expectations_recipe.pywith Great Expectations 1.20.0 (before: failed as expected; after: passed)ruff check .mypy examples/10_great_expectations_recipe.pymkdocs build --strictPYTHONPATH=src pytest -m 'not online and not large' -q(passed; 93.16% coverage)git diff --checkChecklist
mypy) and linter checks (ruff) pass locallyAI Assistance
AI assistance was used to research the current Great Expectations API, draft the example and documentation, and run validation. I reviewed the resulting diff and verified the recipe locally with Great Expectations 1.20.0.
Summary by CodeRabbit
New Features
Documentation