diff --git a/docs/community/comparison-refresh-checklist.md b/docs/community/comparison-refresh-checklist.md new file mode 100644 index 0000000..9f75a95 --- /dev/null +++ b/docs/community/comparison-refresh-checklist.md @@ -0,0 +1,115 @@ +--- +title: Comparison refresh checklist +description: >- + A repeatable, evidence-first checklist for keeping FreshData comparisons with + pandas, PyJanitor, Great Expectations, ydata-profiling, cleanlab, and + OpenRefine current. +keywords: freshdata comparison, contributor checklist, pandas, pyjanitor, great expectations, ydata profiling, cleanlab, openrefine +--- + +# Comparison refresh checklist + +Use this checklist when a compared project publishes a significant release, +when FreshData changes a capability named in the comparison, or before a +FreshData release. The canonical detailed tables live on the +[comparison page](../comparison.md); do not copy them into the README. + +The goal is not to make FreshData win more rows. It is to keep every verdict +current, reproducible, and useful to someone choosing a tool. + +## 1. Record the review scope + +- [ ] Record the review date and the FreshData commit under review in the pull + request description. +- [ ] Record the latest stable version checked for every relevant project. Use + a released version, not an unreleased documentation preview. +- [ ] Read the release notes since the version used by the previous review. +- [ ] List the rows affected by upstream or FreshData changes. Do not rewrite + unrelated rows just to make the table sound more uniform. + +Check these primary sources first: + +| Project | Capability source | Change source | Surfaces to recheck | +|---|---|---|---| +| pandas | [User guide](https://pandas.pydata.org/docs/user_guide/) | [Release notes](https://pandas.pydata.org/docs/whatsnew/index.html) | missing data, duplicates, scaling, and individual DataFrame operations | +| PyJanitor | [Functions API](https://pyjanitor-devs.github.io/pyjanitor/api/functions/) | [GitHub releases](https://github.com/pyjanitor-devs/pyjanitor/releases) | convenience verbs, method chaining, domain modules, and pandas-only assumptions | +| Great Expectations | [GX Core documentation](https://docs.greatexpectations.io/docs/core/introduction/) | [GitHub releases](https://github.com/great-expectations/great_expectations/releases) | expectations, checkpoints, Data Docs, execution backends, and repair boundaries | +| ydata-profiling | [Documentation](https://docs.profiling.ydata.ai/latest/) | [GitHub releases](https://github.com/ydataai/ydata-profiling/releases) | profiling, visual reports, dataset comparison, integrations, and whether data is mutated | +| cleanlab | [Stable documentation](https://docs.cleanlab.ai/stable/) | [GitHub releases](https://github.com/cleanlab/cleanlab/releases) | label issues, Datalab issue types, model-output requirements, and repair workflows | +| OpenRefine | [User manual](https://openrefine.org/docs) | [GitHub releases](https://github.com/OpenRefine/OpenRefine/releases) | interactive curation, reconciliation, operation-history replay, clustering, and scale | + +Official documentation, API references, repositories, and release notes are +primary sources for upstream capabilities. Search results, vendor blog posts, +and third-party comparisons can help locate a claim, but must not be its only +support. + +## 2. Audit every affected row + +For each row that is added or reviewed: + +- [ ] Name one narrow criterion. Split rows that combine independently + testable claims, such as runtime and peak memory. +- [ ] Reproduce the current upstream behavior when the verdict depends on + runtime behavior rather than a documented API boundary. +- [ ] Give the row an explicit **win**, **loss**, **tie**, or **unproven** + verdict using the rubric below. +- [ ] Put a source or a short, checkable rationale in the row's evidence cell. + A bare assertion such as "tool X cannot do this" is not sufficient. +- [ ] Link an upstream claim to the project's official documentation, API, or + release notes. Prefer a versioned page when behavior differs by version. +- [ ] Link a FreshData capability claim to a specific test, implementation, + benchmark artifact, or documentation contract in this repository. +- [ ] For performance or quality claims, record versions, inputs, commands, + environment, repeated measurements, and the committed raw result. Do not + infer a benchmark result from documentation. +- [ ] Compare like with like: the same workload, output contract, materialized + or lazy mode, and optional dependencies. State unavoidable differences. +- [ ] Check that every repository-relative path and external link still works. +- [ ] Remove or downgrade a verdict when its evidence is stale or cannot be + reproduced. **Unproven** is an acceptable result. + +Use this verdict rubric consistently: + +| Verdict | Required support | +|---|---| +| **win** | FreshData satisfies the criterion and the named alternative does not, supported by primary sources or a reproducible head-to-head result. | +| **loss** | The alternative satisfies the criterion better or FreshData lacks the capability; state the limitation plainly. | +| **tie** | Both satisfy the criterion at the stated scope. Do not use a tie to hide meaningful trade-offs. | +| **unproven** | Evidence is absent, stale, incomparable, or inconclusive. Do not turn expectations into a verdict. | + +## 3. Keep evidence reproducible + +- [ ] Put reusable competitor adapters in `benchmarks/baselines/`, not only in + a pull-request comment or an uncommitted notebook. +- [ ] Commit evidence to the tracked destination for its harness: + `benchmarks/cleanbench/results/` for CleanBench, the relevant accepted + `baseline.json` for Gauntlet or TruthBench, or compact + `benchmarks/results/performance/*-summary.json` plus `*-report.md` for the + performance harness. Generic `benchmarks/results//`, Gauntlet run + output, TruthBench `latest.*`, and raw performance cases are intentionally + ignored; keep them as CI artifacts and link the run. Do not force-add an + ignored raw result as permanent evidence. +- [ ] Use committed fixtures or document how generated input is seeded. +- [ ] Confirm that the comparison text matches the stored results; do not copy + rounded numbers from an older run. +- [ ] Note optional extras, paid services, remote APIs, GPUs, or model outputs + required by either tool. +- [ ] Keep cleaning, profiling, validation, and label-quality detection as + separate categories. Similar names do not imply equivalent behavior. + +## 4. Protect the README and final wording + +- [ ] Keep the detailed matrix in `docs/comparison.md` as the single source of + truth. +- [ ] If the README mentions competitors, use a short neutral summary and link + to the comparison page. Do not duplicate the detailed tables there. +- [ ] Re-read the introduction, verdict, and "cannot claim" section after + changing rows so they do not overstate the updated evidence. +- [ ] Use the projects' current names and capitalization: pandas, PyJanitor, + Great Expectations (GX), ydata-profiling, cleanlab, and OpenRefine. +- [ ] Run `mkdocs build --strict` and `ruff check .` before opening the pull + request. + +In the pull request, summarize which upstream versions were checked, which +rows changed, what evidence was added or retired, and any claims deliberately +left **unproven**. diff --git a/docs/comparison.md b/docs/comparison.md index e7fb088..d8a3144 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -1,5 +1,9 @@ # Honest comparison: pandas, PyJanitor, Great Expectations, OpenRefine +Contributors updating these tables should follow the +[comparison refresh checklist](community/comparison-refresh-checklist.md), +including its source, reproducibility, verdict, and README-scope rules. + Every cell is one of **win / loss / tie / unproven**, and every *win* links to committed evidence in this repository. "Unproven" means we believe something but have not measured it — we say so instead of claiming it. Losses are diff --git a/mkdocs.yml b/mkdocs.yml index 717d072..6d4fe78 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -145,4 +145,5 @@ nav: - Overview: contributing.md - First PR guide: community/first-pr.md - Contributor roadmap: community/contributor-roadmap.md + - Comparison refresh checklist: community/comparison-refresh-checklist.md - Project labels: community/labels.md