Skip to content

Fix for epa-airpollutantemission-level1 - #2148

Open
shourya116 wants to merge 5 commits into
datacommonsorg:masterfrom
shourya116:fix_epa-airpollutantemission-level1
Open

Fix for epa-airpollutantemission-level1#2148
shourya116 wants to merge 5 commits into
datacommonsorg:masterfrom
shourya116:fix_epa-airpollutantemission-level1

Conversation

@shourya116

@shourya116 shourya116 commented Aug 10, 2026

Copy link
Copy Markdown

Fixes an issue where 767,314 observations across the 2017 and 2020 observation periods (specifically nonpoint/area sources) were silently dropped during the Cloud Batch import validation pipeline for EPA_AirPollutantEmission_Level1.

Root Cause : Pandas 2.x Type Mismatch: The raw 2017 and 2020 nonpoint CSVs contained empty emissions type code columns, which Pandas loaded as float64. The check 'point' in file_path evaluated to True for ..._nonpoint/... files, triggering df.loc[:, 'emissions type code'] = ''. In Pandas 2.x, assigning a string to a float64 series via .loc raises a TypeError: Invalid value '' for dtype 'float64'. Broad exception handling in _national_emissions() caught the TypeError and returned an empty DataFrame, silently dropping the entire nonpoint dataset (767k rows) during consolidation.

Files Updates: process.py, manifest.json, validation_config.json, golden_files/*

gcs validation_output.csv : https://pantheon.corp.google.com/storage/browser/_details/datcom-import-test/scripts/us_epa/national_emissions_inventory/EPA_AirPollutantEmission_Level1/2026_08_12T02_05_10_395551_07_00/input0/validation/validation_output.csv;tab=live_object?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&e=13803378&mods=-monitoring_api_staging&project=datcom-infosys-dev

cloud batch job : https://pantheon.corp.google.com/batch/jobsDetail/regions/us-west4/jobs/epa-airpollutantemission-level1-shouryasingh-20260812-090213/details?e=13803378&invt=Ab5y7Q&mods=-monitoring_api_staging&project=datcom-infosys-dev

@google-cla

google-cla Bot commented Aug 10, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the resource limits (CPU, memory, and disk) in the manifest configuration and refactors pandas DataFrame operations in process.py to avoid using .loc for simple column assignments and to replace deprecated inplace=True usage. Feedback is provided to remove a redundant .replace('', np.nan) call on the observation column, as empty strings are already converted to NaN earlier in the processing pipeline.

Comment thread scripts/us_epa/national_emissions_inventory/process.py Outdated
shouryasingh1616 and others added 4 commits August 11, 2026 12:30
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants