Fix for epa-airpollutantemission-level1 - #2148
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request increases the resource limits (CPU, memory, and disk) in the manifest configuration and refactors pandas DataFrame operations in process.py to avoid using .loc for simple column assignments and to replace deprecated inplace=True usage. Feedback is provided to remove a redundant .replace('', np.nan) call on the observation column, as empty strings are already converted to NaN earlier in the processing pipeline.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Fixes an issue where 767,314 observations across the 2017 and 2020 observation periods (specifically nonpoint/area sources) were silently dropped during the Cloud Batch import validation pipeline for EPA_AirPollutantEmission_Level1.
Root Cause : Pandas 2.x Type Mismatch: The raw 2017 and 2020 nonpoint CSVs contained empty emissions type code columns, which Pandas loaded as float64. The check 'point' in file_path evaluated to True for ..._nonpoint/... files, triggering df.loc[:, 'emissions type code'] = ''. In Pandas 2.x, assigning a string to a float64 series via .loc raises a TypeError: Invalid value '' for dtype 'float64'. Broad exception handling in _national_emissions() caught the TypeError and returned an empty DataFrame, silently dropping the entire nonpoint dataset (767k rows) during consolidation.
Files Updates: process.py, manifest.json, validation_config.json, golden_files/*
gcs validation_output.csv : https://pantheon.corp.google.com/storage/browser/_details/datcom-import-test/scripts/us_epa/national_emissions_inventory/EPA_AirPollutantEmission_Level1/2026_08_12T02_05_10_395551_07_00/input0/validation/validation_output.csv;tab=live_object?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&e=13803378&mods=-monitoring_api_staging&project=datcom-infosys-dev
cloud batch job : https://pantheon.corp.google.com/batch/jobsDetail/regions/us-west4/jobs/epa-airpollutantemission-level1-shouryasingh-20260812-090213/details?e=13803378&invt=Ab5y7Q&mods=-monitoring_api_staging&project=datcom-infosys-dev