2024-08-19 Pyaerocom Meeting

[lewisblake]

Last meeting

• 2024-08-12 Pyaerocom Meeting #1303

Next milestone: m2024-09

PRs

• Allow trends to run even if a component does not have any data due to min_yrs constraints #1313

Other Issues

• Actris/ebas reader
• new EEA archive reader
• aerovaldb get_profiles #1311
• Implement cloud filtering on EARLINET reader #1310
• pyaerocom usage of pyaro too slow  #1302 / pyaerocom requests pyaro reader to read data twice  #1301
• Implement country filtering for 4D colocated data sets  #1295
• Arguments given in VarWebScaleAndColormap may not be going through pydantic #1291
• Optimize performance of _process_statistics_timeseries #1277
• Model maps shown for EMEP should use the frequency of the Base_*.nc file rather than the colocated data and then resampling #1260

AOB

• Too many tasks for next release: what to push back in order to meet our two most pressing issues

[lewisblake]

1. ACTRIS / EBAS reader

• @jgriesfeller will attend an ACTRIS meeting on Monday 26.08.24 in which the details of the new data set will be discussed. He hopes at this meeting to get the info he needs to implement a reader.
• The previously available information regarding component and matrix seems not be valid anymore. So at the very least we need to recreate the table which maps EBAS variable names to aerocom variable names (how they are stored in pyaerocom).
• Jan says he can work on this alone but can't start until after the meeting at which point it becomes a one-man job, which shouldn't be rushed (it will likely take several iterations with them to solve all the problem we have).
• Completing this task in September is not likely nor the priority, probably this can be completed by November at the latest.

2. EEA reader

• @dulte is writing a EEA reader with pyaro for the new parquet files. The parquet files are one file per station, species, and data set (airbase, validated, unvalidated).
• That said, we should talk about the most efficient way to do this. Does the EEA reader need to download the data? This probably won't be efficient, but could be nice to have as an extra even if not the default behavior.
• @avaldebe is submitting a PR on airbase to download the parquet files in parallel. IMO, it makes more sense to download the EEA separately and do this operationally with a newer version of airbase, then implement pyaro-reader to just read this data from a provided filepath.
• Note, some metadata in the new parquet files are subject to change (i.e., all station codes are going to change. This is relevant to filtering of "bad" stations for specific years.)
• Also with this data, the pyaro-reader will remove invalid values by default.
• UTC timestamp: timestamp is not time-zone aware, but the data should correspond to UTC+1. An inquiry about this has been made.
• API is not totally robust yet, but old service is planned to discontinued by end of the year.