-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CMOR fixes +> clip_timerange interactions before time statistics pre-processors are causing outputs with missing data #2018
Comments
Hi @Neah-Ko , In this recipe you have a way to deal with the issue of the last time point not being CMOR compatible: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/cmorizers/recipe_daily_era5.yml |
Hello, I decided to investigate a bit on what was causing this besides the data being shifted half an interval back in time by the fixes. Now if we look again at the points of the output file time coord:
It is not that the last point is not increasing like the others, in fact it is missing the data for that point. Because Moreover, I think that it might affect the mapping between the time coordinate and the data points, because everything is shifted so we are computing statistics on data that comes from half an interval later and pinning it on the current time. Which for low frequencies is more or less fine, but if we speak of half a month/season/year that sounds more concerning already. I have designed an experimental feature in the pull request attached, that attempts at fixing this problem. |
Describe the bug
Hello,
I am currently trying to build a dataset using
ERA5
data downloaded from the CDS.The requirements are asking for
tasmin
/tasmax
variables, which are not available in daily frequency. However, hourlytas
is available, so I am attempting to usedaily_statistics
preprocessor to build them.Here is a minimal version of the recipe I am using for CMORization:
It produces the expected resulting file. However, when attempting to load them in another recipe, this happens during the run:
I went on to investigate this, and it turns out that the time coordinate of the cube is responsible:
The very last point is not strictly monotonically increasing, and by a difference that is above the accepted margin (i.e.
0.001
) so the last loop of_check_time_coord
fromcheck.py
finds it and raises the error.Now, one normally fixes this kind of issues by tweaking the CMORizer, right ? But in the CMORization run, here is the order of preprocessors actions:
I want to point out here that the checks happens before the daily statistics, and does not go again through it. So I do not have a point where I could plug in a fix for time coordinate before it gets saved.
I thought about doing it in a script, however that means I have to load and realize the data of the cube because saving doesn't work with lazy data. Which sounds like a pretty heavy operation just to add 0.02 to the last point of the time coordinate.
Explanation
What I think is happening
The shift is exactly of
-1/2h
. This is most likely caused by the checking and fixing of hourly data that happens before the daily_statistics is called, then is carried over until saving of the data.I tried using this preprocessor instead, but it turns the time coord in a 'day_of_year' one, so not quite the expected result either.
So what I would like to know is if there is a known way to change frequency of a variable and save the resulting file in a compliant way with the tool ? Else I would like to know if it is possible to explicitly call
cmor_check_metadata
preprocessor in the recipe after daily_statistics ?Redefining it like this, causes the run to fail because it is missing
{'mip', 'cmor_table', 'frequency', 'short_name'}
arguments, but I would like those to be passed over from the last operation.The text was updated successfully, but these errors were encountered: