Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin deps to fix tests, ditch docker for uv in GHA #374

Open
wants to merge 12 commits into
base: develop
Choose a base branch
from

Conversation

omad
Copy link
Member

@omad omad commented Feb 6, 2025

I've broken my own rule and combined a few changes in the one PR, sorry!!

  • This includes the numpy 2 fixes from Resolve numpy2 compatibility issues #373 by @Ariana-B
  • And fixes deprecated usage of pytest.warn that breaks in pytest >= 8
  • And fixes a couple of other warnings in the tests.
  • And ditches gdal as a test dependency. According to the gdal docs, what we were testing for has been remedied since version 3.1 which came out 4 years ago. Overview block size now defaults to the same size as the main images instead of 128.
  • By ditching gdal from testing, I can also ditch docker in GHA, which massively simplifies the test setup. And it's faster. Saves about 1m in GHA.
  • i don't think eo-datasets needs it's own docker image. The DockerHub credentials seem to have been removed, which is lucky, because it was probably pushing GitHub tokens inside the image. So I've consolidated all the GitHub Actions workflows into one file.
  • I'm also trialling running tests in parallel using pytest-xdist, which required removing a usage of sys.argv from one of the data prep scripts, which was only there to give a warning about outdated usage from years ago. Test execution in GHA drops from about 4m to 2.5m.
  • We don't need to run pre-commit linting checks in GHA, they're being run by https://pre-commit-.ci/ . Which is now the recommended approach.

Ariana Barzinpour and others added 11 commits February 6, 2025 16:18
It was only used to test the blocksize of the overviews of COGs being
created, since they had used 128 as a default. However, since gdal 3.1
the default has changed to be the same as the blocksize as the main
image. This should mean this is no longer and issue, and we no longer
need to install gdal just to run one test.
@omad omad requested review from jeremyh and Ariana-B February 6, 2025 06:39
Copy link

codecov bot commented Feb 6, 2025

Codecov Report

Attention: Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.

Project coverage is 84.74%. Comparing base (f2117dd) to head (98dfcf4).
Report is 17 commits behind head on develop.

Files with missing lines Patch % Lines
eodatasets3/scripts/tostac.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #374      +/-   ##
===========================================
+ Coverage    84.44%   84.74%   +0.30%     
===========================================
  Files           26       26              
  Lines         4506     4530      +24     
===========================================
+ Hits          3805     3839      +34     
+ Misses         701      691      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@robbibt
Copy link
Contributor

robbibt commented Feb 6, 2025

@omad Not sure if it's useful, but re parallelization, we currently run our eo-tides uv tests in parallel directly via GitHub actions using a matrix strategy:

https://github.com/GeoscienceAustralia/eo-tides/blob/main/.github%2Fworkflows%2Fmain.yml#L40-L47

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants