Scipy 2024 - Ibis testing talk

[cpcloud]

I'd like to submit a talk similar to the one I submitted for #14: testing all the things and how.

xref: #11

[cpcloud]

Title: Keep calm and test 20 databases on every commit

Abstract:

Ibis is a cross-backend DataFrame API for Python, heavily inspired by many things including
R, SQL, pandas and others. The cross backend nature of Ibis presents a bit of testing
pickle: how on earth can we reliably test 20 analytic query engines while maintaining our
sanity? Can we do this on every commit? Yes we can! Can we also avoid aging ten years in a month
at the same time? Probably!

Description:

In this talk I'll delve into the guts of how we test Ibis across 20 backends on every commit and pull request and techniques for dealing with CI versus local environments.

I'll cover docker, docker compose, a few of the services we use, GitHub Actions, CI versus local development, pytest, even more pytest, and there might even be a bit about pytest in there.

Additionally, I'll cover the limitations to our current approach and how we might address them, or just accept them because all software engineering has tradeoffs.

[ncclementi]

@cpcloud quick question, did you submitted this to the Maintainers and Community track?

The reason I ask, is because when I went to the Scipy proposal OH, they mentioned that this track has strong emphasis on the community aspect, and proposals for this track, should aim to include a community aspect.

For example, in this case, I'll add something about how this approach helps with "code sustainability" (this are very common buzz words for this track), "developer onboarding", and minimizing "maintainers burnout".

If that was the chosen track, the proposal as is, is most likely to get rejected.

You'll also would want to include some of the things mentioned in the guidelines page "Tips for Submitting a Proposal" part,

• In your abstract, be sure to include answers to some basic questions: Who is the intended audience for your talk? What, specifically, will attendees learn from your talk?

Also, you might want to add links to the repo, ibis.org and a blog about it if we have it.

[cpcloud]

Thanks, I will look into it!

[cpcloud]

@ncclementi I submitted it for the Playing Nice: Scientific Computing Across Programming Languages track.

[cpcloud]

New abstract:

Ibis is a cross-backend, beautiful DataFrame API for Python, heavily inspired by many things including R, SQL, pandas and others. The cross-backend nature of Ibis presents a bit of a
pickle: how on earth can we reliably test 20+ analytic query engines while maintaining our
sanity? Can we do this on every commit? Yes we can! Can we also avoid aging ten years in a month at the same time? Probably!

If you're working with databases, CI, complex test setups or anything in between, this talk is for you!

We'll cover many things, but the main takeaway from this talk will be various techniques and pieces of kit that can make complex testing setups manageable.

[ncclementi]

Closing this one as it was rejected. :(