Framework for tracking what data we actually have

[kyleredilla]

We need some kind of tool / method / framework for quickly understanding what data we have. We have discussed this a few times but here is the ticket finally :D This would be most useful for the raw mirrored CMIP6 data, but eventually we will want it for all derived datasets.

The essential information is to know what combinations of time periods / models/ scenarios/ variables/ frequencies/ we have data for. Perhaps a dashboard, perhaps relying on a testing framework, perhaps an audit like what the esgf_holdings.py script does but for our own filesystem, maybe integrate with Google sheets, not sure what is best!

[Joshdpaul]

An internal holdings audit is currently accomplished by running transfers/holdings_summary.ipynb and manually joining the CSV outputs into a Google sheet. A more automated way of running the audit, generating the sheet, and sharing the results would be a great feature to add.

Maybe implement via a scheduled prefect flow? Maybe host the table on ARDAC somewhere for reference?