You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We'll probably just want to start by dumping raw attestation data into a data lake
Describe the solution you'd like
A crawler similar to the open_collective.py example that fetches attestations from the EAS graphql endpoint and dumps that standard data plus a JSON with the decoded schema data into BigQuery.
@ccerv1 Done! I made several adjustments to the PR, but the core contribution was definitely useful. I’ve tested it both locally and on BigQuery, and everything is working well.
On the flip side, I found yet another concurrency bug with Dagster. Apparently, each define_asset_job call can only have one partitioned run, so Open Collective and the new EAS assets clash. I'll create an issue to track and fix it.
Describe the feature you'd like to request
We'll probably just want to start by dumping raw attestation data into a data lake
Describe the solution you'd like
A crawler similar to the open_collective.py example that fetches attestations from the EAS graphql endpoint and dumps that standard data plus a JSON with the decoded schema data into BigQuery.
Here's an example: https://gist.github.com/ccerv1/12a83e4a698a200a48ac17b97b049241
Describe alternatives you've considered
Ask the EAS team if we can get a mirror of their database
The text was updated successfully, but these errors were encountered: