Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Iceberg] AWS Glue Job Runner #190

Open
pdames opened this issue Aug 16, 2023 · 1 comment
Open

[Iceberg] AWS Glue Job Runner #190

pdames opened this issue Aug 16, 2023 · 1 comment
Assignees
@pdames
Labels
iceberg This issue is related to Apache Iceberg catalog support
Milestone
Compaction for Ic...

Comments

@pdames
Copy link
Member

pdames commented Aug 16, 2023 •
edited
Loading

Add a one-click AWS Glue Job Runner that exposes a simplified singular CLI command for creating, configuring, and running an AWS Glue for Ray job against an Iceberg catalog with an integrated local or PyPi build of DeltaCAT.

As part of this story, we also want to ensure that we can read Iceberg tables into both Daft Dataframes and Ray Datasets within our Glue Job.
@pdames pdames mentioned this issue Aug 16, 2023
Open
@pdames pdames added this to the Compaction for Iceberg milestone Aug 16, 2023
@pdames
Copy link
Member Author

pdames commented Aug 16, 2023 •
edited
Loading

Daft integration depends on first resolving #170 and publication of Daft 0.1.12+.

@pdames pdames added the iceberg This issue is related to Apache Iceberg catalog support label Aug 16, 2023
@pdames pdames self-assigned this Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pdames pdames

Labels
iceberg This issue is related to Apache Iceberg catalog support
Projects
None yet
Milestone
Compaction for Iceberg
Development

No branches or pull requests
1 participant
@pdames