Flink: Maintenance - TableManager + ExpireSnapshots #11144

pvary · 2024-09-16T07:39:46Z

TableManager builder implementation along with the first maintenance task to provide context.
https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA/edit#heading=h.yd2vbtnf7z6w

pvary · 2024-09-16T07:43:09Z

@stevenzwu: This PR become quite sizeable. I still think that it is better to keep it as one to provide context for some of the decisions made during the definition of the MaintenanceTaskBuilder.

If you have time we could discuss offline the review strategy, and whether to split this PR to smaller ones.

Thanks,
Peter

.../flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java

...k/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java

.../v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java

...nk/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java

...1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java

...k/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/ExpireSnapshots.java

...nk/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java

.../flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/MaintenanceTaskBuilder.java

stevenzwu · 2024-09-17T17:10:00Z

.../v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java

+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.stream;


I don't know if users would interpret stream sub package as pubic APIs. It is better to use proper Java class scope for that purpose. public classes are public and non-public classes can be package private.

...1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/AsyncDeleteFiles.java

...nk/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java

.../v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/stream/TableMaintenance.java

.../flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestAsyncDeleteFiles.java

...20/flink/src/test/java/org/apache/iceberg/flink/maintenance/stream/TestTableMaintenance.java

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java

.../flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java

.../v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestExpireSnapshots.java

...k/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestMaintenanceE2E.java

....20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java

...nk/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java

pvary · 2024-09-26T21:08:13Z

@netvl: I'm struggling to find the relevant comment (there are too many of them already), but I would like to ask you to check if the proposed MaintenanceTaskBuilder would serve your needs. You can see in the ExpireSnapshots class how it is ment to be used.

Thanks,
Peter

pvary · 2024-09-26T21:08:49Z

@rodmeneses: If you have time, I would like to ask you to review the PR.
Thanks,
Peter

....20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java

....20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java

stevenzwu · 2024-10-24T16:37:20Z

....20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java

+      TableLoader newTableLoader,
+      String defaultUidSuffix,
+      String defaultSlotSharingGroup,
+      int mainParallelism) {


this should also be defaultParallelism?

pvary · 2024-11-04T14:06:48Z

Merged to main.
Thanks for the review @stevenzwu, @rodmeneses and @netvl!

github-actions bot added the flink label Sep 16, 2024

pvary force-pushed the flow branch 3 times, most recently from a1dabe5 to 96322c5 Compare September 16, 2024 09:38

Flink: Maintenance - TableManager + ExpireSnapshots

70e0783

pvary force-pushed the flow branch from 96322c5 to 70e0783 Compare September 16, 2024 11:04

stevenzwu reviewed Sep 16, 2024

View reviewed changes

pvary force-pushed the flow branch from eb8f7be to 33307c3 Compare September 17, 2024 09:02

Steven's comments

5a1516c

pvary force-pushed the flow branch from 33307c3 to 5a1516c Compare September 17, 2024 09:09

stevenzwu reviewed Sep 17, 2024

View reviewed changes

stevenzwu reviewed Sep 18, 2024

View reviewed changes

Steven's second round of comments

ea3366e

pvary force-pushed the flow branch 5 times, most recently from fa56618 to 2403f44 Compare September 18, 2024 20:44

Small fixes

3952792

pvary force-pushed the flow branch from 2403f44 to 3952792 Compare September 18, 2024 21:14

move every publicly accessible class to the api package

fa03a71

pvary force-pushed the flow branch from a2677e1 to fa03a71 Compare September 25, 2024 12:46

stevenzwu reviewed Sep 25, 2024

View reviewed changes

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ExpireSnapshots.java Outdated Show resolved Hide resolved

stevenzwu reviewed Sep 26, 2024

View reviewed changes

Peter Vary added 2 commits September 26, 2024 14:32

Steven's comments

b6dae52

Remove Table from the Trigger

415dc90

stevenzwu reviewed Sep 26, 2024

View reviewed changes

....20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java Outdated Show resolved Hide resolved

...nk/src/main/java/org/apache/iceberg/flink/maintenance/operator/ExpireSnapshotsProcessor.java Show resolved Hide resolved

stevenzwu reviewed Sep 26, 2024

View reviewed changes

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java Outdated Show resolved Hide resolved

Steven's new comments

e3dd6f4

pvary force-pushed the flow branch from c0097a6 to e3dd6f4 Compare September 26, 2024 21:04