[#4545] improvement(paimon-catalog): reduce catalog-lakehouse-paimon libs size from 222MB to 75MB #4547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…aimon libs size from 222MB to 156MB
FANNG1
reviewed
Aug 16, 2024
…ateDependenciesSize
…g-binary-package-size' into apache#4545-Shrink-Paimon-catalog-binary-package-size
|
@caican00 can you please help to review this PR? |
|
|
|
@caican00 can you please help on this, to make this move fast. |
okay. |
caican00
reviewed
Sep 4, 2024
| exclude("org.apache.hive") | ||
| exclude("org.apache.hbase") | ||
| exclude("it.unimi.dsi") | ||
| exclude("org.apache.hadoop") |
There was a problem hiding this comment.
@LiuQhahah here we can not exclude all the hadoop dependencies.
There was a problem hiding this comment.
you can only exclude hadoop yarn dependencies
…imon-catalog-binary-package-size
…g-binary-package-size' into apache#4545-Shrink-Paimon-catalog-binary-package-size
|
@FANNG1 could you help re-trigger this pipeline? Thanks |
done |
@LiuQhahah could you please update the pr title for the real lib size? |
Ok The final size is 74M |
LiuQhahah
changed the title
caican00
reviewed
Sep 9, 2024
| implementation(libs.guava) | ||
| implementation(libs.hadoop2.common) { | ||
| exclude("com.github.spotbugs") | ||
| exclude("com.sun.jersey") | ||
| exclude("javax.servlet") | ||
| } | ||
| implementation(libs.hadoop2.hdfs) { |
There was a problem hiding this comment.
@LiuQhahah gravitino paimon catalog supports FilesystemCatalog as its backend catalog, and therefore we can not remove the hdfs dependency.
There was a problem hiding this comment.
and could you please do a basic test locally? such as built successfully and ran through the unit tests. thanks!
|
LGTM @jerryshao @FANNG1 do you have further comments? |
|
@LiuQhahah can you please list all the jars and paste here on Github for Paimon catalog? You can use |
|
@FANNG1 would you please help to review this? |
Hi @jerryshao |
|
Hi, @LiuQhahah , I did some extra work to exclude |
FANNG1
changed the title
Can we remove these? |
|
Why the paimon format jar is so big, is it correct? |
|
Also, why do we need hdfs jar? I assume the hdfs client jar should be enough. |
it's used by Paimon internally, we'd better to keep it.
Yes, this could be removed |
Paimon format jar packages the dependences in the jar, @caican00 do you know any other alternative jars? |
Kerberos relies on the default HDFS configuration, removing hdfs jar may cause Kerberos operation failed, cc @yuqi1129 |
2 tasks
Yeah, it will make Paimon Kerberos-related tests fail. Let's leave it as is now. |
I have tested it and we can make hdfs as testImplementation. |
The paimon catalog explicitly calls the code in the paimon-format module, so it cannot be replaced with other jars. I exclude more dependencies and ran through the integration tests. ITS:
And deployed it on physical machine for testing.
I have no permission to push code to the repo, please help update the pr. |
FANNG1
approved these changes
Sep 27, 2024
jerryshao
approved these changes
Sep 27, 2024
LiuQhahah
deleted the
#4545-Shrink-Paimon-catalog-binary-package-size
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
remove some unnecessary dependencies
Why are the changes needed?
reduce catalog-lakehouse-paimon libs
Fix: #4545
Does this PR introduce any user-facing change?
No
How was this patch tested?
CI passed