[Lake/Paimon] Create datalake enabled table should also create in lake #640
Conversation
luoyuxia
force-pushed
the
create-lake-table
branch
4 times, most recently
from
ad3e4a7 to
b768922
Compare
| <useTransitiveDependencies>true</useTransitiveDependencies> | ||
| <useTransitiveFiltering>true</useTransitiveFiltering> | ||
| <includes> | ||
| <include>org.apache.flink:flink-shaded-hadoop-2-uber</include> |
There was a problem hiding this comment.
Paimon requires hadoop bundled, soe we include it in paimon plugin dir
https://paimon.apache.org/docs/master/flink/quick-start/
luoyuxia
force-pushed
the
create-lake-table
branch
from
b768922 to
74248a9
Compare
|
Is it ready to review? @luoyuxia |
| } | ||
|
|
||
| // set pk | ||
| if (tableDescriptor.getSchema().getPrimaryKey().isPresent()) { |
There was a problem hiding this comment.
At first, I introduce additional offset and timestamp coumns, that's to enabled Fluss to subscribe the data in lake from a given offset and timestamp via Fluss client. But now, I feel like we can remove these two additional offset and timestamp columns at least for now.
-
Now, we mainly focus on Flink read the historical data in paimon and real-time data in Fluss. The
offsetandtimestampcolumns is not used. Introduce these columns may bring unnecassary complexity in early stage -
subscribe via
offsetandtimestampcolumns only works for log table and only works for paimon with bucket-num specified. But in paimon, it's recommend not to set bucket-num. So,offsetandtimestampcolumns become useless in most cases.
Still, we keep the possibility to support to subscribe the data in lake from a given offset and timestamp in the future. We can then introduce a option to enabled this feature for lake table.
There was a problem hiding this comment.
After discuss, still keep the ability to subscribe via offset/timestamp, so, let's introduce another column bucket to help us to subscribe via bucket + offset.
luoyuxia
force-pushed
the
create-lake-table
branch
from
74248a9 to
569d5e9
Compare
luoyuxia
force-pushed
the
create-lake-table
branch
from
569d5e9 to
8e06422
Compare
Yes, now, it's ready to review. |
|
@wuchong @leonardBang Could you please help review? |
luoyuxia
force-pushed
the
create-lake-table
branch
from
8e06422 to
85e28a2
Compare
Purpose
Linked issue: close #430
Brief change log
LakeStoragePluginSetUpthat load theLakeStoragePluginby datalake formatLakeCatalogto create table in lakeLakeCatalogTests
LakeEnabledTableCreateITCase
API and Format
Documentation