ticdc(pulsar): write DDL events to pulsar #9433

yumchina · 2023-07-26T03:29:09Z

What problem does this PR solve?

write DDL events to pulsar and some unit test
Issue Number: ref #9413

What is changed and how it works?

Check List

Tests

Unit test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

add pulsar ddl sink update deps

add test file and mock create pulsar client and producer fix Makefile for unit test with pulsar.

ti-chi-bot · 2023-07-26T03:29:11Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ti-chi-bot · 2023-07-26T03:29:18Z

Hi @yumchina. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

asddongmen · 2023-07-26T07:46:08Z

/ok-to-test

ti-chi-bot · 2023-07-26T07:46:26Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

asddongmen · 2023-07-28T02:05:04Z

/ok-to-test

cdc/sink/ddlsink/mq/ddlproducer/pulsar_ddl_producer.go

pkg/sink/pulsar/factory_test.go

cdc/sink/dmlsink/mq/manager/pulsar_manager.go

pkg/leakutil/leak_helper.go

pkg/sink/pulsar/factory.go

pkg/sink/pulsar/factory_mock.go

cdc/sink/ddlsink/factory/factory.go

cdc/sink/ddlsink/mq/ddlproducer/pulsar_ddl_mock_producer.go

remove Flush() from pulsar_ddl_producer.go

pkg/sink/pulsar/factory.go

cdc/sink/ddlsink/mq/pulsar_ddl_sink.go

sre-bot · 2023-08-02T06:23:17Z

All committers have signed the CLA.

remove code comment fix bugs by gch4236

TLS config

ti-chi-bot · 2023-08-07T06:16:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [3AceShowHand,asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-08-07T06:16:08Z

[LGTM Timeline notifier]

Timeline:

2023-08-07 06:02:22.766498819 +0000 UTC m=+683626.708847349: ☑️ agreed by asddongmen.
2023-08-07 06:16:07.265749026 +0000 UTC m=+684451.208097554: ☑️ agreed by 3AceShowHand.

asddongmen · 2023-08-07T07:36:46Z

/test all

ref pingcap#9413

yumchina added 5 commits July 21, 2023 11:28

add pulsar ddl producer

d779dc8

add pulsar ddl sink update deps

setup authentication for pulsar

bd704f4

Merge branch 'master' into dev-merge-sinkddl

70f9d7a

Merge branch 'master' into dev-merge-sinkddl

8216c23

add write ddl events to pulsar

4a9d4c3

add test file and mock create pulsar client and producer fix Makefile for unit test with pulsar.

ti-chi-bot bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. contribution This PR is from a community contributor. labels Jul 26, 2023

ti-chi-bot bot added needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 26, 2023

asddongmen requested review from 3AceShowHand and asddongmen July 26, 2023 07:06

asddongmen removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Jul 26, 2023

ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jul 26, 2023

asddongmen changed the title ~~ticdc(pulsar) write DDL events to pulsar~~ ticdc(pulsar): write DDL events to pulsar Jul 26, 2023

ti-chi-bot bot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Jul 26, 2023

fix notes

a17c8fe

asddongmen removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Jul 28, 2023

asddongmen added the component/sink Sink component. label Jul 28, 2023

asddongmen requested a review from Rustin170506 July 28, 2023 06:41

Merge branch 'master' into dev-merge-sinkddl

5fc1b79

asddongmen reviewed Jul 31, 2023

View reviewed changes

cdc/sink/ddlsink/mq/ddlproducer/pulsar_ddl_producer.go Outdated Show resolved Hide resolved

asddongmen reviewed Jul 31, 2023

View reviewed changes

cdc/sink/ddlsink/mq/ddlproducer/pulsar_ddl_producer.go Outdated Show resolved Hide resolved

asddongmen reviewed Jul 31, 2023

View reviewed changes

cdc/sink/ddlsink/mq/ddlproducer/pulsar_ddl_producer.go Outdated Show resolved Hide resolved

3AceShowHand reviewed Jul 31, 2023

View reviewed changes

pkg/sink/pulsar/factory_test.go Outdated Show resolved Hide resolved

asddongmen reviewed Aug 1, 2023

View reviewed changes

cdc/sink/dmlsink/mq/manager/pulsar_manager.go Show resolved Hide resolved

asddongmen reviewed Aug 1, 2023

View reviewed changes

pkg/leakutil/leak_helper.go Outdated Show resolved Hide resolved

asddongmen reviewed Aug 1, 2023

View reviewed changes

pkg/sink/pulsar/factory.go Outdated Show resolved Hide resolved

asddongmen reviewed Aug 1, 2023

View reviewed changes

pkg/sink/pulsar/factory_mock.go Outdated Show resolved Hide resolved

3AceShowHand reviewed Aug 1, 2023

View reviewed changes

cdc/sink/ddlsink/factory/factory.go Outdated Show resolved Hide resolved

3AceShowHand reviewed Aug 1, 2023

View reviewed changes

cdc/sink/ddlsink/mq/ddlproducer/pulsar_ddl_mock_producer.go Outdated Show resolved Hide resolved

remove factory_mock.go factory_test.go

4280181

remove Flush() from pulsar_ddl_producer.go

3AceShowHand reviewed Aug 1, 2023

View reviewed changes

pkg/sink/pulsar/factory.go Show resolved Hide resolved

3AceShowHand reviewed Aug 1, 2023

View reviewed changes

cdc/sink/ddlsink/mq/pulsar_ddl_sink.go Outdated Show resolved Hide resolved

replace log.L()

6e4d1b2

remove code comment fix bugs by gch4236

yumchina force-pushed the dev-merge-sinkddl branch from a5cf376 to 6e4d1b2 Compare August 2, 2023 09:33

yumchina added 4 commits August 2, 2023 17:40

an empty implementation of GetPartitionNum for pulsar

1f0fc09

use lru to producers cache

e3ed5a6

TLS config

format code

6d93b24

add NewAuthenticationTLS for TLS

433bc32

asddongmen approved these changes Aug 7, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Aug 7, 2023

3AceShowHand approved these changes Aug 7, 2023

View reviewed changes

ti-chi-bot bot added the lgtm label Aug 7, 2023

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 7, 2023

format documents

729a64f

Merge branch 'master' into dev-merge-sinkddl

7bf1683

ti-chi-bot bot merged commit e616696 into pingcap:master Aug 7, 2023
4 checks passed

3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this pull request Aug 29, 2023

ticdc(pulsar): write DDL events to pulsar (pingcap#9433)

a199d59

ref pingcap#9413

kennytm mentioned this pull request Feb 14, 2024

cdc/sink/dmlsink: recognize pulsar+ssl:// #10606

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ticdc(pulsar): write DDL events to pulsar #9433

ticdc(pulsar): write DDL events to pulsar #9433

yumchina commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

asddongmen commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

asddongmen commented Jul 28, 2023

sre-bot commented Aug 2, 2023 •

edited

Loading

ti-chi-bot bot commented Aug 7, 2023

ti-chi-bot bot commented Aug 7, 2023

asddongmen commented Aug 7, 2023

ticdc(pulsar): write DDL events to pulsar #9433

ticdc(pulsar): write DDL events to pulsar #9433

Conversation

yumchina commented Jul 26, 2023

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

asddongmen commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

asddongmen commented Jul 28, 2023

sre-bot commented Aug 2, 2023 • edited Loading

ti-chi-bot bot commented Aug 7, 2023

ti-chi-bot bot commented Aug 7, 2023

[LGTM Timeline notifier]

asddongmen commented Aug 7, 2023

sre-bot commented Aug 2, 2023 •

edited

Loading