tiflow v5.1.5

Bug fixes

• Fix data loss that occurs in special incremental scanning scenarios #5468
• Fix the issue that there are no sorter metrics #5690
• Fix excessive memory usage by optimizing the way DDL schemas are buffered #1386

tiflow v6.1.3

Bug fixes

• Fix data loss occurred in the scenario of executing DDL statements first and then pausing and resuming the changefeed #7682 @asddongmen
• Fix the issue that when collation_compatible is set to "strict", DM might generate SQL with duplicated collations #6832 @lance6716
• Fix the issue that DM tasks might stop with an Unknown placement policy error #7493 @lance6716
• Fix the issue that relay logs might be pulled from upstream again in some cases #7525 @liumengya94
• Fix the issue that data is replicated for multiple times when a new DM worker is scheduled before the existing worker exits #7658 @GMHDBJD

Improvements

• Enable transaction split and disable the safe mode of a changefeed in TiCDC by default to improve performance #7505 @asddongmen
• Improve the performance of Kafka protocol encoder #7540, #7532, #7543 @sdojjy @3AceShowHand

tiflow v6.4.0

Improvements

• Remove the useless operate-source update command from dmctl #7246 @buchuitoudegou
• Fix the issue that DM full import fails if the upstream database uses DDL statements that are incompatible with TiDB. You can create the schema of target tables in TiDB manually in advance using DDL statements supported by TiDB to ensure successful import #37984 @lance6716
• Support replicating the exchange partition DDL statements #639 @asddongmen
• Improve non-batch sending performance for the MQ sink module #7353 @hi-rustin
• Improve performance of TiCDC puller when a table has a large number of Regions #7078 #7281 @sdojjy
• Support reading historical data in the downstream TiDB by using the tidb_enable_external_ts_read variable when Syncpoint is enabled #7419 @asddongmen

Bug fixes

• Fix the issue that DM WebUI generates the wrong allow-list parameter #7096 @zoubingwu
• Fix the issue that a DM-worker has a certain probability of triggering data race when it starts or stops #6401 @liumengya94
• Fix the issue that when DM replicates an UPDATE or DELETE statement but the corresponding row data does not exist, DM silently ignores the event #6383 @GMHDBJD
• Fix the issue that the secondsBehindMaster field is not displayed after you run the query-status command #7189 @GMHDBJD
• Fix the issue that updating the checkpoint may trigger a large transaction #5010 @lance6716
• Fix the issue that in full task mode, when a task enters the sync stage and fails immediately, DM may lose upstream table schema information #7159 @lance6716
• Fix the issue that deadlock may be triggered when the consistency check is enabled #7241 @buchuitoudegou
• Fix the issue that task precheck requires the SELECT privilege for the INFORMATION_SCHEMA table #7317 @lance6716
• Fix the issue that an empty TLS configuration causes an error #7384 @liumengya94
• Fix the issue that sasl-password in the changefeed query result is not masked #7182 @dveeden
• Fix the issue that TiCDC might become unavailable when too many operations in an etcd transaction are committed #7131 @asddongmen
• Fix the issue that redo logs might be deleted incorrectly #6413 @asddongmen
• Fix performance regression when replicating wide tables in Kafka Sink V2 #7344 @hi-rustin
• Fix the issue that checkpoint ts might be advanced incorrectly #7274 @hi-rustin
• Fix the issue that too many logs are printed due to improper log level of the mounter module #7235 @hi-rustin
• Fix the issue that a TiCDC cluster might have two owners #4051 @asddongmen

tiflow v6.1.2

Improvements

• Enhance the performance of the region worker by handling resolved TS in a batch #7078 @sdojjy

Bug fixes

• Fix the issue that the cdc server might panic if it receives an HTTP request before the cdc server fully starts #6838 @asddongmen

• Fix the log flooding issue during upgrade #7235 @hi-rustin

• Fix the issue that changefeed's redo log files might be deleted by mistake #6413 @hi-rustin

• Fix the issue that TiCDC might become unavailable when too many operations in an etcd transaction are committed #7131 @hi-rustin

• Fix the issue that data inconsistency might occur when non-reentrant DDL statements in redo logs are executed twice #6927 @hicqu

• Fix the issue that upstream table structure information is lost when DM tasks enter the sync unit and are interrupted #7159 @lance6716

• Fix large transaction errors by splitting SQL statements when saving checkpoints #5010 @lance6716

• Fix the issue that DM precheck requires the SELECT privilege on INFORMATION_SCHEMA #7317 @lance6716

• Fix the issue that DM-worker triggers a deadlock error after running DM tasks with fast/full validators #7241 @buchuitoudegou

• Fix the issue that DM reports the Specified key was too long error #5315 @lance6716

• Fix the issue that latin1 data might be corrupted during replication #7028 @lance6716

tiflow v6.3.0

Improvements

• Improve compatibility with MySQL 8.0 as data source #6448 @lance6716

• Optimize DDL by executing DDL asynchronously when encounter "invalid connection" #4689 @lyzx2001

• Improve compatibility for MySQL 8.0 upstream #6506 @lance6716

• Support logging start ts of DML statements when MySQL sink gets an error #6460 @overvenus

• Enhance the api/v1/health API to return a more accurate health state of a TiCDC cluster #4757 @overvenus

• Implement MQ sink and MySQL sink in the asynchronous mode to improve the sink throughput #5928 @hicqu @hi-rustin

• Delete the deprecated pulsar sink #7087 @hi-rustin

• Improve replication performance by discarding DDL statements that are irrelevant to a changefeed #6447 @asddongmen

Bug fixes

• Fix the issue that DM reports the Specified key was too long error #5315 @lance6716

• Fix goroutine leak when relay meets an error #6193 @lance6716

• Fix the issue that when collation_compatible is set to "strict", DM might generate SQL with duplicated collations #6832 @lance6716

• Reduce the appearance of the warning message "found error when get timezone from binlog status_vars" in DM-worker log #6628 @lyzx2001

• Fix the issue that latin1 data might be corrupted during replication #7028 @lance6716

• Fix the issue that TiCDC returns an inaccurate error for a wrong PD address with a grpc service #6458 @crelax

• Fix the issue that the cdc cause cli changefeed list command does not return failed changefeeds #6334 @asddongmen

• Fix the issue that TiCDC is unavailable when changefeed initialization fails #6859 @asddongmen

tiflow v5.4.3

Improvements

• Reduce performance overhead caused by runtime context switching in multi-Region scenarios #5610

Bug fixes

• Fix the TiCDC panic issue when you set enable-old-value = false #6198
• Fix the issue that DM Worker might get stuck when getting DB Conn #3733
• Fix the issue that DM reports the Specified key was too long error #5315
• Fix the issue that latin1 data might be corrupted during replication #7028
• Fix the issue that DM fails to start when TiDB uses an IPv6 host #6249
• Fix the issue of possible data race in query-status #4811
• Fix goroutine leak when relay meets an error #6193

tiflow v5.3.4

Bug fixes

• Fix the issue that changefeed state is incorrect because a MySQL-related error is not reported to the owner in time #6698

tiflow v6.1.1

Improvements

• Users can manually set the amount of concurrency for lightning loader #5505 @buchuitoudegou
• Add a sink uri parameter transaction-atomicity to support splitting the large transaction in a changefeed. This can greatly reduce the latency and memory consumption of large transactions #5231 @CharlesCheung96
• Reduce performance overhead caused by runtime context switching in multi-Region scenarios #5610 @hicqu
• Enhance the MySQL sink to turn off the safe mode automatically #5611 @overvenus

Bug fixes

• Fix the issue that the txn-entry-size-limit configuration item does not take effect in DM #6161 @ForwardStar
• Fix the issue that the check-task command cannot handle special characters #5895 @Ehco1996
• Fix the issue of possible data race in query-status #4811 @lyzx2001
• Fix the different output format for the operate-schema command #5688 @ForwardStar
• Fix goroutine leak when relay meets an error #6193 @lance6716
• Fix the issue that DM Worker might get stuck when getting DB Conn #3733 @lance6716
• Fix the issue that DM fails to start when TiDB uses an IPv6 host #6249 @D3Hunter
• Fix the wrong maximum compatible version number #6039 @hi-rustin
• Fix a bug that may cause the cdc server to panic when it receives an HTTP request before it fully starts #5639 @asddongmen
• Fix the ddl sink panic issue when the changefeed sync-point is enabled #4934 @asddongmen
• Fix the issue that a changefeed is stuck in some scenarios when sync-point is enabled #6827 @hicqu
• Fix a bug that changefeed API does not work properly after the cdc server restarts #5837 @asddongmen
• Fix the data race issue in the black hole sink #6206 @asddongmen
• Fix the TiCDC panic issue when you set enable-old-value = false #6198 @hi-rustin
• Fix the data consistency issue when the redo log feature is enabled #6189 #6368 #6277 #6456 #6695 #6764 #6859 @asddongmen
• Fix poor redo log performance by writing redo events asynchronously #6011 @CharlesCheung96
• Fix the issue that the MySQL sink can not connect to IPv6 addresses #6135 @hi-rustin

tiflow v6.2.0

For the complete and official release notes, see https://docs.pingcap.com/tidb/v6.2/release-6.2.0.

Improvements

• Reduce performance overhead caused by runtime context switching in multi-Region scenarios #5610 @hicqu

• Optimize redo log performance, and fix meta and data inconsistency problems (#6011 @CharlesCheung96) (#5924 @zhaoxinyu) (#6277 @hicqu)

tiflow v5.4.2

Bug Fixes

• Fix data loss that occurs in special incremental scanning scenarios #5468
• Fix a bug that the redo log manager flushes logs before writing logs #5486
• Fix a bug that the redo log manager flushes logs before writing logs #5486
• Fix a bug that the redo log manager flushes logs before writing logs #5486
• Fix replication interruption that occurs when Region leader is missing and the retry exceeds the limit #5230
• Fix the bug that MySQL Sink may save a wrong checkpointTs #5107
• Fix a bug that may cause goroutine leak in the HTTP server #5303
• Fix the issue that changes in meta Region can lead to latency increase #4756 #4762
• Fix the issue that DM occupies more disk space after a task automatically resumes #5344
• Fix the issue that the uppercase table cannot be replicated when case-sensitive: true is not set #5255