All notable changes to this project will be documented in this file.
- Add English UI for DM-portal
- Add the
--moreparameter in thequery-statuscommand to show complete replication status information
- Fix the issue that
resume-taskmight fail to resume the replication task which is interrupted by the abnormal connection to the downstream TiDB server - Fix the issue that the online DDL operation cannot be properly replicated after a failed replication task is restarted because the online DDL meta information has been cleared after the DDL operation failure
- Fix the issue that
query-errormight cause the DM-worker to panic afterstart-taskgoes into error - Fix the issue that the relay log file and
relay.metacannot be correctly recovered when restarting an abnormally stopped DM-worker process beforerelay.metais successfully written
- When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together
- Add English UI for DM-portal #480
- Add the
--moreparameter in thequery-statuscommand to show complete replication status information #533 - Fix the issue that
resume-taskmight fail to resume the replication task which is interrupted by the abnormal connection to the downstream TiDB server #436 - Fix the issue that the online DDL operation cannot be properly replicated after a failed replication task is restarted because the online DDL meta information is cleared after the DDL operation failure #465
- Fix the issue that
query-errormight cause the DM-worker to panic afterstart-taskgoes into error #519 - Fix the issue that the relay log file and
relay.metacannot be correctly recovered when restarting an abnormally stopped DM-worker process beforerelay.metais successfully written #534 - Fix the issue that the
value out of rangeerror might be reported when gettingserver-idfrom the upstream #538 - Fix the issue that when Prometheus is not configured DM-Ansible prints the wrong error message that DM-master is not configured #438
- Add the command mode in dmctl
- Support replicating the
ALTER DATABASEDDL statement - Optimize the error message output
- Fix the panic-causing data race issue occurred when the full import unit pauses or exits
- Fix the issue that
stop-taskandpause-taskmight not take effect when retrying SQL operations to the downstream
- When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together
- Add the command mode in dmctl #364
- Optimize the error message output #351
- Optimize the output of the
query-statuscommand #357 - Optimize the privilege check for different task modes #374
- Support checking the duplicate quotaed route-rules or filter-rules in task config#385
- Support replicating the
ALTER DATABASEDDL statement #389 - Optimize the retry mechanism for anomalies #391
- Fix the panic issue caused by the data race when the import unit pauses or exits #353
- Fix the issue that
stop-taskandpause-taskmight not take effect when retrying SQL operations to the downstream #400 - Upgrade golang to v1.13 and upgrade the version of other dependencies #362
- Filter the error that the context is canceled when a SQL statement is being executed #382
- Fix the issue that the error occurred when performing a rolling update to DM monitor using DM-ansible causes the update to fail #408
- Generate some config items for DM-worker automatically
- Generate some config items for replication task automatically
- Simplify the output of
query-statuswithout arguments - Manage DB connections directly for downstream
- Fix some panic when starting up or executing SQL statements
- Fix abnormal sharding DDL replication on DDL execution timeout
- Fix starting task failure caused by the checking timeout or any inaccessible DM-worker
- Fix SQL execution retry for some error
- When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together
- Generate random
server-idfor DM-worker config automatically #337 - Generate
flavorfor DM-worker config automatically #328 - Generate
relay-binlog-nameandrelay-binlog-gtidfor DM-worker config automatically #318 - Generate table name list for dumping in task config from black & white table lists automatically #326
- Add concurrency items (
mydumper-thread,loader-threadandsyncer-thread) for task config #314 - Simplify the output of
query-statuswithout arguments #340 - Fix abnormal sharding DDL replication on DDL execution timeout #338
- Fix potential DM-worker panic when restoring subtask from local meta #311
- Fix DM-worker panic when committing a DML transaction failed #313
- Fix DM-worker or DM-master panic when the listening port is being used #301
- Fix retry for error code 1105 #321, #332
- Fix retry for
Duplicate entryandData too long for column#313 - Fix task check timeout when having large amounts of tables in upstream #327
- Fix starting task failure when any DM-worker is not accessible #319
- Fix potential DM-worker startup failure in GTID mode after being recovered from corrupt relay log #339
- Fix in-memory TPS count for sync unit #294
- Manage DB connections directly for downstream #325
- Improve error system by refining error information passed between components #320
- Fix a bug that may cause database connection re-establish too frequent #280
- Fix a potential panic bug when we query-status during subtask unit transforming between different units #274
- Add auto recovery framework
- Make task retryable when encounters a database driver error
- Make task retryable when encounters network issue
- Improve the DDL compatibility in DM
- Fix the bug that has a risk of data loss when the upstream database connection is abnormal
- When upgrading from a previous version, please note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together
- Retry for upstream bad connection #265
- Added some retryable errors in underlying database implementation #256
- Added task auto recovery framework #246
- Unify DB operation into one pkg for load and binlog replication #236
- Fixed a bug in task handler that may cause dm-worker panic #225
- Refactor DM error system #216
- Add strictly config verification in dm-master, dm-worker and task #212
- Limit DM error and log message length #257
- Support case insensitive binlog event filter in binlog replication #188
- Refactor DM log with pingcap/log #195
- Use
INSERTinstead ofREPLACEunder non safe-mode #199
- Remove the restriction of "the next shard DDL statement cannot be executed unless the current shard DDL operation is completely finished in shard merge scene for binlog replication" #177
- Support retry task on the
invalid connectionerror for binlog replication #66 - Support generating
-schema-create.sqlfiles automatically for full migration #186 - Use TiDB SQL Parser to parse and restore DDL statements in binlog query events, and remove the
use dbstatement when replicating to downstream #54 - Support migrating tables with generated column for full migration and binlog replication #42 #60
- Support appending the task name as a suffix to the dumped data directory (
dirin the task configuration file) for full migration #100 - Support resuming tasks automatically after the DM-worker process restarted #88 #116
- Support skipping pre-check items when starting tasks #65
- Support case-insensitive in binlog event type of Binlog Event Filter rules #188
- Optimize relay log mechanism to improve compatibility with upstream MySQL/MariaDB #92 #108 #117 #140 #171
- Optimize RPC framework to limit the concurrency between DM-master and DM-worker #157
- Redirect the log of mydumper into the log of DM-worker #93
- Support using
--print-sample-configcommand flag to show the sample config of DM-worker and DM-master #28 - Support showing the status of sharding DDL replication in the Grafana dashboard and optimize the dashboard #96 #101 #120
- Support setting the
max_allowed_packetparameter for the database connection to the upstream and downstream #99 - Support using
unsafe_cleanupin DM Ansible to clean components of DM cluster #128 - Fix flushing checkpoint wrongly when existing multiple sharding groups #124
- Fix the wrong progress status when loading dumped files #89
- Greatly improve the test to ensure correctness
- Fix many other bugs that don't affect correctness
- Support the full data migration and the incremental data migration from MySQL/MariaDB into TiDB
- Support running multi independent synchronization tasks concurrently on one DM-worker instance
- Support synchronizing a certain table of the upstream MySQL/MariaDB instance to the specified table in TiDB by table routing
- Support only synchronizing or filtering all operations of some databases or some tables by black and white lists
- Support synchronizing or filtering some of the binlog events by binlog event filter
- Support modifying auto-increment primary key fields to resolve the conflicts for shard tables by column mapping
- Support merging the original shard instances and tables into TiDB but with some restrictions
- Support synchronizing data definition changed by online DDL tools (including pt-osc and gh-ost)
- Support handling synchronization trouble caused by DDL which is not supported by TiDB