Skip to main content

​
Migration

AutoMQ is a next-generation Kafka release redesigned based on cloud-native concepts, and it is compatible with Apache Kafka®. This article will detail how to migrate from Apache Kafka® or other Kafka distributions to AutoMQ.

​
Migration Tool

The AutoMQ Open Source recommends using the MirrorMaker2 tool for cluster migration, supporting transitions from Apache Kafka or other Kafka distributions to AutoMQ.
AutoMQ commercial editions provide a Kafka Linking tool, which supports Byte-to-Byte message synchronization (ensuring that message offsets remain completely consistent before and after synchronization) and enables zero-downtime migration. If your business scenario requires a smoother migration solution, you can reach out to the AutoMQ team via this form for assistance and best practices.
MirrorMaker2 migration tool has the following capabilities:
  • Supports batch migration of clusters: MirrorMaker2 facilitates batch migration of Topics and Groups from the source cluster to the target cluster, enabling phased rollout and validation.
  • Supports synchronization of consumption progress: MirrorMaker2 ensures real-time synchronization of consumption progress from the source cluster, allowing consumers to continue consuming from the target cluster after migration.
  • Supports dynamic updates of Topics and Groups: MirrorMaker2 enables real-time monitoring of dynamic updates of Topics and Groups, ensuring that no updates are missed during the migration process.
The technical architecture of MirrorMaker2 is as follows:

​
Core Concepts

MirrorMaker2 defines several Connectors to implement data synchronization tasks between two Apache Kafka® clusters, including necessary configurations and real-time running states.
  • MirrorSourceConnector: Used for synchronizing message data from the source cluster.
  • MirrorCheckpointConnector: Used to synchronize the consumption progress data of the source cluster ConsumerGroup.
  • MirrorHeartbeatConnector: Used for system detection of the synchronization link’s operational status.

​
Deploying MirrorMaker2

For the installation and deployment of MirrorMaker2, refer to the relevant links from the Apache Kafka community:

​
Constraints and Limitations

MirrorMaker2 uses Connectors to synchronize messages between the source and target clusters. This mechanism cannot guarantee complete data consistency between the source and target clusters. Therefore, when using the migration tool, please note the following constraints and limitations:
Minor Message DuplicationConsumer progress data is asynchronously and intermittently synchronized, which may cause offset rollbacks and result in duplicate consumption when switching consumers.Message synchronization is an asynchronous operation, and due to machine anomalies or fault recovery, partial data may be duplicated.Minor Message ReorderingIf offset rollbacks occur when switching consumers, a small number of messages will be reconsumed, leading to disorder in the message sequence.

​
Next Steps

After understanding the background of cluster migration, it is recommended to choose the corresponding solution based on the current cluster situation: