Apache Kafka is extremely resilient and provides flexible ways to distribute, replicate and copy data to multiple locations. All Aiven for Apache Kafka plans, even the smallest startup ones, come with a 3-node distributed architecture. Data is automatically replicated to three different brokers for high availability. By using the rich capabilities of Kafka Connect and MirrorMaker 2, you can build even more resilience to your environment. In the event of an error in one broker, a copy of the data already exists in another broker.
Coupled with the fact that Kafka is typically used for only transient data storage, there is usually no need to back up the "data". Any data in a Kafka system, in the sense of the actual messages stored in Kafka topics, is fleeting and it's replicated anyway.
But what about my...
Your data is safe.
The one issue that may actually cause problems is a bit "meta". If your cluster goes down, whether due to the system power cycling or in the wake of an unplanned incident, you lose the configuration of your cluster.
This means reconfiguring your Kafka cluster and recreating the topics from scratch. This is at best annoying, at worst a reconfiguration nightmare. In any case, the turnaround time to get your Kafka cluster up and running again is longer than you’d think. And then, of course, manual work often introduces errors and misconfigurations.
This is where something that backs up your Kafka configuration would come in really handy.
And guess what: this is exactly what we’ve just added as a feature for all our Kafka customers.
What does the configuration backup do?
Kafka cluster configuration backup automatically restores a Kafka cluster configuration to the previous state after a power off/on cycle or an incident that has caused the cluster to stop working.
It backs up and restores the following information:
- Topics and related configuration parameters (such as retention time)
- Everything in Schema Registry: existing schemas, schema IDs, configurations, subjects and subject versions
- Kafka Connect configurations
(Note that consumer groups and their offsets are not backed up.)
The Kafka configuration backup feature reduces the time required for bringing your Kafka cluster back online to a fraction. The result is happy users (well, users who are less unhappy, because after all there’s just been an outage, although it didn’t last long) and happier Kafka maintenance crew!
Want more info on how the Kafka configuration backup feature works? Check out our help article.
Not using Aiven services yet? Sign up now for your free trial at https://console.aiven.io/signup!