Announcing Aiven for Apache Kafka cluster configuration backup

Is there a need for backing up your Apache Kafka cluster? Well, yes and no. Find out what we mean by that, and how our backups work.

19 September 2021
Auri Poso
Auri Poso RSS Feed
Technical Copywriter at Aiven

Apache Kafka is extremely resilient and provides flexible ways to distribute, replicate and copy data to multiple locations. All Aiven for Apache Kafka plans, even the smallest startup ones, come with a 3-node distributed architecture. Data is automatically replicated to three different brokers for high availability. By using the rich capabilities of Kafka Connect and MirrorMaker 2, you can build even more resilience to your environment. In the event of an error in one broker, a copy of the data already exists in another broker.

Coupled with the fact that Kafka is typically used for only transient data storage, there is usually no need to back up the "data". Any data in a Kafka system, in the sense of the actual messages stored in Kafka topics, is fleeting and it's replicated anyway.

But what about my...

Your data is safe.

The one issue that may actually cause problems is a bit "meta". If your cluster goes down, whether due to the system power cycling or in the wake of an unplanned incident, you lose the configuration of your cluster.

This means reconfiguring your Kafka cluster and recreating the topics from scratch. This is at best annoying, at worst a reconfiguration nightmare. In any case, the turnaround time to get your Kafka cluster up and running again is longer than you’d think. And then, of course, manual work often introduces errors and misconfigurations.

This is where something that backs up your Kafka configuration would come in really handy.

And guess what: this is exactly what we’ve just added as a feature for all our Kafka customers.

What does the configuration backup do?

Kafka cluster configuration backup automatically restores a Kafka cluster configuration to the previous state after a power off/on cycle or an incident that has caused the cluster to stop working.

It backs up and restores the following information:

  • Topics and related configuration parameters (such as retention time)
  • Everything in Schema Registry: existing schemas, schema IDs, configurations, subjects and subject versions
  • Kafka Connect configurations

(Note that consumer groups and their offsets are not backed up.)

Wrapping up

The Kafka configuration backup feature reduces the time required for bringing your Kafka cluster back online to a fraction. The result is happy users (well, users who are less unhappy, because after all there’s just been an outage, although it didn’t last long) and happier Kafka maintenance crew!

Want more info on how the Kafka configuration backup feature works? Check out our help article.

Not using Aiven services yet? Sign up now for your free trial at https://console.aiven.io/signup!

In the meantime, make sure you follow our changelog and blog RSS feeds or our LinkedIn and Twitter accounts to stay up-to-date with product and feature-related news.

Start your free 30 day trial

Test the whole platform for 30 days with no ifs, ands, or buts.

Aiven logo

Let‘s connect

Apache Kafka, Apache Kafka Connect, Apache Kafka MirrorMaker 2, M3, M3 Aggregator, Apache Cassandra, Elasticsearch, PostgreSQL, MySQL, Redis, InfluxDB, Grafana are trademarks and property of their respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.