db-backup: Redis & MySQL backup platform

Context

Backing up a database is a solved problem. Backing up hundreds of databases, running on two different engines, distributed across hundreds of servers, with different retention policies and a requirement to prove that restores work: that is an architecture problem.

Before db-backup, each team used rsync to send backups to shared NAS devices. Each NAS stored backups for multiple databases using the same engine (Redis or MySQL), and the NAS devices were then backed up with NetBackup. The system worked, but used approximately 400 TB of storage across the NAS devices. Storage was only part of the problem. It was also difficult to answer a simple question: Was Team X’s database backed up last night, and do we know how to restore it? When the answer depends on who is available, it is no longer a backup; it is a hope.

The decision: an API for shared service state

A natural response to this problem is to write scripts and deploy them everywhere. But a collection of distributed scripts does not, by itself, provide a centralized, consistent, queryable view of backups.

To see which backups exist, which failed, how much storage they consume, and whether retention ran, the teams needed one service for control, status, and operations.

The platform’s defining architectural decision was to make it a service rather than a simple tool. A REST API provided the single entry point; playbooks, the CLI, rotation, and cleanup all relied on the same contract.

Why an API-first design simplified extension

This point is worth making explicit because it applies well beyond backups: when a system exposes a stable API before its user interfaces, each additional interface can reuse the same orchestration contract. The Go CLI did not require the orchestration to be redesigned; it consumes the same API as the dashboard. The service started with Redis. Once it was stable, MySQL support reused the same system, with a new playbook handling engine-specific operations.

How it works

The architecture separates interfaces, orchestration, execution, and storage with observability. This separation made it possible to move storage from OpenStack to Azure and later the cluster to AKS by adapting the affected components, rather than rewriting the whole platform.

A nightly backup runs as follows:

The sidecar pattern is worth explaining. A failed Kubernetes job exposes status, events, logs, and an exit code, but those signals do not provide workflow-specific notifications on their own. The sidecar is a second container in the same pod whose sole purpose is to observe the first container and report what is happening as it happens. This turns “last night’s backup failed” into “the backup failed during the upload step at 3:12 a.m. with this error.”

The platform I helped evolve

I joined db-backup after the project had already been created and while it was still under development. I contributed to building and deploying its components to production, then assumed primary responsibility for the service following its production rollout.

The main components are best understood by their responsibilities in the service, rather than as an inventory:

Two decisions are worth highlighting because they are not visible in a list of technologies.

The Prometheus exporters are custom-built. System-level metrics alone did not answer the question that mattered: not “is the process running?” but “is this database’s latest valid backup less than N hours old?” Useful metrics describe the state of the backups, not just the state of the process.

The playbooks are tested against real machines, not simulated ones. A dedicated GitLab CI/CD pipeline, triggered automatically every day and every week, uses Terraform to provision actual MySQL and Redis instances, restores an existing backup to them using the db-backup CLI, verifies that the restore genuinely succeeded, then runs another backup from the restored instance before destroying the infrastructure. The daily cycle runs against a test database; the weekly cycle repeats the same scenario using a production backup, providing assurance closer to real-world conditions without incurring that cost every day. These real-machine tests also exercise target-machine state that a simulated playbook cannot validate.

Results

The service launched on OpenStack using Swift storage and eliminated the NetBackup and NAS infrastructure it replaced, generating cost savings of more than US$200,000 per year. Several years later, after moving into an SRE role, I independently led the migration of storage to Azure Blob Storage and then the cluster to AKS, automating the process through a GitLab pipeline.

Over the lifetime of the service, I resolved more than 300 operational incidents tied to the platform (backup failures, credential rotation failures, storage capacity alerts, configuration errors) across the NCSA, EMEA, and APAC regions. That volume of incidents is also what made it possible to continuously improve the platform’s reliability and sharpen its observability.

But the result I am most satisfied with cannot be quantified: the service continued to operate after I left the team. The operational handover included technical documentation, runbooks, operations procedures, diagnostic guides, and video courses covering routine operations and incident debugging. The documentation and training enabled the receiving team to operate the service independently.

Key takeaways

A backup system is not judged by its ability to create backups. It is judged on three things, in this order: its ability to restore, its ability to report whether it is healthy, and its ability to be operated by someone other than its author.

The first point is obvious yet routinely overlooked: a backup that has never been restored is an assumption, not a guarantee. The second explains why observability was not a layer added at the end. The third can only be assessed over time: it shows whether you built an operable service or a dependency on its original author.