Perforce Backup System: replacing NetBackup across ~200 instances

Context

Approximately 200 Perforce instances, ranging from 10 GB to 120 TB each, were backed up by a commercial solution whose monthly cost had become the largest line item in the backup budget. The request had three objectives: move these backups off NetBackup, reduce costs, and do so without sacrificing restore performance or existing functionality such as snapshots, ideally improving restore times.

I was brought in to design the solution based on my experience with Azure Blob Storage and backup systems from the db-backup project. This matters later: the project did not start from scratch; it built on lessons already learned.

The decision: a POC and, above all, the right criterion

Two open-source tools met the requirements without requiring development from scratch: Restic and Kopia. Both support snapshots, encryption, deduplication, and Azure Blob Storage. On paper, Kopia is the more feature-rich option: multiple compression methods, a graphical interface, built-in retention policies, integrated hooks, and cross-host deduplication.

The POC was therefore conducted using real production data, not a test dataset.

What a backup POC should actually measure

Restore time on the POC workloads was the decisive criterion for this fleet.

The backup window mattered, but it was not enough: overnight backup time had to be weighed against the delay a slow restore would impose during an incident.

On these workloads, restore performance carried more weight than a few hours saved during the nightly backup.

Measurements from three POC scenarios:

Backup results were more nuanced: Kopia could match or exceed Restic on very large volumes with high parallelism. During restore, some Kopia tests recorded up to 126 GB of memory use. Beyond a certain point in these tests, more parallelism reduced performance.

In the same tests, setting azure.connections to 120 gave Restic good throughput with manageable resource use. Restic used slightly more CPU and RAM than Kopia in some backup scenarios; that was not a constraint for this fleet.

The paradox of feature richness

Kopia offers more configuration options. Across roughly 200 instances, additional variation would have increased the work of standardizing configurations and the risk of inconsistencies between servers.

Restic offered fewer settings to maintain, which made consistent configuration and operations easier for this fleet.

At the time of the POC, Kopia did not support Azure RBAC as required by the proposed authentication model. I opened a pull request for Kopia to add Azure AD support through Managed Identity, Service Principal, and Azure CLI tokens; it was still under review when we made the decision and could not affect the project schedule. Restic’s more established track record was another factor in reducing risk for this service.

The formal recommendation to use Restic was presented to IT and Security leaders and approved by management.

How it works

Each instance schedules and runs its own backups with a systemd timer and Restic. No single central orchestrator starts jobs across the fleet, so its failure cannot, by itself, stop backup scheduling everywhere.

Each Perforce instance has its own encrypted Restic repository in a dedicated Azure container. This isolation limits the impact of corruption or compromise within one repository, though it cannot remove risks shared by the storage platform or identities. Azure Arc Managed Identity is used to authenticate to Vault without a persistent Azure credential on the servers; Chef generates the systemd timers from declared configuration.

This mechanism warrants more detail because it enables the solution to scale across 200 instances. A single Chef recipe manages the entire fleet: it installs version-pinned Restic and Resticprofile binaries, then reads a data bag for each instance containing its specific settings (paths to back up, scheduling window, and resources) and merges them with defaults defined by the Chef recipe. The recipe uses this information to generate the instance’s Resticprofile configuration and systemd units. Adding an instance or changing its backup window becomes a data bag change rather than a manual intervention on a server.

The recipe also converges toward the declared state: it rewrites a systemd unit only when its contents change, avoiding restarts on every Chef run. Each run compares the declared instances with those on the host and disables orphaned jobs from decommissioned instances.

AI-assisted engineering workflow

I defined the initial architecture, constraints, and implementation specifications, then used Claude Code (through the company’s internal API with Anthropic models) both to challenge selected technical decisions and as a coding agent for implementation, code changes, testing, debugging, and documentation. I performed targeted code reviews, defined acceptance criteria, and validated the resulting behavior before deciding to deploy; I retained ownership of production rollout and operations.

What I built

Four components address observability, validation, identity, and configuration: three custom Python tools and a generated Resticprofile configuration.

The consecutive-failure counter accommodates different schedules: some instances back up daily, others three times a week. A “no backup in the last 24 hours” alert can fire on a normal gap for the latter and react too late for the former. Counting successive failures detects missed runs without imposing the same time window on the whole fleet.

The Vault integration script deserves separate mention because it is the only one of the three that handles sensitive data. The password is emitted only to standard output and is never written to a file or log. Every input, including Azure identifiers and Vault paths, is validated before being passed to a subprocess to prevent command injection. SSL verification cannot be disabled in the production environment, even through an explicit flag.

Observability also includes 4 Grafana dashboards, 19 Prometheus alerting rules backed by recording rules, and highest-severity Azure Monitor alerts integrated with ServiceNow for security events.

Results

The service was deployed to production and then stabilized across the entire fleet, reducing the monthly cost of Perforce backups by approximately 87%, roughly $1M/year.

The project concluded with an operational handover to the Ops team: technical and architectural documentation, operations procedures, a runbook for each alert, an illustrated restore guide, and video training covering maintenance and incident diagnosis.

Key takeaways

The decision criterion matters more than the feature count. In our POC, testing restores on critical scenarios led us to choose Restic despite Kopia’s additional options. That finding applies to our workloads and constraints, not to every comparison of the two tools.

Standardization is an operational choice. Across this fleet, fewer configuration variations reduced the work of keeping instances consistent.

A technical trade-off must be defensible to non-specialists. Presenting the results to IT and Security leaders required translating benchmarks into risk and cost. That translation, not the measurements themselves, ultimately drove the decision.