Perforce Backup Replication: ~940 TB across four Azure regions

Context

Security Risk Management required a Perforce backup copy in a separate Azure region and dedicated subscription, within the designated data geography. A copy confined to the original account and region would not address the specified region-loss or account-compromise scenarios.

For roughly 940 TB of backups and monthly synchronization of a ~35 TB delta, the cost estimate compared managed geo-redundant storage (GRS) with existing LRS storage plus an independent copy. GRS was estimated to add about $15K/month; the chosen pipeline was estimated at about $2,200/month. GRS would have left less software to maintain, but did not provide the same subscription-level separation as the intended copy.

The decision: an independent copy instead of GRS

An Azure Functions application synchronizes each scope from a source account to a destination in another region and a dedicated subscription. The pipeline controls monthly frequency and Cool/Archive tiering.

For this workload, the estimated cost difference justified taking on more code, tests, observability, operations, and recovery work. The trade-off depends on this volume and replication frequency; GRS would have avoided some of that operational burden.

Why Durable Functions instead of a standard Azure function

A transfer cycle can take days, longer than a worker invocation. Splitting the work is not enough if a restart loses track of progress.

Durable Functions persists orchestration state while transfer activities run separately. A worker can be recycled without restarting the whole cycle.

Per-shard checkpoints then determine which portion of an interrupted transfer must run again.

How it works

The pipeline runs once a month and synchronizes changes since the previous cycle. Containers are handled differently based on their size: a container containing a few gigabytes is copied as a single job, while a container containing several dozen terabytes must be split.

The deployment uses two independent source-to-destination region pairs: an EMEA pair covering EMEA and APAC instances, and an NCSA pair dedicated to NCSA instances. Replication therefore remains within the storage geography assigned to each group.

Three design details distinguish a copy script from a system designed to bound concurrent work and resume after interruption without recopying completed shards.

The concurrency window. Starting all containers at once would load the network and workers without a bound. A sliding window capped at 96 AzCopy processes starts new work as processes finish, maintaining high throughput while limiting work in flight.

Prefix-based sharding. Restic stores data blobs under data/00 through data/ff. The pipeline reuses those 256 prefixes as parallelization boundaries rather than inventing a new partitioning scheme; other repository paths still need to be accounted for in the copy.

Granular checkpoints. With container-level checkpoints, an interruption at 90% of a 40 TB container would restart that container. Per-shard checkpoints retain completed shards and resume the remaining work; individual prefixes do not necessarily hold exactly 1/256 of the data.

Destination tier management follows three rules: active Perforce repositories are replicated to Cool, while decommissioned repositories are replicated to Archive; during the first synchronization, if the source snapshot files are more than 30 days old, the destination uses Archive immediately, otherwise it uses Cool; Azure Lifecycle Management then automatically moves any file older than 180 days to Archive.

The lock problem

Repository locks are another case to handle explicitly when resuming transfers. Restic locks its repository during certain maintenance operations, particularly prune. Copying a repository during maintenance could result in an inconsistent copy.

The selected approach requeues locked containers: when a container is locked, it is not marked as failed. Instead, it is placed back in the queue and retried for up to 72 hours. This gives a long-running prune time to complete before the container is copied in a consistent state.

The system therefore does not attempt to bypass the lock: it waits for maintenance to finish, then resumes automatically.

AI-assisted engineering workflow

I defined the initial architecture, constraints, and implementation specifications, then used Claude Code (through the company’s internal API with Anthropic models) both to challenge selected technical decisions and as a coding agent for implementation, code changes, testing, debugging, and documentation. I performed targeted code reviews, defined acceptance criteria, and validated the resulting behavior before deciding to deploy; I retained ownership of production rollout and operations.

What I built

The application is written in Python 3.13 and runs on Azure Functions Flex Consumption, using Durable Functions for orchestration and AzCopy as the transfer engine.

The pipeline uses no persistent application secret to authenticate to Azure: the runtime uses Managed Identity, while CI/CD uses OIDC with workload identity federation. The pipeline issues no SAS tokens for transfers, and a Network Security Perimeter restricts network access to the relevant Azure resources.

These identities remove the need to store, distribute, and rotate static Azure credentials for this pipeline. They do not imply that no secret exists elsewhere in the wider environment.

The infrastructure is defined in Terraform across multiple environments and deployed through a GitLab CI/CD pipeline with a controlled manual trigger: a system operating across approximately 940 TB does not deploy automatically on merge.

Validation uses a 41-test automated suite comprising unit tests that run without Azure and integration tests against Azurite, including six sharding scenarios and six chaos scenarios. The latter exercise simulated worker interruptions and the expected recovery in those scenarios; they do not establish resilience to every production failure.

Recovery

A GitLab CI/CD process handles an instance restore from replicated storage: it rehydrates Archive data when required, transfers the repository to a dedicated storage account, and then restores the Perforce instance with Restic.

Results

The pipeline maintains an independent copy of approximately 940 TB of Perforce backups across four Azure regions, organized into two source-to-destination pairs for the EMEA/APAC and NCSA scopes. Each monthly cycle synchronizes the delta, with explicit control over the storage tier and recovery process.

What I learned

This project raises a concrete operational question: what work can be resumed when a long transfer stops midway?

A worker may be recycled and a repository may be locked during a cycle. Per-shard checkpoints preserve completed work; locked repositories return to the queue within the planned retry window rather than being copied forcibly.

The case for this alternative to GRS was the estimated cost and the need for an independent copy, with an explicit cost of ownership: building and operating our own replication and recovery process.