How do you design a disaster recovery strategy for Bitbucket Data Center, including backup, restoration, and failover procedures?

Think of disaster recovery like insurance for a building. You hope you never need it, but when a fire (disk failure), flood (data corruption), or earthquake (complete data center loss) strikes, the insurance policy (DR strategy) determines whether you rebuild in days or months. The policy must be regularly reviewed (tested), the premiums must be paid (backup infrastructure maintained), and the coverage must match the value of the building (RPO/RTO aligned with business impact).

Bitbucket Data Center's disaster recovery strategy must cover three data stores. The relational database (PostgreSQL or Oracle) stores all metadata: repositories, users, permissions, pull requests, comments, and settings. The shared filesystem (NFS, EFS, or clustered storage) stores the actual Git repository data (bare repositories with all branches, tags, and objects). The application home directory stores configuration files, plugins, and log files. All three must be backed up consistently and restored together; restoring a database without the matching filesystem (or vice versa) results in orphaned references and data inconsistency.

The backup strategy uses coordinated snapshots. Bitbucket Data Center provides a backup-client tool (bitbucket-backup-client.jar) that creates consistent backups by coordinating database dumps and filesystem snapshots. The tool puts Bitbucket into a maintenance mode briefly, takes a database dump, snapshots the shared filesystem, and exits maintenance mode. For large installations, filesystem-level snapshots (ZFS snapshots, EBS snapshots, NetApp snapshots) are preferred over file-copy backups because they complete in seconds regardless of repository size. Database backups use pg_dump for PostgreSQL, producing a consistent dump that can be restored independently. Backup frequency determines the RPO: hourly backups give an RPO of one hour (maximum one hour of data loss).

In production, the DR architecture typically involves a warm standby environment in a separate data center or cloud region. The standby runs the same Bitbucket version on identical infrastructure but remains offline or in read-only mode during normal operation. Backups are replicated to the standby's storage continuously (using rsync, storage-level replication, or cloud cross-region replication). The failover procedure involves: restoring the latest database backup on the standby, mounting the replicated filesystem, updating DNS to point to the standby, and starting Bitbucket. The RTO depends on restoration speed: a 500 GB filesystem restored from snapshots takes minutes, while restoring from file-level backups can take hours. Organizations with strict RTO requirements use storage-level replication (synchronous or near-synchronous) to minimize restoration time.

A critical gotcha: smart mirrors are NOT a disaster recovery solution. Mirrors are read-only replicas that cannot be promoted to primary. If the primary cluster fails, mirrors continue serving cached read requests temporarily but cannot process pushes, pull requests, or any write operations. The mirror's data is also not guaranteed to be consistent with the primary at the exact moment of failure (eventual consistency). Organizations that treat mirrors as DR replicas discover this limitation during an actual disaster, which is the worst time to learn. Another trap is backup verification: teams that never test restoration discover corrupted backups, incompatible database versions, or missing filesystem permissions during an actual disaster. Regular DR drills (at least quarterly) where the team restores from backup to the standby environment and validates functionality are essential for a credible DR strategy.