How does GitLab's High Availability (HA) architecture work, and what are the key components and failure modes in a production GitLab deployment?

Think of GitLab HA architecture like a hospital's critical infrastructure. The hospital cannot have a single entrance (load balancer), a single operating room (application server), a single pharmacy (database), or a single medical records room (repository storage). Instead, there are multiple entrances with triage (load balancers), multiple operating theaters (Rails nodes), a pharmacy with redundant inventory systems (PostgreSQL with Patroni), medical records stored in multiple fireproof vaults (Gitaly cluster), and a dispatch center (Redis Sentinel) that coordinates staff assignments even if one pager system fails. If any single component fails, the hospital continues operating while the failed component is repaired.

GitLab's reference architecture for HA deployments consists of several tiers. The external load balancer (HAProxy, NGINX, or cloud LB) distributes HTTP/HTTPS traffic across multiple GitLab Rails application nodes, which handle the web interface, API, and Git over HTTPS. An internal load balancer routes traffic between internal services. GitLab Rails nodes are stateless and horizontally scalable; you can add more nodes to handle increased traffic. Gitaly is the service responsible for all Git operations (clone, push, pull, diff). In HA mode, Gitaly Cluster uses Praefect as a proxy that replicates repository data across multiple Gitaly storage nodes. PostgreSQL stores all application data (users, projects, issues, merge requests, CI/CD configuration) and runs with Patroni for automatic leader election and failover, backed by PgBouncer for connection pooling. Redis handles caching, session storage, and Sidekiq job queues, running with Sentinel for automatic failover. Sidekiq workers process background jobs (sending emails, updating caches, processing webhooks, CI/CD pipeline scheduling). Object storage (S3, GCS, MinIO) stores large binary data: CI artifacts, LFS objects, package registry files, and container registry blobs.

Internally, Gitaly Cluster is the most complex HA component. Praefect sits between the Rails application and the Gitaly nodes, routing Git operations to the primary node and asynchronously replicating writes to secondary nodes. When the primary Gitaly node fails, Praefect promotes a secondary to primary, assuming it has the latest data. The replication model is eventually consistent: writes are acknowledged after the primary processes them, and replication to secondaries happens asynchronously (or synchronously with strong consistency mode, which waits for replication before acknowledging). PostgreSQL HA uses Patroni, which monitors the PostgreSQL primary and automates failover by promoting a replica if the primary becomes unresponsive. Consul provides distributed locking and service discovery, allowing Patroni nodes to coordinate leader election and allowing Rails nodes to discover the current database primary. PgBouncer sits in front of PostgreSQL to pool connections, preventing the database from being overwhelmed by hundreds of Rails and Sidekiq connections.

In production, a company serving 5,000 developers with GitLab would deploy the 5K reference architecture: 3 external load balancers (for LB redundancy), 5 GitLab Rails application nodes (handling web UI and API traffic), 3 Gitaly Cluster nodes with Praefect (storing all Git repositories with replication), 3 PostgreSQL nodes with Patroni (primary + 2 replicas), 3 Redis nodes with Sentinel (primary + 2 replicas), 4 Sidekiq nodes (processing background jobs with dedicated queues), and object storage via AWS S3 or GCS. The infrastructure is provisioned using Terraform and configured with Ansible or the GitLab Environment Toolkit (GET). Monitoring uses Prometheus (GitLab ships with built-in metrics exporters) and Grafana dashboards that track Rails request latency, Gitaly RPC duration, PostgreSQL replication lag, Sidekiq queue depth, and Redis memory usage. Alerting is configured for critical indicators: PostgreSQL replication lag exceeding 30 seconds, Sidekiq queue depth exceeding 10,000 jobs, Gitaly node becoming unreachable, and Rails 5xx error rate exceeding 1%.

A critical gotcha in HA deployments is the NFS dependency. Older GitLab versions used NFS for shared file storage across Rails nodes, but NFS introduces a single point of failure and performance bottleneck. The recommended approach is to eliminate NFS entirely by using Gitaly Cluster for repositories and object storage for everything else. Another common failure mode is database failover during peak load: when Patroni promotes a new primary, PgBouncer must redirect connections, and there is a brief period (typically 10-30 seconds) where writes fail. Applications and CI/CD pipelines must tolerate this transient failure. Also, Gitaly Cluster's strong consistency mode (required for data safety) adds latency to Git push operations because the write must be replicated before being acknowledged. Test the latency impact before enabling strong consistency on a high-traffic instance.