Disaster recovery for a mission-critical platform
A second data centre that can take over a mission-critical client platform, with database replication and an approval-gated failover, aiming for near-zero recovery time and data loss.
- Linux
- Docker
- Bash
Architecture
- Users → Platform (primary data centre): traffic
- Platform (primary data centre) → Database
- Database → Replica: replication
- Approval → Failover scripts: go
- Failover scripts → Platform (secondary data centre): promote
- Platform (secondary data centre) → Replica
- Users → Platform (secondary data centre): after failover
- Platform (primary data centre): Live
- Platform (secondary data centre): Standby
- Approval: A person approves
- primary data centre: Platform, Database
- secondary data centre: Platform, Replica
Context
A client runs a platform that has to stay available at all times: an outage has an immediate operational impact. Details are kept general on purpose.
Problem
Recovery time and data loss both had to be close to zero. The platform combines several technologies, each with its own way of replicating, and yet they have to fail over together.
Approach
The secondary data centre mirrors production, with the database replicated continuously between the two sites. Every part of the stack has its own replication and backup strategy, chosen for how that technology stores its state.
Failover is scripted end to end, with one deliberate manual step: a person approves the switch before it runs. That keeps the speed of automation without letting a false alarm move production.
Outcome
The aim is for the platform to move to the second data centre almost immediately, with the database already in sync.