Coinbase trading service lacked automated zone failover

What's new

• Coinbase's global trading service lacked automated zone failover.
• A manual process was used for zone failover, requiring engineer intervention.
• The service experienced an outage due to a single Availability Zone (AZ) failure.

Why it matters

Operating a global trading service without automated zone failover introduces significant operational risk. While manual failover can be a deliberate choice to avoid complex automation failures or to enforce human oversight, it directly correlates to longer recovery times during incidents. The incident at Coinbase, where a single AZ failure caused an outage, demonstrates the direct blast radius of this trade-off. For organizations with similar critical, low-latency services, the decision hinges on the acceptable RTO versus the complexity and potential failure modes of automated failover mechanisms. This incident suggests that even at significant scale, the operational simplicity of manual processes can introduce unacceptable downtime for core financial services.

What to watch for

Organizations should evaluate their own critical services for similar gaps. Consider the cost of manual intervention during an incident versus the engineering investment in robust, automated failover. Assess the potential for cascading failures if a manual process itself encounters issues or delays. The complexity of multi-region or multi-AZ failover often requires careful design around data consistency and service discovery, which may not be justified for all workloads.

Bottom line: Manual zone failover for critical services introduces an unacceptable recovery time objective for many organizations.