Why multiple availability zones do not guarantee disaster recovery, how successive failures change the mathematics of resilience, and what the events of October 2026 reveal about managing critical IT infrastructure.

Alex Burlachenko

Microsoft MVP

https://ctrlaltdel.blog/

Introduction. A disaster does not always begin with the first failure

In October 2026, Yandex’s infrastructure faced an emergency that would be difficult to describe as an ordinary availability incident.

Over four days, three of the company’s data centres – in Sasovo, Kaluga and Vladimir – were damaged in succession. Two sites ceased operating entirely, while the third was partially affected.

The consequences extended beyond Yandex’s own services to external Yandex Cloud customers. Restrictions on provisioning resources, degradation of certain managed services, and problems involving databases, Kubernetes, object storage and application availability were reported.

On 23 October, the company reported that access had been restored to 90.47% of the data held on Yandex Cloud, and that a new site had been brought into operation to host some of the data and associated services.

On 29 October, Yandex confirmed that the core functions of all its services had been restored for consumers and businesses. Work to restore full functionality was, however, continuing.

The company also identified improving infrastructure resilience and moving towards a more distributed operating model as priorities.

The physical cause of the incidents is known: damage to the data centres following drone attacks.

For an IT architect, however, that is where the analysis begins rather than ends.

Physical damage explains why infrastructure stopped working. Understanding the duration of recovery requires an entirely different set of questions.

Why did losing one zone lead to restrictions on provisioning resources in the others?

Why did restoring the management components of certain services require additional intervention?

Why did redundancy within a single cloud region fail to ensure continuity for every customer application?

And, ultimately, which architectural decisions make it possible to withstand the successive loss of multiple sites without turning a technical incident into a prolonged business crisis?

Let me establish the limits of this analysis at the outset.

I do not have access to Yandex Cloud’s internal architecture diagrams, consensus logs, replica placement data, actual data-centre utilisation figures or internal root cause analysis (RCA).

Public statements can therefore establish a chronology and describe observed service behaviour. They cannot, on their own, justify categorical claims that a particular failure resulted from the loss of a Raft quorum, a PostgreSQL misconfiguration or a mistake by a particular engineering team.

Moreover, several technical symptoms occurring at the same time do not establish that they shared one underlying cause.

I will therefore distinguish between confirmed facts, technically plausible hypotheses and architectural recommendations.

That distinction is essential if the objective is engineering analysis rather than simply finding someone to blame.

Chapter 1. Four Days of Successive Failures – and Weeks of Recovery

8 October. The first availability-zone failure

During the night of 7–8 October, the data centre in Sasovo sustained serious damage and was taken completely out of operation.

The official Yandex Cloud incident history records the unavailability of zone ru-central1-b and problems affecting a number of cloud-platform services.

In a distributed infrastructure, the loss of one availability zone should not, by itself, take down an entire region – provided that this scenario falls within the architecture’s designed failure model.

This is precisely why high-availability (HA) mechanisms exist: data replication, spare compute capacity, independent network components and automatic workload failover.

But there is an important distinction here.

Preserving data, keeping existing workloads running and retaining the ability to manage infrastructure are three different objectives.

For example, Kubernetes worker nodes may continue running existing containers even when the management components are unavailable.

The cluster’s subsequent behaviour, however, depends heavily on the state of its control plane.

If the Kubernetes API and controllers become unavailable, creating new Pods, scaling, changing configurations and automatically recovering workloads after node failures may all be constrained.

A kubelet on a functioning node can still perform certain local operations, but it does not replace the scheduler or controllers. Nor does the continued execution of existing Pods guarantee application availability if networking, load balancing, DNS, storage or external dependencies have failed.

A similar situation can arise with managed databases: the survival of a replica does not guarantee automatic failover if the conditions for electing a new leader are not met or essential management dependencies are unavailable.

In such an architecture, an application may continue handling some requests while progressively losing its ability to recover from further failures.

This is the fundamental difference between being operational now and being resilient to the next incident.

9 October. Damage to a second site and the restoration of manageability

On the morning of 9 October, the data centre in Kaluga was damaged. Unlike Sasovo, it suffered a partial rather than complete disruption.

That day, Yandex Cloud announced arrangements with other cloud providers to accelerate the placement and reservation of customer resources at external sites.

The platform’s incident history also described additional measures to restore the ability to move the management components of regional Kubernetes clusters from the unavailable ru-central1-b zone to other zones.

This is a significant technical observation.

It shows that, following the first failure, certain clusters required the restoration or relocation of management components.

But it does not follow that the cloud’s entire control plane was unavailable, or that its engineers had failed to design management-plane resilience.

Either conclusion would go beyond what the available evidence supports.

The more useful question is:

Why did certain management-component recovery operations require additional intervention after the zone was lost?

Answering it would require knowledge of the actual control-plane topology, supported cluster configurations, failover mechanisms and dependencies on platform services.

10 October. Capacity management becomes part of incident response

On 10 October, Yandex Cloud announced that quotas for compute and disk resources would be managed manually.

This was one of the most revealing developments of the crisis.

Under normal conditions, cloud infrastructure is perceived as almost infinitely elastic: additional virtual machines can be provisioned, disk resources increased or new clusters deployed when required.

But elasticity exists only while real spare capacity is available to support it.

During a regional emergency, many customers try to perform the same operations simultaneously: recreate virtual machines, restore backups, increase replica counts and relocate workloads.

Demand for resources rises sharply at precisely the point when the infrastructure’s available capacity is shrinking.

The move to manual quotas shows that allocating constrained resources required centralised oversight and prioritisation.

It does not, however, prove that all data centres had previously been running at maximum utilisation to minimise capital expenditure (CAPEX).

Reaching that conclusion would require actual utilisation figures, information about reserve pools and the rules governing emergency resource allocation.

Indeed, on 29 October Yandex reported having made earlier investments in reserve capacity and equipment. The severity of the incident alone does not establish whether those investments were sufficient for this particular extreme scenario.

Nevertheless, the underlying principle is straightforward:

A redundant architecture without the capacity needed to restore it is not comprehensive resilience. It is a hope that the next failure will not arrive too soon.

11 October. The loss of a third site

During the night of 11 October, the data centre in Vladimir was damaged and ceased operating entirely.

Following the third incident, Yandex Cloud stated that the remaining resource configuration was considered unstable, that the platform was operating in emergency mode and that customers were advised to activate alternative recovery plans.

Subsequent updates described different conditions across different services.

Failover was being performed for some HA clusters of managed databases, although certain clusters could remain read-only or unavailable.

Object Storage experienced write restrictions, Compute Cloud was partially operational and Managed Service for Kubernetes operated with limitations.

Some management and auxiliary functions nevertheless remained available.

Write capability in object storage was subsequently restored, although some operations could still take longer than normal.

This uneven picture is particularly important.

It rules out a simplistic account in which the entire cloud platform lost one global quorum and stopped accepting all writes at once.

The actual situation is more complex.

Different services have different storage architectures, failure domains, recovery mechanisms and dependencies on management infrastructure.

One failure may prevent writes in a particular replica group; another may impair network access; a third may prevent additional resources from being allocated.

To the user, all three may look the same: the application is not working.

To an architect, they are fundamentally different classes of failure.

After 11 October. Why recovery took considerably longer

The following days saw cloud and application services restored in stages.

On 14 October, the company reported the restoration of some enterprise AI services. Other capabilities and products returned gradually.

On 23 October, Yandex reported that its Yandex Cloud engineers had restored access to 90.47% of the data held on the platform.

It also reported bringing a new site into service and relocating some data and associated services there.

These figures matter, but must be interpreted carefully.

Restoring access to 90.47% of data does not mean that 90.47% of compute capacity had been restored, that the same proportion of applications was fully operational, or that any universal RTO target had been met.

Nor does access to data necessarily mean that every management mechanism, original performance level or previous level of redundancy has been restored.

On 29 October, Yandex confirmed the restoration of the core functions of all its services for consumers and businesses, while acknowledging that full functionality was still being restored.

A vital distinction follows.

Restoring application availability, restoring data, restoring the original resilience of the infrastructure and completing the recovery of business processes are not the same thing.

A service may begin accepting requests again while running with fewer replicas.

A database may be available while background synchronisation is still under way.

A payment interface may be functional while certain operations still require additional verification.

Recovery time must therefore be measured against specific business functions and their required service levels, not a single moment when an application first loads in a browser.

 

 

Chapter 2. Why Multi-AZ Does Not Guarantee Disaster Recovery

1. Consensus protects consistency; it does not create physical resources

When discussing incidents of this kind, there is a temptation to explain everything through the CAP theorem.

The argument often goes something like this: the system lost its quorum, sacrificed consistency for availability and consequently suffered split-brain.

But CAP is not a universal explanation for every distributed-system failure.

The theorem addresses the limitations on consistency and availability under network partitioning. In this context, Consistency means linearizability, while Availability has a specific formal definition.

It cannot simply be used to divide all databases into ‘CP’ and ‘AP’ categories without specifying their guarantees and failure models.

A network partition must also be distinguished from the physical loss of members of a consensus group.

A majority may be lost when nodes physically fail, even if the remaining components have no network partition between them.

Consider a simplified system with three voting replicas using a consensus protocol such as Raft.

Suppose the replicas are distributed across zones A, B and D.

A quorum of two members is needed to commit decisions.

If B is lost, A and D remain. The majority is preserved, and the system can continue committing writes provided the other necessary conditions hold.

But suppose A also fails before redundancy has been restored.

Only D remains.

One replica out of three does not constitute a majority and cannot independently commit new decisions under the original configuration.

This is not a flaw in the algorithm.

On the contrary, refusing to commit without a quorum is part of the safety guarantees provided by consensus.

Nor does this scenario automatically imply split-brain. A correctly implemented consensus protocol is specifically designed to prevent conflicting committed histories.

The limits of this model must nevertheless be understood.

It describes a particular replicated group with three voting members.

It does not establish that YDB, PostgreSQL clusters or Object Storage used precisely this configuration during the October incident.

YDB has its own distributed architecture. Managed PostgreSQL clusters may combine leader-election mechanisms with different replication modes. Object storage cannot reasonably be described as one global Raft cluster either.

The model is therefore useful for understanding a class of risks, but it is not a substitute for examining the actual implementation.

2. Restoring a quorum is not simply a matter of copying a third replica

Suppose that, after losing zone B, we want to create a new replica in zone E.

On an architecture diagram, this looks straightforward:

Before the failure: A + B + D.

After the failure: A + D.

After recovery: A + D + E.

In a real distributed system, however, four distinct stages must be considered:

1. Creating the new replica instance.
2. Synchronising the required state and logs.
3. Safely changing the membership of the voting group.
4. Actually restoring the ability to survive the next anticipated failure.

While A and D retain a majority, safe reconfiguration is possible, subject to the requirements of the specific implementation.

But the mere appearance of a new replica does not immediately restore the original level of resilience.

We must verify that the new replica has received the required state, correctly applied the operation log and is ready to participate in the protocol.

Changes to voting membership must preserve the consensus protocol’s safety guarantees.

Depending on the implementation, safe membership changes may use joint consensus or other reconfiguration protocols.

An unavailable node cannot simply be removed and a new one added arbitrarily, disregarding the protocol’s requirements for quorum intersection and the history of committed entries.

The situation becomes particularly dangerous once the majority itself has been lost.

Under those conditions, ordinary reconfiguration through the previous quorum may no longer be possible.

Emergency recovery requires separate procedures to establish the admissible data history, exclude conflicting participants and prevent incompatible configurations from operating simultaneously.

These procedures are technology-specific. They may entail reduced availability and, if sufficient committed data have not survived, a risk of data loss.

The appropriate architectural objective is therefore not merely to ‘rebuild three replicas automatically within a few hours’.

It is to restore the required redundancy safely, within a defined and measured period, while preserving consistency and data-integrity guarantees.

That is the outcome which disaster-recovery exercises must test.

3. Capacity shortages can make recovery impossible

Even a perfectly implemented consensus protocol is of little help if there is nowhere to place a new replica.

Recovery requires CPU, RAM, storage capacity, IOPS and network throughput.

Consider three identical zones, each with a compute capacity of 100 notional units.

Total capacity is 300 units.

If demand is 195 units, average utilisation is 65%.

After losing one zone, 200 units of capacity remain.

On paper, the original load of 195 units can fit into the surviving zones.

But doing so brings their combined utilisation to 97.5%.

Even with perfectly flexible workload redistribution, that leaves only 2.5% spare capacity.

Such a margin provides almost no room for workload peaks, housekeeping operations, additional replicas or data restoration.

Furthermore, the feasibility of redistributing workloads depends on architectural constraints.

CPU may be available while RAM is already insufficient.

Compute capacity may remain free while storage IOPS or network links are saturated.

Anti-affinity policies, replica-placement rules or restrictions imposed by particular hardware platforms may also prevent recovery.

The rule of thumb that ‘35% of servers should be kept idle’ cannot therefore be treated as a universal architectural standard.

Reserve capacity must be calculated separately for different resource dimensions and specific failure scenarios.

Under an N+1 model, it must be demonstrated that, after losing the largest anticipated failure domain, the remaining infrastructure can sustain the required load within acceptable operating limits.

Even verified N+1 resilience, however, does not automatically imply preparedness for the successive loss of two failure domains.

For storage, the time needed to recreate data copies is especially important.

If 1 PB of data must be transferred over a link delivering an effective throughput of 100 Gbit/s, the theoretical transfer time alone is:

T = (10¹⁵ × 8) / (100 × 10⁹) = 80,000 seconds.

That is approximately 22 hours and 13 minutes, using the decimal definition of a petabyte.

The actual transfer will take longer if 100 Gbit/s denotes nominal line speed rather than effective useful throughput.

Protocol overhead, competing traffic, storage-system bottlenecks, load balancing and validation of data state all add to the duration.

Consequently, restoring the original level of redundancy may take a substantial amount of time even when spare servers exist.

In a distributed system, capacity planning is not merely a question of infrastructure cost. It is part of the resilience model.

4. Time to restore redundancy is a separate architectural metric

This brings us to a measure that is often missing from conventional SLAs and RTO discussions.

I propose assessing Time to Restore Redundancy (TTRR) separately.

By this I mean the measured interval between losing the designed level of redundancy and actually restoring it to a previously defined architectural state.

This is a practical engineering metric, not a universally standardised measure.

Its value lies in describing the period during which the system remains unusually vulnerable.

RTO defines the target time for restoring a function or service.

TTRR describes how long the system operates without its original resilience margin.

The end of TTRR should not be declared merely because a new virtual machine or container has appeared.

The recovered components must demonstrably be capable of fulfilling their architectural roles:

• participating in consensus;
• accepting and committing writes;
• serving the expected workload;
• supporting failover when the next failure occurs;
• meeting the required storage and network performance characteristics.

An application may become available again within 30 minutes while replica synchronisation continues for a further two days.

From a business user’s perspective, the service is working.

From an architectural perspective, it remains in a higher-risk state.

That distinction is crucial in scenarios involving successive failures.

A system may survive the first incident, restore most functionality and nevertheless remain vulnerable to a second.

This is why I believe we should measure not only the time taken to restore availability, but also the time taken to rebuild the system’s resilience margin.

 

 

Chapter 3. Control Plane: Infrastructure Can Keep Running but Lose the Ability to Recover

One of the most underestimated weaknesses in modern cloud architecture is the dependence of recovery on management services.

We are accustomed to discussing redundancy in compute nodes, disks and network devices.

But a cloud also has another critical layer: the control plane.

It manages resource configuration and lifecycles and interacts with identity, metadata, orchestration and automation services.

The data plane, by contrast, directly handles user operations and traffic.

This is a logical distinction, not a guarantee of physical independence.

Imagine a situation in which existing virtual machines continue running but new ones cannot be created.

Or Kubernetes applications continue executing while the management API is unavailable and controllers cannot make the changes they need to make.

This condition is particularly dangerous during successive failures.

After the first incident, the system may preserve most business functions. But the loss of effective management prevents it from rebuilding the redundancy that has been lost.

When the second failure occurs, the resilience margin may already have been exhausted.

For that reason, I consider it essential to assess the control plane as a critical dependency in its own right.

What a sound model should look like

The management architecture should provide:

• no unacceptable single points of failure;
• resilient placement of its own components;
• documented mechanisms for restoring metadata;
• independent routes for administrative access;
• secure recovery of IAM, DNS, secrets management and essential network components;
• the ability to carry out emergency operations when the primary platform is partially unavailable.

That does not mean the entire control plane should simply be duplicated in another region.

Doing so may itself introduce new dependencies and consistency risks.

Instead, it is necessary to determine which functions must remain operational locally, which can be recovered independently and how management is to be performed during a regional disaster.

Fencing is a separate concern.

When switching a leader or resource owner, the system must ensure that the previous owner can no longer perform conflicting operations.

Depending on the architecture, mechanisms may include STONITH, storage-level fencing, lease-based fencing and other approaches.

But the existence of a fencing mechanism does not, in itself, prove that it is correct.

It must be shown that exclusive ownership is maintained during network delays, loss of connectivity, partial management-component failures and the return of previously unavailable participants.

For lease-based fencing, assumptions about timing, participant behaviour and lease validation are critical. Some implementations require fencing tokens that allow a resource to reject operations from its former owner.

Revoking a TLS certificate is not a universal substitute for STONITH.

Removing a network route does not necessarily guarantee that a node has been excluded from operation either.

Failover safety must be established from the properties of the specific mechanism and its behaviour under network partitioning.

This matters particularly for automation.

Automatic failover that compromises data consistency is worse than controlled, temporary unavailability.

 

 

Chapter 4. High Availability and Disaster Recovery Serve Different Purposes

Let us now address the central architectural question.

Why does having multiple availability zones within one region not guarantee that the entire system will survive?

Because high availability and disaster recovery address different scenarios and different outcomes.

HA reduces the probability and duration of disruption caused by component failures within the designed failure model.

DR provides recovery following a disaster whose impact exceeds what ordinary HA mechanisms can handle.

The boundary between them is defined not by the name of a technology, but by the architecture, failure model and business requirements.

Synchronous replication between nearby sites may offer low latency and prevent the loss of acknowledged transactions under certain failure conditions.

But geographical proximity can also increase exposure to shared physical threats.

This does not mean that any data centres a few hundred kilometres apart automatically belong to the same failure domain.

That assessment requires examining shared dependencies: power supply, backbone networks, management systems, telecommunications providers and external threats.

Equally, separating two regions by 2,000 kilometres does not automatically make the standby region independent.

If both regions depend on the same critical IAM, DNS, routing or administrative-management components, geographical separation may not eliminate the common risk.

A two-region architecture: local HA and geo-DR

For business-critical systems, it is sensible to consider two layers.

The first layer is local high availability.

Multiple availability zones provide resilience to anticipated local failures, with replication and failover according to the guarantees of the technology concerned.

The second layer is regional recovery.

An independent region holds the standby data, compute resources, configurations and management capabilities needed to restore business functions if the primary region is lost.

Yet merely having two regions does not guarantee that the required RTO and RPO will be achieved.

Every essential dependency must be proven to work, including identity services, name resolution, encryption keys, network routing and management of standby resources.

There is a fundamental trade-off here.

Synchronous replication over long distances increases write-acknowledgement latency and may affect availability during network problems.

Asynchronous replication reduces the impact of inter-region latency on normal operations, but allows a non-zero loss of the most recently acknowledged data if the source suffers a catastrophic failure.

RPO and RTO targets should therefore not be chosen because they look attractive on an architecture diagram.

They should follow from the business impact analysis (BIA).

RTO, RPO and the minimum acceptable level of service

Recovery Point Objective (RPO) specifies the target recovery point for information and the acceptable extent of data loss, usually expressed as a time interval.

Recovery Time Objective (RTO) specifies the target period within which an activity, service or resource must be restored to a defined level after disruption.

There is another important distinction.

Meeting the RTO does not necessarily require restoring the entire infrastructure to its original performance, complete functionality and former level of redundancy.

Business continuity planning must define the minimum acceptable level of service that the organisation must sustain during disruption.

The term Minimum Business Continuity Objective (MBCO) is used for this purpose.

For example, a financial system may resume critical postings while temporarily restricting secondary reporting and analytics functions.

Such an outcome may satisfy the approved continuity strategy if the critical business requirements have been met.

Meeting the RTO does not, however, mean that every activity required to restore the platform is complete.

That is why RTO, RPO and TTRR should be assessed separately.

If a bank’s ledger cannot tolerate the loss of acknowledged postings, asynchronous replication on its own cannot guarantee RPO = 0.

An appropriate transaction-commit architecture, independent journalling or another mechanism that meets the business requirement must then be designed.

For less critical services, an acceptable RPO may be measured in seconds or minutes.

Similarly, an RTO of 30 minutes is a perfectly plausible design objective for some systems, but it is not an inherent characteristic of a remote DR region.

Its achievability must be demonstrated through regular testing.

 

 

Chapter 5. Backups: Having a Backup Does Not Mean You Can Restore It

Every major disaster raises a question that is often much more important than the choice of cloud platform:

Where is the latest usable backup, and where are the tools required to restore it?

Backup, replication and disaster failover are different mechanisms.

Replication can make an up-to-date state available quickly in a standby environment.

But it may propagate erroneous deletions, data corruption and the effects of incorrect application operations alongside legitimate changes.

Backups with a controlled version history make it possible to restore an earlier state.

A backup held inside an inaccessible region and dependent on the same control plane may, however, prove unusable at the very moment it is most needed.

This does not establish that all affected Yandex Cloud customers had organised their backups in this way.

It is nevertheless a risk that must be examined in any architecture.

Why one weekly full backup is insufficient

Suppose a full backup is taken once a week.

If the system loses data immediately before the next scheduled backup, the potential loss of changes may approach seven days, assuming that no usable intermediate backups or logs exist.

For a transactional database, that will often be unacceptable.

PostgreSQL can combine base backups with Write-Ahead Log (WAL) archiving to support point-in-time recovery (PITR).

PITR, however, depends on the preservation of the necessary continuous sequence of logs and a usable base backup.

Furthermore, continuous archiving does not automatically guarantee an RPO of a few seconds.

Actual data loss depends on transfer delays, archive state, acknowledgement mechanisms and the availability of the last required log record.

Change Data Capture (CDC) should not be treated as a direct replacement for WAL archiving. The mechanisms can complement one another, but serve different purposes.

The 3-2-1-1-0 rule

For critical systems, the practical 3-2-1-1-0 model is useful:

• at least three copies of the data;
• two different media types or storage environments, where applicable;
• one copy outside the primary site;
• one immutable or isolated copy;
• zero undetected errors following the prescribed backup and recovery checks.

The 3-2-1-1-0 approach should be understood as a practical data-protection model, not an independent guarantee of compliance with RTO, RPO or any particular standard.

It is also important not to confuse immutable storage, WORM and an air gap.

They provide different protections.

Immutable storage restricts modifications to objects according to the applicable access model and retention policies.

WORM is a write-once, read-many model offering defined immutability guarantees.

An air gap isolates the backup environment from particular access paths, but its effectiveness depends on how connectivity, management and operations are implemented in practice.

None of these mechanisms guarantees successful recovery on its own.

Independence of backup data and encryption keys

The dependence of recovery on KMS, IAM and secrets deserves particular attention.

A geographically remote backup may prove inaccessible if its decryption requires a key that has been destroyed, is unavailable or has been compromised.

Independence of the backup environment does not, however, justify uncontrolled copying of encryption keys or granting broad administrative privileges.

It is necessary to establish in advance:

• which keys and secrets are needed for recovery;
• where they are held and in what protected form;
• who is authorised to initiate emergency access;
• how essential privileges can be recovered without the primary IAM system;
• how unauthorised use of backup materials will be prevented;
• what actions will follow if keys or the management environment are compromised.

These procedures must be exercised in practice.

The most important measure of backup quality is not whether a backup job finished successfully.

It is whether the necessary business function can be restored within the permitted time and with an acceptable loss of data.

 

 

Chapter 6. Financial Systems: An Available Interface Is Not the Same as Transaction Integrity

The recovery of financial services deserves separate consideration.

Financial technology infrastructure differs from an ordinary web application in more than its availability requirements.

Its defining challenge is preserving a correct financial state in the face of partial failures, repeated requests, network delays and uncertain operation outcomes.

Consider a typical scenario.

A customer initiates a payment.

The application sends the instruction to a payment processor.

The processor completes the operation, but its response is lost because of a network timeout.

From the calling service’s perspective, the outcome is unknown.

The operation may have succeeded. It may not have been executed. Or it may have entered an intermediate state requiring further investigation.

If the application simply retries the request as a new operation, duplicate execution becomes possible.

In distributed systems, this is known as an ambiguous outcome.

The scenario is technically realistic.

We have no evidence, however, that it explains the prolonged recovery of Yandex’s financial services in October 2026.

Establishing that would require operation logs, request traces, queue-state information, reconciliation data and the findings of an internal investigation.

Idempotency must exist at the business-operation level

Idempotency is one mechanism for controlling this risk.

Each logical operation is assigned a stable, unique identifier that is reused for every retry.

UUIDv7, for example, is a suitable identifier format because of its time component and uniqueness properties.

But UUIDv7 is not deterministic by definition.

Generating a new UUID for each retry does not provide idempotency.

The identifier must be generated once, stored reliably and reused for the same logical operation.

And the identifier alone is not enough.

The system must store processing results reliably, preserve the atomicity of the required changes and prevent a retry from executing the business action again.

Storing idempotency state only in an unreliable cache may create additional risks.

There is another limitation to consider.

API-level idempotency does not guarantee exactly-once execution across an entire chain of distributed systems.

An external payment processor, a message queue and an internal ledger may have different transactional boundaries.

A sound design therefore combines idempotent operations, deduplication, durable state storage, appropriate retry handling and subsequent reconciliation.

Ledgers and financial postings

Critical financial accounting benefits from a model in which posted transactions are not altered arbitrarily; instead, corrections are recorded through new compensating entries.

An append-only ledger with appropriate integrity invariants can form part of such an architecture.

Event sourcing alone, however, does not guarantee financial correctness.

The architecture also needs atomic accounting rules, balanced-entry checks, uniqueness of business operations, reconciliation with external processors and reliable reconciliation procedures.

Technical message deduplication must also be distinguished from preserving business invariants.

The fact that a single HTTP request has not been processed twice does not establish that a complex distributed workflow cannot produce a duplicate financial posting.

Likewise, correctly balanced accounting entries do not necessarily prove that a transaction matches the customer’s original instruction.

Card authorisation, capture or debit, bank postings, clearing and final settlement must also be distinguished.

These are different stages, which may complete at different times and have their own reversal or compensation mechanisms.

Why restoring financial services can take longer

After a serious infrastructure failure, starting a virtual machine and confirming that an application responds to HTTP requests is not sufficient.

It is necessary to verify that the internal general ledger, the state of customer liabilities and the records held by external payment systems agree.

For distributed workflows, the fate of operations interrupted during execution must be established.

Some operations may have been confirmed by an external party but not recorded in the local application. Others may still be awaiting confirmation. Some may require reprocessing or compensation.

Recovery may therefore involve reconciliation, event replay, workflow-state correction and the gradual re-enablement of individual functions.

The duration of this work does not, in itself, prove database inconsistency or split-brain.

A financial system may be right to refuse certain operations precisely because preserving integrity matters more than immediate availability.

 

 

Chapter 7. The Management Perspective: Resilience Architecture Begins with BIA

Technical components do not exist independently of management decisions.

Who determines the acceptable duration of downtime?

Who approves the maximum permissible loss of data?

Who decides how much reserve capacity should be maintained?

Who approves the cost of an independent DR region?

And who verifies that the approved requirements are actually being met?

These are questions of IT governance, enterprise architecture and business continuity management.

This is where ISO 22301, ISO/IEC 27031 and NIST SP 800-34 are particularly important.

Their respective purposes must, however, be understood.

ISO 22301 sets out requirements for a business continuity management system. It supports governance processes, the allocation of responsibilities and systematic organisational preparation for disruption.

ISO/IEC 27031:2025 addresses the readiness of information and communication technology to support business continuity. It connects technical recovery capabilities to wider business continuity requirements.

NIST SP 800-34 Rev. 1 provides guidance on information-system contingency planning, including prioritisation, strategy development, plan documentation and testing.

These documents complement one another but are not interchangeable.

They do not specify a ready-made distributed-database configuration, but they help structure the definition of requirements, recovery planning and readiness assurance.

Business impact analysis instead of an arbitrary ‘four-nines’ availability target

Before investing in additional data centres, an organisation must determine the consequences of interrupting particular business functions.

For each critical process, the financial, operational, contractual and regulatory consequences of unavailability should be assessed.

The organisation should then define the maximum tolerable disruption, the required RTO and RPO, critical dependencies and the minimum level of operation during a disaster.

In this context, MBCO specifies the minimum level of products, services or activities needed to sustain business continuity.

Not every system needs the same recovery targets.

An internal analytics service may tolerate an outage lasting several hours.

Operational banking ledgers, payment infrastructure and critical control systems may have considerably stricter requirements.

But assigning every system an RTO of 30 minutes simply because the figure looks impressive in a presentation serves no useful purpose.

Its feasibility and cost must be evaluated.

Target measures must also be supported by actual measurements.

An approved RTO does not prove that the system can, in fact, be restored within that time.

That requires realistic tests with transparent success criteria.

Capacity management must address catastrophic scenarios

Capacity planning normally focuses on demand growth, performance and the efficient use of resources.

For critical infrastructure, that is not enough.

Models must account for the loss of the largest failure domain, successive failures before redundancy can be restored, simultaneous surges in resource demand and throughput constraints.

Correlated failures must also be considered.

A conventional model that treats component failures as independent may underestimate risk when several sites face the same external threat or rely on shared critical dependencies.

Calculating the failure probability of individual servers is therefore insufficient to assess the resilience of the infrastructure as a whole.

Otherwise, an organisation may find that a standby architecture exists on paper but cannot be scaled during a disaster.

Economic justification matters too.

Maintaining a fully capable standby region can be expensive.

The decision to provide one must nevertheless be based on the BIA, the likelihood and severity of relevant scenarios, residual risk and comparisons between available risk-reduction options.

Not every organisation needs to build a geographically remote data centre of its own.

For some, an independent provider, hybrid infrastructure or contracted disaster-recovery capacity may be more appropriate.

The essential requirement is that the chosen strategy meets business needs and has been tested in practice.

Can we conclude that Yandex’s management misjudged the risks?

Not conclusively, on the basis of public information.

Such a judgement would require access to the company’s internal risk model, approved continuity requirements, existing recovery plans, test results and decisions about managing residual risk.

Investments in reserve capacity do not automatically prove adequate protection against every anticipated or unanticipated scenario.

Equally, the severity of a disaster does not in itself prove a management failure.

Any assessment must consider the original threat model, applicable business requirements, the reasonableness of the decisions taken and the infrastructure’s actual constraints.

Nevertheless, the events of October raise fundamental questions for the management of any large technology organisation.

Has the possibility of losing several sites exposed to a shared regional risk been considered?

How long would it take to restore full redundancy after the first incident?

Would spare capacity be sufficient to withstand another failure?

Could critical services be recovered without the original control plane?

And what would happen if the disaster affected not only the applications but also the infrastructure needed to manage their recovery?

If an organisation cannot answer these questions with measurable test results, its readiness for a disaster remains unproven.

 

 

Chapter 8. How to Test an Architecture for Genuine Disaster Resilience

I would identify five essential areas of testing.

1. Successive failures across multiple zones

Testing the shutdown of one virtual machine or Kubernetes worker node is not enough.

The loss of an entire availability zone must be simulated, the time needed to restore the original redundancy assessed, and the consequences of a further failure examined.

Particularly valuable are tests in which a second failure occurs before recovery from the first has been completed.

These scenarios expose the difference between resilience to a single failure and resilience to successive infrastructure degradation.

The measurements should include not only failover time but also TTRR – the period during which the system lacks its designed resilience margin.

Performance limits and available capacity while operating in a degraded state should also be recorded.

2. Recovery without the primary control plane

Tests should establish whether critical resources can be deployed, access to data, configurations and secrets restored, and users redirected when the primary management environment is unavailable.

If the DR procedure depends on a functioning API belonging to the platform assumed to have been lost, those dependencies need to be reconsidered.

It is also important to verify that emergency access does not rely exclusively on an unavailable IAM service, the original DNS service or a single administrative channel.

3. Independence of recovery data

Testing should include restoration from an external environment, independent access to the necessary encryption keys, log validation and the ability to restore to the required point in time.

The scenario in which the original system no longer exists is particularly important.

It is not enough to establish that backup files can be read. The organisation must demonstrate that they can be used to restore a functioning service, including its necessary configurations, permissions and dependencies.

4. Verification of financial invariants

For transactional systems, testing should include lost acknowledgements, repeated message delivery, partial execution of distributed operations, failures of external processors and recovery from ambiguous outcomes.

The success of an API response is not the only criterion.

Tests must verify preservation of financial invariants, avoidance of unjustified duplicate execution and the ability to complete reconciliation.

They should also establish that idempotency survives all critical retry and reprocessing boundaries, rather than existing only within a single HTTP service.

5. Realistic disaster-recovery exercises

Chaos engineering and disaster-recovery testing complement one another, but are not the same thing.

Chaos engineering investigates how a system behaves when failures are deliberately introduced under controlled conditions.

DR exercises assess the organisation’s ability to restore business functions against defined objectives.

High-risk scenarios require an impact assessment in advance, a constrained blast radius, agreed criteria for terminating the test and a route back to a safe state.

Simply pulling the plug on a production data centre without these safeguards is not evidence of mature engineering practice.

The purpose of testing is to validate architectural guarantees, not to cause another incident.

Which measures should be recorded?

In my view, the results of DR exercises should include, at a minimum:

• actual recovery time for the critical business function compared with its RTO;
• the actual data-recovery point compared with the target RPO;
• whether the minimum acceptable service level was achieved;
• time to restore the required redundancy (TTRR);
• the available compute and network capacity after the failure;
• the state of critical dependencies, including the control plane, IAM, DNS and KMS;
• whether the system can withstand the next anticipated failure;
• deviations requiring corrective action.

Each measure must have a defined method of measurement in advance.

For example, the moment at which an application is regarded as recovered should not be determined solely by a successful HTTP health check.

It must be demonstrated that the relevant business function operates at the required service level and preserves the necessary data invariants.

Similarly, redundancy cannot be considered restored merely because a specified number of replicas have been started.

Their actual readiness must be verified.

 

 

Conclusion. Real Resilience Is Not Measured by Replica Count

The events of October 2026 were a severe test for Yandex’s infrastructure and its cloud customers.

The physical damage to the sites resulted from an external event of extraordinary scale.

For the wider IT industry, however, the more important lesson is this: distributed architecture must be assessed not only by whether it survives one failure, but also by whether it can restore its resilience margin before the next.

Three replicas do not guarantee survival after the successive loss of two sites.

Multiple availability zones are not equivalent to an independent DR region.

Having a backup does not mean it can be restored when the primary cloud is unavailable.

Automatic failover does not guarantee preservation of every business invariant.

And a published SLA is no substitute for verified recovery outcomes.

There is one further point worth emphasising.

Restored availability does not always mean restored resilience.

A system may once again serve users, process transactions and meet minimum business requirements while still operating with fewer replicas, limited spare capacity or incomplete data synchronisation.

It is precisely this interval between restoring operations and restoring the required redundancy that becomes critical when failures occur in succession.

Alongside RTO and RPO, we therefore need to assess the time taken to restore the resilience margin and the platform’s ability to withstand the next anticipated incident.

At the same time, it would be technically unsound to attribute every consequence of the October crisis to one consensus failure, inadequate CAPEX or a particular defect in banking microservices.

The public evidence does not support such a definitive root cause analysis.

It does, however, support a central architectural principle.

Resilience is a system’s ability to continue critical operations, or restore them within acceptable limits, after the loss of anticipated resources, dependencies and sites. That ability must be demonstrated through testing.

For a CTO or IT Director, the question is not how many standby servers appear on an architecture diagram.

The real question is this:

If the largest failure domain is lost tomorrow, which business functions will remain available, how much data might be lost, how quickly can we recover, and what happens if another failure occurs before the first recovery is complete?

If these questions cannot be answered with verified evidence, the organisation may have a design for a resilient architecture.

What it does not yet have is demonstrated disaster resilience.

Disclaimer

This article reflects my personal professional views only. It is based on publicly available information, technical documentation and general principles of designing resilient IT systems.

It is unrelated to my employer’s activities and does not represent my employer’s official position, internal processes, architectural decisions or the condition of its IT infrastructure.

In connection with this article, I will not discuss or comment on matters relating to my employer’s internal processes, IT infrastructure, architecture, business continuity arrangements, potential vulnerabilities, risks or the possible impact of similar incidents.

Any attempt to draw parallels between this analysis and my employer’s infrastructure is entirely speculative and must not be interpreted as confirmation of any facts on my part.

Sources and Technical References

Public sources on the events of October 2026

1. Yandex. ‘The Iron Heart of Yandex: What Is Happening to Our Data Centres’. 9 October 2026, updated 11 October 2026. https://yandex.ru/company/news/09-10-2026-01
2. Yandex Cloud Status. Incident history, including incidents Nos 2092 and 2136. https://status.yandex.cloud/ru/timeline ; https://status.yandex.cloud/ru/incidents/2092 ; https://status.yandex.cloud/ru/incidents/2136
3. Vedomosti. ‘Yandex Cloud Restores Some AI Services Following Outage’. 14 October 2026. https://www.vedomosti.ru/technology/news/2026/10/14/1237072-yandex-cloud
4. Yandex. ‘Yandex Restores Access to More Than 90% of Yandex Cloud Customer Data’. 23 October 2026. https://yandex.ru/company/news/23-10-2026-01
5. Yandex. Third-quarter 2026 financial results, section on significant events after the reporting date. 29 October 2026. https://yandex.ru/company/news/29-10-2026-01
6. Interfax. ‘Yandex Restores All Services Following Attacks on Data Centres’. 29 October 2026. https://www.interfax.ru/business/1125700

Technical publications and standards

7. Gilbert, S. and Lynch, N. Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. ACM SIGACT News, 2002.
8. Ongaro, D. and Ousterhout, J. In Search of an Understandable Consensus Algorithm (Raft). USENIX ATC, 2014.
9. RFC 9562. Universally Unique IDentifiers (UUIDs). IETF, 2024. https://www.rfc-editor.org/rfc/rfc9562.html
10. ISO 22301:2019. Security and resilience – Business continuity management systems – Requirements.
11. ISO/IEC 27031:2025. Cybersecurity – Information and communication technology readiness for business continuity. https://www.iso.org/standard/27031
12. NIST SP 800-34 Rev. 1. Contingency Planning Guide for Federal Information Systems.

Note: The architectural examples in this article are analytical models. They do not purport to describe the actual internal configuration of Yandex’s infrastructure. Conclusions about possible failure mechanisms are distinguished from the confirmed chronology of events. TTRR is used as a practical engineering measure proposed by the author; it is not presented as a separate normative metric under any of the standards listed above.