A facility I was working on took a few minutes to fail over when its primary network path disappeared.
The network engineer in me knew that result could be better.
There were technical paths available to reduce the amount of time the site spent waiting for the network to recover. Maybe we could tune the routing protocol. Maybe we could change some less obvious attributes. Maybe a different routing design would converge faster.
However, the more useful question was did it need to be better?
If the organization can tolerate a few minutes of disruption at that site, I can spend engineering time making the number smaller and still fail to create more meaningful value.
The faster failover would be technically better.
I’m not sure it would be the better architecture.
I keep coming back to this when organizations talk about five-nines availability.
Five nines sounds more precise than it is
99.999% availability allows for roughly five minutes of downtime in a year.
Sometimes that really is the requirement.
There are systems where a few additional minutes of downtime have enormous consequences. There are services sitting underneath thousands of other applications and businesses. There are environments where safety, financial exposure, regulatory obligations, production, or customer impact make extremely high availability worth pursuing.
But five nines can also become shorthand for something much less precise:
Make this really reliable.
That sounds reasonable. I want the applications and infrastructure I depend on to be available all the time too.
The problem starts when the number arrives before the problem it is supposed to solve.
If you ask an infrastructure team for 99.999% availability, they can probably build you something very expensive.
We asked for five-nines availability, so we bought redundant routers, redundant firewalls, redundant carriers and redundant power.
But… the application has a four-hour recovery objective.
What exactly did we make more reliable?
That is where five nines stops being a useful technical target by itself.
Available what?
The routers?
The WAN?
The application?
A customer transaction?
An entire facility?
And from whose perspective?
Those answers can produce very different architectures.
Reliability has an operating cost
The obvious cost of higher availability is infrastructure.
Another router costs money. So does another firewall, circuit, access point, power source, data center, cloud region or application instance.
The less obvious cost is everything required to keep that redundancy useful.
A second path has to be monitored. Redundant equipment has to be patched and upgraded. Failover needs to be tested. Failback needs to work. People need to understand both sides of the design. Business continuity and disaster recovery exercises consume time. Procedures need to survive employee turnover. Dependencies change. Applications change. Someone has to notice when the backup path quietly stopped being a backup six months ago.
The more complex the architecture becomes, the more expertise the organization has to maintain just to preserve the reliability it purchased.
This does not mean redundancy is bad.
It means redundancy is not free after the purchase order.
That matters because reliability decisions can look very different when the business consequence of failure is known.
If a warehouse can continue operating for thirty minutes while a network recovers, the architecture required there may be different from one where every minute of lost connectivity stops production.
If an internal application can be restored in four hours without meaningful business impact, building every infrastructure layer underneath it for only a few minutes of annual downtime may be hard to justify.
The technically strongest design is not automatically the one most appropriate for the organization.
Engineers can contribute to the problem
I understand how this happens because the engineering side of it is appealing.
If a network takes three minutes to converge and I know how to make it converge in thirty seconds, leaving it alone can feel uncomfortable.
There is a problem I know how to solve.
Solving it is technically satisfying.
There is also an organizational version of this.
Someone may have already established the availability target. Leadership aligned around it. Funding was approved. The project arrived with five nines written in the requirements.
At that point, it may be considerably easier for an engineer to build toward the target than to reopen the conversation about why the target exists.
Not everyone feels comfortable doing that anyway.
So we over-engineer the network.
The resulting network may be excellent.
It can still be solving the wrong reliability problem.
Part of architecture is creating enough context for engineering teams to understand why the difficult requirement exists in the first place.
If the site truly cannot tolerate several minutes of routing convergence, that changes the conversation. Now tuning the protocol, changing the topology, introducing another technology, or spending more money may be entirely appropriate.
The technical work did not change.
Our ability to explain why it matters did.
Highly available is not the same as recoverable
There is another trap in assuming redundancy solves reliability broadly.
High availability and disaster recovery are related, but they are not the same problem.
A highly available application can still need to be recovered.
A bad deployment can propagate across redundant systems. Data can become corrupted. A shared dependency can fail. A software defect can exist in every region. An operational mistake can affect both sides of the architecture.
Sometimes redundancy just introduces the failure at scale.
Geography creates similar questions.
Two facilities may be far enough apart to protect against one failure and still depend on the same regional infrastructure. Moving them farther apart might reduce one risk while introducing latency or replication concerns somewhere else.
The point is not that every architecture needs to account for every imaginable failure.
It cannot.
A disaster recovery exercise proves the scenario that was exercised. It does not prove that every possible disaster will behave the same way.
Reliability design works within those limits too.
We have to decide which failures matter enough to design around and what recovery looks like when the highly available architecture is not enough.
The same facility type may not need the same reliability
Standardization makes this more complicated.
Imagine an organization with several facilities that it considers the same type.
One operates 24 hours a day, seven days a week.
Another operates primarily during normal business hours.
A third may have a process where a network outage immediately stops production.
Architecturally, there is an argument for treating them differently.
Build the more resilient design only where the operating requirement justifies it. Spend less where the consequences of downtime are smaller.
There is also a good argument for doing exactly the opposite.
A common architecture can make the environment easier to administer. Engineers learn one design. Lifecycle activities become more consistent. Spares, standards, monitoring, automation, troubleshooting and operational procedures can all become simpler.
Now the organization is making a real architecture decision.
Do we spend more money giving some facilities reliability they may never need, or do we accept additional architectural variation because the business requirements are genuinely different?
Neither answer is obviously correct.
That is what makes it an architecture problem instead of an availability percentage.
Start with the failure
I would rather begin a reliability conversation with what happens when something stops working.
Does the facility actually need to operate 24 hours a day?
What happens to production during an outage?
Can employees continue working?
Can the application fail over along with the infrastructure underneath it?
Is there a safety consequence?
Does an outage create compliance exposure?
How much customer impact is acceptable?
How much revenue is actually at risk?
How quickly does the organization need the service restored?
Those consequences eventually influence financial decisions, even when the consequence itself is not purely financial.
Safety has a cost.
Employee downtime has a cost.
Reputation has a cost.
Delayed production has a cost.
Engineering time has a cost too.
The goal is not necessarily to turn every reliability decision into a spreadsheet and find the cheapest architecture.
It is to understand the problem well enough that the trade-offs are intentional.
That also helps avoid copying reliability architectures from organizations solving very different problems.
A CDN, ISP or cloud provider has reasons to obsess over certain kinds of availability. Their service may sit directly underneath enormous numbers of customers.
A manufacturer has reliability problems too, but they may be distributed across physical facilities, production systems and operational processes.
Healthcare and financial organizations have their own consequences and constraints.
None of those architectures has to be wrong.
They are solving different problems.
The architecture everyone is talking about online may be excellent for the organization that built it and unnecessary for yours.
Five nines can absolutely be the answer
The title of this article is intentionally a little more absolute than my actual position.
Five nines can be a requirement.
If the organization has thought through what needs to remain available, what an outage means, what failures it needs to tolerate, what the application can actually deliver, and what it is willing to spend and operate to achieve that level of reliability, then 99.999% may be exactly the right target.
What I am skeptical of is five nines as the beginning of the conversation.
These phrases are often intended to give teams direction. They create a vision. “We need five-nines availability” certainly sounds more inspiring than a discussion about routing convergence, application recovery objectives and the cost of maintaining redundant infrastructure.
The trouble begins when someone asks what achieving the vision actually requires.
Then the cost appears.
The application limitations appear.
The operational burden appears.
Different groups have different opinions about how much complexity is justified.
The organization can stagnate around a philosophical argument about reliability when it never clearly defined the problem in the first place.
Bring the actual problem forward.
What has to keep working?
What happens when it does not?
How long can that condition last?
Which failures are important enough to design around?
What are we willing to spend, operate and maintain to make those failures less likely?
Maybe the answer really is five nines.
If it is, the engineering teams now have a reason to do the hard work required to achieve it.
If it is not, that too is useful.
A reliability target should come from thinking through the problem.
It should not be a substitute for doing it.