I keep thinking about how much of the work entering operations could have been addressed before it arrived there.
It’s natural to try to make a steady stream of alarms, tickets, escalations, and repeated investigations easier to manage. Better event correlation, cleaner dashboards, automated routing, more useful runbooks, and AI-assisted triage can all help.
I’ve been on teams responsible for that work, and I understand the value of having people who can see across systems, coordinate incidents, and investigate problems when nobody is quite sure where they belong.
And while not every event can be prevented, equipment fails after all, I still think there’s a difference between improving how efficiently we process operational work and examining why we’re producing so much of it in the first place.
The problem can exist before the alert
Consider a facility with two WAN paths. The primary path is working, and the applications are available. From a customer perspective, there may be nothing wrong.
Now suppose the route that should make the secondary path usable is missing. The facility is operating, but it isn’t operating in the resiliency state the organization intended.
If the primary path fails, monitoring will detect a genuine problem. The operations team may troubleshoot connectivity, bring engineers into the incident, and eventually discover the missing route. Event correlation might even help relate the resulting alarms to a common failure.
That would be useful work. It just wouldn’t be the earliest opportunity to find the problem.
If the organization has defined where particular routes should be advertised, which paths must remain available, and what constitutes an acceptable degraded state, it has something to check before the primary path fails. It can compare those expectations with routing policy and operational state, potentially model the behavior under a failure, and correct discrepancies while the site is still functioning.
That’s the part of compliance I find interesting. Not checking a box for an audit, but continuously asking whether the environment still conforms to the engineering decisions we’ve made.
The comparison might involve software versions, access controls, network paths, monitoring coverage, or architectural relationships. Configuration collection can answer some questions; routing and forwarding models may be needed for others. Either way, the evidence has to be trustworthy and current.
None of those checks guarantees availability. They create an opportunity to correct a known defect before it becomes a more disruptive one.
Compliance with what?
The word compliance comes with baggage. It often suggests regulatory obligations, security findings, or reports prepared for an audit.
NIST’s guidance on continuous security monitoring, for example, treats ongoing visibility into assets and control effectiveness as information for risk decisions, not merely a periodic checklist. That doesn’t turn every architecture standard into a security control, but the underlying discipline is familiar.
I’m thinking more broadly about conformance to intended state.
We need to know what the organization decided should exist, what is actually running, and whether the differences matter. That distinction is part of why I’ve become cautious about calling one discovered inventory a single source of truth. A platform can accurately describe the environment and still describe an environment we don’t want.
The reverse is also possible. The intended design may be outdated or simply wrong.
Imagine that an engineer makes an emergency routing change to restore a service. The change works, but it violates the approved network pattern. A continuous check reports the deviation.
I would consider that unresolved. I wouldn’t necessarily consider reverting the change the right answer.
The team might discover that the original design no longer fits the environment. Updating the architecture may be the appropriate correction. Or the exception might need to remain until a larger change is funded.
Either way, there should be a decision. Automatically treating discovered state as approved state can institutionalize technical debt. Automatically forcing the old design back into production can create a different kind of risk.
Reconciliation isn’t always an automated correction. Sometimes it’s an architectural review, an accepted and time-bounded exception, or a request for investment that has finally been supported by evidence.
Not every finding is an incident
There’s an obvious problem with continuous checking: it can generate another queue of things for people to investigate.
Suppose a conformance process finds dozens of deviations. Some expose meaningful security risk. Some affect failover. Some are harmless inconsistencies. Others may be known exceptions waiting on a replacement project.
If we turn all of them into urgent operational alerts, we’ve probably recreated the fatigue we were trying to avoid.
The purpose of identifying a deviation shouldn’t be to make another dashboard red. It should be to make the condition understandable and give someone a reasonable way to act on it.
It may be an engineering task, a planned remediation, an architectural decision, or an exception with an owner and review date. It could also warrant an incident. What matters is whether the work can be evaluated and resolved.
Priority matters. A discrepancy affecting a safety boundary, an exposed management interface, or a critical failover path deserves different treatment from a naming inconsistency. Effort, likelihood, business consequence, and what the organization has already accepted all influence the decision.
I’m not suggesting that a deviation should remain open indefinitely just because the business hasn’t noticed it. A system can be available while being less secure, less recoverable, or more difficult to operate than intended.
But I also wouldn’t measure success by how many compliance findings we generate or how quickly we turn them into tickets. The better test is whether we understand and reduce the consequential gaps over time.
This is where a mature software team might reasonably say that I’m describing testing and continuous reconciliation. In many ways, I am. Unit and integration tests, deployment gates, canary releases, and reconciliation loops exist for good reasons.
Infrastructure can apply similar thinking, but the distinction I’m interested in is what happens outside a particular deployment. The environment continues to change through failures, emergency work, manual intervention, lifecycle events, and dependencies beyond the last change window. We need ways to evaluate the installed system even when nobody is actively deploying something.
I’ve also written before about validating network changes. Here, the larger question is what the results of ongoing validation should change about the work we send downstream.
The attraction of fixing the queue
When an operations team says it needs less noise, it’s tempting to begin with the place where the noise is most visible.
Consolidate the telemetry. Correlate related events. Suppress duplicates. Route tickets more intelligently. Put an AI system around the stream and ask it to identify what matters.
There are situations where I would support every one of those investments.
At enterprise scale, multiple systems can report legitimate effects of the same failure. Someone needs to connect the infrastructure symptom, the application behavior, and the business consequence. A centralized operations function may be better positioned than any individual service team to see across those boundaries.
The trouble starts when the correlation platform becomes the preferred answer to problems that should have been corrected where they originated.
An application that emits several redundant alerts for one condition might need better instrumentation. An inventory that omits deployed infrastructure might need a more reliable lifecycle process. A recurring configuration violation might need a standard, a usable deployment path, and a check that tells us when the standard stops holding.
A correlation layer can make those problems easier to live with. It may not make them less common.
I see a parallel with network segmentation. We can spend a great deal of time choosing and deploying firewalls without first deciding which systems should communicate. The firewall still has value. It just becomes the place where the missing policy work shows up, often as a growing collection of exceptions.
Operational intelligence can inherit the same problem. If we haven’t done the engineering work needed to explain what good looks like, we’re asking the downstream system to infer the answer from everything that happens to be observable.
That seems like an expensive way to discover intent.
Some work belongs in operations
Better engineering won’t eliminate operations, and not every failure is evidence of an upstream mistake.
A redundant router can fail even when the design and configuration are correct. I would be comfortable treating that as an incident requiring repair, while recognizing that successful failover may mean there was little or no customer impact. The priority and response should reflect both the failed component and the remaining resilience.
Security investigations are even less suited to a claim that conformance checking can prevent everything. An attacker doesn’t have to wait for our standards to drift, and detecting suspicious behavior is valuable even when the environment appears compliant.
There will also be unfamiliar failures and combinations of events that nobody could reasonably have encoded as an expected-state check. During those moments, centralized visibility, experienced operators, incident coordination, and correlation are not wrappers around bad engineering. They’re necessary capabilities.
Google’s SRE guidance on monitoring distinguishes conditions that warrant urgent human interruption from information better handled through other channels. Its work on eliminating toil also makes a useful case for changing the system that generates repeated manual work, rather than only making that work easier to perform.
Those ideas are compatible. We need to respond well to failures we cannot prevent and keep improving the engineering that determines how many avoidable problems enter the response process.
Correction might belong to service engineering, SRE, network operations, or a centralized team with the right authority. Moving the same work between boxes on an organization chart isn’t the improvement.
What the recurring work should tell us
There is one more thing that continuous conformance changes: it gives repeated operational problems a way back into architecture and engineering decisions.
If the same exception keeps appearing, perhaps the deployment workflow is incomplete. Perhaps a standard is difficult to implement consistently. Perhaps the architecture requires a capability the organization hasn’t funded. Or perhaps the standard itself needs to be reconsidered.
Those are not all problems a central operations team can solve, however capable its people are. They require the domain context, decision rights, and capacity to change the underlying system.
The same is true of service data. If other teams depend on our monitoring signals, inventories, and descriptions of dependencies, maintaining them is part of delivering the service. Handing over a stream of technically valid but poorly understood alerts isn’t quite enough.
None of this tells us how much money to spend on engineering cleanup versus event intelligence. That decision depends on scale, risk, existing capability, and the cost of the work being avoided. Continuous assurance has its own costs, and chasing every low-consequence exception could be a poor use of engineering time.
But when we review another proposal to improve alert correlation or triage, I think there’s a companion question worth asking.
Which recurring conditions are creating this work, and what would it take to correct them before the next alert arrives?