The Moment I Realized Our CMDB Was a Liability
Salil Kulkarni is a senior IT executive with over 15 years of experience leading infrastructure and operations teams at enterprise organizations. He writes about IT governance, change management, and the operational realities of maintaining CMDB accuracy at scale.
A stale CMDB — one updated through change-ticket processes but never validated by continuous discovery — creates dangerous gaps between documented and actual infrastructure. CIOs approving changes based on unvalidated CMDB dependency counts are approving the wrong blast radius. In one production incident, 44 documented dependencies masked 62 actual ones. Eighteen undiscovered CIs triggered an unplanned outage, violated two SLA commitments, and cost $47,000 to remediate. The data was 11 months old.
I have described my CMDB to many audiences over the years, in board presentations and vendor conversations. In all of those descriptions, I used variations of the same phrase: “We have a CMDB. We keep it current.” I believed that. My infrastructure team maintained it. We had a process for updating it after changes. We had a field in our change ticket template that required engineers to confirm the CMDB had been updated before closing a ticket. What I am about to describe is a CIO’s production incident caused by stale CMDB data — and it is more common than most change advisory boards admit.
What I did not know, until a Saturday morning storage array replacement, was that “keeping it current” was not the same as it being accurate. A CMDB that is updated after documented changes still accumulates gaps at three key points: changes that happen outside formal documentation, new services deployed without CMDB entries, and decommissions not followed up in the system. Over 11 months in a segment of my infrastructure, that accumulation had grown to 18 missing CI records. Those 18 CIs were invisible to me when I approved the change.
The change that revealed the gap
It was a routine storage array replacement in a segment of our data center that had been flagged in a hardware lifecycle review. The array was 5.5 years old, past its recommended replacement window, and had experienced two minor fault events in the preceding quarter.
The change advisory board reviewed the request. The CMDB showed 44 systems with direct dependencies on the array. All 44 teams had been notified with the required 72-hour advance notice. The maintenance window was Saturday from 1 AM to 5 AM, a four-hour window that the infrastructure team had sized based on the 44-system dependency scope and the documented data migration requirements.
I reviewed the change brief on Friday afternoon. Everything looked appropriate: the blast radius matched what I would expect for a storage array in that segment, the team notifications were confirmed, the rollback plan was documented, and the maintenance window was correctly sized. I approved the change.
| How does stale CMDB data make dangerous changes appear safe? |
|---|
| A CMDB that does not reflect the current infrastructure shows a smaller blast radius than actually exists. CIOs reviewing change requests see 44 dependent systems when there are 62, approve a four-hour maintenance window when eight hours are needed, and notify 44 teams when 62 need to know. The change looks well-managed because the data behind it looks complete. The data is the problem, not the process. |
What happened at 2:14 AM
The maintenance window started at 1 AM. By 1:47 AM, the data migration from the old array to the new array was proceeding on schedule. At 2:14 AM, the infrastructure team began switching storage paths from the old array to the new array.
Within four minutes of the path switch, 18 systems that had not appeared in the change notification list began showing storage access errors. Application teams for those systems started receiving automated alerts. On-call engineers for services that were not supposed to be in maintenance started investigating what they thought was an unrelated incident.
By 2:30 AM, my infrastructure lead called me: “We have 18 systems that weren’t on the dependency list. They’re all hitting the old array paths. We think they’re microservices that got deployed in the last several months without a CMDB update.”
By 3:10 AM, the team had identified all 18 systems and restored their storage paths to the new array. The last of the 18 was restored at 5:32 AM, 92 minutes past the planned maintenance window end.
Three of the 18 systems were customer-facing services. What started as a routine maintenance window became an unplanned outage for those three services. Two had SLA commitments that required 72-hour advance notice for maintenance. Both SLA obligations were violated.
| What is the business cost of approving a change based on an incomplete CMDB? |
|---|
| Beyond the direct remediation time, the costs include: SLA penalty exposure for any customer-facing services not in the change notification, regulatory risk if the unplanned outage affects compliance-relevant systems, and the post-incident investigation cost of tracing which deployments created the undocumented CIs. In practice, those secondary costs often exceed the direct incident response cost by a factor of two to three. |
How often does your CMDB get validated against discovery? See how Virima’s automated discovery keeps CI records current →
The post-incident analysis
The formal investigation that followed would take four days and change how we approach every major change window. The central finding: the CMDB data for the affected storage segment had not been validated against discovery in 11 months. During that 11-month period, 18 new microservices had been deployed by product engineering teams that used the same storage array path without submitting CMDB update requests. None of those deployments had been declined or flagged. The process for CMDB updates was a voluntary checklist item, not an enforced gate.
The CMDB record for the storage array was not wrong because of negligence. It was wrong because the process for keeping it current relied on engineers remembering to update it, and 18 times in 11 months, that did not happen.
The distinction I walked away with: a CMDB updated only through change-ticket processes will always have gaps where changes happen outside the formal change process. In a modern development environment, new deployments that affect infrastructure happen constantly, many of them below the threshold that triggers a formal change request. Each of those deployments can add a new CI or a new dependency without creating a CMDB record. Discovery is the only mechanism that catches those gaps, because discovery does not depend on engineers remembering.
Agentless network-based scanning — which reaches every system on the network regardless of whether a change ticket was submitted — would have surfaced all 18 microservices sharing that storage array path before the change executed. The ViVID™ service maps showing those storage path relationships would have made the dependency gap visible before the change advisory board voted.
What I changed
Three policy changes followed the post-incident review.
Mandatory pre-change discovery validation
For any change affecting a tier-1 or tier-2 infrastructure component, a fresh IT discovery scan of the affected CI segment must run within 7 days of the change advisory board vote. The discovery-sourced count is then documented alongside the CMDB count in the change request. If the two counts diverge by more than 10%, the change window must be resized to account for the actual dependency scope.
Maximum CI age for blast radius assessment
Any CI record used as the basis for blast radius assessment in a change request must have been discovery-validated within 30 days. CIs older than 30 days are flagged as “freshness-unverified” in the change ticket. The change sponsor must then either run a fresh scan or explicitly accept the staleness risk in writing.
We set a simple governance gate: if discovery finds the dependency scope more than 10% larger than the CMDB shows, the change window is resized before approval. That threshold is now a governance requirement, not a courtesy check.
30-day automated discovery refresh
A 30-day automated discovery scan cycle for all infrastructure segments — not a manual update cycle. Discovery identifies new CIs, flags changed dependencies, and marks stale records for review. Trusted Runtime Truth requires that the data reflect what the infrastructure is producing, not what engineers remembered to document. CMDB with Automated Discovery for Hybrid and Cloud Environments
The three policy changes required approximately six months to fully implement. In the 18 months since implementation, the change advisory board has caught four additional cases where the pre-change discovery scan found dependency counts materially different from the CMDB, one by 28%, one by 17%, and two by 10-12%. All four were caught before the change executed. None became incidents.
The storage array incident cost approximately $47,000 in remediation time, SLA exposure, and engineering overtime. The four subsequent catches, at a conservative estimate, avoided four incidents that would have cost an average of $35,000 each. The policy changes paid for themselves within a year.
| What policy change has the highest impact on reducing CMDB-related change incidents? |
|---|
| Mandatory pre-change discovery validation: requiring that a fresh discovery scan of the affected infrastructure segment be run within 7 days of any major change window, with the discovery-sourced dependent system count documented in the change request alongside the CMDB count. This single gate catches the gap between what the CMDB shows and what is actually running before the change executes. |
For CIOs who recognize this scenario — a CMDB maintained but not validated, a change process that depends on voluntary updates, a dependency count that looks complete but is not — the path starts with discovery, not a CMDB redesign project. See how Virima’s continuous discovery validates CMDB accuracy before change windows →
