The Challenge
A GIS platform built on an auto-scaling architecture was degrading under load. Despite increasing demand, the Auto Scaling Groups were not expanding as expected. The system was running below intended capacity without any obvious indication of why.
From the outside, it looked like a performance problem. The underlying cause turned out to be something most people would not think to check first.
What I Found
Analysing the scaling behaviour and instance logs pointed toward an infrastructure constraint rather than a configuration error or application issue. Deeper investigation revealed that the VPC had accumulated a large number of unused Elastic Network Interfaces - close enough to the service limit that new EC2 instances could not acquire private IP addresses during scale-out events.
The Auto Scaling Group was trying to launch instances. The instances were failing silently. And because the failure mode was infrastructure-level rather than application-level, it was not surfacing in the places the team was looking.
The broader environment told a similar story. Unused AMIs, orphaned EBS volumes and outdated snapshots had accumulated over time - not through negligence, but through gaps in the deployment pipeline and operational processes that had never been closed.
How I Approached It
The immediate fix was clear: clean up the unused ENIs to bring the VPC back within usable headroom. This restored scaling functionality immediately.
To prevent the same condition recurring, I implemented a lightweight automation script to regularly identify and remove unused ENIs before they could accumulate to service-limit levels again.
The cost side required a different approach. Rather than simply deleting orphaned resources, I worked with the system owner to trace the root cause - understanding why those resources existed in the first place. That led back to specific gaps in pipeline logic and governance controls. Fixing the pipeline meant the problem would not keep reappearing; cleaning up the backlog without fixing the source would just delay the next occurrence.
What Changed
Auto-scaling was fully restored, and the platform returned to operating at its intended capacity. The customer also gained a clearer picture of how their operational processes were contributing to infrastructure drift.
The combined effect of the ENI cleanup, the orphaned resource removal, and the pipeline corrections reduced overall cloud spend by approximately one-third - a significant outcome from work that began as a performance investigation.
Lessons for Enterprise Cloud Platform Management
Service limits are invisible until they are not. AWS service limits rarely come up in architecture reviews, but they are real constraints that can cause failures in ways that do not surface clearly in standard monitoring. For platforms running at scale with high instance churn, tracking ENI usage, IP address availability and other VPC resource limits deserves to be part of standard operational hygiene.
Silent failures are the most dangerous kind. The auto-scaling was not throwing visible errors. It was just not working. This is a good reminder that absence of alerts is not the same as absence of problems. Observability needs to include capacity and infrastructure-layer metrics, not just application health.
Automation prevents recurrence; root cause prevents repetition. The ENI cleanup script addressed the symptom reliably. But the pipeline and governance work addressed why it happened. Both matter: one buys time, the other closes the gap.
Operational drift is a cost problem. Orphaned resources do not announce themselves. They accumulate quietly across deployments and over time they become real spend. Building cleanup and validation into delivery pipelines - rather than relying on periodic manual reviews - is the only sustainable way to manage this at scale.