IaC Horror Stories: When Infrastructure Code Goes Wrong

A cautionary look at infrastructure automation for data engineers

---

Consider a Monday morning at a growing startup. The CTO's message lands in Slack in all caps: "WHO RAN WHAT ON AWS LAST NIGHT? WE HAVE A $47,000 BILL."

That scenario is what happens when Infrastructure as Code (IaC) goes wrong.

Terraform, CloudFormation, and "infrastructure automation" are often treated as DevOps concerns. But as data teams increasingly own their cloud infrastructure, knowing what can go wrong with IaC matters directly to data engineers.

The cases below are the kind of real-world failures that make a careful engineer review every Terraform plan before running apply.

What Infrastructure as Code Is

Infrastructure as Code means managing cloud resources - databases, servers, storage buckets, networks - through code files instead of clicking around in the AWS, Azure, or GCP consoles.

Instead of manually creating a Snowflake warehouse, S3 buckets, and an Airflow cluster by pointing and clicking, you write a configuration file declaring "I want these resources with these settings," and tools like Terraform create them automatically.

For data teams, this means you can:

• Spin up entire data environments in minutes
• Keep dev, staging, and prod identical
• Version-control infrastructure changes
• Automatically tear down expensive resources when they aren't needed

The same automation that builds environments quickly can also break them quickly.

Horror Story #1: The $43,000 Weekend

A data engineer at a mid-sized e-commerce company was setting up a new analytics environment. The deadline was Monday morning, so the work happened over the weekend.

The engineer had used Terraform for a few weeks and wrote a configuration to create EC2 instances for a Spark cluster. The problem was a small typo in the instance count variable: instead of count = 3, the config read count = 30. It also requested r5.24xlarge instances instead of r5.large.

When terraform apply ran at 11 PM on Saturday, Terraform spun up 30 large instances, each costing about $5.50 per hour. By Monday morning the bill had passed $43,000 for a weekend of unused compute.

The avoidable part: a terraform plan review would have caught this. The plan output showed "30 to add" instead of "3 to add," but it went unread under deadline pressure.

The lesson: always review plan output. Terraform shows exactly what it will create, modify, or destroy. A two-minute review can save tens of thousands of dollars.

Horror Story #2: The Disappearing Data Lake

A senior data engineer at a healthcare analytics firm was modernizing infrastructure. The team had managed AWS resources manually, and leadership approved a move to Infrastructure as Code for better governance.

The migration moved an existing S3 data lake - three years of properly anonymized patient analytics data - into Terraform management. The plan was straightforward: import existing resources, then manage them through code.

A mistake in Terraform state management broke that plan. On the first terraform apply after the import, Terraform didn't recognize the existing S3 bucket properly. Instead of updating the bucket configuration, it tried to recreate it.

The result: Terraform deleted the production S3 bucket containing 2.4 TB of processed healthcare data.

The aftermath: backups existed, but restoring and reprocessing three years of data took two weeks and cost the company a major client who needed real-time access to their analytics dashboard.

The hidden problem: importing existing resources into Terraform requires careful state management. One mismatch between the code and the existing resource, and Terraform assumes it needs to replace everything.

The lesson: when importing existing infrastructure, test on non-critical resources first. Use Terraform's -target flag to apply changes to specific resources, and keep a complete backup before touching production data stores.

Horror Story #3: The Open Database

A data engineer at a fintech startup was tasked with building Terraform modules other engineers could use to provision databases consistently as the company grew.

The PostgreSQL RDS module looked solid - clean configuration, thorough documentation, smooth initial testing. It was shared across the data team, and soon everyone used it to create databases for different projects.

Three months later, a routine security audit found that 12 production databases were accessible from the public internet with default passwords. Customer financial data, transaction histories, and personal information sat on publicly accessible databases.

How it happened: the module's default setting allowed public access, with the intent that developers would override it for production. But the default was publicly_accessible = true with a weak default password, and most engineers using the module didn't realize they needed to override these settings explicitly.

The damage: regulatory fines, emergency security audits, customer notification requirements, and a six-month infrastructure review. The startup's Series B funding round was delayed by eight months while investors assessed the security implications.

The lesson: secure defaults matter. When building reusable infrastructure modules, assume people will use the defaults. Make the secure option the easy option, not something people must remember to configure.

Horror Story #4: The Cascading Failure

A data platform engineer at a media company had used Terraform successfully for months, with code reviews, testing environments, and approval processes in place.

On a Friday afternoon, the engineer deployed what looked like a routine update - changing the instance type for Airflow workers to handle increased workload. The change had been tested in staging and approved by the team.

The configuration had an overlooked dependency chain. The Airflow workers connected to an Auto Scaling Group, which connected to a Launch Template, which referenced a Security Group. When Terraform tried to update the instance type, it determined it needed to recreate the Launch Template.

Recreating the Launch Template triggered the Auto Scaling Group to cycle all instances. The new instances couldn't connect to the database because the security group update hadn't propagated properly. The Auto Scaling Group kept terminating "unhealthy" instances and spinning up new ones that also couldn't connect.

Within 20 minutes, the team had:

• No functioning Airflow workers
• 47 failed EC2 instances piling up costs
• All scheduled data pipelines failing
• Real-time dashboards going dark

The cost: a full weekend spent manually fixing the cascading failures and rolling back changes, plus explaining to executives why customer-facing analytics were down for 18 hours.

The lesson: understand the dependency chains in your infrastructure. Small changes can have large ripple effects. Keep a tested rollback plan, and avoid deploying significant infrastructure changes on Fridays.

The Common Threads

These engineers were competent, experienced professionals. They made mistakes that many engineers could make.

The recurring patterns:

Time pressure creates shortcuts. Every one of these failures happened during a rush - weekend deployments, tight deadlines, quick fixes. Those conditions are where careful review gets skipped.

Infrastructure has hidden complexity. Where an application bug might crash one function, an infrastructure mistake can cascade across entire systems. One misconfigured security group can expose dozens of databases.

Defaults are dangerous. Most IaC tools optimize for flexibility, not security. The easy path is often the insecure or expensive path.

State management is unforgiving. Unlike application deployments that usually roll back easily, infrastructure changes can permanently delete data or create security gaps that are hard to detect.

Prevention Strategies That Work

The strategies that consistently prevent these failures:

The Two-Person Rule

Don't apply infrastructure changes alone. Have someone else review your terraform plan output. Fresh eyes catch things you've stared at for hours. Some teams require two approvals for any production infrastructure change.

The Friday Freeze

Adopt a policy of no infrastructure changes after Wednesday unless it's a genuine emergency. That leaves buffer time to fix problems without losing weekends.

Cost Alerts That Actually Alert

Set billing alerts at multiple thresholds. If your monthly bill is usually $5,000, set alerts at $7,000, $10,000, and $15,000. Route them to phone numbers, not just email.

Test Everything in Staging First

Often skipped despite being obvious. Staging should be as close to production as possible. If you can't replicate production fully, at least test the riskiest components.

Make Incremental Changes

Instead of deploying 15 changes at once, deploy them one by one. Use Terraform's -target flag to apply changes to specific resources first. If something breaks, you know exactly what caused it.

Back Up Before You Touch

Before importing existing resources or making major changes, snapshot databases, back up S3 buckets, and document the current configuration. Recovery is much faster with a known good state to return to.

The Bottom Line: Infrastructure Code Is Still Code

Infrastructure as Code is becoming a core skill for data professionals, not just a DevOps tool. As data teams take more ownership of their infrastructure, safely managing cloud resources through code is becoming as important as writing SQL or Python.

But unlike a pipeline bug that might delay a report, infrastructure mistakes can cost tens of thousands of dollars, expose sensitive data, or bring down entire systems.

Most of these disasters are preventable with simple practices: careful review, incremental changes, proper testing, and healthy paranoia about what could go wrong.

These stories aren't meant to discourage adoption of Infrastructure as Code - they show why it deserves respect. Most engineers who have automated their infrastructure have a few near-miss stories of their own.

A short checklist:

• If you already use IaC, review your cost monitoring and backup strategies.
• If you plan to adopt IaC, start with non-critical resources and build confidence gradually.
• Either way, treat infrastructure mistakes as often irreversible, and plan accordingly.

The ability to spin up entire data environments with a few commands is a real advantage. The practices above are what keep it from becoming the next horror story.