Md Nayeem Hossain | Software Engineer

Infrastructure as Code with Terraform: Lessons from Production#

The first production server I managed was configured by hand over SSH. It worked perfectly until it needed to be rebuilt, and nobody could remember every package, firewall rule and environment variable that made it work. That experience made me a believer in infrastructure as code.

Terraform is the tool I reach for most. Writing Terraform is easy. Running it safely with a team, for years, takes some hard-won habits.

Why infrastructure as code matters#

When infrastructure is defined in code:

It is reviewable. A change to a security group goes through a pull request like any other change.
It is reproducible. You can create an identical staging environment from the same code.
It is documented by definition. The code is the source of truth for what exists.
It is recoverable. Rebuilding after a disaster is running a pipeline, not an archaeology project.

Remote state with locking, from day one#

Terraform tracks what it manages in a state file. On a laptop, that file is a liability: it can be lost, it can drift between teammates, and it often contains secrets.

Store state remotely with locking, so two people can never apply at the same time:

hcl

1terraform {2  backend "s3" {3    bucket         = "acme-terraform-state"4    key            = "production/network.tfstate"5    region         = "ap-southeast-1"6    dynamodb_table = "terraform-locks"7    encrypt        = true8  }9}

Restrict access to the state bucket tightly. Anyone who can read state may be able to read database passwords and keys stored in it.

Split state by blast radius#

One giant state file for everything means every plan takes minutes and every apply can touch everything. Split it along lines of change frequency and risk:

text

1infra/2├── global/        # DNS zones, IAM, rarely changes3├── network/       # VPCs, subnets, routing4├── data/          # databases, caches — high risk, changes rarely5└── services/      # app services — changes often

A mistake while editing a service should not be able to destroy the production database.

Modules: reuse without over-engineering#

Modules let you define a pattern once — "a web service with a load balancer, autoscaling and alarms" — and reuse it per environment. Keep them small and focused, version them, and resist building one mega-module with fifty variables. A module that tries to handle every case becomes harder to use than plain resources.

Pass environment differences as variables, not as copy-pasted code:

hcl

1module "api" {2  source        = "../modules/web-service"3  name          = "api"4  instance_type = var.environment == "production" ? "t3.large" : "t3.small"5  min_instances = var.environment == "production" ? 3 : 16}

Plan in CI, apply deliberately#

The workflow that has served me best:

A pull request runs terraform fmt -check, validate and plan.
The plan output is posted to the pull request for review.
A reviewer reads the plan, not just the code. The code says what you intended; the plan says what will actually happen.
After merge, apply runs from CI with the reviewed plan, never from a laptop.

Pay special attention to any line that says a resource will be destroyed or replaced. Changing a seemingly harmless attribute can force Terraform to recreate a database.

Protect what must not be deleted#

For critical resources, add guardrails in code:

hcl

1resource "aws_db_instance" "main" {2  # ...3  deletion_protection = true4 5  lifecycle {6    prevent_destroy = true7  }8}

Combine this with backups that live outside Terraform's control, so even a catastrophic apply cannot remove your ability to recover.

Drift: when reality diverges from code#

Someone changes a setting in the cloud console during an incident. Now the real infrastructure no longer matches the code, and the next apply will silently revert the fix.

Run a scheduled plan against each environment and alert when it shows unexpected changes.
Make console access read-only for most people.
When an emergency change is made by hand, follow up with a pull request the same day.

Secrets do not belong in code#

Never put secrets in .tf files or variable files committed to the repository. Reference them from a secret manager, generate them inside Terraform with random providers, or inject them at runtime. And remember that values Terraform handles may still end up in state, which is why state access must be restricted.

Conclusion#

Terraform turns infrastructure into something you can review, reproduce and recover. Use remote state with locking, split state by blast radius, keep modules small, review the plan rather than the code, protect critical resources from deletion and watch for drift. Your infrastructure becomes as dependable and well understood as your application code.