DevOps Folio
Back to articles

Infrastructure as Code in Practice: Benefits, Trade-offs, and Tooling

By tvignoli DevOps Folio·Published on August 30, 2024

IaC is more than "turning clicks into code". It is the contract between platform teams and developers, enabling reproducible environments, automated compliance, and codified knowledge. I have seen IaC rollouts succeed when leaders treat it as a product, and fail when treated as a weekend migration. After managing IaC transformations across fintech, healthcare, and e-commerce organizations processing thousands of infrastructure changes monthly, I've distilled the patterns that deliver reliability, security, and velocity at scale.

Benefits, Risks, and Anti-Patterns

Benefits: versioning, PR-based reviews, drift detection, ephemeral test environments, and the ability to bolt policy engines (OPA, Terraform Cloud, cf-guard) directly into CI/CD. Risks: state corruption, poorly scoped IAM policies, and "copy/paste modules" that become unmaintained snowflakes. Anti-patterns include letting every squad fork the same Terraform module, or granting CI runners admin roles without guardrails.

The most critical benefit is auditability. When a production incident occurs, you can trace infrastructure changes through git history, identify the exact commit that introduced the issue, and rollback with confidence. I've seen organizations reduce mean time to recovery (MTTR) from hours to minutes simply by having infrastructure changes versioned and reviewable. However, this requires discipline: every change must go through IaC, with zero manual console modifications.

# Example: Terraform module with embedded policy checks
module "secure_s3_bucket" {
  source = "git::https://github.com/company/terraform-modules//s3-bucket?ref=v2.1.0"
  
  bucket_name = "prod-artifacts-${var.environment}"
  versioning  = true
  encryption  = "aws:kms"
  
  # Policy checks enforced via Sentinel
  tags = {
    Environment = var.environment
    CostCenter   = var.cost_center
    ManagedBy    = "terraform"
  }
}

# Sentinel policy (enforced in Terraform Cloud)
# main = rule {
#   all s3_buckets as _, buckets {
#     all buckets as bucket {
#       bucket.versioning.enabled is true
#     }
#   }
# }

Tool Comparison: Terraform, CloudFormation, Pulumi

Terraform remains the de facto multi-cloud option, with mature workflows (remote state, Sentinel, CDK for Terraform). CloudFormation integrates deepest with AWS features like ChangeSets, StackSets, and Drift Detection. Pulumi targets developer-first teams that prefer TypeScript/Python/Go and want to reuse existing libraries or apply imperative logic.

Terraform's strength lies in its ecosystem: 3,000+ providers covering virtually every cloud and SaaS platform. The HCL language is declarative and readable, though it can feel verbose for complex logic. CloudFormation's native AWS integration means you get features like StackSets for multi-account deployments and ChangeSets for previewing changes before apply. Pulumi's programmatic approach shines when you need loops, conditionals, or integration with existing codebases.

// Pulumi: S3 bucket with encryption and lifecycle policies
import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("artifacts", {
  serverSideEncryptionConfiguration: {
    rule: {
      applyServerSideEncryptionByDefault: {
        sseAlgorithm: "aws:kms",
        kmsMasterKeyId: kmsKey.id,
      },
      bucketKeyEnabled: true,
    },
  },
  versioning: { enabled: true },
  lifecycleRules: [
    {
      enabled: true,
      expiration: { days: 30 },
      noncurrentVersionExpiration: { days: 7 },
      abortIncompleteMultipartUploadDays: 1,
    },
  ],
  publicAccessBlockConfiguration: {
    blockPublicAcls: true,
    blockPublicPolicy: true,
    ignorePublicAcls: true,
    restrictPublicBuckets: true,
  },
});

// CloudFormation equivalent (YAML)
// Resources:
//   ArtifactsBucket:
//     Type: AWS::S3::Bucket
//     Properties:
//       VersioningConfiguration:
//         Status: Enabled
//       BucketEncryption:
//         ServerSideEncryptionConfiguration:
//           - ServerSideEncryptionByDefault:
//               SSEAlgorithm: aws:kms
//               KMSMasterKeyID: !Ref KMSKey
# Terraform: Reusable module pattern
# modules/vpc/main.tf
variable "environment" {
  type        = string
  description = "Environment name (dev, staging, prod)"
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "cidr_block" {
  type        = string
  description = "CIDR block for VPC"
  default     = "10.0.0.0/16"
}

resource "aws_vpc" "main" {
  cidr_block           = var.cidr_block
  enable_dns_hostnames = true
  enable_dns_support   = true
  
  tags = {
    Name        = "vpc-${var.environment}"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

# Usage in root module
module "production_vpc" {
  source = "./modules/vpc"
  
  environment = "prod"
  cidr_block  = "10.1.0.0/16"
}

State Management: The Critical Foundation

State management is where most IaC initiatives stumble. Local state files are fine for learning, but production requires remote backends with locking. Terraform Cloud, S3 + DynamoDB, or Pulumi ESC provide state locking, versioning, and encryption at rest. Without locking, concurrent applies can corrupt state, leading to hours of recovery work.

Best practice: store state remotely from day one, enable versioning on the state bucket, and use DynamoDB for locking. Encrypt state at rest using KMS, and restrict access via IAM policies. For multi-account setups, use Terraform Cloud workspaces or AWS Organizations with separate state per account.

# terraform.tf - Remote backend configuration
terraform {
  backend "s3" {
    bucket         = "company-terraform-state-prod"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789012:key/abc123"
    dynamodb_table = "terraform-state-lock"
    
    # Prevent accidental overwrites
    versioning = true
  }
  
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# State recovery runbook snippet
# 1. Check DynamoDB for stale locks: aws dynamodb scan --table-name terraform-state-lock
# 2. If lock is stale (>1 hour), delete: aws dynamodb delete-item --table-name terraform-state-lock --key '{"LockID":{"S":"..."}}'
# 3. Restore state from S3 versioning if corruption detected
# 4. Run terraform refresh to sync with actual infrastructure

Case Study: Terraform at Scale

A travel company with 40 squads adopted Terraform using a platform-owned registry of opinionated modules. Each module had embedded policy checks and surfaced semantic versioning. Rollouts happened behind a single "infra-apply" GitHub Action workflow, ensuring state locking and drift detection were centralised. The result was a median provisioning time of 12 minutes versus 2+ hours previously with tickets.

The key to their success was a centralized module registry with semantic versioning. Each module (VPC, ECS cluster, RDS instance) was versioned independently, allowing squads to adopt updates at their own pace. Policy-as-code via Sentinel prevented common mistakes: no public S3 buckets, required tags, encryption at rest. The platform team maintained a "golden path" module for each resource type, reducing variance across 200+ microservices.

# GitHub Actions: Terraform apply workflow with policy checks
name: Infrastructure Apply
on:
  pull_request:
    paths:
      - 'infrastructure/**'
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  terraform-plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.6.0
          terraform_wrapper: false
      
      - name: Terraform Init
        run: |
          cd infrastructure
          terraform init             -backend-config="bucket=${{ secrets.TF_STATE_BUCKET }}"             -backend-config="key=${{ github.ref_name }}/terraform.tfstate"
      
      - name: Terraform Validate
        run: |
          cd infrastructure
          terraform validate
          terraform fmt -check
      
      - name: Terraform Plan
        id: plan
        run: |
          cd infrastructure
          terraform plan -out=tfplan -no-color
        continue-on-error: true
      
      - uses: hashicorp/terraform-github-actions@master
        with:
          tf_actions_version: 1.6.0
          tf_actions_comment: true
      
      - name: Policy Check (Sentinel)
        if: github.event_name == 'pull_request'
        run: |
          # Run Sentinel policies via Terraform Cloud
          terraform plan -out=tfplan
          terraform show -json tfplan | sentinel apply policies/
      
  terraform-apply:
    needs: terraform-plan
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      
      - name: Terraform Apply
        run: |
          cd infrastructure
          terraform init
          terraform apply -auto-approve

Drift Detection and Remediation

Infrastructure drift—when actual resources diverge from IaC definitions—is inevitable. Someone makes a manual change in the console, a script modifies resources directly, or a third-party tool updates configurations. Drift detection must be automated and run regularly, not just during deployments.

Terraform Cloud and AWS Config both offer drift detection. The key is deciding how to handle drift: auto-remediate (risky), alert-only (safe but requires manual action), or selective remediation based on resource type. For production, I recommend alert-only with a weekly review process. Critical resources (IAM roles, security groups) should auto-remediate, while compute resources can be reviewed before remediation.

# Python script: Automated drift detection and reporting
import boto3
import json
from datetime import datetime
from typing import List, Dict

cloudformation = boto3.client('cloudformation')
config = boto3.client('config')

def detect_drift(stack_name: str) -> Dict:
    """Detect CloudFormation stack drift."""
    response = cloudformation.detect_stack_drift(StackName=stack_name)
    drift_id = response['StackDriftDetectionId']
    
    # Wait for detection to complete
    while True:
        status = cloudformation.describe_stack_drift_detection_status(
            StackDriftDetectionId=drift_id
        )
        if status['DetectionStatus'] == 'DETECTION_COMPLETE':
            break
        time.sleep(2)
    
    # Get drift details
    drift_details = cloudformation.describe_stack_resource_drifts(
        StackName=stack_name
    )
    
    return {
        'stack_name': stack_name,
        'drift_status': status['StackDriftStatus'],
        'drifted_resources': [
            {
                'logical_id': drift['LogicalResourceId'],
                'resource_type': drift['ResourceType'],
                'drift_type': drift['StackResourceDriftStatus'],
                'property_differences': drift.get('PropertyDifferences', [])
            }
            for drift in drift_details['StackResourceDrifts']
            if drift['StackResourceDriftStatus'] != 'IN_SYNC'
        ]
    }

def remediate_drift(stack_name: str, resource_logical_id: str):
    """Remediate drift by updating stack from IaC."""
    # This would trigger a Terraform apply or CloudFormation update
    # Only run for non-critical resources after approval
    pass

# Scheduled Lambda: Run daily drift detection
def lambda_handler(event, context):
    stacks = cloudformation.list_stacks(
        StackStatusFilter=['CREATE_COMPLETE', 'UPDATE_COMPLETE']
    )
    
    drift_report = []
    for stack in stacks['StackSummaries']:
        drift = detect_drift(stack['StackName'])
        if drift['drift_status'] != 'IN_SYNC':
            drift_report.append(drift)
            # Send to SNS for alerting
            sns.publish(
                TopicArn=DRIFT_ALERT_TOPIC,
                Message=json.dumps(drift, indent=2),
                Subject=f"Drift detected in {stack['StackName']}"
            )
    
    return {'drifted_stacks': len(drift_report), 'details': drift_report}

Testing Infrastructure Code

Testing IaC is non-negotiable for production workloads. Unit tests validate variable validation and module logic. Integration tests spin up real resources in ephemeral environments, validate they work correctly, then tear them down. Compliance tests ensure security policies are enforced.

Tools like Terratest (Go), Kitchen-Terraform (Ruby), and Pytest with moto (Python) enable integration testing. The pattern: write tests that create infrastructure, validate it behaves correctly, then destroy it. Run these in CI/CD before merging to main. For compliance, use tools like Checkov, tfsec, or cfn-nag to scan for security misconfigurations.

// Terratest example: Testing VPC module
package test

import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)

func TestVPCModule(t *testing.T) {
    terraformOptions := &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "environment": "test",
            "cidr_block":  "10.0.0.0/16",
        },
        NoColor: true,
    }
    
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    
    // Validate VPC exists
    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcId)
    
    // Validate DNS is enabled
    vpc := getVPC(t, vpcId)
    assert.True(t, *vpc.EnableDnsHostnames)
    assert.True(t, *vpc.EnableDnsSupport)
    
    // Validate subnets exist
    publicSubnets := terraform.OutputList(t, terraformOptions, "public_subnet_ids")
    assert.Len(t, publicSubnets, 2) // Expect 2 AZs
    
    // Validate security groups
    sgId := terraform.Output(t, terraformOptions, "default_security_group_id")
    assert.NotEmpty(t, sgId)
}
# Pytest + moto: Testing CloudFormation templates
import boto3
import pytest
from moto import mock_cloudformation, mock_s3
from botocore.exceptions import ClientError

@mock_cloudformation
@mock_s3
def test_s3_bucket_template():
    """Test CloudFormation template creates S3 bucket correctly."""
    cf = boto3.client('cloudformation', region_name='us-east-1')
    s3 = boto3.client('s3', region_name='us-east-1')
    
    # Load template
    with open('templates/s3-bucket.yaml') as f:
        template_body = f.read()
    
    # Create stack
    stack_name = 'test-bucket-stack'
    cf.create_stack(
        StackName=stack_name,
        TemplateBody=template_body,
        Parameters=[
            {'ParameterKey': 'BucketName', 'ParameterValue': 'test-bucket'}
        ]
    )
    
    # Validate bucket exists
    buckets = s3.list_buckets()
    assert any(b['Name'] == 'test-bucket' for b in buckets['Buckets'])
    
    # Validate versioning enabled
    versioning = s3.get_bucket_versioning(Bucket='test-bucket')
    assert versioning['Status'] == 'Enabled'
    
    # Cleanup
    cf.delete_stack(StackName=stack_name)

# Checkov: Security scanning
# checkov -d infrastructure/ --framework terraform
# checkov -f template.yaml --framework cloudformation

Multi-Account and Multi-Region Patterns

Enterprise organizations require multi-account strategies for isolation, billing, and compliance. Terraform workspaces, CloudFormation StackSets, and Pulumi stacks enable managing infrastructure across accounts. The key is a consistent module library and centralized state management.

Pattern: Use AWS Organizations with separate accounts per environment (dev, staging, prod). Each account has its own Terraform state, but modules are shared via a private registry. Use assume-role authentication to deploy from a central CI/CD account. For multi-region, parameterize regions in modules and deploy the same stack to multiple regions.

# Multi-account deployment pattern
# terraform.tfvars per account
# accounts/dev/terraform.tfvars
account_id    = "111111111111"
environment   = "dev"
region        = "us-east-1"
kms_key_alias = "alias/dev-terraform-state"

# accounts/prod/terraform.tfvars
account_id    = "999999999999"
environment   = "prod"
region        = "us-east-1"
kms_key_alias = "alias/prod-terraform-state"

# main.tf - Assume role for cross-account access
provider "aws" {
  region = var.region
  
  assume_role {
    role_arn = "arn:aws:iam::${var.account_id}:role/TerraformDeploymentRole"
  }
  
  default_tags {
    tags = {
      Environment = var.environment
      ManagedBy   = "terraform"
      AccountId   = var.account_id
    }
  }
}

# GitHub Actions: Deploy to multiple accounts
# jobs:
#   deploy-dev:
#     environment: dev
#     steps:
#       - run: terraform apply -var-file=accounts/dev/terraform.tfvars
#   deploy-prod:
#     needs: deploy-dev
#     environment: production
#     steps:
#       - run: terraform apply -var-file=accounts/prod/terraform.tfvars

Adoption Strategy & Best Practices

1) Start with one platform team owning shared modules. 2) Enforce code reviews and automated plan checks. 3) Use remote backends with locking (S3 + DynamoDB, Terraform Cloud, or Pulumi ESC). 4) Document runbooks for state recovery and cross-account bootstrapping. 5) Provide developer enablement—brown bags, templates, and office hours do more than top-down mandates. 6) Track metrics like "time to environment" and "incidents due to drift" to justify further investments.

The migration path matters. Don't try to convert everything at once. Start with net-new infrastructure, then gradually migrate existing resources using terraform import or CloudFormation drift detection. Create a "golden path" module for each resource type, then require all new infrastructure to use these modules. Over time, teams will naturally migrate to avoid maintaining custom code.

Measure success with metrics: time to provision environments (target: <15 minutes), infrastructure change lead time (target: <1 day), and drift incidents (target: <1 per month). These metrics justify continued investment and help identify bottlenecks in the IaC workflow.

# Example: Golden path module with sensible defaults
# modules/ecs-service/main.tf
variable "service_name" {
  type        = string
  description = "Name of the ECS service"
}

variable "cpu" {
  type        = number
  default     = 256
  description = "CPU units (256 = 0.25 vCPU)"
}

variable "memory" {
  type        = number
  default     = 512
  description = "Memory in MB"
}

variable "desired_count" {
  type        = number
  default     = 2
  description = "Desired number of tasks"
}

# Enforce best practices
resource "aws_ecs_service" "main" {
  name            = var.service_name
  cluster         = var.cluster_id
  task_definition = aws_ecs_task_definition.main.arn
  desired_count   = var.desired_count
  
  # Always use Fargate (no EC2 management)
  launch_type = "FARGATE"
  
  # Health checks
  health_check_grace_period_seconds = 60
  
  # Auto-scaling (enforced)
  enable_execute_command = true # For debugging
  
  network_configuration {
    subnets          = var.subnet_ids
    security_groups  = [aws_security_group.ecs.id]
    assign_public_ip = false # Always private
  }
  
  # Load balancer (required)
  load_balancer {
    target_group_arn = aws_lb_target_group.main.arn
    container_name   = var.service_name
    container_port   = 8080
  }
  
  tags = {
    Name        = var.service_name
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

# Usage - teams just provide service name
module "api_service" {
  source = "git::https://github.com/company/modules//ecs-service?ref=v1.2.0"
  
  service_name = "user-api"
  cpu          = 512
  memory       = 1024
  # All other settings come from module defaults