TECH **Terraform `local-exec` & `remote-exec` Provisioners: The Good, The Bad, and The Ugly**

Terraform local-exec & remote-exec Provisioners: The Good, The Bad, and The Ugly

(A Hyper-Practical, Zero-Fluff Guide for Engineers Who Need to Ship)

1. What This Is & Why It Matters

You’re a cloud engineer, and your team just inherited a Terraform codebase that deploys a fleet of EC2 instances. The previous team used remote-exec to run a bash script that installs Docker, configures a user, and pulls a private container image—every single time terraform apply runs. Now, your CI/CD pipeline is failing because the script occasionally times out, and you’re getting paged at 3 AM because a production instance didn’t bootstrap correctly.

This is what local-exec and remote-exec provisioners do:
- They let you run arbitrary commands on your local machine (local-exec) or on a newly created resource (remote-exec) after Terraform provisions it.
- They’re not idempotent—they run every time, even if nothing changed.
- They’re not declarative—they’re imperative scripts bolted onto Terraform, which breaks the IaC paradigm.

Why this matters in production:
- Reliability: If the script fails, Terraform marks the resource as "tainted" and destroys/recreates it on the next run (hello, downtime).
- Security: Hardcoded secrets in scripts, SSH keys in plaintext, or sudo commands with no audit trail.
- Cost: Unnecessary runs waste compute time (e.g., re-installing Docker on every apply).
- Debugging: When a remote-exec fails, you’re left with a half-configured instance and no logs (unless you explicitly captured them).

Real-world scenario:
You’re deploying a Kubernetes cluster with Terraform, and the last step is to install Helm charts. The "easy" way is to use remote-exec to run helm install on the control plane. The right way is to use a null_resource with triggers (or better, a dedicated tool like Argo CD). This guide will show you why—and how to avoid the pitfalls.

2. Core Concepts & Components

1. local-exec Provisioner

• Definition: Runs a command on the machine where Terraform is executed (e.g., your laptop or CI/CD runner).
• Production insight: Useful for post-processing (e.g., generating a config file, calling an API), but never for critical infrastructure setup—it’s not part of the resource lifecycle.
• Example use case: After creating an S3 bucket, run a local-exec to update a DNS record via AWS CLI.

2. remote-exec Provisioner

• Definition: Runs a command on a remote resource (e.g., an EC2 instance) via SSH or WinRM.
• Production insight: Avoid this for anything mission-critical. If the script fails, Terraform destroys the instance and tries again (which may fail again).
• Example use case: Installing a monitoring agent on a VM (but even this is better handled by user_data or a configuration management tool like Ansible).

3. connection Block

• Definition: Defines how Terraform connects to the remote resource (SSH/WinRM credentials, IP, port).
• Production insight: Never hardcode credentials. Use IAM roles, SSH keys from a secrets manager, or dynamic lookups.
• Example:
  hcl connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip }

4. when = destroy

• Definition: Runs the provisioner only when the resource is destroyed (e.g., cleanup tasks).
• Production insight: Rarely used, but useful for deregistering instances from a load balancer before deletion.
• Example:
  hcl provisioner "local-exec" {
when = destroy
command = "aws elb deregister-instances --load-balancer-name my-lb --instances ${self.id}" }

5. on_failure = continue

• Definition: If the provisioner fails, Terraform ignores the error and continues.
• Production insight: Almost always a bad idea. If the script fails, your resource is in an unknown state.
• Example (anti-pattern):
  hcl provisioner "remote-exec" {
on_failure = continue # ❌ Never do this in production!
inline = ["sudo apt update && sudo apt install -y nginx"] }

6. null_resource

• Definition: A Terraform resource that doesn’t manage any real infrastructure but can trigger provisioners.
• Production insight: Useful for workarounds (e.g., running a script after a resource is created), but prefer native Terraform resources where possible.
• Example:
  hcl resource "null_resource" "example" {
triggers = {
instance_id = aws_instance.web.id
}
provisioner "local-exec" {
command = "echo 'Instance ${aws_instance.web.id} created!'"
} }

7. triggers (in null_resource)

• Definition: A map of values that, when changed, force the null_resource to re-run.
• Production insight: Critical for idempotency. Without triggers, null_resource runs on every apply.
• Example:
  hcl triggers = {
script_hash = filebase64sha256("install.sh") # Re-run if the script changes }

8. user_data (vs. remote-exec)

• Definition: Cloud-init script that runs once when an EC2 instance boots (idempotent, managed by AWS).
• Production insight: Always prefer user_data over remote-exec for EC2 bootstrapping. It’s more reliable, auditable, and doesn’t block Terraform.
• Example:
  hcl resource "aws_instance" "web" {
user_data = file("user_data.sh") }

3. Step-by-Step Hands-On: Deploying an EC2 Instance with remote-exec (and Why You Shouldn’t)

Prerequisites

• AWS account with IAM permissions to create EC2 instances.
• Terraform installed (>= 1.0).
• An SSH key pair in ~/.ssh/id_rsa (or specify another key).
• A VPC with a public subnet (or use the default VPC).

Task:

Deploy an EC2 instance and use remote-exec to install Nginx. Then, break it intentionally to see why this is a bad idea.

Step 1: Write the Terraform Config

Create main.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"  # Ubuntu 20.04 LTS
  instance_type = "t3.micro"
  key_name      = "your-key-pair-name"      # Replace with your SSH key name

  # Allow SSH and HTTP
  vpc_security_group_ids = [aws_security_group.web.id]

  # Connection block for remote-exec
  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = self.public_ip
  }

  # ⚠️ Anti-pattern: remote-exec for bootstrapping
  provisioner "remote-exec" {
    inline = [
      "sudo apt update -y",
      "sudo apt install -y nginx",
      "sudo systemctl start nginx"
    ]
  }
}

resource "aws_security_group" "web" {
  name        = "allow_web_traffic"
  description = "Allow HTTP and SSH"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # ⚠️ Restrict this in production!
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

output "public_ip" {
  value = aws_instance.web.public_ip
}

Step 2: Deploy the Instance

terraform init
terraform apply -auto-approve

Expected output:

aws_instance.web: Creating...
aws_instance.web: Still creating... [10s elapsed]
aws_instance.web: Provisioning with 'remote-exec'...
aws_instance.web (remote-exec): Connecting to remote host via SSH...
aws_instance.web (remote-exec):   Host: 54.210.123.45
aws_instance.web (remote-exec):   User: ubuntu
aws_instance.web (remote-exec):   Password: false
aws_instance.web (remote-exec):   Private key: true
aws_instance.web (remote-exec):   Certificate: false
aws_instance.web (remote-exec):   SSH Agent: false
aws_instance.web (remote-exec):   Checking Host Key: false
aws_instance.web (remote-exec): Connected!
aws_instance.web (remote-exec): Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
aws_instance.web (remote-exec): Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
aws_instance.web: Still creating... [20s elapsed]
aws_instance.web (remote-exec): Setting up nginx (1.18.0-0ubuntu1.4) ...
aws_instance.web (remote-exec): Processing triggers for systemd (245.4-4ubuntu3.22) ...
aws_instance.web: Creation complete after 1m10s [id=i-0123456789abcdef0]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

public_ip = "54.210.123.45"

Step 3: Verify Nginx is Running

curl http://$(terraform output -raw public_ip)

Expected output:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</html>

Step 4: Break It Intentionally (See Why remote-exec is Dangerous)

1. Edit main.tf and add a typo to the remote-exec script:
   hcl
provisioner "remote-exec" {
inline = [
"sudo apt update -y",
"sudo apt install -y nginxx", # ❌ Typo: "nginxx" instead of "nginx"
"sudo systemctl start nginx"
]
}
2. Run terraform apply again:
   bash
terraform apply -auto-approve
3. Observe the failure:
   aws_instance.web (remote-exec): E: Unable to locate package nginxx
aws_instance.web (remote-exec): sudo: systemctl: command not found
╷
│ Error: remote-exec provisioner error
│
│ with aws_instance.web,
│ on main.tf line 25, in resource "aws_instance" "web":
│ 25: provisioner "remote-exec" {
│
│ timeout - last error: SSH command error: Process exited with status 100
╵
4. Check the instance state:
   bash
terraform state show aws_instance.web
   Output:
   status = "tainted" # ⚠️ Terraform marked it as tainted!
5. Next terraform apply will destroy and recreate the instance:
   bash
terraform apply -auto-approve
   Result: Downtime, wasted time, and a frustrated on-call engineer.

Step 5: Fix It the Right Way (Using user_data)

1. Replace remote-exec with user_data:
   hcl
resource "aws_instance" "web" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t3.micro"
key_name = "your-key-pair-name"
user_data = file("user_data.sh") # ✅ Better: cloud-init script
vpc_security_group_ids = [aws_security_group.web.id]
}
2. Create user_data.sh:
   bash
#!/bin/bash
sudo apt update -y
sudo apt install -y nginx
sudo systemctl start nginx
3. Run terraform apply:
   bash
terraform apply -auto-approve
4. Verify it works:
   bash
curl http://$(terraform output -raw public_ip)
   Key differences:
5. user_data runs once at boot (idempotent).
6. If it fails, the instance still exists (no tainting).
7. Logs are in /var/log/cloud-init-output.log.

4. ? Production-Ready Best Practices

Security

• ❌ Never hardcode secrets in remote-exec or local-exec. Use:
• AWS Secrets Manager (aws_secretsmanager_secret_version).
• HashiCorp Vault (vault_generic_secret).
• Environment variables (TF_VAR_*).
• ❌ Avoid sudo in scripts unless absolutely necessary. Use user_data with sudo privileges.
• ✅ Restrict SSH access in security groups. Use a bastion host or AWS Session Manager instead.

Cost Optimization

• ❌ remote-exec runs on every apply, even if nothing changed. This wastes time and money.
• ✅ Use triggers in null_resource to run only when needed: hcl resource "null_resource" "example" {
triggers = {
script_hash = filebase64sha256("script.sh")
}
provisioner "local-exec" {
command = "./script.sh"
} }

Reliability & Maintainability

• ❌ remote-exec is not idempotent. If the script fails, Terraform taints the resource.
• ✅ Prefer declarative tools:
• EC2: user_data (cloud-init).
• Kubernetes: Helm charts, Argo CD.
• General config: Ansible, Chef, Puppet.
• ✅ Use null_resource with triggers for one-off tasks: hcl resource "null_resource" "db_migration" {
triggers = {
db_endpoint = aws_db_instance.default.endpoint
}
provisioner "local-exec" {
command = "python migrate.py --db ${aws_db_instance.default.endpoint}"
} }

Observability

• ❌ remote-exec logs are ephemeral. If the script fails, you lose the output.
• ✅ Capture logs explicitly:
  hcl provisioner "remote-exec" {
inline = [
"sudo apt update -y 2>&1 | tee /var/log/bootstrap.log",
"sudo apt install -y nginx 2>&1 | tee -a /var/log/bootstrap.log"
] }
• ✅ Use AWS CloudWatch Logs for user_data: bash #!/bin/bash exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 sudo apt update -y sudo apt install -y nginx

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "Which provisioner runs on the Terraform execution machine?"
2. ✅ local-exec

3. ❌ remote-exec

4. "What happens if a remote-exec script fails?"

5. ✅ The resource is tainted and will be destroyed/recreated on the next apply.

6. ❌ Terraform continues without error.

7. "How do you run a script only when a resource is destroyed?"

8. ✅ provisioner "local-exec" { when = destroy ... }

9. ❌ There’s no way to do this.

10. "What’s the most reliable way to bootstrap an EC2 instance?"

11. ✅ user_data (cloud-init)
12. ❌ remote-exec

Key Trap Distinctions

Scenario-Based Question

"You need to install a monitoring agent on 100 EC2 instances. What’s the most reliable approach?"
- ❌ Use remote-exec in Terraform.
- ✅ Use user_data with a cloud-init script.
- ✅ Use AWS Systems Manager (SSM) to run the agent install.
- ✅ Use a configuration management tool like Ansible.

Why?
- remote-exec is not scalable (Terraform would SSH into 100 instances sequentially).
- user_data is idempotent and runs at boot.
- SSM/Ansible are designed for this and provide better logging.