IaC - Deploying AWS EC2 user-data with Terraform

Posted on Aug 11, 2020

When we launch an EC2 instance, we can pass user data to the instance for performing common automated configuration tasks or running scripts after the instance boot.

With Terraform, like using the console, we could ‘paste’ the script we would like to use in user-data.

Here is a sample user-data embedded into tf file:

resource "aws_instance" "my-instance" {
 	ami = "ami-04169656fea786776"
 	instance_type = "t2.nano"
 	user_data = << EOF
 		#! /bin/bash
        apt-get update
 		apt-get install -y apache2
 		systemctl start apache2
 		systemctl enable apache2
 		echo "<h1>Deployed via Terraform</h1>" | sudo tee /var/www/html/index.html
 	EOF
 	tags = {
 		Name = "Terraform"	
 	}
 }

But how if the script is quite long?

So we prefer to use file() function.

Here is the code example of installing metricbeat on Amazon Linux.

ec2.tf

# https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/
data "aws_ssm_parameter" "amazonlinux2" {
  name = "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
}

resource "aws_instance" "metricbeat" {
  ami                         = data.aws_ssm_parameter.amazonlinux2.value
  associate_public_ip_address = false
  iam_instance_profile        = aws_iam_instance_profile.ec2_metricbeat_instance_profile.name
  instance_type               = var.ec2_instance_type
  user_data = file("metricbeat.sh")
  subnet_id = var.private_subnet
  timeouts { create = "30m" }

}

metricbeat.sh

#!/bin/bash -xe
 cd /tmp/
 yum update -y
 curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.8.0-x86_64.rpm
 rpm -iv metricbeat-7.8.0-x86_64.rpm
 metricbeat modules disable system
 rm /etc/metricbeat/metricbeat.yml
 touch /etc/metricbeat/metricbeat.yml
 
 cat > /etc/metricbeat/metricbeat.yml << EOM
 ##################### Metricbeat Configuration Example #######################
 
 # =========================== Modules configuration ============================
 metricbeat.config.modules:
 # Glob pattern for configuration loading
   path: /etc/metricbeat/modules.d/*.yml
 
 # Set to true to enable config reloading
 reload.enabled: false
 
 # ======================= Elasticsearch template setting =======================
 setup.template:
 name: "metricbeat--inframetrics"
 pattern: "metricbeat--inframetrics-*"
 setup.ilm.enabled: true
 setup.ilm.rollover_alias: "metricbeat--inframetrics"
 setup.ilm.policy_name: "metricbeat-*" # should be ALWAYS <beatsname>-*
 setup.ilm.pattern: "{now/M{yyyy.MM}}-001"
 # =================================== Kibana ===================================
 setup.kibana:
   host: "https://"
   space.id: "dev"
 # ---------------------------- Elasticsearch Output ----------------------------
 output.elasticsearch:
   hosts: ["https://"]

 # Authentication credentials - either API key or username/password.
   api_key: "key"
 # ================================= Processors =================================
 
 # Configure processors to enhance or manipulate events generated by the beat.
 processors:
   - add_host_metadata: ~
   - add_docker_metadata: ~
   - add_kubernetes_metadata: ~
 
 logging.level: debug
 EOM
 
 rm /etc/metricbeat/modules.d/aws.yml.disabled
 touch /etc/metricbeat/modules.d/aws.yml
 
 cat > /etc/metricbeat/modules.d/aws.yml << \EOM
 - module: aws
   period: 2m
   region: eu-west-1
   metricsets:
     - cloudwatch
   metrics:
     - namespace: AWS/SQS
 - module: aws
   period: 2m
   region: eu-west-1
   metricsets:
     - cloudwatch
   metrics:
     - namespace: AWS/ECS
 
 - module: aws
   period: 300s
   region: eu-west-1
   metricsets:
     - sqs
 
 - module: aws
   period: 86400s
   region: eu-west-1
   metricsets:
     - s3_daily_storage
     - s3_request
 EOM
 
 systemctl start metricbeat
 systemctl enable metricbeat

Furthermore, at some point, we would like to pass terraform attributes or variables to the script.

How do we do that?

templatefile() function to the rescue.

templatefile reads the file at the given path and renders its content as a template using a supplied set of template variables.

ec2.tf

# https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/
data "aws_ssm_parameter" "amazonlinux2" {
  name = "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
}

resource "aws_instance" "metricbeat" {
  ami                         = data.aws_ssm_parameter.amazonlinux2.value
  associate_public_ip_address = false
  iam_instance_profile        = aws_iam_instance_profile.ec2_metricbeat_instance_profile.name
  instance_type               = var.ec2_instance_type
  user_data = templatefile("${path.module}/metricbeat.tpl", {
  space = "${var.short_name}_${var.environment}", api_key = var.elk_api_key })  
  subnet_id = var.private_subnet
  timeouts { create = "30m" }

}

metricbeat.tpl

#!/bin/bash -xe
 cd /tmp/
 yum update -y
 curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.8.0-x86_64.rpm
 rpm -iv metricbeat-7.8.0-x86_64.rpm
 metricbeat modules disable system
 rm /etc/metricbeat/metricbeat.yml
 touch /etc/metricbeat/metricbeat.yml
 
 cat > /etc/metricbeat/metricbeat.yml << EOM
 ##################### Metricbeat Configuration Example #######################
 
 # =========================== Modules configuration ============================
 metricbeat.config.modules:
 # Glob pattern for configuration loading
   path: /etc/metricbeat/modules.d/*.yml
 
 # Set to true to enable config reloading
 reload.enabled: false
 
 # ======================= Elasticsearch template setting =======================
 setup.template:
 name: "metricbeat--inframetrics"
 pattern: "metricbeat--inframetrics-*"
 setup.ilm.enabled: true
 setup.ilm.rollover_alias: "metricbeat--inframetrics"
 setup.ilm.policy_name: "metricbeat-*" # should be ALWAYS <beatsname>-*
 setup.ilm.pattern: "{now/M{yyyy.MM}}-001"
 # =================================== Kibana ===================================
 setup.kibana:
   host: "https://"
   space.id: ${space}
 # ---------------------------- Elasticsearch Output ----------------------------
 output.elasticsearch:
   hosts: ["https://"]

 # Authentication credentials - either API key or username/password.
   api_key: ${api_key}
 # ================================= Processors =================================
 
 # Configure processors to enhance or manipulate events generated by the beat.
 processors:
   - add_host_metadata: ~
   - add_docker_metadata: ~
   - add_kubernetes_metadata: ~
 
 logging.level: debug
 EOM
 
 rm /etc/metricbeat/modules.d/aws.yml.disabled
 touch /etc/metricbeat/modules.d/aws.yml
 
 cat > /etc/metricbeat/modules.d/aws.yml << \EOM
 - module: aws
   period: 2m
   region: eu-west-1
   metricsets:
     - cloudwatch
   metrics:
     - namespace: AWS/SQS
 - module: aws
   period: 2m
   region: eu-west-1
   metricsets:
     - cloudwatch
   metrics:
     - namespace: AWS/ECS
 
 - module: aws
   period: 300s
   region: eu-west-1
   metricsets:
     - sqs
 
 - module: aws
   period: 86400s
   region: eu-west-1
   metricsets:
     - s3_daily_storage
     - s3_request
 EOM
 
 systemctl start metricbeat
 systemctl enable metricbeat

IaC - Deploying AWS EC2 user-data with Terraform

But how if the script is quite long?

How do we do that?

Thanks for visiting the blog.

See you in the next post.