Automate Zero Downtime Deployment with Amazon ECS and Lambda

Continuing the Continuous Integration and Continuous Deployment topic from my last post, this post will answer question like “How can you monitor the changes done to the EC2 Container Service?”, “How to automate deployment whenever an updated Docker Image is uploaded to the EC2 Container Registry?” and similarly monitor all API Calls done to EC2 Container Service.

The main focus of the post will be on Automating the Deployment Process but to make the post general, I will cover the basic steps to make your application live to the world with Amazon ECS and ELB.

Architecture design:-

Let’s start building the Architecture with Amazon ELB.

Elastic Load Balancer —Elastic Load Balancer distributes incoming application traffic across multiple EC2 instances which are attached to the Auto Scaling Group. For this post, I’ll be using Classic Load Balancer. My Docker Image will be running a simple “Hello World” Nginx Container on Port 80 of the Instance. So port 80 of the Instance is mapped to port 80 of the ELB. ELB provides us with a DNS name which a user can use to access the API. ELB performs a Health check on the attached Instances, ELB ping the instances withPing Path=/index.html & Ping Port= 80 and accordingly show the health check status of the Instances.

To create an ELB go to Load Balancer under the EC2 section of Amazon Web Services.

Click on Create Load Balancer and give the name to your ELB and assign a VPC. I am using Default VPC to reduce the complexity for those being new to these services.

Create ELB

Next, configure Security group for your application, see example below. I opened port 80 of ELB as my application will be running on port 80.

Configure Security Group

Configure Health Check is an important step, configure it as per your application requirement. Ping Path should be given a path which will give Status 200 OK response whenever the application is live.

On the next page Add Instances, leave the same as it is because Instances will be registered in ELB once the ECS Cluster is up.

Add Tags is optional but it helps to identify our services related to the Application.

Then click on Review and Create to Create the ELB.

Next, Setup the application on the EC2 Container Service.

Amazon ECS has three main components:-

1. Clusters
2. Task Definitions
3. Repositories

Upload the Docker Image to ECR

First, upload the docker image to ECR so that we can use it with Task Definition. Create a Repository on ECS and follow the push commands which appear after the Repo is created.

Create a Cluster:-

To run tasks through ECS, we place them on a cluster, which is a logical grouping of EC2 instances. Here we specify the Number of Instances, Instance Type, VPC and Security Group for our EC2 Instances. Create the cluster by going to ECS Console

Let’s keep the number of Instances 2 as I’ll be running 2 tasks.

Create a Task Definition:-

This is where we will give the path of our uploaded image to a container. Under the Add Container option, we can provide image location, port mapping, volume mount and many other specifications which we want to attach with our container. Additionally, we can specify a list of volumes that will be passed to the Docker daemon on a container instance. Do read about all the given options when setting up the container as it will help in deploying an application which perfectly suits your needs.

Add Service to the Task Definition

• Under the Service Section, we are provided with an option to attach ELB and Auto Scaling Group to our Task. A service lets you specify how many copies of your task definition to run and maintain in a cluster but attaching an Auto Scaling group gives us more options to define policies like Scale in and Scale out, so it's better to attach an Auto Scaling group.
• Assign your Service a name and also pass the name of Task Definition, Cluster and Number of tasks as 2
• Do read about Minimum healthy percent and Maximum percent as these two will decide the number of tasks running during the Deployment stage. Minimum healthy percent takes care of the minimum number tasks and similarly, Maximum Percent takes care of the Maximum number of tasks when a service gets updated. Assign Minimum Percent as 50 so when the service gets updated it will stop one task among the two running tasks and run a new task with updated service. Once this task is up and everything is running then it will stop the previous second task too so as to update it. Through this way, the service will get updated without any need of extra resources. Our aim is to automate this whole process whenever a new image is pushed to ECR. Give Maximum Percent = 200.
• There is a section for Task Placement under Create Service. Here we can specify two things:-
• Placement Strategy: - A placement strategy defines how tasks are placed by spread, bin pack, or random across Instances.
• Constraint: - Constraints allow you to filter the instances used for your placement strategies using built-in or custom attributes. Example:- `attribute:ecs.instance-type =t2.micro`
• Under Configure ELB , choose type as Classic Load Balancer and give ELB Name as the one which you created in the previous step. Configure Auto Scaling Group with proper Scale In and Scale Out policies.

Click on Create Service and the cluster will be up in few minutes.

So, now when the Application is up, let’s setup the Automation part.

Automate Deployment whenever Docker Image gets uploaded on ECR

EC2 Container does not write logs directly to CloudWatch but CloudTrail records API activity for ECS. The flow of automation will be like this: Whenever a Upload occurs to ECR, CloudTrail records PutImage event and can write it to CloudWatch Logs. An Alarm can be triggered whenever a PutImage event is written in CloudWatch Logs which can further trigger a Lambda Function through SNS. Lambda Function Contains the code to Create Task Definition Revision and Update Service with latest Task Definition.

Let’s get into detailed working and complete the Pipeline.

CloudTrail: - When CloudTrail logging is enabled, API calls made to Amazon ECS actions are tracked in log files. Amazon ECS records are written together with other AWS service records in a log file. CloudTrail delivers an event within 15 minutes of the API call. CloudTrail integration with CloudWatch Logs is enabled from the CloudTrail console by specifying a CloudWatch Logs log group and an IAM role. We need to create a Trail so that ECS activities can be recorded. Go to CloudTrail console and Create New Trail

Create New Trail

Click On Create and once it is created click on it and Configure CloudWatch Logs.

SNS: - CloudWatch Logs are configured to send a notification whenever an alarm is triggered for a specific API activity. SNS further triggers Lambda. Whenever the CW alarm updates from INSUFFICIENT_DATA to ALARM, the alarm publishes to a SNS topic. The SNS then triggers Lambda function to bring updates in ECS Cluster. Go to SNS and Create Topic.

CloudWatch: - Using logs, we can put CloudWatch alarms whenever a specific API activity occurs. Go to CloudWatch Logs, Select the Log Group you created in the earlier step and click on Create Metric Filter. Now we want to filter PutImage event for the Repository we Created in the previous step. Set the filter pattern as given below:-

{ ($.eventSource = ecr.amazonaws.com) && ($.eventName = PutImage) && ($.requestParameters.repositoryName = “<YourRepoName>”) && ($.errorCode NOT EXISTS) }

Go forward, set the rest parameters and Create Metric Filter. Once it is created, you’ll see an option to Create Alarm, click on it and fill in the details:-

CW Alarm for upload to ECR

Whenever a PutImage event occurs, one entry is added into this CloudWatch Filtered metric which further triggers a CloudWatch Alarm. CloudWatch alarm integrates with Simple Notification Service to trigger Lambda.

Lambda: - A lambda functions which is triggered by SNS is written in Python2.7 with the use of the Boto3 library. The Function first Create revision of the Task Definition and then update service to start tasks with revised task definition. The code for Lambda function is here. Set trigger for Lambda as SNS topic which we created in the previous step.

Finally, you are ready to test the Automated Deployment. Upload an updated Docker Image to the ECR and look closely to the changes happening in the ECS Cluster. You’ll notice tasks are updating one by one to serve updated Docker Image.
If you found the post helpful then please do Recommend/Clap :)