CI/CD from GitHub to AWS EC2

Simplified and updated instructions for setting up an automatic deployment pipeline, including tips and tricks

Photo by Christian Wiediger on Unsplash

Setting up AWS services had always been difficult for me. Without the paid developer support (too expensive for a small studio like mine), I had to rely on Google searches that often turn up results that are slightly dated (AWS continues to improve or add services, which is a good thing).

Recently I’ve gotten back to setting up a deployment pipeline on AWS. I had previously done this using CodePipeline + CodeDeploy + CodeCommit and gullibly thought that it should be familiar, but I still ended up spending days fiddling with things.

In this article I will compile the entire sequence of steps and include notes and tips where I can remember. I hope it will be helpful to you.

Setting up EC2

I will break up the steps into three sections — Setting up EC2, Setting up CodeDeploy and Setting up GitHub. In the Setting up EC2 segment, we will talk about starting up a new EC2 instance, pulling code manually from the GitHub repository, and running the application — in this article we will be running a Node.js web application.

1. Create an IAM role for EC2 (if you haven’t done so already)

Deployment via CodeDeploy needs some IAM roles to work, and one of it is AmazonEC2RoleforAWSCodeDeploy, which we need to attach to our EC2 instance.

Tip: I will advise you to do this first — I had assigned the IAM after my EC2 instance was already running, and wasted a lot of time debugging because I did not know that I had to reboot in order for the instance to recognise the newly attached IAM role.

This role only needs to be created once per account as it can be reused, so if you already have this you can skip to the next step.

Otherwise, please choose Services > IAM. Click on Roles > Create role.

Under Select trusted entity, check AWS service and under Use case, check EC2. Click Next.

Options for our new role that we need to attach to the EC2 instance. (image by author)

Under Add permissions, search for AmazonEC2RoleforAWSCodeDeploy. Click Next.

Give a Role name, e.g. CodeDeployForEC2. Click Create role.

2. Start an EC2 instance

Click Services > EC2. You might also want to take a quick glance at the top navigation bar and make sure that you are creating the instance in your intended region.

Tip: It is advisable to take a mental note of the region where you had started up your services. It is possible that a service is not available yet, or need to be in a specific region in order to work. Often, the configuration command is slightly different, which you will see in Step 7.

Click Launch instance.

i) Name and tags

Give your instance a name, e.g. staging.

ii) Application and OS Images

Select Amazon Linux (or the other options for your use case).

iii) Instance type

Select t2.micro if you have Free Tier. Otherwise you can select any of the other newer versions.

Tip: I recommend using a micro instance at least. I started with a nano (because I was setting up a staging pipeline), and it didn’t work well. The CodeDeployAgent (see later sections) froze every time I ran the status command, but this did not happen after I swapped to a micro instance.

iv) Key pair (login)

Click Create new key pair unless you wish to reuse an existing pair.

A quick Google search suggests that ED25519 is better, so I am using that instead.

For file format — you can select .pem if you are working on a Mac, or .ppk if you are working on a Windows.

Tip: Save this file in a safe location, possibly locations. It was quite a traumatic experience when I took over an EC2 with no keys and had to recreate one.

v) Network settings

Select Create security group unless you have an existing security group that you wish to reuse.

Under Allow SSH traffic from, change Anywhere to My IP.

Check Allow HTTPS traffic from the internet and Allow HTTP traffic from the internet, because we are creating a web server in this example.

vi) Advanced details

Under IAM instance profile, select the IAM role, CodeDeployForEC2, that we had created in Step 1 above.

Click Launch Instance.

The instance should appear in the EC2 > Instances page, and it should say Running under the instance state column.

vii) Verify the instance

From the EC2 > Instances page, click on your newly created instance to view the Instance summary.

Click on the Security tab and click on the Security group that’s attached to your instance. Make sure that HTTP and HTTPS are allowed on their respective ports for all incoming traffic.

Click Edit inbound rules and add one more TCP rule for the port that your application is listening to.

Important note: you will need to add another custom TCP rule at the port that your Node.js application is listening to. In my example, it is 3000. Without this step, your website will not display, even if the server is running correctly.

Ensure that you have one more TCP rule for the port your application is listening to. (Image by author)

3. Log in and Configure your instance

i) Change the permissions of the key:

chmod 600 my_key.pem

Tip: I would advise you to create a duplicate of the key before this step, just in case the key gets damaged/corrupted. It sounds paranoid, but it had happened to me years ago during a key conversion from .ppk to .pem. Also, this step is very necessary, before you even try to log in for the first time.

ii) Log into your instance (from a CLI):

ssh -i my_key.pem ec2-user@123.123.123.123

123.123.123.123 is just an example dummy IP. You can find your instance’s IP address from Public IPv4 address in the Instance Summary screen.

iii) Do an update

Update your instance by running the following commands:

sudo yum update

sudo yum upgrade

iv) Install Node.js on the instance

AWS has a documentation for this step, which is the reference I’m using.

First, install node version manager (nvm):

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

Activate nvm:

. ~/.nvm/nvm.sh

Install the latest version of node.js:

nvm install --lts

Verify your node.js installation:

node -v

Also verify that npm (node package manager) is installed:

npm -v

4. Manual Git clone

At this point our EC2 instance is initialised. We will do a manual git clone first to make sure that the EC2 instance is working well before introducing the other complicated steps.

I assume that you already have a Node.js project in GitHub, otherwise you may wish to refer to my earlier article.

i) Install git in the instance

sudo yum install -y git

ii) Clone your repository

Navigate to the root folder:

cd /home/ec2-user

Clone the repository:

git clone https://github.com/mygithubaccount/myrepo.git

The URL here is a dummy. You can find your URL from your GitHub repository page. Click on the green <> Code button > HTTPS.

A prompt will appear that asks for your GitHub username and password. The password here is a Personal access token. Do not enter the password that you use to log into GitHub.

Tip: Password access via the CLI is no longer allowed by GitHub, we will need to create and use a personal token instead. You can refer to my other article, GitHub Passwords on the CLI, here.

iii) Run the app

Navigate to your newly cloned repository folder, e.g.

cd myrepo

Install necessary packages:

npm install

Run the app.js file:

node app.js

A print out should appear in the CLI (if you have added console.log in your code, like I did).

You can also view your page on the browser via http://123.123.123.123:3000, where 123.123.123.123 is your instance’s public address and 3000 is your application’s port number. Take note that at this point you can only use http and not https.

If your page does not load, please ensure that your security group has a rule for your application’s port number.

5. Create Elastic IP

An Elastic IP ensures that the public IP address of your instance remains the same when your instance is rebooted.

Under Services > EC2 > Network & Security, click on Elastic IPs. Click Allocate Elastic IP address.

After a new Elastic IP is created for you, click on it to view its summary and click on Associate Elastic IP address.

Select instance under Resource type, and choose your newly created instance.

Check Allow this Elastic IP address to be reassociated. Click Associate.

Tip: Take note that any Elastic IP address that is not associated to any instance will be charged.

Associating your Elastic IP (image by author)

Go back to your EC2 instance summary page. Notice that the public IP address has now changed to your Elastic IP. You can now use this address when you log in via SSH or view your page through the browser.

6. Install process manager, pm2

pm2 is the process manager for Node.js that helps to ensure that our application can automatically restart itself when the instance is rebooted.

You can refer to my other article, Installing pm2 for Node.JS, for more detailed instructions on how to set it up.

7. Install the CodeDeploy Agent

This step is more straightforward, and I will just include the link from AWS’ documentation for your reference.

Tip: You need to be very careful and double check your region identifier accordingly, before you run the command to get the install file.

Setting up CodeDeploy

In this stage, we talk about setting up CodeDeploy — we will create a CodeDeploy application and trigger a manual deployment from our GitHub repository to our EC2 instance.

8. Create another IAM role for CodeDeploy (if you haven’t done so already)

We need another IAM role, AWSCodeDeployRole, for the deployment to work. You can skip this step if your account already has this role, as it can be reused. Otherwise, please continue with the following:

i) Create Role

Click Services > IAM. Click Roles > Create role.

Under Select trusted entity, check AWS service and under Use case, check EC2. Click Next.

Search for AWSCodeDeployRole, click Next.

Give a role name, e.g. CodeDeployRole and click Create role.

ii) Change Trust Relationship

Click on the newly created role to open up the Role summary.

Click the Trust relationships tab. Click Edit trust policy and change the value for “Service”:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Principal": {
    "Service": "codedeploy.amazonaws.com"
   },
   "Action": "sts:AssumeRole"
  }
 ]
}

Click Update policy.

9. Add scripts to your project

We will need to add a few scripts in the root of our project for CodeDeploy to pick up the configuration. One of them is the CodeDeploy AppSpec file.

i) Create appspec.yml in the root of your project

version: 0.0
os: linux
files:
  - source: /
    destination: /home/ec2-user/myrepo
hooks:
# Install:
  AfterInstall:
    - location: scripts/after_install.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: scripts/application_start.sh
      timeout: 300
      runas: root
# ValidateService:

This file tells CodeDeploy where to look for our source code (ensure that your path is correct), and what to do after installation and how to start the application. As you can see, it is looking for two .sh scripts in the /scripts folder that we will create in the next steps below:

ii) Create /scripts/after_install.sh

#!/bin/bash
echo 'run after_install.sh: ' >> /home/ec2-user/myrepo/deploy.log

echo 'cd /home/ec2-user/myrepo' >> /home/ec2-user/myrepo/deploy.log
cd /home/ec2-user/myrepo >> /home/ec2-user/myrepo/deploy.log

echo 'npm install' >> /home/ec2-user/myrepo/deploy.log 
npm install >> /home/ec2-user/myrepo/deploy.log

This script tells the program to go to the source code folder and install any necessary node.js packages.

iii) Create /scripts/application_start.sh

#!/bin/bash

echo 'run application_start.sh: ' >> /home/ec2-user/myrepo/deploy.log
# nodejs-app is the same name as stored in pm2 process
echo 'pm2 restart nodejs-app' >> /home/ec2-user/myrepo/deploy.log
pm2 restart nodejs-express-app >> /home/ec2-user/myrepo/deploy.log

Note: Please ensure that you have replaced myrepo to your folder’s name. Also notice that nodejs-app is the same name that we have used in the pm2 process earlier.

Push the scripts to your GitHub repository.

10. Create a deployment

Go to Services > CodeDeploy. Click Getting started > Create application.

Creating a new CodeDeploy application. (image by author)

11. Create a deployment group

After your CodeDeploy application is created, it should automatically route you to the Application details screen as follows:

CodeDeploy application details (image by author)

Click Create deployment group.

Enter a deployment group name, e.g. nodejs-app-staging.

Enter a service role. The role that we have created earlier in Step 8 should automatically appear as an option when the field is clicked.

Select your Deployment type. I use In-place.

Under Environment configuration, select Amazon EC2 instances. Under Tag group 1, Enter Name for Key and the name of your instance for Value, e.g. staging from Step 2i) above.

Leave the default settings under Agent configuration with AWS Systems Manager.

Select your preference under Deployment settings. I use CodeDeployDefault.AllAtOnce, but it doesn’t really matter here because I only have one instance.

I also uncheck Enable load balancing under Load balancer.

Click Create deployment group.

12. Test a deployment

After the Deployment group is successfully created, you should be automatically routed to the Deployment group details page.

We can create a manual deployment to test our CodeDeploy has been correctly set up.

i) Making changes and Getting the Commit ID

In order to see that the deployment is working, please make a change to your code like changing the Hello World text to Greetings. Ensure that you have committed the changes to the repository.

Tip: Please make the changes to a visible part of the project. console.log statements will not be visible if your server is running with pm2.

From your GitHub screen, click on the latest commit’s hash.

The commit hash (image by author)

In this example, it’d be the e9ef5dd string. Click on this string, and you should see details about the changes you have done. Copy the full hash from the url address, which looks something like this:

https://github.com/mygithubaccount/myrepo/commit/e9ef5ddabcdefghijklmnopqrstu

The hash will be the text after commit/. Take note of this hash and we will be using it in the next step.

ii) Create deployment

Click Create deployment.

Under Deployment group, select our newly created Deployment group from the step above, nodejs-app-staging.

Under Revision type, select My application is stored in GitHub. Because this is the first time we are connecting to GitHub here, we will need to enter our GitHub username into the text field. Click Connect to GitHub.

When the application is successfully bound to the GitHub token, enter your repository name. Take note that it is in the format GitHub_user/repo_name like this:

mygithubaccount/myrepo

Paste the hash from i) above into the Commit ID field.

Under Additional deployment behaviour settings, check Overwrite the content.

Click Create deployment.

We should be automatically routed to the Deployments page and be able to see the deployment progress and result.

Refresh your browser to view your updates!

iii) Troubleshooting

Unfortunately, it is very likely that we will run into errors, especially if this is the first time we are setting up. To troubleshoot, we can log into the instance via SSH and view the log data for CodeDeploy.

Common errors will be codedeploy agent is not found. This is due to the missing IAM during the EC2 startup step.

At this point you should be able to deploy from your Github repo, albeit manually. Let’s proceed to the last step, Setting up GitHub, and look at using Github Actions to trigger the deployment automatically.

Setting up GitHub

We will be largely referring to the documentation provided by AWS.

13. Create a IAM identity provider for GitHub

Click Services > IAM. Click on Identity providers > Add provider.

Under Configure provider > Provider type, select OpenID Connect.

Under Provider URL, enter:

https://token.actions.githubusercontent.com

and click Get thumbprint.

Under Audience, enter:

sts.amazonaws.com

Click Add provider.

14. Create a IAM role for GitHub

Click on our newly created Identity provider and click on Assign role.

Select Create a new role and click Next.

Create a new role for our Identity provider (image by author)

You will notice that some of the fields are already pre-filled for you.

Under Audience, select sts.amazonaws.com.

Under GitHub organisation, enter your GitHub username.

Click Next.

In the Add permissions page, check AWSCodeDeployFullAccess, click Next.

Under the Create role page, enter a Role name, e.g. GitHubAction-AssumeRoleWithAction.

Edit the trusted entities json to restrict repository access:

"StringLike": {
    "token.actions.githubusercontent.com:sub": [
          "repo:mygithubaccount/myrepo:ref:refs/heads/main"
    ]
}

You can change /main to /development if you are setting up for the development branch.

IMPORTANT NOTE: If you are planning to set this up for multiple branches, change /main to /* at this step, before the role is created. It is not possible to simply edit the role later — you will need to delete the identity (even though the ARN will remain the same), and repeat the steps.

Click Create role.

13. Create the workflows script

Go to GitHub > Select your repository and Click Add File.

Create a file at .github/workflows/deploy.yml:

# This is a basic workflow to help you get started with Actions
# name:Connect to an AWS role from a GitHub repository

# Controls when the action will run. Invokes the workflow on push events but only for the main branch
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  
  AWS_REGION : "ap-southeast-2" #Change to reflect your Region

# Permission can be added at job level or workflow level    
permissions:
      id-token: write   # This is required for requesting the JWT
      contents: read    # This is required for actions/checkout
jobs:
  AssumeRoleAndCallIdentity:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v3
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@v1.7.0
        with:
          role-to-assume: arn:aws:iam::123456789:role/GitHubAction-AssumeRoleWithAction #change to reflect your IAM role’s ARN
          role-session-name: GitHub_to_AWS_via_FederatedOIDC
          aws-region: ${{ env.AWS_REGION }}
      # Hello from AWS: WhoAmI
      - name: Sts GetCallerIdentity
        run: |
          aws sts get-caller-identity

      # Step 3 - check the application-name and deployment group name
      - name: Create CodeDeploy Deployment
        id: deploy
        run: |
          aws deploy create-deployment \
          --application-name nodejs-app \ 
          --deployment-group-name nodejs-app-staging \
          --deployment-config-name CodeDeployDefault.AllAtOnce \
          --github-location repository=${{ github.repository }},commitId=${{ github.sha }}

There are a few fields that you need to change:

i) branch — if you are setting up a staging server, you might want to use another branch e.g. development instead,

ii) AWS REGION,

iii) role-to-assume — copy the ARN of your role created in Step 14,

iv) the application-name and deployment-group-name as in Steps 10 and 11 above.

14. Testing your setup!

Finally, you are ready to test your auto deployment. Push some changes to your repository.

From your GitHub page, click on your repository and click Actions.

You should be able to see the progress and result of the deployment. Refresh your browser to see the updates!

References:

Connect to your Linux instance from Linux or macOS using SSH

Tutorial: Setting Up Node.js on an Amazon EC2 Instance

Install the CodeDeploy agent for Amazon Linux or RHEL

View log data for CodeDeploy EC2/On-Premises deployments

Use IAM roles to connect GitHub Actions to actions in AWS | Amazon Web Services

GitHub - saasscaleup/nodejs-aws-codedeploy-pipeline