ECS Runner Task Definition

Tony Tannous

/

This entry is part 8 of 12 in the series Scalable Self-Hosted GitHub Runners on AWS Cloud

Scalable Self-Hosted GitHub Runners on AWS Cloud

Architecture Overview and Infrastructure Components

ECR Runner Image Repository

Self-hosted GitHub Runner(s) Registration Token

Hosting the Runner Docker Artifacts on CodeCommit

Build/Push Runner Image using CodeBuild

Scalable ECS Cluster

EventBus and Schema Discover for Webhook Events

ECS Runner Task Definition

Lambda Function URL

GitHub Webhook

EventBridge Rule

Testing the Final Infrastructure

The ECS task definition for the runner depends on the following resources:

  1. Store the runner registration token in AWS SecretManager
  2. Task IAM role
  3. Task Execution IAM role
  4. Runner image has been pushed to ECR

Store Runner Registration Token in AWS SecretsManager

Store the runner registration token, in Secretsmanager with a name github-actions-runner-registration-token:

 aws --region us-east-1 secretsmanager create-secret \
	 --name github-actions-runner-registration-token \
	 --description "GitHub Actions PAT used for self-hosted runner registrations" \
	 --secret-string "XXXXXXXXX"

Task IAM role

Using the following trust policy $HOME/runner_task_role_trust_policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

create the task role with name github-actions-self-hosted-runner-debian-iam-task-role:

aws iam create-role \
    --role-name github-actions-self-hosted-runner-debian-iam-task-role \
    --assume-role-policy-document file://$HOME/runner_task_role_trust_policy.json

Create a file for storing the IAM policy definition :

$HOME/github-actions-self-hosted-runner-debian-iam-task-role-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:Describe*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Attach this policy, assigning it a name of github-actions-self-hosted-runner-debian-iam-task-role-policy, to the role.

aws iam put-role-policy \
    --role-name github-actions-self-hosted-runner-debian-iam-task-role \
    --policy-name github-actions-self-hosted-runner-debian-iam-task-role-policy \
    --policy-document file://$HOME/github-actions-self-hosted-runner-debian-iam-task-role-policy.json

Task Execution IAM role

Create the following file, containing the trust policy definition.

$HOME/runner_task_execution_role_trust_policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Create the role with name github-actions-self-hosted-runner-debian-iam-task-execution-role and attach the trust policy

aws iam create-role \
    --role-name github-actions-self-hosted-runner-debian-iam-task-execution-role \
    --assume-role-policy-document file://$HOME/runner_task_execution_role_trust_policy.json

Create a file with the following policy definition:

$HOME/github-actions-self-hosted-runner-debian-iam-task-execution-role-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxx:secret:github-actions-runner-registration-token-??????"
            ]
        }
    ]
}

Attach this policy, naming it github-actions-self-hosted-runner-debian-iam-task-execution-role-policy, to the role.

aws iam put-role-policy \
    --role-name github-actions-self-hosted-runner-debian-iam-task-execution-role \
    --policy-name github-actions-self-hosted-runner-debian-iam-task-execution-role-policy \
    --policy-document file://$HOME/github-actions-self-hosted-runner-debian-iam-task-execution-role-policy.json

ECS Task Definition

Create a task definition for an ephemeral runner using the sample $HOME/runner_task_definition.json shown below.

{
    "family": "github-actions-self-hosted-runner-debian-task",
    "containerDefinitions": [
        {
            "name": "github-actions-self-hosted-runner-debian-container",
            "image": "xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/github-actions-self-hosted-runner-debian:latest",
            "cpu": 0,
            "portMappings": [],
            "essential": true,
            "environment": [
                {
                    "name": "ORGANIZATION",
                    "value": "foo-organisation"
                },
                {
                    "name": "RUNNER_CONFIG_ARGS",
                    "value": "--unattended --replace --disableupdate --no-default-labels --labels <detail_org>-target_env-<detail_repo_name>-<wf_job_run_id>-<wf_job_run_attempt> --name <detail_org>-target_env-<detail_repo_name>-<wf_job_run_id>-<wf_job_run_attempt> --runnergroup <detail_org>-target_env --ephemeral"
                },
                {
                    "name": "RUNNER_LABELS",
                    "value": ""
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "scratch",
                    "containerPath": "/var/lib/docker"
                }
            ],
            "volumesFrom": [],
            "linuxParameters": {
                "initProcessEnabled": true
            },
            "secrets": [
                {
                    "name": "RUNNER_ADMIN_TOKEN",
                    "valueFrom": "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxx:secret:github-actions-runner-registration-token"
                }
            ],
            "stopTimeout": 120,
            "privileged": true,
            "dockerLabels": {
                "Env": "sandpit",
                "Name": "github-actions-self-hosted-runner-debian"
            },
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "github-actions-self-hosted-runner-debian-container",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "self-hosted-runner-debian"
                }
            },
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "[ $(curl -s -I  https://api.github.com | grep -i '^x-github-request-id' | cut -d ':' -f 2- | sed 's/\\n//g; s/\\s//g' | wc -m) -gt 0 ] || echo failed"
                ],
                "interval": 300,
                "timeout": 60,
                "retries": 3,
                "startPeriod": 120
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/github-actions-self-hosted-runner-debian-iam-task-role",
    "executionRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/github-actions-self-hosted-runner-debian-iam-task-execution-role",
    "volumes": [
        {
            "name": "scratch",
            "dockerVolumeConfiguration": {
                "scope": "task",
                "driver": "local",
                "labels": {
                    "scratch": "space"
                }
            }
        }
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "256",
    "memory": "512",
    "tags": [
        {
            "key": "Env",
            "value": "sandpit"
        },
        {
            "key": "App_Category",
            "value": "ecs_task"
        }
    ]
}

Listed below are the key parameters/values that appear in the task definition:

  • image: ECR image URI
  • environment variables:
    • ENTERPRISE: name of Enterprise (used only for Enterprise runners)
    • ORGANIZATION: name of the GitHub organization (used for Organization runners)
    • REPOSITORY: name of Repository (use for Repository runners)
    • RUNNER_CONFIG_ARGS: a single string of arguments/values to pass to the runner’s ./config command, i.e:
      • name: name of the new runner
      • runnergroup: group that the new runner should join
      • labels: array of custom labels to assign to the runner
      • work_folder: working directory (relative to where the runner agent has been installed to)
      • no-default-labels: do not assign default labels (such as self-hosted) to the runner
      • ephemeral: this is an ephemeral runner
      • replace: if a runner with the same name/label exists, then replace it
      • disable: prevent automatic updates of the runner agent
    • RUNNER_ADMIN_TOKEN: secret that contains the runner registration token
  • taskRoleArn: IAM task role arn
  • executionRoleArn: IAM task execution role arn
  • initProcessEnabled: set to true, to enable graceful termination of containers on ECS-optimised instances
  • the task vCPU/memory combination, respectively chosen as 256MB and 512MB, need to be allocated in context with the EC2 instance size(s)—spawning several tasks with larger allocations may exceed the total memory capacity of the target instance(s), and cause containers to fail with out-of-memory errors

Note: The following parameter has been set to to true within the task definition to allow for docker-in-docker support, with each container runner instance hosting its own docker daemon.

"privileged": true,

This introduces obvious security implications and is only used in examples throughout for demonstration purposes. More emphasis on security hardening is required before adopting this approach within a controlled production environment setting

Register the ECS task:

aws ecs register-task-definition \
    --cli-input-json file://$HOME/runner_task_definition.json

Scalable Self-Hosted GitHub Runners on AWS Cloud

EventBus and Schema Discover for Webhook Events Lambda Function URL