- Architecture Overview and Infrastructure Components
- ECR Runner Image Repository
- Self-hosted GitHub Runner(s) Registration Token
- Hosting the Runner Docker Artifacts on CodeCommit
- Build/Push Runner Image using CodeBuild
- Scalable ECS Cluster
- EventBus and Schema Discover for Webhook Events
- ECS Runner Task Definition
- Lambda Function URL
- GitHub Webhook
- EventBridge Rule
- Testing the Final Infrastructure
The following diagram shows the various architecture components in a sample solution for implementing GitHub self-hosted runners.
Green numbered squares are used to break down the architecture into manageable segments. This introductory post provides an overview of each segment, with the aim of addressing the finer build details in subsequent articles.
Self-hosted GitHub Runner(s) Registration Token
- Generation of token to allow ECS containers to register themselves as self-hosted runners within either a GitHub Enterprise/Organization/Repository
ECR Runner Image Repository
- Provision ECR repository for storing self-hosted running image
Hosting Runner Docker Artifacts on CodeCommit
- Create source code repository for the GitHub runner docker image artifacts
- The Dockerfile used to build the image would include the Installation of GitHub runner agent
Build and Push Runner Image using CodeBuild
- Create a CodeBuild project (using the CodeCommit repo from previous section as a source) to build and push the Github runner image to ECR
- Updates to runner agent are released regularly via GitHub’ official public repository: https://github.com/actions/runner
- Spawning a self-hosted runner using an outdated agent would result in the runner terminating with deprecation messages being issued
- To ensure the runner agent is kept up-to-date, include a trigger for the CodeBuild project to rebuild and push the image to ECR when new agent releases are available
Scalable ECS Cluster
- New dedicated cluster with Amazon ECS cluster auto scaling enabled
- Using this approach, scaling of EC2 instances is managed by Amazon
ECS Runner Task Definition
- Create ECS task definition for self-hosted runner
Event Bus and Schema Discover for Webhook Events
- Create an EventBus for receiving Github webhook events
- Enable schema auto discovery for the EventBus
- The events received on the Event bus are inferred into a type, and automatically added to a schema registry
Lambda Function URL
- Lambda Function URL provides an HTTP(s) endpoint which is used to configure a Github webhook (refer to next section)
- Invocation of the Lambda Function occurs for event types configured through the Github webhook
- The purpose of the function is to forward the event to Eventbridge, only after webhook payload has been successfully validated (in the case of scaling self-hosted runners, we forward through events related to workflows jobs)
GitHub Webhook
- Using the Lambda URL, configure an appropriate GitHub webhook – either Enterprise/Organization/Repository
Event Rule
- Create an EventBridge rule to invoke an ECS runner task (event target) when a workflow queued event is received
Testing the Final Infrastructure
- Create an organization repository with test workflows
- Ensure the runner name is set appropriately within each workflow definition
- Commit and push a change
- This should trigger an ECS ephemeral container for each workflow job