Buildkite Docker Image with Mock AWS Endpoints

Some reasons for opting to build a custom Buildkite agent image, as opposed to using the Docker hub release:

Running the agent container with userid other than root
Including additional tools, such as the AWS CLI, Terraform
Agent container can be networked with other services as part of a local Docker Compose stack, for example, mock AWS standalone Moto server
With this setup, you can achieve an integrated virtual cloud/Buildkite environment for testing

Image OS

The agent Docker image is derived using the following basic details:

Base OS: ubuntu:22.04
Agent container user is identical to the current host user
docker group is also created as a mirror of the host docker group
Bundles in the AWS CLI

Clone Sample Repo

To get up and running, use the following repository as a starting point. It contains all the required files.

$ cd $HOME
$ git clone https://github.com/tonys-code-base/buildkite-agent-custom.git

Building the Image

Build the image:

$ cd $HOME/buildkite-agent-custom/docker

$ docker build -t buildkite-custom-agent \
--build-arg="USER_ID=$(id -u $(whoami))" \
--build-arg="GROUP_ID=$(id -g $(whoami))" \
--build-arg="HOST_USERNAME=$(whoami)" \
--build-arg="TARGETOS=linux" \
--build-arg="TARGETARCH=amd64" \
--build-arg="AGENT_USER_HOMEDIR=/home/$(whoami)" \
--build-arg="DOCKERHOST_GID=$(getent group docker | cut -d: -f3)" .

Buildkite Agent Token

Login to Buildkite and create an organization
Retrieve the agent token
Agents > Docker > “Reveal Agent Token“

Run Local Agent with Docker Compose

The docker-compose.yml provided within the repository includes definitions for:

Moto standalone server (motoserver), accessible from other containers via http://motoserver:5000
Buildkite agent (buildkite-agent)

AWS credentials and region are passed to buildkite-agent via environment variables., i.e

  - AWS_ACCESS_KEY_ID=testing
  - AWS_SECRET_ACCESS_KEY=testing
  - AWS_SECURITY_TOKEN=testing
  - AWS_SESSION_TOKEN=testing
  - AWS_DEFAULT_REGION=us-east-1

version: "3.8"

services:
  motoserver:
    image: motoserver/moto:latest
    container_name: moto
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "5000:5000"
    networks:
      - buildkite-net
    environment:
      - MOTO_PORT=5000

  buildkite-agent:
    image: buildkite/agent:ubuntu-lts
    container_name: buildkite-agent
    user: ${HOST_USERNAME}
    volumes:
      - /home/${HOST_USERNAME}/.ssh:/home/${HOST_USERNAME}/.ssh
      - /home/${HOST_USERNAME}/.aws:/home/${HOST_USERNAME}/.aws
      - /var/run/docker.sock:/var/run/docker.sock
      - ./buildkite:/home/${HOST_USERNAME}/.buildkite-agent
    networks:
      - buildkite-net
    environment:
      - BUILDKITE_AGENT_TOKEN=${BUILDKITE_AGENT_TOKEN}
      - BUILDKITE_BUILD_PATH=/home/${HOST_USERNAME}/.buildkite-agent/builds
      - BUILDKITE_AGENT_NAME=buildkite-agent-ubuntu-jammy
      - BUILDKITE_BIN_PATH=/home/${HOST_USERNAME}/.buildkite-agent/bin
      - BUILDKITE_HOOKS_PATH=/home/${HOST_USERNAME}/.buildkite-agent/hooks
      - BUILDKITE_PLUGINS_PATH=/home/${HOST_USERNAME}/.buildkite-agent/plugins
      - AWS_ACCESS_KEY_ID=testing
      - AWS_SECRET_ACCESS_KEY=testing
      - AWS_SECURITY_TOKEN=testing
      - AWS_SESSION_TOKEN=testing
      - AWS_DEFAULT_REGION=us-east-1
networks:
  buildkite-net:
    name: buildkite-net

Pipelines can access the AWS moto-hosted mock services by passing "--endpoint-url http://motoserver:5000“. We will see how this works by walking through an example in the next section.

Run the agent, replacing “<your buildkite agent token>“ with your agent token:

$ cd $HOME/buildkite-agent-custom
$ HOST_USERNAME=$(whoami) BUILDKITE_AGENT_TOKEN="<your buildkite agent token>" docker compose up -d

Accessing AWS Mock Services from Buildkite Agent

The following sections describe the procedure for setting up a pipeline to create an S3 bucket, s3://mytestbucket, using our mock Moto server.

Before starting, ensure that pipelines are set to use YAML steps.

Create Pipeline

You can either create a new pipeline, or edit Buildkite’s “Starter Pipeline” for testing purposes.
To use the “Starter Pipeline”, navigate to “Pipelines > Starter Pipeline > Settings > Steps“
Add the following code in for Steps

steps:
  - label: ":buildkite: Upload steps"
    command: "buildkite-agent pipeline upload .buildkite/pipeline.yml"
  - label: ":buildkite: create s3 bucket"
    command: "aws --endpoint-url http://motoserver:5000 s3 mb s3://mytestbucket"

Execute the pipeline by selecting “Save and Build”
When prompted to provide a message, enter “test local agent“, followed by “Create build”

If the pipeline executes successfully, you should expect to see the following in the output log:

Inspecting the Moto container logs should also confirm the Motoserver mock endpoint was also

$ docker logs moto

output:

WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
....
"PUT /mytestbucket HTTP/1.1" 200 -