CI/CD for Robotics: GitHub Actions, Merge Queue, Docker

The pipeline is the product. A robot you can't reliably ship is a liability, and a CI that takes 25 minutes on every PR is a pipeline nobody runs. This post is the anatomy of the otonav CI/CD stack: one GitHub Actions workflow, one native merge queue, one multi-stage Dockerfile, one release pipeline to GHCR. Twelve PRs went through it. Two were deliberately broken to prove the gate holds. One was the first to ride the merge queue live.

This is Part 3 of 7. Part 1 made the case for SIL. Part 2 covered the architecture. Here we go inside the machines.

CI Workflow Anatomy

One workflow, three triggers, two jobs. Everything lives in .github/workflows/ci.yml. The triggers are the contract:

name: CI
on:
  pull_request:
    branches: [main]
  merge_group:
  workflow_dispatch:
permissions:
  contents: read
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

pull_request runs on every PR opened against main. merge_group runs when a PR is admitted to the merge queue — this is the real gate, because it's the last CI that runs before the commit lands on main. workflow_dispatch is the escape hatch for manual reruns without pushing an empty commit. The concurrency block cancels superseded runs on the same ref so a force-push doesn't burn minutes.

Two jobs: pre-commit (advisory) and lint-build-test (the required gate). Only lint-build-test is a required status check.

The lint-build-test job

Runs in a container, not in a Docker build. That decision is the single most important CI design choice in the whole project, and we'll come back to it. The job declaration:

  lint-build-test:
    runs-on: ubuntu-22.04
    container: ros:humble-ros-base
    timeout-minutes: 10
    env:
      MUJOCO_VERSION: "3.3.7"
      MUJOCO_DIR: /opt/mujoco-3.3.7
      CCACHE_DIR: ${{ github.workspace }}/.ccache
      ROS_LOCALHOST_ONLY: "1"
      RMW_IMPLEMENTATION: rmw_fastrtps_cpp

timeout-minutes: 10 is Hard Rule #8, encoded as YAML. If a run ever exceeds 10 minutes, GitHub kills it and the PR goes red. There is no soft warning. The env block pins MuJoCo 3.3.7, isolates ROS to localhost-only (no DDS leaking onto the runner network), and forces FastRTPS as the RMW so we don't accidentally test Cyclone on luck.

The 9 steps

After the test step, two upload steps follow: JUnit XML (always — feeds the GitHub test panel and the merge queue's ALLGREEN check), and a rosbag upload on if: failure(). The rosbag is the post-mortem: when SIL breaks in CI, you don't reproduce locally, you download the bag.

flowchart TD
    A[PR opened/updated] --> B[pre-commit job]
    A --> C[lint-build-test job]
    C --> C1[container: ros:humble-ros-base]
    C1 --> C2[checkout]
    C2 --> C3[cache MuJoCo SDK]
    C3 --> C4[cache ccache]
    C4 --> C5[rosdep install]
    C5 --> C6[colcon build Release]
    C6 --> C7[colcon test]
    C7 -->|pass| C8[upload JUnit]
    C7 -->|fail| C9[upload rosbag]
    C8 --> D{required check}
    D -->|green| E[PR eligible for merge queue]
    D -->|red| F[PR blocked]

    style C7 fill:#fdd,stroke:#c00
    style D fill:#dfd,stroke:#0a0
    style F fill:#fdd,stroke:#c00

Budget reality: ~80 seconds cached, ~6 minutes cold. The first PR after a cache eviction is the cold run; every PR after that rides the ccache hit. The 10-minute ceiling has never been hit on a green build. PR #6 (deliberately broken test) and PR #8 (deliberately broken gain) both ran the test step to completion and failed fast — the budget is for builds, not for infinite hangs.

Why container: Instead of docker build

The obvious pattern is wrong for CI. A naive robotics CI does docker build -f Dockerfile . and runs tests inside the built image. I rejected that for three reasons:

1. Direct artifact access. With container:, the workspace is just a directory. actions/upload-artifact sees the JUnit XML, the rosbag, the ccache dir, the build logs — no docker cp, no volume gymnastics. The rosbag-on-failure upload is a one-liner.
2. Faster iteration. docker build rebuilds every layer above the changed one. With container:, the base image is pulled once and cached by the runner; the build happens in it, with ccache spanning runs. The cold-to-warm ratio is ~4.5× in favor of container:.
3. Test what you ship. The container: base is ros:humble-ros-base. The release Dockerfile's builder stage is also FROM ros:humble-ros-base. Same apt universe, same colcon, same MuJoCo. The CI environment and the release build environment are the same image — the only difference is what we do in it.

The Dockerfile still exists (see below) — it's for the release artifact, not for CI. CI builds in the same base; release ships a slimmed image built from that base.

Caching Strategy

Two caches, two keys, one purpose: stay under 10 minutes.

MuJoCo SDK cache. Keyed mujoco-3.3.7-linux-x86_64 — a fixed string, because the SDK version is pinned in env. A cache miss costs ~30 seconds (download + extract). A hit is free. This is a actions/cache@v4 step with path: /opt/mujoco-3.3.7 and the fetch step gated on if: steps.cache-mujoco.outputs.cache-hit != 'true'.

ccache cache. Keyed ccache-humble-${{ github.sha }} with restore-keys: ccache-humble-. The precise key means each commit gets its own cache slot after the run; the restore-key prefix means a new commit restores from the most recent matching cache (its parent commit, usually) before the build. This is the trick that gets us to 80 seconds: a warm ccache turns colcon build from a 4-minute C++ compile into a 20-second link.

flowchart LR
    A[new PR commit sha=abc] --> B[restore ccache-humble-*]
    B --> C[ccache hit ratio ~90%]
    C --> D[colcon build 80s]
    D --> E[write ccache-humble-abc]
    E --> F[next PR sha=def]
    F --> G[restore ccache-humble-abc]
    G --> D

    style C fill:#dfd,stroke:#0a0
    style D fill:#dfd,stroke:#0a0

Why the 10-minute budget matters more than it looks. This CI runs on every PR and every push to a PR. With 12 PRs in the project and an average of 4 pushes per PR, that's ~50 runs. At 25 minutes each, that's 20+ hours of CI time and a contributor-waiting-on-CI tax that kills momentum. At 80 seconds, you check CI like you check Slack.

Merge Queue — the 3-Act War Story

Native merge queue is the single biggest CI reliability win in this project. It's also the one that fought back the hardest. The story is in three acts.

Act 1: Personal private repo → REST API 422

I enabled the merge queue via the GitHub REST API: PUT /repos/{owner}/{repo}/merge-group with the documented body. Response: 422 Unprocessable Entity, body {}. No error message. No field name. Just an empty object and a status code. I spent an afternoon convinced I had the schema wrong.

Act 2: Made the repo public — still 422

I flipped the repo to public, thinking it was a visibility constraint. Same 422. Same empty body. At this point I filed the fallback: documented the behavior in an ADR, kept the branch protection ruleset active (required check on lint-build-test), and accepted that PRs would merge directly on green without queue serialization. The gate still held — it just wasn't serialized.

Act 3: Transferred to an org — it worked

I moved the repo under an organization. Same API call. 200 OK. UI and REST both accepted the configuration. The constraint was account type, not visibility. Native merge queue requires an org-owned repo. This is undocumented in the error path (the empty 422 body is the bug), but it's the behavior.

PR #12 was the first PR to pass through the queue. Its merge_group CI run fired — visible in the Actions log — and the PR merged via the queue, not via the direct merge button. That run is the live proof that ADR-6 (merge queue adoption) is implemented, not just designed.

sequenceDiagram
    participant PR as PR #12
    participant Q as Merge Queue
    participant CI as merge_group CI
    participant main as main branch
    PR->>Q: Add to queue (green PR)
    Q->>CI: trigger merge_group event
    CI->>CI: lint-build-test on merge_group ref
    CI-->>Q: green (ALLGREEN)
    Q->>main: SQUASH merge
    main-->>PR: closed (merged)

Merge queue settings

Branch Protection Ruleset

The gate is active even with zero approvals. I'm the sole maintainer, so required_approving_review_count: 0. But the required status check lint-build-test is non-negotiable, and so are the structural rules:

• main: PR required, required check lint-build-test, no force-push, no deletion
• allow_force_pushes: false
• allow_deletions: false

I tested this directly: git push origin main on a local commit. GitHub rejected it with refusing to allow an OAuth App to push updates to main without a pull request. The gate works at the transport layer, not just the UI layer. That matters — it means a misconfigured local git client can't bypass the pipeline by accident.

Multi-Stage Docker

The release image is two stages: a fat builder and a slim runner. The builder is the same base as CI; the runner is ros:humble-ros-core plus only what the built artifacts need.

flowchart TD
    B[FROM ros:humble-ros-base AS builder] --> B1[install build-essential cmake wget colcon rosdep libgl1 libglew-dev]
    B1 --> B2[download MuJoCo SDK 3.3.7]
    B2 --> B3[COPY 5 ROS packages]
    B3 --> B4[rosdep install --from-paths src]
    B4 --> B5[colcon build --merge-install]
    B5 --> B6[colcon test unit]
    B6 --> R
    R[FROM ros:humble-ros-core AS runner] --> R1[install libgl1 runtime]
    R1 --> R2[COPY MuJoCo libs from builder]
    R2 --> R3[COPY /ws/install from builder]
    R3 --> R4[COPY entrypoint.sh]
    R4 --> R5[ENTRYPOINT /entrypoint.sh]
    R5 --> R6[CMD ros2 launch otonav_bringup ...]

    style B fill:#eef,stroke:#33a
    style R fill:#efe,stroke:#3a3

The builder does the heavy work: full toolchain, MuJoCo SDK, colcon build --merge-install (single install/ tree, simpler COPY), and unit tests to catch anything that slipped past CI. The runner is ros:humble-ros-core — about 200 MB smaller than ros-base — plus libgl1 for MuJoCo rendering, the MuJoCo runtime libs, and the install/ tree from the builder.

The entrypoint trap

source is not exec. A naive entrypoint does source install/setup.bash && ros2 launch .... That fails because the source only affects the shell, and the CMD runs in a child process that never sees the sourced environment. The fix is entrypoint.sh:

#!/usr/bin/env bash
set -euo pipefail
source /ws/install/setup.bash
exec "$@"

exec "$@" replaces the shell with the command, so the sourced environment is inherited by the CMD (here ros2 launch otonav_bringup ...). This is a 4-line file and it took a full evening to debug. Worth documenting.

The Dockerfile COPY trap (Bug 5)

Bug 5 was a missing package in the COPY list. The Dockerfile's builder stage copies 5 ROS packages from the build context: otonav_common, otonav_sim, otonav_nav, otonav_control, otonav_bringup. An earlier version of the Dockerfile only copied 4 — otonav_nav was missing. colcon build in the container silently produced an image without navigation. The image built green. The launch failed at runtime with package 'otonav_nav' not found.

The fix was PR #10: add otonav_nav to the COPY list. The lesson is structural: colcon build in Docker does not fail when a source package is missing — it builds what it has. A release image can be green and broken. This is why the release workflow also runs colcon test in the builder stage before COPY-ing install/ to the runner: a missing package shows up as a failing test, not as a runtime crash on the robot.

Release Workflow

release.yml fires on tag. A single workflow handles the path from git tag v0.1.2 to a multi-tagged image on GHCR:

• Trigger: on: push: tags: ['v*.*.*']
• Job: docker/build-push-action@v5 with context: ., file: docker/Dockerfile
• Cache: type=gha (GitHub Actions cache for Docker layers)
• Tags: four of them, in order of specificity

The sha-* tag is the one that matters for incident response. When something breaks on the robot, the operator reads the image tag, and the tag points at exactly one commit. No ambiguity, no "which image was running?" — the tag is the commit.

The zero-change org port. The release workflow uses ${{ github.repository }} for the image name, not a hardcoded owner/repo. When the repo moved from personal account to org (Act 3 above), the release workflow kept working without a diff. The image path updates automatically because ${{ github.repository }} resolves at run time.

The 12-PR Picture

Two PRs were deliberately broken. PR #6 had a failing unit test — the gate blocked the merge, the PR was closed. PR #8 had a broken control gain that passed unit tests but failed SIL — the gate blocked the merge, the rosbag was uploaded as an artifact, the PR was closed. Both were demonstrations that the gate holds at two different levels: syntactic (unit) and semantic (SIL).

Lesson: Document the Fallback, Don't Stop

When you hit a constraint, the worst move is to block. The merge queue 422 could have stopped the project for a week while I waited for GitHub support. Instead: I documented the 422 in an ADR, recorded the fallback (branch protection ruleset alone, no queue serialization), kept shipping, and scheduled the permanent fix (org transfer) as its own PR (#12). The queue went live the day the fix landed. Zero lost PRs in between.

The pattern, generally:

1. Hit the constraint. Read the error (or the empty body, in this case).
2. Document it. ADR, comment in the workflow, anything future-you can find.
3. Define the fallback. A working state that's worse but unblocks.
4. Ship on the fallback. Don't wait for the permanent fix.
5. Plan the permanent fix as its own unit of work. PR #12 was only the org transfer + queue enablement. No scope creep.

What's Next

Part 4 opens the black box that made all of this trustworthy: the testing strategy. Unit tests, gtest, ament lint, headless SIL with MuJoCo, sim-time determinism, the rosbag-on-failure pipeline, and why PR #8's broken gain was caught by SIL and not by unit tests. The gate is only as good as the tests behind it — Part 4 is about those tests.