Why I Built a ROS 2 SIL Platform from Scratch

In every robotics team I've worked with, we kept hitting the same problems. Code passes in simulation but fails on hardware. A PR is green on a feature branch but breaks main the moment it merges. Tests are flaky in CI because they depend on wall-clock timing, and trust in the pipeline erodes week by week. This is endemic in robotics, and we shouldn't accept it.

This post is the first in a 7-part series documenting how I built a production-grade Software-in-the-Loop (SIL) platform for a ROS 2 robot — from scratch, from first principles, with discipline about scope. The goal isn't to impress with robot complexity. The goal is to prove that a simple robot backed by a bulletproof pipeline is worth more than a sophisticated robot held together by duct tape and luck.

The Problem: Pain We Felt in the Team

The same failure modes appeared on every project. Each one had a name we whispered in standups like a curse:

• Sim passes, hardware fails. We'd merge a change validated in Gazebo, then watch the robot veer off a 2-meter corridor on the real rig. Debugging meant booking the hardware bench, fighting for a slot, and reproducing a race condition that only showed up at 30% battery. Expensive. Slow. Sometimes impossible.
• PRs "passed on my branch" but break main. A contributor runs tests locally, sees green, opens a PR. CI is green. Merge to main — and the build fails because something about the merge order or a transitive dependency changed. The PR author swears it worked ten minutes ago. They're not lying.
• Tests are wall-clock dependent. A test does sleep(2) and checks a node's state. On the developer's laptop, fine. Under CI load, the scheduler stalls, the two seconds pass before the node is ready, and the assertion fires. Flaky. Erodes trust. People start ignoring red.
• "Works on my machine." The validation environment is not the ship environment. Local Ubuntu 24.04 with ROS built from source; production is a Docker image on Ubuntu 22.04 from binaries. The gap is invisible until it bites.
• Release image ≠ test image. When someone asks "which code is running in production?", nobody can answer with certainty. The release image was built last Tuesday from an unknown commit; the test image was rebuilt this morning. They differ by three merges and a hotfix.

flowchart TD
    A[Code change on branch] --> B[Local sim: PASS]
    B --> C[PR CI: PASS]
    C --> D[Merge to main]
    D --> E{Build on main}
    E -->|Fails| F[main red — who broke it?]
    E -->|Passes| G[Release image built]
    G --> H[Deploy to hardware]
    H --> I[Hardware fails]
    I --> J[Book bench time]
    J --> K[Reproduce? Maybe.]
    K --> L[Debug on hardware — slow, expensive]

    style F fill:#fdd,stroke:#c00
    style I fill:#fdd,stroke:#c00
    style L fill:#fdd,stroke:#c00

These are not edge cases. They are the default experience of robotics teams that ship. We accepted them as the cost of doing robotics for years. They are not the cost — they are the symptom of a missing integration layer.

Thesis: Simple Robot, Production-Grade Pipeline

The real cost in robotics is not in the algorithm. It's in integration, testing, release, and the gap between "it works on my bench" and "it works on every robot we ship." Most teams invest in sophisticated robot software and tolerate a failing pipeline. I inverted that bet.

I deliberately kept the robot simple — a differential-drive platform with a lidar and an IMU, no Nav2, no SLAM, no ros2_control — and invested everything in the integration infrastructure. Simple robot plus bulletproof pipeline beats sophisticated robot plus failing pipeline, every time, because the pipeline is what compounds. A great algorithm you can't reliably ship is a liability.

What Is Software-in-the-Loop?

SIL means running the entire software stack against a physics simulator instead of real hardware. The same binaries, the same launch files, the same nodes — only the hardware abstraction layer swaps its backend from a CAN bus to MuJoCo. Deterministic. Repeatable. Runnable in CI on every push.

SIL is not a demo. A demo is something you show to convince a stakeholder the robot moves. SIL is a gate: a test that must pass before any change is allowed near real hardware. It validates every commit, every PR, every release candidate — without booking bench time, without risking a $40k rig, without the thermal noise that makes hardware debugging a lottery.

The key property is determinism. Given the same commit and the same seed, the simulation produces the same trajectory to the microsecond. That means a flaky test in SIL is a real bug, not a CI hiccup — and you can reproduce it on your laptop by running the exact same seed.

Scope Discipline: What We Did Not Do

Every "didn't do" was a conscious choice. Scope discipline is the only way to ship a pipeline this deep. Here is what I left out, and why:

• No Nav2. The navigation stack is a sprawling dependency surface. It would have consumed the testing budget before the pipeline was even interesting. I wrote a simple go-to-goal P-controller with a lidar stop-on-obstacle guard — enough to exercise the full stack, small enough to reason about.
• No SLAM. Mapping adds an entire perception subproblem with its own failure modes. There is no map — the robot navigates by direct goal coordinates. SLAM is a future concern, not a Phase 1 one.
• No ros2_control. It's powerful but heavy, and the hardware abstraction I needed was simpler than what it offers. A thin RobotHardwareInterface abstract class with a MujocoInterface implementation and a CanBusInterface stub was enough to draw a clean HAL boundary.
• No visual optimization. No camera calibration, no AprilTag tuning, no point-cloud filtering. The sensor suite is a 13-beam lidar and an IMU. Period.

Narrow scope is not a limitation. Narrow scope is what made a deep pipeline possible. The energy I didn't spend on Nav2 went into the merge queue, the sim-time discipline, the Docker layering, and the release traceability.

Technology Choices

Every row in that table was written down in an ADR before the code was written. The project has 6 Architecture Decision Records, and they are the reason the pipeline stayed coherent.

Phase Plan

The build was staged in five phases. Phases 0 and 2 are non-negotiable — skip either and the whole thing collapses.

flowchart LR
    P0[Phase 0: Repo hygiene<br/>format, lint, ADRs] --> P1[Phase 1: Robot + bridge<br/>MuJoCo, rclcpp, HAL]
    P1 --> P2[Phase 2: Sim-time CI<br/>deterministic smoke + SIL]
    P2 --> P3[Phase 3: Merge queue<br/>squash, ALLGREEN]
    P3 --> P4[Phase 4: Release<br/>Docker, GHCR, traceability]

    style P0 fill:#dfd,stroke:#060
    style P2 fill:#dfd,stroke:#060

What We Achieved

By the end of Phase 4, the numbers told the story:

• 45 tests across 4 layers — linters, gtest unit tests, launch smoke tests, and full SIL integration tests.
• CI budget: under 10 minutes per PR. With caching, the median run was ~80 seconds. That's fast enough that no one is tempted to skip CI.
• 6 real bugs caught by SIL before they ever touched hardware — a silent QoS mismatch, a physics-induced false obstacle detection, a Dockerfile missing a package, and more. Each one a different failure class that unit tests alone could never catch.
• A release image that is the test image. Same Dockerfile, same layers, tagged by commit SHA. "Which code is in production?" is now a one-line answer.

Series Roadmap

This is Part 1 of 7. The series walks the stack layer by layer:

1. Part 1 — Why I Built a SIL Platform (this post): the problem, the thesis, the scope.
2. Part 2 — Architecture: package layout, the HAL boundary, sim time, and 6 ADRs.
3. Part 3 — CI/CD: the GitHub Actions workflow, the merge queue war story, multi-stage Docker, GHCR release.
4. Part 4 — Testing: the 4-layer test pyramid, sim-time discipline, why no test touches wall-clock.
5. Part 5 — Bugs: 6 real bugs SIL testing caught, each with symptom, diagnosis, fix, and lesson.
6. Part 6 — Release: multi-stage Docker, GHCR, SHA traceability, the "release image = test image" invariant.
7. Part 7 — Lessons: process lessons, cultural lessons, and why the pipeline — not the robot — is what keeps hardware safe and teams fast.

Next up, Part 2: Architecture. I'll open the repo layout — the five packages, the HAL boundary that lets the same binary run against MuJoCo or a real CAN bus, and the sim-time plumbing that makes determinism possible. If you've ever stared at a 12-package monorepo and wondered where the boundaries should be, that's the post for you.