From Integration Pain to Production Pipeline

Why This Project Matters

The integration pain robotics teams feel is real and endemic — not theoretical. Every robotics engineer I know has a story about a "small change" that bricked a robot on the bench, a merge that silently regressed a controller, or a release that worked on one machine and failed on its twin. We treat these as war stories. They are actually process failures wearing a costume.

This project turned that pain into a concrete, reproducible solution. The OtoNav SIL CI/CD platform is not a framework, not a whitepaper, not a vision deck. It is a repository you can clone, a workflow you can read, and a gate you can run. Every layer from lint to release is wired end-to-end.

The repo is living documentation: every PR, every ADR, every bug is a recorded decision. Nothing here is retroactively narrated. The ADRs were written when the trade-offs were uncertain. The bug fixes in Part 5 were filed against real failing pipelines. If you want to understand why a choice was made, you read the commit, not a blog post written six months later pretending we knew the answer.

The SIL platform is not a demo — it's a gate before real hardware. Code does not touch the robot until it passes gtest, smoke, and a full SIL run against a simulated vehicle in sim time. That ordering is enforced by CI, not by convention. Reducing hardware risk and bench time is the whole point.

The Full Pipeline at a Glance

Seven posts, one pipeline. Here is everything this series covered, compressed into one diagram:

flowchart TD
    subgraph "Part 1 · Architecture"
        A1[Developer laptop] -->|git push| A2[GitHub remote]
    end

    subgraph "Part 2 · HAL Boundary + Sim Time"
        B1[hardware_interface] -->|abstraction| B2[Real Robot / Sim Robot]
        B3["/clock authority"] --> B2
    end

    subgraph "Part 3 · Merge Queue"
        C1[PR opened] --> C2[Linter + gtest]
        C2 -->|green| C3[Merge queue]
        C3 -->|re-test vs latest main| C4[Merge to main]
    end

    subgraph "Part 4 · 4-Layer Test Pyramid"
        D1[Linters] --> D2[gtest unit]
        D2 --> D3[Smoke launch]
        D3 --> D4[SIL end-to-end]
    end

    subgraph "Part 5 · Bugs Caught"
        E1[6 real bugs] -->|each = missing check| E2[Stricter gate]
    end

    subgraph "Part 6 · Release Chain"
        F1[Release digest] --> F2[Git tag]
        F2 --> F3[Commit SHA]
        F3 --> F4[PR + Issue refs]
        F4 --> F5[Container image]
    end

    A2 --> B1
    B3 --> D4
    C4 --> D4
    D4 --> E1
    E2 --> F1
    F5 --> G1[Deployed image: robot-ready]

    style A2 fill:#1f2937,color:#fff
    style B1 fill:#374151,color:#fff
    style D4 fill:#065f46,color:#fff
    style F5 fill:#7c2d12,color:#fff
    style G1 fill:#be123c,color:#fff

Read it top to bottom: a push enters at the top, and only a traced, tested, containerized artifact leaves at the bottom. Nothing skips a box.

Process Lessons

"Simple robot, production-grade pipeline" was the thesis, and it holds in real life. OtoNav is a differential-drive robot with no Nav2, no SLAM, no ros2_control. None of that mattered to the pipeline's value. What mattered was that every change was tested, traced, and shipped safely.

Scope discipline is a feature, not a constraint. Every "didn't do" was a conscious decision, recorded in an ADR. Nav2 was out because it adds a perception stack we did not need to validate the pipeline. SLAM was out because localization is orthogonal to CI. ros2_control was deferred — documented as a stretch goal in ADR-4 — because migrating a working HAL boundary mid-project would have cost us the thread.

A phase plan with cut lines saved the project. Before writing any code, I wrote down what was non-negotiable (HAL boundary, sim time, merge queue, SIL gate, release traceability) and what was negotiable (multi-arch builds, noise injection, industrial_ci). When time ran out, the cut lines told me what to drop without re-litigating.

ADRs are living documentation of decisions, not afterthoughts. The moment I felt a trade-off getting fuzzy, I opened an ADR. Writing the ADR forced me to name the alternatives I rejected and why. Six months later, the ADR is more useful than any commit message.

Technical Lessons, Summarized

Cultural Lessons

"Works on my machine" can be fatal in robotics. A robot is not a webapp; a bad merge can damage hardware, injure a bystander, or burn a bench day that the team could not afford. Validating every change without touching hardware is not a convenience — it is a safety and cost requirement.

Flaky tests erode trust, and trust erodes fast. Once a test suite flakes, teams start ignoring it. Once they ignore it, bugs slip through. The fix is not "re-run until green" — it is deterministic sim time, no sleep(), and a hard policy that flaky tests block merges until fixed or quarantined.

Integration without a merge queue is "hope it works" engineering. A green PR at 9 AM is not green at 11 AM if three other PRs landed in between. The merge queue re-tests against the actual main your change will land on. Without it, you are shipping a probability, not a guarantee.

The pipeline keeps hardware safe and teams fast. Those two goals are usually framed as a trade-off. They are not. The same gate that protects the robot is the gate that lets a junior engineer ship on Friday without a review marathon.

Next Steps and Stretch Goals

The platform is done; the work is not. The documented stretch path includes:

• ros2_control migration — replacing the hand-rolled HAL boundary with the canonical ROS 2 control architecture, scoped in ADR-4.
• Multi-arch image builds — arm64 targets for embedded compute, so the image that passed SIL is the image that runs on the robot.
• industrial_ci adoption — standardized ROS CI conventions instead of a bespoke workflow, lowering the onboarding cost for new contributors.
• Dependabot + rosdep automation — dependency bumps tracked in PRs, not in someone's head.
• Noise injection in sim — sensor and actuator noise models in SIL, pushing the gate closer to real-world failure modes without touching hardware.

Each of these is a future item, already scoped, already cut-lined. The pipeline is designed to absorb them without restructuring.

Closing

Building this taught me that integration engineering is not about writing the smartest algorithm — it is about making sure every change is tested, traced, and shipped safely. The robot is a vehicle for the pipeline. The flashy part — the autonomy, the control loops, the sensors — is downstream of the gate that decides whether any of it gets to run.

In robotics, the pipeline is what keeps hardware safe and teams fast. Not the planner. Not the estimator. The pipeline. Main stays green, releases are traceable, bugs are caught before they touch metal, and a junior engineer can ship on Friday. That is the deliverable.

Seven posts, one thesis: simple robot, production-grade pipeline. The integration pain was real. The pipeline is the answer.

This is Part 7 of 7 in the OtoNav SIL CI/CD series. Start at Part 1 — Why I Built a ROS 2 SIL Platform from Scratch.