6 Real Bugs That SIL Testing Caught

Every bug is a missing check. These 6 are the gold stories — each a real integration problem where a test caught something that would have exploded on hardware.

This is Part 5 of a 7-part series on building a CI/CD platform for a small ROS 2 robot (otonav). The previous parts covered the test pyramid, the toolchain, and the SIL harness. Here I want to slow down and show what that harness actually found — because a test suite that never catches a real bug is just theater.

Every bug below is real. I pulled each one from the commit log, the failing CI run, or the debug session that produced it. Each follows the same structure: Symptom → Diagnosis → Fix → Lesson. Some are embarrassing. All are useful.

Debug methodology

Before the bugs, the method. When a test fails I don't reach for print() — I reach for this loop:

flowchart TD
    A[Read the symptom fully] --> B[Which test/layer caught it?]
    B --> C[Reproduce live<br/>launch_test, logs, sim]
    C --> D[Isolate hypothesis<br/>physics? QoS? build?]
    D --> E{Root cause found?}
    E -- no --> C
    E -- yes --> F[Fix root cause<br/>not symptom]
    F --> G[Validate in shipped env]
    G --> H[Write test that would have caught it]

The single most important rule: fix root cause, not symptom. Step 5 — "validate in the shipped environment" — is where Bug 5 below was hiding for weeks.

Bug 1 — Silent QoS mismatch (Phase 2)

Symptom. test_odom_smoke.py hung for 45 seconds, then printed:

AssertionError: topics not publishing: [imu, scan]

/clock and /odom arrived on time. /imu and /scan never showed up. No error, no exception, no DDS warning — just silence until the deadline expired.

Diagnosis. The bridge publishes /imu and /scan with SensorDataQoS (best-effort, volatile). The test subscriber used the default QoS profile (reliable, keep-last). In DDS, a RELIABLE reader against a BEST_EFFORT writer is an incompatible combination — zero messages, zero errors. The 45s deadline expiry was the only clue. Reliable topics arrived instantly; best-effort ones never came.

I confirmed by adding a QoS probe node and printing the actual ReliabilityPolicy on both ends. Mismatch confirmed.

Fix. One line:

self.create_subscription(
    Imu, '/imu', self.cb, 10,
    qos_profile=qos_profile_sensor_data,  # was default
)

Same for /scan. Test dropped from 45s-timeout to green in ~2s.

Lesson. ROS 2's most insidious bug class. Sensor streams are best-effort; commands and /odom are reliable. QoS mismatch is silent — DDS will not log it, and your subscriber will simply never receive a frame. Always pass qos_profile_sensor_data when subscribing to sensor topics, and assert QoS in smoke tests.

Bug 2 — Pitch → false stop-on-obstacle (Phase 3) ⭐

This is the bug I show people when they ask "why do you bother with simulation?"

Symptom. test_goal_reach.py FAIL. The robot reached x = 2.68 (target 3.0) and stopped. Logs:

[obstacle_guard] obstacle ahead < 0.40m — halting

The real obstacle was at (1.5, 0.5) — already passed. The "obstacle" in front was fake. No object existed in the lidar's true field of view.

Diagnosis. The robot had no front support: drive wheels at x = 0, a single caster at the back. On forward acceleration the body pitches nose-down. Front lidar beams tilt toward the ground, read a close distance, and the obstacle guard triggers.

I confirmed by running pip-mujoco manually and logging front-beam ranges alongside body pitch angle. Pitch climbed to ~3° during the acceleration phase; front beams dropped below 0.40m at exactly that moment. The guard logic was correct — it was protecting against a real-looking signal.

Fix. Two changes:

1. Added a second front caster → 4-point stance. Pitch dropped to ~0.02° → front beams read -1 (no hit).
2. Increased wheel servo kv from 0.5 to 5.0 → faster convergence. Robot now covers 3m in ~6s sim time (budget 20s).

Verified with a pip-mujoco time-to-3m sweep across kv ∈ {0.5, 1, 2, 5, 10}.

Lesson. This is exactly what SIL testing is for. Unit tests passed — the guard logic was correct, the lidar was correct, the controller was correct. The bug was in the interaction between physics and algorithm. Only a headless SIL scenario reproducing the full dynamics could catch it. A unit test on the guard function would have returned false (no obstacle). A hardware run would have crashed the robot. SIL is the layer in between.

Bug 3 — cpplint include order (Phase 3)

Symptom. otonav_nav cpplint FAIL:

Found C system header after C++ system header  [build/include_order] [4]

Diagnosis. cpplint treats <gtest/gtest.h> — an angle-include ending in .h — as a C-system header. <cmath> and <vector> are C++-system headers. The expected order is: related → C-system → C++-system → other. So <gtest/gtest.h> must come before <cmath>.

That's counterintuitive — gtest is obviously C++ — but it's how cpplint's regex classifies it.

Fix. Separate gtest into its own include block at the top of the test file:

#include <gtest/gtest.h>

#include <cmath>
#include <vector>

clang-format with IncludeBlocks: Preserve maintains block order, so the two-block structure survives formatting.

Lesson. ROS 2 linter conventions are opinionated and not always intuitive. In gtest test files, gtest goes first — period. Document the rule in your CONTRIBUTING so the next contributor doesn't fight the linter.

Bug 4 — pre-commit untracked trap (Phase 3)

Symptom. Locally:

$ pre-commit run --all-files
clang-format.............................................................Passed

On CI, the pre-commit job FAILED — it wanted to reformat the same files that were "Passed" on my machine.

Diagnosis. pre-commit --all-files only processes git-tracked files. The new otonav_nav/ files were untracked at the time → skipped → false "Passed". After I committed them (now tracked), CI ran pre-commit on the real tracked set and found formatting drift.

Fix. git add to stage, re-run pre-commit run, commit the real reformatting.

Lesson. Local green ≠ CI green. --all-files is a lie when files aren't tracked yet. The fix is workflow hygiene: git add before you run pre-commit locally, so your local set matches CI's set.

Bug 5 — Dockerfile missing COPY (Phase 4) ⭐

Symptom. The v0.1.0 release workflow failed:

ERROR: rosdep: Cannot locate rosdep definition for [otonav_nav]

Diagnosis. The Dockerfile's explicit COPY list was written in Phase 1. otonav_nav was added in Phase 3 but never COPY'd into the image. otonav_bringup depends on otonav_nav, so rosdep couldn't resolve it.

Why did this hide for so long? Local validation used cp -a of the entire repo — including otonav_nav. CI used actions/checkout + colcon build — also including it. The Dockerfile only runs during release, so the missing COPY surfaced there and nowhere else.

Fix. Added one line:

COPY otonav_nav src/otonav_nav

Verified with a full docker build --target runner producing all 5 packages. Released as v0.1.1.

Lesson. "Works on my machine" validation that bypasses the real build path is misleading. My local cp -a shortcut hid the gap for an entire phase. Validation environment must equal ship environment. If you ship via Docker, your CI must run docker build — not a parallel colcon path that happens to also work.

Bug 6 — Merge queue API 422 (Phase 0 → solved 2026-06-12)

Symptom. Adding a merge_queue rule to the repository ruleset:

HTTP 422 — empty error message

Not "unauthorized", not "feature not available" — just 422 with no body.

First attempts. I assumed visibility — made the repo public. Still 422. I documented a fallback (branch-protection gate) and kept moving, but logged the issue for a permanent fix.

Root cause. It was not repo visibility — it was account type. Native GitHub merge queue requires an organization repo. A personal repo returns 422 with an empty body. Once I transferred the repo to the ozkanceylan-dev org, the UI toggle worked and the same API call was accepted (set to SQUASH).

Lesson (two layers).

1. When error messages are empty or meaningless, narrow your hypotheses and test them one by one — visibility? plan? account type? The first reasonable hypothesis may be wrong. Don't fixate.
2. When you hit a constraint, document the fallback and don't stop. Plan the permanent fix, but don't blindly retry the same call hoping it works. Understand the boundary, then change your path.

Bug summary

A few patterns worth naming:

• Silent failures (Bug 1, Bug 6) are the hardest. No error, no log, just a deadline or an empty 422. Always instrument the "nothing happened" path.
• Interaction bugs (Bug 2) only appear at the integration layer. No unit test will find them.
• Environment drift (Bug 4, Bug 5) is when "works locally" and "works in CI" diverge. The cure is making the local path a strict subset of the CI path — never a parallel one.

Six bugs, six categories, six fixes — each one a check that now exists in CI and will never silently break again.

What's next

Part 6 leaves the bug log and moves to release engineering — how the v0.1.0 → v0.1.1 path above got codified into a tagged, semver-driven, Docker-image-published release pipeline. We'll cover the release workflow, the version-bump automation, and what it takes to ship a ROS 2 image that a teammate can docker pull and actually run.

Release engineering is where all the layers in this post finally meet the user.