Architecture: HAL Boundary, Sim Time, and 6 ADRs

The entire project is one control loop. nav publishes a Twist, kinematics converts v,ω → ωL,ωR, the HAL writes it into MuJoCo, mj_step advances the world, sensors flow back, and nav recomputes. Everything else in this post — the HAL boundary, sim time, the six ADRs — exists to keep that loop honest across sim and real, across a developer laptop and a CI runner.

One Data Loop

flowchart LR
  nav["otonav_nav<br/>P-controller + lidar stop"]
  cmd["/cmd_vel<br/>(geometry/Twist)"]
  kin["otonav_control<br/>diff-drive kinematics"]
  hal["RobotHardwareInterface<br/>(HAL boundary)"]
  mj["MuJoCo<br/>mj_step ×N"]
  sens["sensordata<br/>qpos/qvel/rangefinder"]
  odom["/odom /scan /imu"]
  clock["/clock<br/>(sim time)"]

  nav -->|"/cmd_vel"| cmd
  cmd --> kin
  kin -->|WheelCommand| hal
  hal -->|"ctrl[]"| mj
  mj --> sens
  sens --> hal
  hal --> odom
  odom -->|use_sim_time:=true| nav
  clock --> nav
  clock --> hal

The bridge owns /clock and is the sim-time authority. Every other node runs with use_sim_time:=true and reads the clock from /clock. There is no wall-clock in this loop in CI — only sim time, which advances exactly as fast as mj_step is called. On a fast runner that's 100 Hz of sim time per 100 Hz of wall; on a loaded runner it's still 100 Hz of sim time, just delivered slower in wall time. The tests don't care which.

The HAL Boundary — Sim-to-Real in One Box

This is the heart of the project, and it's a single abstract class. RobotHardwareInterface defines the contract between the ROS world above and the physics world below:

classDiagram
  class RobotHardwareInterface {
    <<abstract>>
    +write(WheelCommand)
    +step()
    +sim_time() double
    +read_wheels() WheelState
    +read_imu() ImuSample
    +read_scan() RangeScan
  }
  class MujocoInterface {
    -mjModel* m
    -mjData* d
    +write(WheelCommand)
    +step()
    +sim_time() double
    +read_wheels() WheelState
    +read_imu() ImuSample
    +read_scan() RangeScan
  }
  class CanBusInterface {
    <<stub>>
    +write(WheelCommand)
    +step()
    +read_wheels() WheelState
    +read_imu() ImuSample
    +read_scan() RangeScan
  }
  RobotHardwareInterface <|-- MujocoInterface
  RobotHardwareInterface <|-- CanBusInterface

The interface lives in otonav_mujoco_bridge/include/otonav_mujoco_bridge/robot_hardware_interface.hpp. MujocoInterface is the only real implementation; CanBusInterface is a stub waiting for the day we bolt on real motors.

Hard Rule #4: MuJoCo types are confined to a single .cpp file. The header mujoco_interface.hpp forward-declares mjModel and mjData as opaque pointers. Nothing above the HAL ever #includes mujoco.h. The control code, the nav code, the test code — none of them know MuJoCo exists. They know WheelCommand, WheelState, ImuSample, RangeScan, and that's it.

Sim-to-real is then literally a swap of one box in the diagram. Replace MujocoInterface with CanBusInterface and the entire stack above — kinematics, nav, bringup, all 45 tests — stays byte-identical. That's the thesis: the abstraction isn't cosmetic, it's the load-bearing wall. If a real-hardware change ever forces a change in otonav_control or otonav_nav, the HAL failed and we rethink it.

Why not ros2_control? It's the standard ROS 2 answer, and we deliberately did not take it. ADR-4 documents this as a time-boxed choice: building a thin HAL ourselves cost a weekend and gave us a contract tuned to our robot. Migrating to ros2_control later is a documented stretch path — the interface philosophy is identical, the adapter is mechanical. We chose speed-to-CI over ecosystem fit, and we wrote down the trade.

Sim Time (ADR-2)

Wall-clock timeouts are flaky tests waiting to happen. A GitHub Actions runner has unpredictable CPU: sometimes it's snappy, sometimes it's a throttled vCPU shared with a noisy neighbor. A test that asserts "the robot reaches the goal in 5 seconds" against wall time passes on your laptop and fails intermittently in CI for reasons that have nothing to do with your code.

flowchart LR
  subgraph Bridge["Bridge node (use_sim_time:=false)"]
    wall["wall_timer @ 100Hz"]
    step["mj_step ×N"]
    pubclk["publish /clock"]
  end
  subgraph Nodes["All other nodes (use_sim_time:=true)"]
    sub["subscribe /clock"]
    logic["test logic<br/>asserts on sim time"]
  end
  wall --> step --> pubclk --> sub --> logic

The bridge publishes /clock; every node consumes it with use_sim_time:=true. Tests assert on sim time, which advances deterministically with mj_step regardless of how fast the host CPU is. A test that says "reach goal by t=5.0s" means 5.0 seconds of simulation, not 5.0 seconds of wall.

The subtlety that bit us: the bridge that produces /clock cannot consume it. If the bridge ran use_sim_time:=true, it would wait for a clock message that it itself has to publish — a deadlock. So the bridge runs use_sim_time:=false and drives the loop with a wall_timer at 100 Hz, calling mj_step N times per cycle and publishing /clock afterward. It's the one node that lives in wall time so everything else can live in sim time.

No sleep() in tests. This is a hard rule in the test suite. Conditional waits only — spin_until_future_complete on a future, or a predicate loop that checks "has the goal been reached?" against sim time. A rclcpp::sleep_for(2s) would couple the test to wall-clock speed and reintroduce the exact flakiness sim time was meant to kill.

ADR-1: Headless via Runtime Flag, Not #ifdef

mj_step has zero OpenGL dependency. The physics step doesn't render; it integrates the equations of motion. The viewer is a separate, optional concern.

One binary serves both the developer's screen and the CI runner. Launch with gui:=true and you get the viewer window; launch with gui:=false in CI and the viewer is simply never instantiated. Same binary, same code path, same physics.

#ifdef HEADLESS would create two divergent binaries — one you develop with, one CI tests. That directly violates "test what you ship." If offscreen rendering is ever needed (e.g., for a vision-based test), MUJOCO_GL=egl flips the GL backend at runtime without a recompile. The decision is encoded in the environment, not the compiler.

ADR-3: DDS in CI — Simplest Thing First

GitHub-hosted runners block multicast but loopback works fine. ROS 2's default DDS profile assumes multicast discovery, which a shared CI runner won't allow. The fix is layered:

We don't ship Layer 2 as the default because Layer 1 works for ~95% of runs and adding a custom profile is one more thing to maintain. But the fallback is written, committed, and documented — when a runner image flips its network policy, we flip one env var, not the architecture.

ADR-5: Multi-Stage Docker with a Correct Entrypoint

Two stages, one image family. The builder carries the full toolchain — ROS 2 base, MuJoCo SDK, colcon, every build dep — and runs colcon build. The runner stage starts from ros-core, installs only runtime libraries, and copies the install/ tree from the builder. The final image is small and has nothing a compiler needs.

The critical line is in entrypoint.sh: source /opt/ros/humble/setup.bash && source /install/setup.bash && exec "$@". Note source, not exec — setup.bash exports environment variables into the current shell. If you exec it, the variables vanish with the child process. Getting this wrong produces the classic "command not found: ros2" inside the container, and it's a 30-minute waste of a CI cycle.

ccache is volume-mounted and reused across runs in CI. The builder stage hits the cache on anything short of a clean rebuild, taking cold-build times from minutes to seconds on incremental changes.

ADR-6: Merge Queue as Integration Gate

The textbook integration-engineer problem: two PRs pass CI in isolation, both land on main, main is red. Classical SE calls this the integration problem and solves it with a dedicated integration branch and a person. We solve it with platform tooling.

GitHub's native merge queue re-tests each PR against the latest main before merging. When you add a PR to the queue, GitHub updates it with the tip of main, reruns the required checks, and only merges if they pass. "Passed on my branch, broke on main" becomes a category that structurally cannot happen — the queue is the integration engineer, and it doesn't sleep.

Package Layout

otonav_control having zero ROS dependencies is deliberate. It's pure math: Twist → wheel velocities → odometry. That purity is what lets us unit-test it with a plain gtest binary, no rclcpp spin, no nodes, no DDS — the fastest and most reliable tier in the test pyramid.

ADR Summary

Six decisions, all reversible, all written down. The point of an ADR isn't the decision — it's the why. When someone (including future me) asks "why didn't we just use ros2_control?", the answer is one grep away.

What's Next

The architecture is settled, the contracts are drawn, and the ADRs are filed. Part 3 takes this skeleton and wraps it in the CI/CD pipeline that actually enforces it — the GitHub Actions workflow, the Docker build, the merge queue configuration, and the exact sequence of jobs that turns a git push into a green check or a red X. That's where the abstract promises of this post meet the concrete YAML that keeps them.