Back to Projects
completed

Reinforcement Learning for Autonomous Robot Navigation

A TurtleBot3 that learns to navigate from scratch using Q-Learning — no pre-programmed paths, pure trial-and-error learning with LIDAR sensing and 20 hyperparameter configurations.

Reinforcement LearningROSGazeboQ-LearningPythonTurtleBot3
Source
Specifications
  • —Q-Learning agent learns autonomous navigation from scratch — 144 discrete states, 3 actions, zero prior knowledge
  • —20 experiment configurations: Softmax vs ε-greedy exploration, random vs fixed start, 5 hyperparameter scenarios
  • —~85% crash-free navigation after 200 training episodes with smooth obstacle avoidance
  • —Full ROS Noetic + Gazebo 11 pipeline with Docker containerization for reproducibility
  • —Clean class hierarchy: Lidar → QLearning → Control → MapLearner — each layer extends the previous

A robot that learns to navigate — from scratch

A TurtleBot3 starts with zero knowledge about its environment. Through thousands of episodes of trial and error, it learns when to go forward, when to turn, and how to avoid obstacles — no pre-programmed paths, no map, just pure reinforcement learning.

This was my Master's thesis project at the University of Europe, Berlin. The goal: implement and systematically evaluate Q-Learning for autonomous robot navigation in a simulated Gazebo environment.

TurtleBot3 in Gazebo simulation — learning to navigate through obstacles

How it works

The robot uses a 360° LIDAR sensor to perceive its surroundings. The continuous sensor data is discretized into 144 states based on four spatial features:

Feature Region Zones Purpose
x1 Left (0-75°) 3 Left obstacle proximity
x2 Right (285-360°) 3 Right obstacle proximity
x3 Front-left (0-75°) 4 Front-left sector detail
x4 Front-right (285-360°) 4 Front-right sector detail

Three simple actions: forward (0.08 m/s), turn left (+0.4 rad/s), turn right (-0.4 rad/s).

The reward function balances progress against safety:

R = R_action + R_obstacle + R_change

Forward: +0.4    Turn: -0.2
Closer to obstacle: -0.4    Away from obstacle: +0.4
Oscillation (left→right→left): -0.4
Crash: -100 (episode terminates)

Exploration strategies

Two strategies were implemented and compared across all experiments:

  1. Softmax (Boltzmann) — Probabilistic action selection based on Q-values with temperature decay (T_init=25, decay=0.95)
  2. ε-greedy — Random exploration with decaying epsilon (ε_init=0.6, decay=0.998)

Experimental design

20 configurations were trained and evaluated:

Exploration Start position Scenarios Episodes each
Softmax Random 5 200
Softmax Fixed 5 200
ε-greedy Random 5 200
ε-greedy Fixed 5 200

Hyperparameter scenarios

Scenario α (learning rate) γ (discount) Strategy
1 — Moderate 0.5 0.9 Balanced baseline
2 — Fast/High Explore 0.7 0.8 Quick learning, lots of exploration
3 — Fast/Low Explore 0.7 0.85 Quick learning, focused
4 — Slow/High Explore 0.3 0.95 Patient learning, thorough exploration
5 — Slow/Low Explore 0.2 0.7 Conservative approach

Results

After training, the agent achieves:

  • ~85% crash-free navigation in test scenarios
  • Smooth obstacle avoidance without oscillation
  • Generalization to unseen starting positions

Training results — reward convergence over 200 episodes (Softmax, moderate scenario)

Performance plot — steps per episode and cumulative reward trend

Comparing exploration strategies

The ε-greedy strategy with moderate hyperparameters (α=0.5, γ=0.9) shows steady learning with occasional exploration spikes:

ε-greedy training results — moderate scenario

Fast learning with high exploration (α=0.7, γ=0.8) converges faster but with more variance:

Fast learning, high exploration scenario

Slow learning with thorough exploration (α=0.3, γ=0.95) produces the most stable convergence:

Slow learning, high exploration scenario

Project overview

Infographic — Q-Learning navigation system summary

Architecture

The system follows a clean inheritance hierarchy — each layer extends the previous, adding a specific concern:

Lidar (lidar_scanner.py)
  ↓ inherits
QLearning (q_learning.py)
  ↓ inherits
Control (controller.py)
  ↓ inherits
MapLearner (learning_manager.py)
  • Lidar: Raw 360° sensor processing and state discretization
  • QLearning: Q-table operations, Softmax/ε-greedy exploration, reward calculation
  • Control: ROS /cmd_vel publishing, position management, action execution
  • MapLearner: Episode management, training loop, data logging

ROS topics

Topic Type Direction Purpose
/scan sensor_msgs/LaserScan Subscribe 360° LIDAR data
/odom nav_msgs/Odometry Subscribe Robot pose
/cmd_vel geometry_msgs/Twist Publish Velocity commands
/gazebo/set_model_state gazebo_msgs/ModelState Publish Robot reset

Reproducibility

The entire pipeline runs in Docker — no local ROS installation needed:

docker-compose up --build

All 20 experiment configurations, Q-tables, training curves, and analysis notebooks are included in the repository.

Stack

PyTorch · ROS Noetic · Gazebo 11 · Python 3.8+ · Docker · NumPy · Pandas · Matplotlib

Project: rl-turtlebot-navigation
ReturnTerminate Session