Back to Projects
completed

NormaCore Hackathon: AI-Powered Robot Control

SmolVLA fine-tuning pipeline for autonomous ElRobot manipulation — teleop recording, Nebius GPU training, and hybrid Claude+SmolVLA deployment at CIC Berlin.

VLASmolVLAImitation LearningElRobotNormaCoreClaude VisionHackathon
SourceDeployment
Specifications
  • —47 teleop demos recorded across 6 waffle positions with 2-camera setup
  • —SmolVLA fine-tuned on Nebius L40S GPU — loss converged to ~0.05
  • —Hybrid architecture: Claude orchestrates + verifies, SmolVLA executes
  • —Deployed on real ElRobot arm via Station API with walk-mode hardening

Overview

Event: AI & Robotics Hackathon, Jun 20–21 2026, CIC Berlin Track: NormaCore — AI-Powered Robot Control Team: Team 4 (4-person team) Prize: 1 ElRobot unit

The challenge: make the ElRobot 7+1 DoF robotic arm perform a task autonomously from a natural-language instruction and camera feed, using imitation learning.

Architecture

Teleop (master→follower) → Record demos → SmolVLA fine-tune → run_policy (camera + text) → Autonomous task

Pipeline

Step Tool What
Record st3215-remote-teleop-py Human moves leader; follower mirrors. Station streams inference/normvla (2 cams + joint state/goal)
Dataset bin/dataset-generator (Go) Reads queue range → splits into episodes → parquet (task string stamped per frame)
QC bin/dataset-mp4 (Go) Renders parquet as grid mp4 for visual inspection
Train smolvla_py/scripts/train.py Fine-tunes lerobot/smolvla_base on Nebius L40S GPU
Deploy smolvla_py/scripts/run_policy.py --auto Camera + state + task → action → ST3215 command on follower = autonomous

Key Decisions

  • VLA-first, not Claude-only. Claude Vision alone is too imprecise for real manipulation (grasping). With working master→follower teleop, learning from demonstrations is the right tool.
  • NormaCore's existing pipeline. No reinvention — their tooling covers the full flow: teleop → dataset-generator → smolvla_py/train.py → run_policy.py.
  • Hybrid control. Claude orchestrates + verifies, SmolVLA executes. Claude (with camera access) issues the instruction and checks success; SmolVLA does the manipulation.
  • 8 joints. ElRobot has 7 arm joints + 1 gripper → state_dim=action_dim=8, --motor-ids 1,2,3,4,5,6,7,8.
  • 2 cameras required. Pipeline expects exactly 2 images per frame (cam0 + C270 cam1); frames without both are silently dropped.

Task

"move the waffle to the black side" — one task, one instruction string, used verbatim at record time and inference.

Results

  • 47 teleop episodes recorded (8 joints, 2 cameras) across 6 waffle start positions
  • SmolVLA fine-tuned on Nebius L40S (10k steps, batch 64, 12 workers) and local RTX 4050 backup
  • Loss converged to ~0.05
  • Published checkpoints to HuggingFace
  • Deployed on real follower arm via run_policy (PC runs model, Pi runs Station over LAN)
  • Walk-mode hardening: run_policy_walk.py glides toward policy target instead of dead-locking when arm starts far from demo pose

Lessons Learned

  • Camera pipeline matters. normvla cameras, calibration, and OOD start poses were the biggest deploy-day gotchas.
  • Start near demo pose. Best deploy conditions: match scene/lighting and start position to training distribution.
  • Use highest-step checkpoint. More training steps consistently improved robustness.
  • Walk-mode saved the demo. Without it, the arm would dead-lock when starting from a slightly different pose.

Tech Stack

  • Robot: ElRobot (7+1 DoF, ST3215 servos)
  • Platform: NormaCore Station (Go binary, TCP/WebSocket API)
  • Model: SmolVLA (450M params, SmolVLM2-500M backbone + flow-matching action expert)
  • Training: Nebius L40S GPU (remote) + local RTX 4050 (backup)
  • Orchestration: Claude Code + Claude Vision (hybrid layer)
  • Data: Parquet format, 2×224px JPEG cameras, 8-joint state/action

Resources

Project: normacore-hackathon
ReturnTerminate Session