Writing

Robot-Learning Research at Home: What Four Months Taught Me

The setup (AMD and NVIDIA GPUs side by side), the ROCm gotchas, and the habits that keep a solo reinforcement-learning-for-robotics project honest: check success not reward, verify against raw logs, decide stopping rules before you see the result.

6 min read

In July 2026 I started a reinforcement-learning-for-robotics project at home. The long-term aim is to work the way the robot-learning labs I admire work: reproduce published results first, then look for something new where learning-based control meets the control-theory side of the field (safety filters, barrier functions, robustness guarantees).

Four months and roughly 490 code commits later, most of what I’ve learned isn’t about any single result. It’s about how to run research alone without fooling yourself. This post collects those lessons. I’m writing up specific results separately.

The setup: two GPUs, two different jobs

The workstation is a 16-core Ryzen 9 9950X3D with two GPUs:

  • an AMD Radeon AI PRO R9700 (RDNA4, 32 GB VRAM) on ROCm 7.1, which trains the PyTorch policies for my MuJoCo work;
  • an NVIDIA GeForce RTX 5080 (16 GB), added in September for Isaac Lab, NVIDIA’s GPU-parallel robot simulator, which only runs on CUDA.

The rest of the stack is deliberately boring:

  • MuJoCo 3.10 through Gymnasium 1.3 for physics: locomotion (Ant, HalfCheetah, Hopper, Humanoid), manipulation (Metaworld, a simulated SO-101 arm) and a quadcopter.
  • Stable-Baselines3 2.9 (PPO and SAC), TensorBoard for logs.
  • PyTorch from Debian’s ROCm package, not from pip.

For its first six weeks the project ran on the AMD card alone, and getting PyTorch working on it cost me the most time. Three gotchas, in case they save you an evening:

  1. Don’t pip install torch. The PyPI wheels are CUDA-only, and the generic ROCm wheels didn’t include kernels for this GPU (gfx1201). Either one silently shadows a working system build. I build the virtualenv with --system-site-packages so it reuses the apt-packaged ROCm PyTorch instead.
  2. A stray user-level torch still wins. Even with that flag, a CUDA torch installed under ~/.local shadowed the system one. Setting PYTHONNOUSERSITE=1 in the venv’s activate script fixed it.
  3. torch.cuda.is_available() can lie. Device queries worked, but the first real tensor op failed with HIP error: invalid argument. The cause was an LLVM symbol clash between PyTorch and AMD’s code-object compiler; preloading the system libLLVM fixed it. I now bake that into a bootstrap module that re-launches the process with the right environment, so I never have to remember it.

Three structural lessons from running both:

  • MuJoCo physics is CPU-bound. Stepping runs on the CPU, so for that work throughput comes from running many environments across cores, not from the GPU.
  • GPU-parallel simulation changes which questions you can afford. My earlier robustness experiments ran 1–4 environments at a time on the CPU, at a median of roughly 300 simulation steps per second. On the RTX 5080, Isaac Lab runs about 291,000 steps per second across 16,384 parallel environments. Some questions I had parked as “unresolved” were never short of ideas; they were short of compute.
  • Some papers can’t be run as published. Before the NVIDIA card, a few results I wanted to reproduce were locked to NVIDIA-only simulators, so I reimplemented the technique from the paper instead. That turned out to be valuable in its own right: you only discover what a method really depends on when you have to rebuild it.

Lesson 1: a rising reward curve is not success

This is the most important thing I learned, and I learned it more than once.

On Metaworld’s button-press task, PPO produced a reward curve that looked like textbook learning: clean, steadily climbing. When I actually counted successes on fresh episodes, the policy pressed the button 2 times out of 30 (6.7%). Two harder Metaworld tasks were at 3.3% and 0% despite improving rewards.

So the project has a standing rule: no manipulation result gets reported without a direct success count on held-out episodes. When the simulated SO-101 arm learned to reach, I didn’t trust its converging curve either. A 30-episode check with randomized targets gave 27/30 (90%), and only then did it count as done.

Reward is what the optimizer sees. Success is what you care about. They diverge more often than you’d expect, and the optimizer will happily exploit the gap.

Lesson 2: verify against raw logs, not the last printed line

My research log has one rule at the top: it only records results I’ve checked against the raw logged data (TensorBoard event files and saved evaluation arrays), never just a script’s final summary line.

That rule has caught real problems:

  • A double-scaling bug: an environment wrapper rescaled actions that were already in the actuator’s range. Training ran fine and the reward even improved, but the policy was learning around a bug.
  • A mislabeled evaluation: a robustness sweep I believed applied a perturbation actually ran without it. I had briefly read the unperturbed numbers as robustness. Before running a decision-relevant sweep, I now confirm what the function actually varies.
  • A reference repository that wasn’t real code: a public repo I planned to build on turned out to be a scaffold. Its robot was a single jointless box and its action function a documented no-op. Reading the code before trusting the README saved weeks.

Lesson 3: statistics a reviewer would accept

A single training run in RL is close to meaningless; seed-to-seed variance is large. My defaults now:

  • Multiple seeds per condition, scaled to the question. Early reads at small sample sizes have evaporated more than once when I added seeds.
  • Welch’s t-test for continuous metrics (it doesn’t assume equal variance between conditions) and Fisher’s exact test for rates like crash or success counts.
  • Multiple-comparison correction (Bonferroni) when sweeping many held-out conditions. A real failure mode hits the same conditions every time you re-measure; noise produces a different “worst case” each time.
  • Treat sign flips as a warning. If an effect reverses direction between two comparable checkpoints, it’s almost certainly noise, not a small real signal.

None of this is advanced. It’s simply what turns “my run looked better” into a claim someone else can trust.

Lesson 4: decide the stopping rule before you see the result

The hardest discipline is knowing when to stop. Without a rule, a disappointing result invites one more tweak, then another, until either something works by chance or you quietly drop the question.

So before a follow-up experiment, I write down how many attempts it gets and what outcome would end it. When that limit is reached, the result gets recorded as it is, positive or negative. Deciding the rule before seeing the second result is what makes a negative result trustworthy rather than a moving goalpost.

Lesson 5: a negative result is a result, but name it precisely

One transfer experiment, moving a robustness technique from one robot to another, ended without an answer: the stressor the technique defends against never produced a measurable effect on the second robot, at any realistic magnitude I tested. It would have been easy to call that a success (“no gap, so it’s robust!”) or a failure. It was neither. The precondition for the test was never established, and the log says exactly that.

Lesson 6: re-read your own notes before choosing the next experiment

A humbling one. I once picked a follow-up experiment based on fresh reasoning, without re-reading my own earlier write-up. That write-up had already worked through the same question more carefully and recommended a different, sharper experiment. I ran the weaker one first, then had to run the recommended one anyway.

The fix is cheap: before deciding what to run next, re-read what you already concluded. I now also keep a short, always-current status section at the top of the research log, so the state of every open question is readable in two minutes rather than buried in thousands of lines of history.

What’s next

The project is moving from simulation toward hardware, in stages: a pendulum, then a low-cost SO-101 robot arm, then a quadruped. The prerequisite for each step is making simulated policies survive realistic variation in physics, sensing and timing (domain randomization), which is exactly where most of the hard problems in sim-to-real transfer live.

I’ll write up specific results as they mature. If you work on robot learning, safe RL or sim-to-real transfer, and especially if you’re doing it on AMD hardware, I’d be glad to compare notes: milav.dabgar@gmail.com.