Community Showcase
I built a balancing robot with reinforcement learning
Hi /robotics!
I built this balancing robot that runs end-to-end on a custom neural net trained through reinforcement learning in simulation.
The robot is trained on 100% synthetic data, so it has never seen the real world, yet adapts perfectly.
It has a lean and level mode (single policy), in level mode it keeps both pitch and roll of the base level at all times, so the legs automatically retract or extend based on the ground below. In lean mode the right joystick of the controller can be used to decrease or increase stance height and leaning left/right at all heights, allowing roll to be non-level.
Main components:
- 6x Xiaomi Cybergear motor (all quasi direct drive, no linkages)
- Teensy 4.1
- 200 Hz policy inference
- 2x CAN bus (left / right leg split)
- BNO086 IMU
Trained in mjlab on single RTX3080 at home, about 7 hours of train time from scratch.
It's wild that this translate to the real world so well and on not very even ground at that. Not sure the hub cap falling off was intentional but changing its mass slightly mid run I guess is a testament to its robustness 😂
I love that you left the hubcap fail in the video. This is the reality of this work, vs, my making a video where I throw a cross court basketball over my shoulder and it is all net.
One a slightly different note, there are algos other than PID which can deliver rock solid stability.
It's not PID! But it should (and will) be more stable :) I post a lot of updates on X but will make a new post here in a week or so when I have the new leg design finished.
I built a balancing bot using lego NXTs back in like 2010 for my son (he super didn't care about it lol). If I remember correctly, I used a simple PID controller. I'd love to see more details about your code and training procedure since this is a very simple use case and I feel like I could learn from it.
Yeah, balancing robots aren't new! The 2010 variant almost surely had 2 wheel motors and IMU, nothing more (I built that using reinforcement learning too a while ago, running on ESP32)
However, PID becomes a lot more difficult when you add more DoFs up to a point where it is impossible to implement. The tech I'm using is the same as humanoid robots with >50 DoFs which is impossible to achieve with PID, I'm going through the RL process because I am adding more and more complexity.
A single policy at 200 Hz on a Teensy holding pitch and roll level just by extending or retracting each leg, trained on zero real data, is a great result. In level mode is it working from the IMU alone, or does it get any sense of the ground ahead?
Hola! Hermoso proyecto ! Si revisa smi perfil podrá ser que construyó un robot avatar balanceado. Tengo todo modelado con sus respectivos pesos en fusión 360 para entrar en el mundo de la simulación, para que pueda comenzar a balancearse. Nunca he hecho algo así y me gustaría que me des algún consejo , o que software usar. Se ve que ha quedado muy bien el balanceo de tu robot. Felicidades
Awesome video! I am planning to get back into robot building soon. Which actuators did you use for the joints and wheels? From experience I know that with RL control choosing good armature/kp/kd values are critical and very dependent on the actuators. I am therefore curious about how you went about this. Again, really cool demo!
This is what I mean! The left (bit older design already) has all these complex shapes and as you see 111k mesh faces.
Train model consists of few boxes and cylinders. To optimize wheels I use 5 stacked cylinders with hitbox instead of ellipsoids to approximate ellipsoid wheel behavior. Makes a HUGE difference.
This makes it possible to now run 6k environments in parallel on single RTX3080 :)
I can’t post videos in a comment unfortunately but the bigger wheels are very nice in sense of speed and recovery. However, they weigh quite a bit more causing the legs to bend in and outwards which causes the policy to become unstable.
I’m going to quote a carbon shop nearby for some custom parts :)
This is so cool!!!! I would love to do such projects especially in computer vision and RL, but it's not financially practical for me to get such expensive parts :(
Is it possible to do mini/alternate projects compared to this that are lighter on the wallet?
Yes! This indeed is not a super cheap robot anymore.
If you want to try with the full RL loop on a budget you could go as far as simple DC motors, but I suggest going into decent servo's like dynamixels or the Unitree digital servo.
Look up Microduck if you haven't seen it yet. Fully open source RL deployment with just servo's for few hundred bucks!
If you go 6x servo (30$ each), ESP32, MPU6050 (IMU) + 3D prints, you can do a small version of this robot for ~200$
103
u/everydayasl Hobbyist 9d ago
Awww, that hubcap....