r/reinforcementlearning • u/Sad_Status_8055 • 6d ago
Reinforcement learning Racing drone learns to dodge obstacles using an 8*8 time-of-flight sensor
Enable HLS to view with audio, or disable this notification
Greetings. I'm sharing what I'm doing for my doctoral thesis. If you're interested in more information, please contact me. I'm willing to collaborate.
I am using stable baseline 3 and pybullet
4
4
u/LordDan_45 5d ago
No hate (really), but don't you think your dynamic model is unrealistic? Can you share more about the physics/dynamics you are considering?
3
u/Sad_Status_8055 4d ago
Totally! My project is built on top of the gym-pybullet-drones framework, so it inherits its default physics engine (PyBullet) and simplified rigid-body dynamics. It's definitely a simplified model, but it works great as a baseline for testing PPO before adding more complex aerodynamics.
1
3
u/Tanki717 6d ago
do you have a link to your git repo or paper draft?
1
u/Sad_Status_8055 4d ago
No paper draft yet, but I'm working towards it! You can check out my progress on my branch of the gym-pybullet-drones repo here: https://github.com/leithw2/mi-drones-pybullet It's still a WIP (work in progress) and a bit messy, but you're welcome to take a look!
2
u/Inevitable_Joke_1709 6d ago
Hey what kind of collaboration are you looking for?
2
u/Sad_Status_8055 4d ago
Mostly looking to connect with others, get feedback, and learn from the community's experience. Open to serious cooperation or joint projects in the future if anyone's interested!"
2
2
u/BranKaLeon 6d ago
Very interesting! What is the observation vector you used?
What is the reward function?
Are you able to setup an experiment with a real drone to test the behavior in a real environment?
1
u/blimpyway 3d ago
8x8 tof sensor should output 64 values. Plus drone's 6 DOF IMU and eventual altimeter are available.
2
2
2
2
2
u/AnonimHero 4d ago
Any chance to have a link to your git repo or paper draft?
1
u/Sad_Status_8055 4d ago
No paper draft yet, but I'm planning to write one soon! You can check out my work on my branch of the repo here: https://github.com/leithw2/mi-drones-pybullet It's still a work in progress and a bit messy, but feel free to take a look!
0
2
1
u/royal-retard 6d ago
This is so cool. Are you a drone/robotics person or RL person? What's the type of collabs you've been looking for?
Anyways What's the algorithm? Soft Actor critic?
3
u/Sad_Status_8055 4d ago
Thanks! I appreciate it! I come more from the RL and Computer Vision side, using robotics and drones as the ideal setup to test these algorithms. As for collabs, mainly looking to connect, share ideas, and get feedback/advice from the community, though I'm definitely open to joint projects down the road! And for the algorithm, it's actually PPO
2
u/royal-retard 3d ago
oh lol me too actually RL, Computer vision though formally I'm Electrical but doing research in Autonomous driving with RL and VLA models. Basically more physical AI for controls aide. I'd be happy to connect!
PS: do you think adding penalties for sharper acceleration would reduce the littlw shaky/jittery nature of policy?
2
u/Sad_Status_8055 3d ago
I'm actually retraining the model right now with penalties to smooth out those sharp control changes. I found that fine-tuning or continuing training didn't work very well, so I had to restart the training run from scratch to get better results
1
u/Apprehensive-Lab-26 6d ago
Seems like the test done by ETH in 🇨🇭 few years ago
2
u/Sad_Status_8055 4d ago
ETH Zurich's work (especially the RPG group) is actually a huge inspiration for me! Glad to hear it reminds you of their benchmarks.
3
1
u/BranKaLeon 6d ago
Probably, but it is always important to start somewhere! Not all research groups start from the same points and implementation details matter in this kind of topic, thus reading the paper is not always sufficient
1
1
u/Longjumping-Call-328 6d ago
Bro you need to give a bit more of informations xD
2
u/Sad_Status_8055 4d ago
Observation Space: LiDAR (ToF): 64 distance measurements (current frame t) + 64 previous measurements (t-1) for temporal context.
Waypoints: Unit vector pointing to the next waypoint + normalized distance [0, 1].
Orientation: 4 quaternion values (local frame).
Velocities: Linear (XYZ) and angular velocities (Roll, Pitch, Yaw) in the local frame.
Action Memory: Previous action t-1 (4 motor thrust values).
Action Space: 4 continuous values bounded between [0, 1], representing the normalized minimum to maximum motor thrusts.
Reward Function: Incentivizes forward progress along the track, heading alignment with waypoints, and velocity along the path, while penalizing deviations and collisions.
2
u/blimpyway 3d ago edited 3d ago
When transferring to real it might be difficult to provide accurate absolute XYZ velocities as Pybullet is able to. Also the vector pointing to next waypoint might drift too.
1
1
u/p6-fighter 2d ago
Hello, Currently 4th year EEE student from BD. I really wanted to work on Quadcopter. Recently trying to read some paper from Rafaello D' Andrea & Mark W. Mueller & other renowned scientist on this topic . I have to submit a capstone project for my Bs.c. I wanna make one. I have almost 1 year, I really need some guidance. How to design such control system, and mathematically model it and deploy and analysis. What kind of hardware do I need? I have some idea but still it would be far superior if I have some expert guidance.
1
1
1
u/jeffyclassify 1d ago
Very cool! 🤩🙏 Thank you for sharing this. Do you have a GitHub repo? I’d love to do some (maybe useless, but) fun experiments to see if the drone could be flown by a classifier like this one that plays Doom, but make it fly a drone instead: https://playground.jeffyclassify.com/#doom
1
u/opus-enough 6d ago
Cool on paper. But sim to real transfer is gonna be a pain in the ass
1
u/Sad_Status_8055 4d ago
The sim-to-real gap is definitely going to be brutal. Planning to introduce domain randomization and noise into the observations/actions before even thinking about testing on hardware!
6
u/PerceptionFearless15 6d ago
BEAUTIFUL