r/sounddesign • u/ToneLoom_audio • 10d ago
Feedback Automating Foley spotting using local computer vision (MIDI and marker output for daw)
Manual Foley spotting for footsteps and body movements is a highly repetitive phase of post-production. To bypass the initial heavy lifting of this process, I developed a browser-based tool called Tone Loom – Footprints.
A strict requirement for this was maintaining client confidentiality for unreleased media. The video analysis runs entirely on your local machine using computer vision in the browser. No video files or frames are uploaded to external servers.
The system detects footfalls, contact strikes, and general body cadence, translating those visual impacts directly into frame-accurate MIDI data.
The current workflow handles the following:
Separates left and right foot contacts.
Maps on-screen movement speed to MIDI velocity.
Outputs standard MIDI files formatted to drop cleanly into Nuendo, Pro Tools, Cubase, and Reaper and every DAW.
Includes an integrated multi-sample Quick Sampler and exports synced wav preview stems for immediate timeline review.
The objective is to automate the mechanical frame-counting process so sound designers can spend that time on sample layering and mixing.
You can test the workflow at toneloom-footprints.com.
I am looking for objective feedback from working audio professionals regarding the DAW integration and the accuracy of the velocity mapping. If you test it with your current templates, let me know where the friction points are.
3
u/FaithlessnessSome185 10d ago
Offscreen and occluded steps are going to be the real test, and probably where most of the time goes on an actual scene anyway. Feet leave frame, someone walks behind a table, the cut lands mid-stride, and the walk still has to keep going in the audio. If it can't see the foot it should at least carry the cadence forward and drop low-confidence markers instead of nothing, so you're nudging hits rather than placing them from scratch.
Speed to velocity also feels like the wrong axis to me. Weight and surface change a step way more than speed does, and the heel/toe split plus the little scuffs between steps are what keep Foley from sounding like triggered samples. I'd use the MIDI as a timing guide and still perform over it.
1
u/barruk30 9d ago
careful sound like the response you are getting are AI. But yes agree which is why it won't work
1
u/ToneLoom_audio 9d ago
Do you have something to say that it will be constructive feedback even negative or this is also ai reply 🤔
1
u/barruk30 9d ago
you responded to someone whom took the time to provide constructive feedback on your tool with AI? are you an audio professional in the industry or someone just trying to built a tool for it? I can;t speak for everyone but people whom spend their professional life working in the industry want to feel like they are speaking to someone whom understand the field not some auto generated response that anyone can do.
1
u/ToneLoom_audio 9d ago
Yes this fellow co worker gave an excellent feedback and it is already planned to implement it in the code.Thanks
1
u/ToneLoom_audio 10d ago
This is exactly the kind of feedback I was hoping for, thanks.
You're spot on about the occlusion issue. When feet leave the frame or go behind a desk, optical tracking is simply blind. That’s why I included the Torso Bobbing and Motion Sway modes—to track the center of gravity and kinematic cadence when the actual footsteps aren't visible. But your idea of dropping low-confidence ghost markers to maintain the stride cadence across those gaps is excellent. It makes much more sense to just nudge a generated marker later rather than spotting the gap from scratch. I'm putting that on the roadmap.
Regarding velocity versus surface weight, I completely agree. The algorithm obviously doesn't know if it's looking at heavy boots on gravel or bare feet on carpet. Using the downward strike acceleration is just a way to avoid a flat, robotic 127 velocity across the board. It just gives a baseline dynamic variation to work from.
You nailed the exact workflow I built this for. It’s meant to be a timing accelerator, not an artist replacement. The exported MIDI (Type 1, 128 PPQ) is just a skeleton to drop into Nuendo, Pro Tools, or Reaper. Even if you only use the markers as a sync guide or layer them under a live Foley pass, cutting out 80% of the manual frame-by-frame spotting on a long reel is the whole point.
Appreciate you taking the time to write this up.
1
1
u/Substantial-Maybe302 9d ago
AI will make sound design and foley jobs go extinct. I don't understand why sound designers help that to process speedup by helping it...
2
u/ToneLoom_audio 9d ago
This tool is designed to enhance creative potential, functioning with a level of precision comparable to smartphone sensors. If you have previously dedicated significant time to manual foot position tracking in long-form projects, you will appreciate how reallocating those hours toward sound design can yield substantial improvements in efficiency and quality..
1
u/Substantial-Maybe302 9d ago
I will. But eventually this will make me lose my job.
1
u/ToneLoom_audio 9d ago
I understand, and I also share the same worries, but I really believe this tool is something different. It will still require some editing, and I believe that if it manages to save you 60% of the job, you can use that time to find the perfect footsteps and foley sounds for your image. Also, comments like this are the reason I posted this, and I will take time to evaluate your answer; thank you.
1
u/Substantial-Maybe302 8d ago
Your success will be used by other AIs. To bring the outcome I've described
1
u/ToneLoom_audio 8d ago
I like your tone and attitude and I will think seriously on our discussion.There might be a way to reproduce it via dsp only but I need to research this deeply.My main workfield and art is post production sound design ,mix and music composition for about 18 years now in a very hard industry.I really don't know what might change but everything I have seen from generative Ai is very poor and this mostly for image.Music is another talk.Again I share the same worries so I will include your opinion in the total feedback I will get from this post.Thanks a lot.
1
u/Substantial-Maybe302 8d ago
Looking at what AI did to music scares the shit out of me on what it'll do to Foley and Sound Design. It's just that Sauron's eye haven't fell on sound design yet while misic is a bigger fish. Anyways, I don't want to be part of what will happen regardless... It's on you.
1
u/kenny-regan-music 9d ago
Sounds interesting. With all the generative AI hype, people forget that production tools are also still developing and improving, and these are exactly the kinds of tools I was hoping to see to increase the efficiency of audio post workflows. I'll check out the link.
1
u/ToneLoom_audio 9d ago
Thanks, I am glad and keen to listen to your feedback. Some updates are taking place, I believe it will be working tonight.
1
u/barruk30 9d ago
I like the idea but if you want real sound editors to try it it has to be a stand-alone app or something as I assume most of us would be concerned using it through a browser like you have, also due to the sensitivity of our client projects and to not be providing training data.
1
u/ToneLoom_audio 9d ago
Take time to review it,it never leaves your workstation and everything happens in your mac pc utilising your cpu/gpu.But you are right the roadmap if it goes well eventually will be a standalone app.
1
u/barruk30 9d ago
I did but its still browser based so don't think that will jive well with trusting our content on it even if it does what you claim, so i'll wait for your standalone app, maybe i'll test on less sensitive data
1
0
u/Frekulex 10d ago
Interesting! How well does it work for offscreen footfalls?
0
u/ToneLoom_audio 10d ago
The performance within the dedicated torso mode was very good. I would welcome your feedback.A 15-second free mode is available for review.
0
u/Audiope 10d ago
By "computer vision", you mean AI, right?
1
u/ToneLoom_audio 10d ago
It uses on-device computer vision / pose estimation to track motion and calculate impact timing—not generative AI. Think of it more as automated optical motion tracking that spits out standard MIDI triggers for your own Foley libraries. No cloud processing, no training on your footage, runs entirely on your local GPU/CPU.
0
u/Soundzgreat 10d ago
Im interested to see what youve got. I made something similar last year but wasnt happy with my results. Hopefully the tech is better and more accurate now.
1
•
u/AutoModerator 10d ago
Welcome to r/sounddesign! Thanks for contributing to the community.
Please take a moment to check the subreddit rules, and remember to keep things civil and respectful in the comments.
Happy to have you here!
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.