18 ms·

Show HN: AirBending – Hand gesture based macOS app MIDI controller

92 points by bepitulaz 1y ago

johng 1y ago
This is really cool, thanks for sharing.
Animats 1y ago
Theremin. Here's someone playing a theremin who's reasonably good.[1] [1] https://www.youtube.com/watch?v=K6KbEnGnymk https://www.youtube.com/watch?v=K6KbEnGnymk
CharlesW 1y ago
Imogen Heap demonstrating her Mi.Mu gloves: https://www.youtube.com/watch?v=ci-yB6EgVW4 https://www.youtube.com/watch?v=ci-yB6EgVW4 Using the gloves during an NPR Tiny Desk concert: https://www.youtube.com/watch?v=3QtklTXbKUQ&t=555s https://www.youtube.com/watch?v=3QtklTXbKUQ&t=555s
vunderba 1y ago
From the article: "When using AirBending for pitch control, you can lock your gestures to specific musical scales and keys. This ensures every note you play is perfectly in tune with your composition" Reminds me of the Moog Theremini - that was a fun bit of kit. https://en.wikipedia.org/wiki/Theremini https://en.wikipedia.org/wiki/Theremini
singularity2001 1y ago
Great work but wouldn't the iPhone with the lidar depth sensor be a better device?
- bepitulaz 1y ago
  It’s on the plan to expand this app for iPhone. But, I haven’t tried lidar, so I decided to release for macOS first. Also with iPhone, I have to think how to transmit MIDI data to DAW on laptop. Well, most likely via USB or network.
  - import 1y ago
    I do this with "Apple Lightning to USB Camera Adapter". iPad basically talks to the midi sequencer via USB.
  - docEdub 1y ago
    Apple devices can do MIDI over Bluetooth. I've used this in the past to send VisionPro hand tracking data as MIDI.
blacksignal 1y ago
Not just a ‘theremin’ — that totally downplays the power of midi. Someone else mentioned the mimu gloves, and I love the idea of a vision based controller - almost everyone has a phone, tablet or laptop with a camera, especially if theyre making music. I also love that this could blur the lines of music playing and dance. Great job OP, thanks for sharing.
- whilenot-dev 1y ago
  The Leap Motion Controller came out in 2014 already (11 years ago, wow!) and isn't very expensive. The SDK was lacking in the beginning if I recall correctly, but a webcam seems to be inferior. Technology isn't the limiting factor for a quite some time now. I'm sure many projects existed to translate gestures to MIDI, some less polished, some more polished[0][1]. Reminds me... I even used two PlayStation Eyes (EUR 5 each) with OpenCV and the EVM algorithm[2] on a ThinkPad X230 for a dance performance piece back in 2015. Movements rather than gestures and OSC instead of MIDI, but it worked great! [0]: https://midipaw.com/ https://midipaw.com/ [1]: https://uwyn.com/geco/ https://uwyn.com/geco/ [2]: https://people.csail.mit.edu/mrub/evm/ https://people.csail.mit.edu/mrub/evm/
abalaji 1y ago
Seems like the same software could be used as a soundtrack for Tai Chi exercises. Would be pretty neat.
- bepitulaz 1y ago
  It seems possible, since Apple’s Vision framework can read body pose too. Maybe I can try it for the next update.
Obscurity4340 1y ago
This instruments timbre and tone are literally dream shit to me, so wavy and I dont know—unearthly/worldly
- greenpizza13 1y ago
  This is not an instrument and does not have a timbre. He is used a midi instrument in a DAW and using this software as a midi controller. This is not a theremin, it’s a programmable controller which you can tie to various midi inputs. Think “slider” or “dial”.
irvanputra 1y ago
I wonder if Linux version will be available.
- bepitulaz 1y ago
  Since I developed it using Vision framework from Apple, the current focus is still for Apple devices. So, not in the near future to develop for Linux and Windows.
  - irvanputra 1y ago
    Thanks for the confirmation!
bjelkeman-again 1y ago
That looks great! Do you have any suggestion for how to learn how to hook this up to Logic for anyone who hasn’t used midi before? I wonder how this would perform under live stage lighting conditions, i.e coloured strong lights and high contrast.
- bepitulaz 1y ago
  Open the app and open Logic Pro. Create a MIDI track on Logic, try waving to the app it should automatically receive MIDI message from all channels and all MIDI devices. Then if you want to filter the track to receive specific MIDI channel from specific device, for example AirBending channel 2, then find it in the dropdown in the MIDI inspector section in the same MIDI track.
  - bjelkeman-again 1y ago
    Very cool. I’ll have to test that.
drcongo 1y ago
I would love this as an AUv3 for iPadOS, any plans?
- bepitulaz 1y ago
  There's a plan for iOS and iPadOS. Thanks for the suggestions, I will try exploring AUv3 when building it.
colecut 1y ago
i grabbed this to play with it, it's pretty cool I seem to only be able to map each axis to a single finger? It seems like if it is detecting two fingers at a time I should be able to use the X and Y on each of them simultaneously?
andy_nguyen 1y ago
[flagged]
xnx 1y ago
I don't have a Mac, so I'm not sure how AirBending works, but here's a free open source one that works in the browser: https://gemini.google.com/share/e2ee937a688d https://gemini.google.com/share/e2ee937a688d (Prompt: "Make an in browser midi theremin using Mediapipe") There's a Web MIDI API, so might be able to reproduce all the functionality in a way that works across devices and operating systems.
uxamanda 1y ago
This is so fun. I built this into a live performance tool I am using this weekend. Thank you for sharing! Minor UX feedback - I find the internal scroll of setting up the presets limiting. Because the whole top half is fixed it is hard to see all my settings at once. Could either the whole panel scroll? - In dark mode, the "Save" button is black on dark grey and looks disabled. - The delete button is scarily prominent and no confimation. Can that be moved inside the preset panel so i dont accidently hit it while performing?

92 points by bepitulaz 1y ago

This is really cool, thanks for sharing.

Theremin. Here's someone playing a theremin who's reasonably good.[1] [1] https://www.youtube.com/watch?v=K6KbEnGnymk https://www.youtube.com/watch?v=K6KbEnGnymk

CharlesW 1y ago

Imogen Heap demonstrating her Mi.Mu gloves: https://www.youtube.com/watch?v=ci-yB6EgVW4 https://www.youtube.com/watch?v=ci-yB6EgVW4 Using the gloves during an NPR Tiny Desk concert: https://www.youtube.com/watch?v=3QtklTXbKUQ&t=555s https://www.youtube.com/watch?v=3QtklTXbKUQ&t=555s

vunderba 1y ago

From the article: "When using AirBending for pitch control, you can lock your gestures to specific musical scales and keys. This ensures every note you play is perfectly in tune with your composition" Reminds me of the Moog Theremini - that was a fun bit of kit. https://en.wikipedia.org/wiki/Theremini https://en.wikipedia.org/wiki/Theremini

singularity2001 1y ago

Great work but wouldn't the iPhone with the lidar depth sensor be a better device?

bepitulaz 1y ago
It’s on the plan to expand this app for iPhone. But, I haven’t tried lidar, so I decided to release for macOS first. Also with iPhone, I have to think how to transmit MIDI data to DAW on laptop. Well, most likely via USB or network.
- import 1y ago
  I do this with "Apple Lightning to USB Camera Adapter". iPad basically talks to the midi sequencer via USB.
- docEdub 1y ago
  Apple devices can do MIDI over Bluetooth. I've used this in the past to send VisionPro hand tracking data as MIDI.

blacksignal 1y ago

Not just a ‘theremin’ — that totally downplays the power of midi. Someone else mentioned the mimu gloves, and I love the idea of a vision based controller - almost everyone has a phone, tablet or laptop with a camera, especially if theyre making music. I also love that this could blur the lines of music playing and dance. Great job OP, thanks for sharing.

whilenot-dev 1y ago
The Leap Motion Controller came out in 2014 already (11 years ago, wow!) and isn't very expensive. The SDK was lacking in the beginning if I recall correctly, but a webcam seems to be inferior. Technology isn't the limiting factor for a quite some time now. I'm sure many projects existed to translate gestures to MIDI, some less polished, some more polished[0][1]. Reminds me... I even used two PlayStation Eyes (EUR 5 each) with OpenCV and the EVM algorithm[2] on a ThinkPad X230 for a dance performance piece back in 2015. Movements rather than gestures and OSC instead of MIDI, but it worked great! [0]: https://midipaw.com/ https://midipaw.com/ [1]: https://uwyn.com/geco/ https://uwyn.com/geco/ [2]: https://people.csail.mit.edu/mrub/evm/ https://people.csail.mit.edu/mrub/evm/

abalaji 1y ago

Seems like the same software could be used as a soundtrack for Tai Chi exercises. Would be pretty neat.

bepitulaz 1y ago
It seems possible, since Apple’s Vision framework can read body pose too. Maybe I can try it for the next update.

Obscurity4340 1y ago

This instruments timbre and tone are literally dream shit to me, so wavy and I dont know—unearthly/worldly

greenpizza13 1y ago
This is not an instrument and does not have a timbre. He is used a midi instrument in a DAW and using this software as a midi controller. This is not a theremin, it’s a programmable controller which you can tie to various midi inputs. Think “slider” or “dial”.

irvanputra 1y ago

I wonder if Linux version will be available.

bepitulaz 1y ago
Since I developed it using Vision framework from Apple, the current focus is still for Apple devices. So, not in the near future to develop for Linux and Windows.
- irvanputra 1y ago
  Thanks for the confirmation!

bjelkeman-again 1y ago

That looks great! Do you have any suggestion for how to learn how to hook this up to Logic for anyone who hasn’t used midi before? I wonder how this would perform under live stage lighting conditions, i.e coloured strong lights and high contrast.

bepitulaz 1y ago
Open the app and open Logic Pro. Create a MIDI track on Logic, try waving to the app it should automatically receive MIDI message from all channels and all MIDI devices. Then if you want to filter the track to receive specific MIDI channel from specific device, for example AirBending channel 2, then find it in the dropdown in the MIDI inspector section in the same MIDI track.
- bjelkeman-again 1y ago
  Very cool. I’ll have to test that.

drcongo 1y ago

I would love this as an AUv3 for iPadOS, any plans?

bepitulaz 1y ago
There's a plan for iOS and iPadOS. Thanks for the suggestions, I will try exploring AUv3 when building it.

colecut 1y ago

i grabbed this to play with it, it's pretty cool I seem to only be able to map each axis to a single finger? It seems like if it is detecting two fingers at a time I should be able to use the X and Y on each of them simultaneously?

andy_nguyen 1y ago

[flagged]

xnx 1y ago

I don't have a Mac, so I'm not sure how AirBending works, but here's a free open source one that works in the browser: https://gemini.google.com/share/e2ee937a688d https://gemini.google.com/share/e2ee937a688d (Prompt: "Make an in browser midi theremin using Mediapipe") There's a Web MIDI API, so might be able to reproduce all the functionality in a way that works across devices and operating systems.

uxamanda 1y ago

This is so fun. I built this into a live performance tool I am using this weekend. Thank you for sharing! Minor UX feedback - I find the internal scroll of setting up the presets limiting. Because the whole top half is fixed it is hard to see all my settings at once. Could either the whole panel scroll? - In dark mode, the "Save" button is black on dark grey and looks disabled. - The delete button is scarily prominent and no confimation. Can that be moved inside the preset panel so i dont accidently hit it while performing?