7 ms·

I've been working on a karaoke app called Nightingale. You point it at your music folder and it turns your songs into karaoke - separates vocals from instrument

by rzzzzru 6mo ago

I've been working on a karaoke app called Nightingale. You point it at your music folder and it turns your songs into karaoke - separates vocals from instrumentals, generates word-level synced lyrics, and lets you sing with highlighted lyrics and pitch scoring. Works with video files too. Everything runs locally on your machine, nothing gets uploaded. No accounts, no subscriptions, no telemetry. It ships as a single binary for Linux, macOS, and Windows. On first launch it sets up its own isolated Python environment and downloads the ML models it needs - no manual installation of dependencies required. My two biggest drivers for the creation of this were: The lack of karaoke coverage for niche, avant-garde, and local tracks. Nostalgia for the good old cheesy karaoke backgrounds with flowing rivers, city panoramas, etc. Some highlights: Stem separation using the UVR Karaoke model (preserves backing vocals) or Demucs Automatic lyrics via WhisperX transcription, or fetched from LRCLIB when available Pitch scoring with player profiles and scoreboards Gamepad support and TV-friendly UI scaling for party setups GPU acceleration on NVIDIA (CUDA) and Apple Silicon (CoreML/MPS) Built with Rust and the Bevy engine The whole stack is open source. No premium tier, no "open core" - just the app. Feedback and contributions welcome.

defrost 6mo ago
Struggled somewhat with Tjamuku Ngurra by the Tjintu Desert Band, absolutely nailed Mariah Carey's Ken Lee.
- philsnow 6mo ago
  ( https://knowyourmeme.com/memes/ken-lee https://knowyourmeme.com/memes/ken-lee )
antihero 6mo ago
This looks like awesome awesome fun! Will let you know how it runs. What a wonderful idea <3
solstice 6mo ago
Excited to try this out. How well does WhisperX deal with lyrics in say Mandarin or Cantonese? Does it output Hanzi?
- rzzzzru 6mo ago
  I haven't tried Mandarin and Cantonese, but tried Japanese. back at that time, it performed poorly. however, I've tweaked a bunch of settings since then, so maybe it has changed. Hanzi is a supported font and can be output, but the transcript/alignment quality might not be the best
whilenot-dev 6mo ago
Just tried it with B.E.D - Walk Away[0], unfortunately it lost track of the lyrics after 30 secs (Model is "large-v3"). Will play around a bit more, as it would be great to have a working karaoke generator. Some quick feedback: - Needs a way to skip for-/backwards during playback to validate the result - Sentences seem to be recognized (first letter has uppercasing), but periods aren't added - Needs an option to edit results from the track analysis Thanks for keeping it FOSS! [0]: https://www.youtube.com/watch?v=_MFT4H3VoNE https://www.youtube.com/watch?v=_MFT4H3VoNE
- djtango 6mo ago
  Periods in song lyrics?
  - gaudystead 6mo ago
    I'm guessing they mean punctuation in general?
    - ghostly_s 6mo ago
      lyrics do not have punctuation.
- rzzzzru 6mo ago
  hey mate! thanks for your feedback. indeed, I'm running to two problems on the analyzer side: 1. align model sliding off (especially w/ chorus/back vocals present) 2. transcript skipping parts of lyrics in lyrics-heavy tracks (I tried a lot of russian rap, lol) happy for contributions as I'm not that experienced w/ machine learning side of the project, mostly it was emperical "tweak the parameters and look what is changed"
  - rzzzzru 6mo ago
    also model only affects the transcript job (I need to make it clearer in the UI). For the alignment, it's a single model provided by whisperx
evanjrowley 6mo ago
Amazing work! I am thrilled someone was motivated to approach this problem and develop a creative solution like this. There are very limited options for Karaoke, especially in the FOSS space. Most Karaoke apps are super limited and that's driven many Karaoke enjoyers I know to YouTube in search of the songs they want to sing. This solution would give them the power to do even more songs, even better than what's out there now! Questions for you: 1. What CUDA capability level is necessary for Nvidia GPU accelleration to work? 3. Are there any plans to support iGPU/NPU accelleration on AMD and Intel? Asking because those chips are most common in the mini computers sold at low cost these days. My family members who love Karaoke and will be happy to try this. Looking forward to it!
- rzzzzru 6mo ago
  hi! 1. Maxwell+ should work well 3. I would need to explore, you can join the discord or the mailing list on the website! cheers!
throwaway743 6mo ago
Just tried No_4mat's 1992... unfortunately it didn't work :(
samtp 6mo ago
I just want to say how much I love that you used Dean Blunt in the example video
- rzzzzru 6mo ago
  one of my favorite artists and this one is one of my favorite tracks in general. cheers!
jasonfarnon 6mo ago
how's this different from ultrastar singer? https://github.com/rakuri255/UltraSinger https://github.com/rakuri255/UltraSinger has anyone compared the quality of the two?
QuantumNoodle 6mo ago
I studied signal processing in university and my career evolved to not use what I studied. Decades ago, giving an algorithm a sound file and isolating tracks was difficult. How does your implementation accomplish this? Were you involved or did you use something off the shelf? Edit: ah, using neural nets, demucs. I wonder if there is pure math approach that can compete?
djmips 6mo ago
Can you bypass the LRCLIB lookup and always use Whisper?

by rzzzzru 6mo ago

defrost 6mo ago

Struggled somewhat with Tjamuku Ngurra by the Tjintu Desert Band, absolutely nailed Mariah Carey's Ken Lee.

philsnow 6mo ago
( https://knowyourmeme.com/memes/ken-lee https://knowyourmeme.com/memes/ken-lee )

antihero 6mo ago

This looks like awesome awesome fun! Will let you know how it runs. What a wonderful idea <3

solstice 6mo ago

Excited to try this out. How well does WhisperX deal with lyrics in say Mandarin or Cantonese? Does it output Hanzi?

rzzzzru 6mo ago
I haven't tried Mandarin and Cantonese, but tried Japanese. back at that time, it performed poorly. however, I've tweaked a bunch of settings since then, so maybe it has changed. Hanzi is a supported font and can be output, but the transcript/alignment quality might not be the best

whilenot-dev 6mo ago

Just tried it with B.E.D - Walk Away[0], unfortunately it lost track of the lyrics after 30 secs (Model is "large-v3"). Will play around a bit more, as it would be great to have a working karaoke generator. Some quick feedback: - Needs a way to skip for-/backwards during playback to validate the result - Sentences seem to be recognized (first letter has uppercasing), but periods aren't added - Needs an option to edit results from the track analysis Thanks for keeping it FOSS! [0]: https://www.youtube.com/watch?v=_MFT4H3VoNE https://www.youtube.com/watch?v=_MFT4H3VoNE

djtango 6mo ago
Periods in song lyrics?
- gaudystead 6mo ago
  I'm guessing they mean punctuation in general?
  - ghostly_s 6mo ago
    lyrics do not have punctuation.
rzzzzru 6mo ago
hey mate! thanks for your feedback. indeed, I'm running to two problems on the analyzer side: 1. align model sliding off (especially w/ chorus/back vocals present) 2. transcript skipping parts of lyrics in lyrics-heavy tracks (I tried a lot of russian rap, lol) happy for contributions as I'm not that experienced w/ machine learning side of the project, mostly it was emperical "tweak the parameters and look what is changed"
- rzzzzru 6mo ago
  also model only affects the transcript job (I need to make it clearer in the UI). For the alignment, it's a single model provided by whisperx

evanjrowley 6mo ago

Amazing work! I am thrilled someone was motivated to approach this problem and develop a creative solution like this. There are very limited options for Karaoke, especially in the FOSS space. Most Karaoke apps are super limited and that's driven many Karaoke enjoyers I know to YouTube in search of the songs they want to sing. This solution would give them the power to do even more songs, even better than what's out there now! Questions for you: 1. What CUDA capability level is necessary for Nvidia GPU accelleration to work? 3. Are there any plans to support iGPU/NPU accelleration on AMD and Intel? Asking because those chips are most common in the mini computers sold at low cost these days. My family members who love Karaoke and will be happy to try this. Looking forward to it!

rzzzzru 6mo ago
hi! 1. Maxwell+ should work well 3. I would need to explore, you can join the discord or the mailing list on the website! cheers!

throwaway743 6mo ago

Just tried No_4mat's 1992... unfortunately it didn't work :(

samtp 6mo ago

I just want to say how much I love that you used Dean Blunt in the example video

rzzzzru 6mo ago
one of my favorite artists and this one is one of my favorite tracks in general. cheers!

jasonfarnon 6mo ago

how's this different from ultrastar singer? https://github.com/rakuri255/UltraSinger https://github.com/rakuri255/UltraSinger has anyone compared the quality of the two?

QuantumNoodle 6mo ago

I studied signal processing in university and my career evolved to not use what I studied. Decades ago, giving an algorithm a sound file and isolating tracks was difficult. How does your implementation accomplish this? Were you involved or did you use something off the shelf? Edit: ah, using neural nets, demucs. I wonder if there is pure math approach that can compete?

djmips 6mo ago

Can you bypass the LRCLIB lookup and always use Whisper?