LOADING_
The engine behind the chord sheets: a Python ML service turning an mp3 or a YouTube link into chords, timed lyrics and sections (Demucs, key-aware Viterbi chords, faster-whisper, spectral section clustering), and a local Tauri Mac app with a Synthesia-style player that trains on unique sections only — all offline.
Bhajans and kirtans are learned by ear: lyrics in Sanskrit and Bengali, chords passed hand to hand, "sheet music" existing at best as someone's phone recording. The web library of chord sheets is BhajanApp. But where do the sheets themselves come from when there are recordings and no notation?
"A tool that turns any recording into a draft chord sheet. AI gives the draft — a person quickly finishes it."
So two more products appeared in the same repository: the audio-service ML pipeline, turning an mp3 or a YouTube link into chords, timed lyrics and sections, and the local Mac app BhajanApp Desktop — a personal trainer that computes everything on your own machine and sends nothing out. 160 commits since July 2025.
Python + FastAPI: an mp3 or a YouTube link → stems, MIDI, chords, timed lyrics, sections. The pipeline tolerates missing heavy dependencies: a "light" tier on librosa and a full one with Demucs and MuScriptor.
The project's main lesson: don't multiply inaccuracies. Four ML steps used to stack into noise ("178 chord changes in 8 minutes"). Now chords are computed the way a musician writes them: key estimation, beat tracking, one chord per bar, a diatonic bias against maj↔min flicker, and Viterbi with a change penalty.
# chords.py — one chord per bar, key-aware, Viterbi with a change penalty
tonic, is_minor = _estimate_key(chroma)
in_key = _diatonic_labels(tonic, is_minor)
bonus = np.array([0.12 if labels[k] in in_key else 0.0 for k in range(len(labels))])
_tempo, beats = librosa.beat.beat_track(y=y, sr=sr, hop_length=hop)
bounds = sorted({0, *beat_list[::beats_per_bar], chroma.shape[1]}) # bars, not beats
seg_chroma = [np.median(chroma[:, a:b], axis=1) for a, b in zip(bounds, bounds[1:])]
scores = templates @ normalize(np.stack(seg_chroma, axis=1)) + bonus[:, None]
path = _viterbi_path(scores, settings.chord_change_penalty) # hold the chord until there's a reason to change
def _viterbi_path(scores, change_penalty):
"""'Stay' is free, 'switch' costs change_penalty — O(K·T) via the best previous state."""
K, T = scores.shape
dp = np.full((K, T), -1e18); back = np.zeros((K, T), dtype=int); dp[:, 0] = scores[:, 0]
for t in range(1, T):
prev = dp[:, t - 1]; best = int(prev.argmax()); switch = prev[best] - change_penalty
stay = prev >= switch
dp[:, t] = np.where(stay, prev, switch) + scores[:, t]
back[:, t] = np.where(stay, np.arange(K), best)
...
Lyrics are timed with faster-whisper word timestamps (output romanised across scripts); vocal MIDI comes from MuScriptor or pyin with a median filter, key snapping and beat quantisation.
Verse and chorus are found by self-similarity: beat-synchronous CQT, a recurrence matrix, Laplacian segmentation and spectral clustering — repeats of one section land next to each other in eigenvector coordinates. Each section carries a group — a self-similarity group id shared by repeats. The trainer builds on it: the route plays only unique sections, not the whole song with its repeats.
# sections.py — computed per beat so a repeated verse matches itself even at a different tempo
cqt_sync = librosa.util.sync(cqt, beats, aggregate=np.median)
rec = librosa.segment.recurrence_matrix(cqt_sync, width=3, mode="affinity", sym=True)
rec = librosa.segment.timelag_filter(median_filter)(rec, size=(1, 7))
affinity = mu * rec + (1 - mu) * rec_path # long-range similarity + local continuity
lap = scipy.sparse.csgraph.laplacian(affinity, normed=True)
_evals, evecs = scipy.linalg.eigh(lap)
seg_ids = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(evecs[:, :k] / norms)
A personal practice tool: everything runs on your machine, models live on disk, nothing leaves. Tauri v2 (a ~10 MB Rust shell) spawns the same FastAPI as a sidecar on 127.0.0.1. Inside: mp3/YouTube import, a waveform track with a section-boundary editor, a Synthesia-style player with falling notes on a virtual keyboard, a piano-roll over the recognised MIDI with note playback and .mid export. Bundled as a .dmg, launched with a double-click.
A kirtan recording becomes a draft chord sheet in minutes instead of an evening of working it out by ear, and practice follows the unique sections rather than the whole song with its repeats. The main lesson: don't multiply inaccuracies — fewer ML steps, but reliable ones, with manual editing as part of the product rather than an admission of defeat.