sync_audio_audio_simple.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Demo: Audio-audio synchronization with chroma features and MrMsDTW\n",
    "\n",
    "In this notebook, we'll show a minimal example for the use of the SyncToolbox for music synchronization. We will take two recordings of the same musical piece (the first song of Franz Schubert's \"Winterreise\"), compute chroma representations of both recordings and align them using classical dynamic time warping (DTW) and multi-resolution multi-scale DTW (MrMsDTW). We will also compare the runtimes of the two algorithms.\n",
    "\n",
    "For an explanation of chroma features and DTW, see [1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Loading some modules and defining some constants used later\n",
    "import time\n",
    "import librosa.display\n",
    "import matplotlib.pyplot as plt\n",
    "import IPython.display as ipd\n",
    "from libfmp.b.b_plot import plot_signal, plot_chromagram\n",
    "from libfmp.c3.c3s2_dtw_plot import plot_matrix_with_points\n",
    "\n",
    "from synctoolbox.dtw.core import compute_warping_path\n",
    "from synctoolbox.dtw.cost import cosine_distance\n",
    "from synctoolbox.dtw.mrmsdtw import sync_via_mrmsdtw\n",
    "%matplotlib inline\n",
    "\n",
    "Fs = 22050\n",
    "N = 2048\n",
    "H = 1024\n",
    "feature_rate = int(22050 / H)\n",
    "\n",
    "figsize = (9, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading two recordings of the same piece\n",
    "\n",
    "Here, we take recordings of the song \"Gute Nacht\" by Franz Schubert from his song cycle \"Winterreise\" in two performances (versions). The first version is by Gerhard Hüsch and Hanns-Udo Müller from 1933. The second version is by Randall Scarlata and Jeremy Denk from 2006.\n",
    "\n",
    "### Version 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "audio_1, _ = librosa.load('data_music/Schubert_D911-01_HU33.wav', Fs)\n",
    "\n",
    "plot_signal(audio_1, Fs=Fs, ylabel='Amplitude', title='Version 1', figsize=figsize)\n",
    "ipd.display(ipd.Audio(audio_1, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Version 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "audio_2, _ = librosa.load('data_music/Schubert_D911-01_SC06.wav', Fs)\n",
    "\n",
    "plot_signal(audio_2, Fs=Fs, ylabel='Amplitude', title='Version 2', figsize=figsize)\n",
    "ipd.display(ipd.Audio(audio_2, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining chroma representations of the recordings using librosa\n",
    "\n",
    "For most Western classical and pop music, chroma features are highly useful for aligning different versions of the same piece. Here, we use librosa to calculate two very basic chroma representations, derived from STFTs. The plots illustrate the chroma representations of the first 30 seconds of each version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "chroma_1 = librosa.feature.chroma_stft(y=audio_1, sr=Fs, n_fft=N, hop_length=H, norm=2.0)\n",
    "plot_chromagram(chroma_1[:, :30 * feature_rate], Fs=feature_rate, title='Chroma representation for version 1', figsize=figsize)\n",
    "plt.show()\n",
    "\n",
    "chroma_2 = librosa.feature.chroma_stft(y=audio_2, sr=Fs, n_fft=N, hop_length=H, norm=2.0)\n",
    "plot_chromagram(chroma_2[:, :30 * feature_rate], Fs=feature_rate, title='Chroma representation for version 2', figsize=figsize)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning chroma representations using full DTW\n",
    "\n",
    "The chroma feature sequences in the last cell can be used for time warping. As both versions last around five minutes, an alignment can still be computed in reasonable time using classical, full DTW. In the next cell we use the SyncToolbox implementation of DTW to do this. Each feature sequence consists of around 7000 frames, meaning that the matrices computed during full DTW become quite huge - around 14 million entries each! - leading to high memory consumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "C = cosine_distance(chroma_1, chroma_2)\n",
    "_, _, wp_full = compute_warping_path(C=C)\n",
    "# Equivalently, full DTW may be computed using librosa via:\n",
    "# _, wp_librosa = librosa.sequence.dtw(C=C)\n",
    "\n",
    "plot_matrix_with_points(C, wp_full.T, linestyle='-',  marker='', aspect='equal',\n",
    "                        title='Cost matrix and warping path computed using full DTW',\n",
    "                        xlabel='Version 2 (frames)', ylabel='Version 1 (frames)', figsize=(9, 5))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning chroma representations using SyncToolbox (MrMsDTW)\n",
    "\n",
    "We now compute an alignment between the two versions using MrMsDTW. This algorithm has a much lower memory footprint and will also be faster on long feature sequences. For more information, see [2]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "_ = sync_via_mrmsdtw(f_chroma1=chroma_1,\n",
    "                     f_chroma2=chroma_2,\n",
    "                     input_feature_rate=feature_rate,\n",
    "                     verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Runtime comparison\n",
    "\n",
    "We now compare the runtime of both algorithms. During their first call, they may create function caches etc. So, after running the previous cells, we can now test their raw performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "start_time = time.time()\n",
    "C = cosine_distance(chroma_1, chroma_2)\n",
    "compute_warping_path(C=C)\n",
    "end_time = time.time()\n",
    "print(f'Full DTW took {end_time - start_time}s')\n",
    "\n",
    "start_time = time.time()\n",
    "sync_via_mrmsdtw(f_chroma1=chroma_1,\n",
    "                 f_chroma2=chroma_2,\n",
    "                 input_feature_rate=feature_rate,\n",
    "                 verbose=False)\n",
    "end_time = time.time()\n",
    "print(f'MrMsDTW took {end_time - start_time}s')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[1] Meinard Müller: Fundamentals of Music Processing – Audio, Analysis, Algorithms, Applications, ISBN: 978-3-319-21944-8, Springer, 2015.\n",
    "\n",
    "[2] Thomas Prätzlich, Jonathan Driedger, and Meinard Müller: Memory-Restricted Multiscale Dynamic Time Warping,\n",
    "In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 569–573, 2016."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}