Recommended
Recommended
More Related Content
Similar to PhD Thesis Marius Miron - Source Separation Methods for Orchestral Music
Similar to PhD Thesis Marius Miron - Source Separation Methods for Orchestral Music (20)
Recently uploaded
Recently uploaded (20)
PhD Thesis Marius Miron - Source Separation Methods for Orchestral Music
- 1. Source separation methods for orchestral music: timbre-informed and score-informed strategies Universitat Pompeu Fabra, Barcelona, Music Technology Group Thesis supervisors: Dr. Jordi Janer Dr. Emilia Gómez Marius Miron Thesis defence committee: Dr. Emmanuel Vincent Dr. Xavier Serra Dr. Maximo Cobos Doctoral thesis presentation
- 2. Outline 2 ApplicationsDatasetsIntroduction Deep learningMatrix decomposition Conclusions
- 3. Outline 3 Applications PART V Datasets PART II Introduction PART I Deep learning PART IV Matrix decomposition Conclusions PART III
- 4. Introduction
- 5. Music source separation 5 Audio sources + Music source separation Estimated audio sources
- 7. Orchestral music source separation 7
- 8. Orchestral music source separation 8
- 9. Informed source separation 9 Audio sources + Music source separation Estimated audio sources Musical score Sources’ timbre
- 10. Timbre-informed source separation 10 Frequency (Hz) STFTmagnitude
- 11. Score-informed source separation 11 Frequency(Hz) STFT magnitude spectrogram Time (s) Mahler’s Symphony no. 1 Mixture French horns
- 12. Score-informed source separation 12 Frequency(Hz) STFT magnitude spectrogram Time (s) Mahler’s Symphony no. 1 Mixture French horns
- 13. Score-informed source separation 13 Frequency(Hz) STFT magnitude spectrogram Time (s) Mahler’s Symphony no. 1 Mixture French horns
- 14. Informed source separation 14 Audio sources + Music source separation Estimated audio sources Musical score Isolated samples Score follower Timbre learning
- 15. Context 15 Performances as Highly Enriched aNd Interactive Concert eXperiences Score following Concert notes Conductor gestures Visualization: chords
- 19. Challenges 19 DSD100 dataset Bach10 dataset
- 20. Challenges 20 Multi-microphone recordings: leakage between sources
- 21. Challenges 21 Large number of sources of similar timbre
- 22. Challenges 22 Less disjoint representations Time (s) Mahler’s Symphony no. 1 Frequency(Hz)
- 23. Challenges 23
- 25. Opportunities 25
- 26. Opportunities 26
- 27. Opportunities 27 Tempo Dynamics Timbre Local timing Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance: The state of the art. JNMR, 33(3), 203-216.
- 31. Note refinement 31 Score follower Niedermayer, B. (2012). Audio-to-score alignemnt: data acquisition in the context of computational musicology. PhD Thesis, Johannes Kepler University, Linz
- 33. Opportunities 33 Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance: The state of the art. JNMR, 33(3), 203-216. Part IV Tempo Dynamics Timbre Local timing
- 34. Opportunities 34
- 35. Contributions • Note refinement using image processing to fix local misalignments • Deep learning framework for classical music source separation • Deep learning score-informed source separation 35
- 36. Contributions • Reproducible research (datasets, frameworks, code, results) • Orchestral dataset with score annotations • Cloud-based source separation framework on Repovizz. • Deep learning repository for audio applications. 36
- 37. Datasets
- 38. PHENICX-Anechoic 38 Piece Dura tion Period No. sources Total no. instruments Max. instruments/ source Mozart 3min 47s classical 8 10 2 Beethoven 3min 11s classical 10 20 4 Mahler 2min 12s romantic 10 30 4 Bruckner 1min 27s romantic 10 39 12
- 41. Other dataset Bach10 Dataset 10 Bach chorales, 20-40 seconds each Perfectly and automatically aligned scores 41 Duan, Z. and Pardo B.,(2011), Bach10 dataset
- 43. Source separation using matrix decomposition
- 44. Score-informed source separation 44 Logfrequency (1/4semitoneres) Time (s) Trained timbre bases Time gains initialised with score Magnitude
- 45. Note refinement 45 Score follower Niedermayer, B. (2012). Audio-to-score alignemnt: data acquisition in the context of computational musicology. PhD Thesis, Johannes Kepler University, Linz
- 46. Note refinement for source separation 46 Logfrequency (1/4semitoneres) Time (s) Trained timbre bases Time gains initialised with score Magnitude Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
- 47. Note refinement using NMF gains 47 Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015 Time gains after NMF Time (s) Blob detection for a single note
- 48. Note refinement using NMF gains 48 Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015 Time gains after NMF Time (s) Blob detection for a single note
- 49. Note refinement using pitch salience 49 Miron et al (2014). Audio-to-score alignment at the note level for orchestral recordings, ISMIR 2014 frequencyrelativetonote'sf0(centbins) time (seconds) ➩ A B time (seconds)
- 50. Note refinement using NMF gains 50Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015 Evaluation with Bach10 dataset: alignment NMF gains score follower NMF gains pitch salience pitch salience score follower Error threshold (seconds) Alignmentrate
- 51. Note refinement using NMF gains 51 Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015 Source separation evaluation metrics - BSS EVAL : • Signal to Distortion Ratio (SDR) • Signal to Interference Ratio (SIR) • Signal to Artefacts Ratio (SAR) • Image to Spatial Distortion Ratio (ISR)
- 52. Note refinement using NMF gains 52 Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015 Time gains after NMF Time (s) Blob detection for a single note
- 53. Note refinement using NMF gains 53 Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015 A B C D E F ground truth misaligned refined pitch salience refined NMF gains implicit refined NMF gains submatrix time & frequency Evaluation with Bach10 dataset: source separation
- 54. Multi-channel source separation 54 Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 55. Multi-channel source separation 55 Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016 Ref1
- 56. Multi-channel source separation 56 Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016 Ref2
- 57. PARAFAC gains estimation 57 [ ]… Ref2 Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 58. Multi-channel source separation 58 GT Ali Ext Ref1 Ref2 ground truth refined gains extended refined gains multi submatrix aligned Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 59. Multi-channel source separation 59 SDR(dB) Mozart Beethoven Mahler Bruckner Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 60. Multi-channel source separation 60 SDR(dB) Mozart Beethoven Mahler Bruckner Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 61. Multi-channel source separation 61 SIR(dB) Mozart Beethoven Mahler Bruckner Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 62. Multi-channel source separation 62 SAR(dB) Mozart Beethoven Mahler Bruckner Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings, Journal of Electrical Engineering and computer science 2016
- 63. Source separation using deep learning
- 64. Deep learning source separation 64 (J,T,F) (T,F) Neural network
- 65. Deep learning source separation 65 (J,T,F) (T,F) Neural network Advantages: - faster processing - non-linear, multi-layered Disadvantages: - Slower training - Dependent on training data
- 66. Deep learning source separation 66 conv1 fshape(1,30) stride(1,4) conv2 fshape(20,1) stride(1,1) dense1 256 inverse conv2 inverse conv1 (1,T,F) (J,T,F) (30,T,F) 11 (30,T,F) 2 2 dense2 Jx30xTxF 1 1 = with (J,T,F) (J,T,F) (T,F) Jx(30,T,F) 11 Jx(30,T,F) 2 2 Chandna et al,(2017). Monaural source separation using convolutional neural networks LVA/ICA 2017
- 67. Data driven methods 67 DSD100 dataset Bach10 dataset We do not have training data for orchestral music
- 68. Data generation method 68 Tempo Dynamics Timbre Local timing Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance: The state of the art. JNMR, 33(3), 203-216.
- 69. Data generation method 69 Tempo Dynamics Timbre Local timing Audio Isolated Instruments
- 70. Data generation method 70 Miron et al,(2017). Generating data to train neural networks for classical music source separation SMC 2017 Score Augmented Synthesis Audio Augmentation Multi-track Renditions
- 71. Data generation method 71 Miron et al,(2017). Generating data to train neural networks for classical music source separation SMC 2017 STFT Data Processing Multi-track Renditions CNN Training Trained model
- 72. Data generation method 72 Miron et al,(2017). Generating data to train neural networks for classical music source separation SMC 2017 STFT Data Processing Multi-track Renditions CNN Training Trained model STFT Data Processing Target piece CNN Separation Separated Sources
- 73. Score-informed separation with CNN 73 Score-based binary matrices Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks. ISMIR 2017
- 74. 74 Score-based soft masks Score-informed separation with CNN
- 75. 75 STFT magnitude spectrogram Score-informed separation with CNN
- 77. Score-informed separation with CNN 77 Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks. ISMIR 2017 conv1 fshape(1,30) stride(1,4) conv2 fshape(20,1) stride(1,1) dense1 256 inverse conv2 inverse conv1 (J,T,F) (J,T,F) (30,T,F) 11 (30,T,F) 2 2 = with (J,T,F) (J,T,F) (T,F) (30,T,F) 11 (30,T,F) 2 2 dense2
- 78. Experiments CNN autoencoder NMF 78 Multi-source filter model Score informed Trained on RWC vs Score-informed Trained on renditions synthesised with RWC Duan, Z. and Pardo B.,(2011), Bach10 dataset
- 79. Experiments PA 79 Automatically aligned score Tolerance window around -Onsets -Offsets andPerfectly aligned score Duan, Z. and Pardo B.,(2011), Bach10 dataset
- 80. Results 80 Mean(dB) SDR SIR SAR Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks. ISMIR 2017
- 81. Results 81 Mean(dB) SDR SIR SAR Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks. ISMIR 2017
- 82. Results 82 Mean(dB) SDR SIR SAR Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks. ISMIR 2017
- 83. Results How much data we need? 83 No. of instances SDR(dB) Sampling: Bootstrapping with replacement Fixed samples Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks. ISMIR 2017
- 84. Applications
- 85. Applications 85 Repovizz server Source separation server Cloud-based source separation
- 88. Applications 88
- 89. Conclusions
- 90. Contributions 90
- 91. Contributions 91
- 92. Contributions 92 blob note before blob note after t=0.2 t=1 t=1.4 time(analysis windows) centbins
- 95. Contributions 95
- 96. Contributions 96 Code: https://github.com/MTG/DeepConvSep Data: .wav and .mat DOIDOI 10.5281/zenodo.100913610.5281/zenodo.1009136 Reproducible research
- 97. Contributions 97
- 100. Future work • Instrument recognition using deep learning • Deepconvsep is already a starting point and a baseline • Better databases : NSynth to generate training data • Explore other music traditions, genres, and a more flexible context 100
- 101. Publications by the author 101 Peer-reviewed journals • Miron, M., Carabias-Orti, J.J., Bosch, J.J., Gomez, & Janer, J. (2016). Score-Informed Source Separation for Multichannel Orchestral Recordings. Journal of Electrical and Computer Engineering Full articles in peer-reviewed conferences • Miron, M., Gomez, & Janer, J. (2017). Monaural score-informed source separation for classical music using convolutional neural networks. In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR) • Miron, M., Gomez,E., & Janer, J. (2017). Generating data to train convolutional neural networks for classical music source separation. In Sound and Music Computing Conference (SMC) • Martel, H., Miron, M. (2017). Data augmentation for deep learning source separation of HipHop songs. In 10th International Workshop on Machine Learning and Music (MML) • Chandna, P.,Miron, M., Janer, J., & Gomez, E. (2017). Monoaural Audio Source Separation Using Deep Convolutional Neural Networks. In 13th International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA) • Miron, M., Carabias-Orti, J.J., & Janer, J. (2015). Improving score-informed source separation for classical music through note refinement. In 16th International Society for Music Information Retrieval Conference (ISMIR) • Miron, M., Carabias-Orti, J.J., & Janer, J. (2014). Audio-to-score alignment at the note level for orchestral recordings. In 15th International Society for Music Information Retrieval Conference (ISMIR)
- 102. Source separation methods for orchestral music: timbre-informed and score-informed strategies Universitat Pompeu Fabra, Barcelona, Music Technology Group Thesis supervisors: Dr. Jordi Janer Dr. Emilia Gómez Marius Miron Thesis defence committee: Dr. Emmanuel Vincent Dr. Xavier Serra Dr. Maximo Cobos Doctoral thesis presentation