1. Rui Pedro Paiva
Melody Detection in
Polyphonic Audio
9º Encontro da APEA
October 20, 2007
Centro de Informática e Sistemas da
Universidade de Coimbra
2. 2 Melody Detection in Polyphonic Audio
Outline
Introduction
Overview
Multi-Pitch Detection
From Pitches to Notes
Identification of Melodic Notes
Evaluation Procedures
Overall Results
Conclusions and Future Work
3. 3 Melody Detection in Polyphonic Audio
Introduction
Motivation and Objectives
• Goal
- Extract a symbolic representation of melody from a polyphonic
audio musical signal
• Broad range of applications
- MIR, music education, plagiarism detection, metadata
• No general-purpose, robust, accurate solution
developed so far
Overall Approach
• Focus on melody, no matter what other sources are
present (figure-ground separation)
4. 4 Melody Detection in Polyphonic Audio
Introduction
Melody Definition
• Subjective concept
- Different people may define and perceive melody in
different ways
- Proposals addressing cultural, perceptual, emotional
or musicological facets of melody
• Context of our work
- “Melody is the individual dominant pitched line in a
musical ensemble”
5. 5 Melody Detection in Polyphonic Audio
Overview
Raw Musical
Signal Melody Notes
Multi-Pitch
Detection
1) Cochleagram
2) Correlogram
3) Pitch Salience
Curve
4) Salient Pitches
Determination of
Musical Notes
1) Pitch Trajectory Construction
2) Trajectory Segmentation
(with onset detection directly
on the raw signal)
Identification of
Melodic Notes
1) Elimination of Ghost Octave
Notes
2) Selection of the Most Salient
Notes
3) Melody Smoothing
4) Elimination of False Positives
Pitches in 46-msec Frames Entire Set of Notes
6. 6 Melody Detection in Polyphonic Audio
Multi-Pitch Detection
0.21 0.22 0.23 0.24 0.25
-0.4
-0.2
0
0.2
0.4
Time (s)
x(t)
a) Sound waveform
Time (s)
Frequencychannel
b) Cochleagram frame
0.21 0.22 0.23 0.24 0.25
20
40
60
80
Time Lag (ms)
Frequncychannel
c) Correlogram frame
0 5 10 15 20
20
40
60
80
0 5 10 15 20 25
0
0.1
0.2
0.3
0.4
Time Lag (ms)
Pitchsalience
d) Summary correlogram
7. 7 Melody Detection in Polyphonic Audio
From Pitches to Notes
Goal
• Quantize temporal sequences of detected
pitches into MIDI notes
Approach
• Pitch Trajectory Construction [Serra]
• Frequency-Based Track Segmentation
• Salience-Based Track Segmentation
8. 8 Melody Detection in Polyphonic Audio
From Pitches to Notes
Pitch Trajectory Construction
1 2 nn-1n-2
Frame Number
MIDInotenumber
9. 9 Melody Detection in Polyphonic Audio
From Pitches to Notes
Frequency-Based Track Segmentation
• Approximation of frequency curves by
piecewise-constant functions (PCFs)
- 1) MIDI quantization
- 2) Merging of PCFs: oscillation,
delimited sequences, glissando
- 3) Time correction
- 4) Assignment of final MIDI note numbers:
tuning compensationb) Filtered PCFs
a) Original PCFs
10. 10 Melody Detection in Polyphonic Audio
From Pitches to Notes
Frequency-Based Track Segmentation
0 100 200 300 400 500
4800
4900
5000
5100
5200
5300
5400
Time (msec)
f[k](cents)
a) Eliades Ochoa's excerpt
0 200 400 600 800
6100
6200
6300
6400
6500
6600
6700
6800
Time (msec)
f[k](cents)
b) Female opera excerpt
11. 11 Melody Detection in Polyphonic Audio
From Pitches to Notes
Salience-Based Track Segmentation
- 1) Candidate Segmentation Points
- 2) Onset Detection on the audio signal
[Scheirer; Klapuri]
- 3) Validation of Segmentation Points by
Onset Matching
0 0.3 0.6 0.9 1.2
20
40
60
80
100
Time (sec)
sF[k]
12. 12 Melody Detection in Polyphonic Audio
Identification of Melodic Notes
Goal
• Separate the melodic notes from the
accompaniment
Approach
• Elimination of Ghost Harmonically-Related
Notes
• Selection of the Most Salient Notes
• Melody Smoothing
• Elimination of Spurious Accompaniment
Notes
• Note Clustering
13. 13 Melody Detection in Polyphonic Audio
Identification of Melodic Notes
Elimination of Ghost Harmonically-
Related Notes
• Delete harmonically-related notes that satisfy
the “common fate” principle
30 60 90 120 150 180 210
0.94
0.96
0.98
1
1.02
1.04
1.06
Time (msec)
NormalizedFrequency
14. 14 Melody Detection in Polyphonic Audio
Identification of Melodic Notes
0 1 2 3 4 5 6
40
50
60
70
80
90
Time (sec)
MIDInotenumber
0 1 2 3 4 5 6
40
50
60
70
80
90
Time (sec)
MIDInotenumber
Selection of the Most
Salient Notes
Melody Smoothing
Time
MIDInotenumber
74
72
60
62
64
66
68
70
R1
a2
R3
a2
R2
a1
R4
a3
15. 15 Melody Detection in Polyphonic Audio
Identification of Melodic Notes
Elimination of Spurious Accompaniment
Notes
- 1) Delete deep valleys in the salience contour
- 2) Delete isolated abrupt duration transitions
0 5 10 15 20 25 30 35
40
50
60
70
80
90
Note Sequence
AveragePitchSalience
16. 16 Melody Detection in Polyphonic Audio
Identification of Melodic Notes
Elimination of Spurious Accompaniment
Notes
0 5 10 15
40
50
60
70
80
90
Time (s)
PitchSalience
17. 17 Melody Detection in Polyphonic Audio
Identification of Melodic Notes
Note Clustering
• Feature Extraction
- Spectral shape, harmonicity, attack transient,
intensity, pitch-related features
• Feature Selection and
Dimensionality Reduction
- Forward feature selection
- Principal Component Analysis
• Clustering
- Gaussian Mixture Models
– 2 clusters: melodic and “garbage” cluster
18. 18 Melody Detection in Polyphonic Audio
Evaluation Procedures
Ground Truth Data
ID Song Title Category Solo Type
1 Pachelbel’s “Kanon” Classical Instrumental
2 Handel’s “Hallelujah” Choral Vocal
3 Enya – “Only Time” New Age Vocal
4 Dido – “Thank You” Pop Vocal
5 Ricky Martin – “Private Emotion” Pop Vocal
6 Avril Lavigne – “Complicated” Pop/Rock Vocal
7 Claudio Roditi – “Rua Dona Margarida” Jazz/Easy Instrumental
8 Mambo Kings – “Bella Maria de Mi Alma” Bolero Instrumental
9 Eliades Ochoa – “Chan Chan” Son Vocal
10 Juan Luis Guerra – “Palomita Blanca” Bachata Vocal
11 Battlefield Band – “Snow on the Hills” Scottish Folk Instrumental
12 daisy2 Pop Vocal
13 daisy3 Pop Vocal
14 jazz2 Jazz Instrumental
15 jazz3 Jazz Instrumental
16 midi1 Pop Instrumental
17 midi2 Folk Instrumental
18 opera female 2 Opera Vocal
19 opera male 3 Opera Vocal
20 pop1 Pop Vocal
21 pop4 Pop Vocal
19. 19 Melody Detection in Polyphonic Audio
Evaluation Procedures
Evaluation Metrics
• Pitch Contour Accuracy
- Melodic Raw Pitch Accuracy (MRPA)
- Overall Raw Pitch Accuracy (ORPA)
- Melodic Raw Note Accuracy (MRNA)
- Overall Raw Pitch Accuracy (ORNA)
• Note Extraction Accuracy
- Percentage of correct frames
• Melody/Accompaniment Discrimination
- Recall
- Precison
21. 21 Melody Detection in Polyphonic Audio
Conclusions and Future Work
Main Contributions
• Note determination
• Identification of melodic notes in a mixture
Conclusions
• Adequate MPD approach in a melody context
• Note determination with satisfactory accuracy
• Identification of melodic notes successful in medium
SNR conditions
Future Work
• Larger musical corpus
• Reduce computational time for pitch detection
• Melody/Accompaniment discrimination