3. 3
Overview
Suggest novel methods to apply NLP approaches to music domain
Introduce MusicBERT, a large-scale pretrained model for symbolic music understanding
Evaluate the performance on four tasks
4. 4
Overview
Suggest novel methods to apply NLP approaches to music domain
Introduce MusicBERT, a large-scale pretrained model for symbolic music understanding
Evaluate the performance on four tasks
5. 5
Contributions
Construct a large-scale symbolic music corpus
β Million MIDI Dataset(MMD)
Design some mechanisms to enhance
pre-training with symbolic music data
(OctupleMIDI Encoding & Masking Strategies)
Achieve the state-of-the-art results on
four music understanding tasks
: Melody Completion, Accompaniment Suggestion,
Genre Classification, and Style Classification
6. 6
Related Work
Symbolic Music Understanding Symbolic Music Encoding
Masking Strategies in Pre-
training
Word2vec models for
music:
β’ Huang et al., 2016
β’ Madjiheurem et al.,
2016
Divide music pieces
ο¨ Fixed duration music
slices
β’ Herremans et al., 2017
β’ Chuan et al., 2020
Small NN models &
Only a few music tokens
as inputs
MIDI-based
β’ MIDI
β’ REMI (Huang and Yang,
2020)
β’ CP (Hsiao et al., 2021)
Pianoroll-based
β’ Brunner et al., 2018
β’ Ji et al., 2020
Still need long input tokens
Application of masking
strategies for music
domain
β’ MASS (Song et al., 2019)
β’ SpanBERT (Joshi et al., 2020)
Not considering the
difference between
NLP & music
8. 8
Model Overview
MusicBERT, a large scale Transformer model for symbolic music understanding
Based on Transformer encoder (Vaswani et al., 2017)
9. 9
Model Overview
MusicBERT, a large scale Transformer model for symbolic music understanding
A novel encoding method, OctupleMIDI, to encode the music sequence more
efficiently
10. 10
Model Overview
MusicBERT, a large scale Transformer model for symbolic music understanding
Predict music tokens as output
15. 15
OctupleMIDI Encoding
OctupleMIDI,
a compact symbolic music encoding method
β’ Encode 6 notes into 6 tokens
β’ Much shorter than REMI & CP
β’ Apply to various kinds of music
Each Octuple token:
β’ Correspon to a note
β’ Contain 8 elements
16. 16
OctupleMIDI Encoding
Time Signature
Tempo
Bar and Position
A fraction (e.g., 2/4):
β’ Length of a beat (note duration ο¨ e.g., a quarter note in
2/4),
β’ Number of beats in a bar (e.g., 2 beats in 2/4)
Beats per minute (BPM)
β’ Pace of music
β’ From 16 to 256 for OctupleMIDI
On-set time of a note
β’ 256 bars in a music piece (0 to 255)
β’ 1/64 note to represent the on-set time of a note (from
0)
17. 17
OctupleMIDI Encoding
Instrument
Pitch
Duration
Velocity
Follow MIDI format
β’ 129 tokens to represent instruments
β’ 0 to 127: different general instruments (e.g., piano and
bass)
β’ 128: special percussion instrument (e.g., drum)
Note pitches for general instruments
β’ 128 tokens to represent pitch values (follow MIDI
format)
Note pitches for percussion instruments
β’ 128 pitch tokens to represent percussion type
Note duration
β’ 128 tokens (percussion: all set to 0)
Quantize the velocity of a note into 32 different values
β’ Interval of 4 (e.g., 2, 6, 10, β¦ , 122, 126)
18. 18
Masking Strategy
Bar-level masking strategy:
Elements with the same type in the same bar & mask simulaneously
ο¨ Avoid information leakage & Learn the contextual representation well
19. 19
Pre-training Corpus
Table 2. Size of different music datasets
OctupleMIDI encoding is universal
ο¨ Most MIDI files can be converted
without noticeable loss of musical
information
ο¨ Cleaning and deduplication
ο¨ Obtain Million-MIDI Dataset (MMD):
1.5 million songs with 2 billion octuple
tokens (musical notes)
20. 20
Experiments & Results
Pre-training Setup Fine-tuning MusicBERT Method Analysis
Table 4. Model configurations of MusicBERT
Small MusicBERT
To compare with baselines (similar data
size)
Base MusicBERT
To achieve the SOTA results
21. 21
Experiments & Results
Pre-training Setup Fine-tuning MusicBERT Method Analysis
Four downstream task
Melody Completion Genre & Style Classification
Accompaniment Suggestion
Table 3. Results of different models on the four downstream tasks
22. 22
Experiments & Results
Pre-training Setup Fine-tuning MusicBERT Method Analysis
Four downstream task
Melody Completion Genre & Style Classification
Accompaniment Suggestion
Table 3. Results of different models
on the four downstream tasks
Task Find the most matched consecutive phrase
in a given set of candidates for a given melodic
phrase
Evaluation The rate of correctly chosen phrase
in the top k candidates
Best Performance ππ’π πππ΅πΈπ ππ ππππ , ππ’π πππ΅πΈπ ππππ π
23. 23
Experiments & Results
Pre-training Setup Fine-tuning MusicBERT Method Analysis
Four downstream task
Melody Completion Genre & Style Classification
Accompaniment Suggestion
Table 3. Results of different models
on the four downstream tasks
Task To find the most related accompaniment phrase
in a given set of harmonic phrase candidates for a
given melodic phrase
Evaluation The rate of correctly chosen phrase
in the top k candidates
Best Performance ππ’π πππ΅πΈπ ππ ππππ , ππ’π πππ΅πΈπ ππππ π
24. 24
Experiments & Results
Pre-training Setup Fine-tuning MusicBERT Method Analysis
Four downstream task
Melody Completion Genre & Style Classification
Accompaniment Suggestion
Table 3. Results of different models
on the four downstream tasks
Task To classify the genre and style
Dataset TOP-MAGD for genre, MASD for style
Evaluation F1-micro score
Best Performance ππ’π πππ΅πΈπ ππ ππππ , ππ’π πππ΅πΈπ ππππ π
28. 28
Conclusion
Propose OctupleMIDI encoding & bar-level masking strategy for music
domain
Develop MusicBERT, a large-scale pre-trained model
for symbolic music understanding
Achieve state-of-the-art performance on
all four evaluated symbolic music understanding task
29. 29
For my research
Acquire some baseline models & datasets to review
Understand new symbolic music representation method
Learn how to design experiments to measure each feature of a model
