Recent EPSRC funded research by audio technology staff at GCU has resulted in the development of links with researchers at the cutting edge of music emotion and technology research. The seminar will consist of three talks on this exciting cross-disciplinary area. "Is Music Ever Sad?" Barry Maguire MA, Department of Philosophy, Princeton University "Music Emotion Recognition, A State of the Art Review", Dr Scott Beveridge, Semantic Music Technologies, Fraunhofer Institute of Digital Media Technology "Practical Applications of Semantic Music Technologies", Jakob Abesser Dipl.-Ing. Semantic Music Technologies, Fraunhofer Institute of Digital Media Technology

1. Music Emotion Recognition
A State of the Art Review
Dr Scott Beveridge
Interdisciplinary Perspectives on Music, Emotion and Technology
Glasgow Caledonian University, 25th June 2012
© Fraunhofer IDMT

2. Outline
 Why emotion?
 Definition of Music Emotion Recognition
 History and motivation for interdisciplinary approach
 Current experiments
© Fraunhofer IDMT

3. Why emotion?
Emotion and Meaning in Music
© Fraunhofer IDMT

4. Emotion and Meaning in Music
 Humans treat music with great importance
 Music has a very powerful effect
 One of the primary reasons behind
humans enjoyment of music
 Henry is an example of the listener
perspective
 Music is useful for those who
compose and perform also
 Method of communication and
expression
© Fraunhofer IDMT

5. Emotion and Meaning in Music
 Emotion is a great way of creating information which facilitates browsing and
organization of music
 Vast number of audio tracks online
 Why are we doing this?
 MER Applications
 Music organization and browsing – personal and commercial
 Academic – Music Digital Libraries (MDL), archiving
 Health – music therapy, pain management (Knox 2011)
© Fraunhofer IDMT

6. Emotion and Meaning in Music
 Pin down some aspects of the affective process to make the problem
computationally tractable
 Expression versus Induction
 Most work in MER focusses on expression
 „What the music is trying to say to you‟
 This is easier to decide on with some types of music than others
 Film music
© Fraunhofer IDMT

7. A Definition of Music Emotion
Recognition (MER)
 MER has two steps
1. Identification, recognition and extraction of musical characteristics
which express emotions in music
2. Modelling these characteristics in order to make prediction on
emotions expressed by „new‟ music
© Fraunhofer IDMT

8. Emotion Recognition Happy = Fast Tempo,
Supervised Learning Major Mode
Sad = Slow Tempo,
Happy Minor Mode
Sad
?
© Fraunhofer IDMT

9. What can machines learn?
Conceptualization of Emotion
 Assign emotion labels (Classification)
 Previous example
 Exuberant, Anxious, Depressed, Content
 Define a point in 2D space (Numeric Prediction)
 Predict emotions which vary over time (Time-continuous
prediction)
© Fraunhofer IDMT

10. A Brief History of MER
Background
Music Psychology Engineering
Cate Hevner 1935
1988 Kayatose (Symbolic)
Patrik Juslin 2001
1935
2003 Feng (Signal-based)
NOW
© Fraunhofer IDMT

11. A Brief History of MER
Popular Music
 Popular music is becoming…..popular!
© Fraunhofer IDMT

12. Popular Music
Challenges
 By definition popular music is
 Made commercially
 Limits the scope of expressed emotions
 Made using ever-changing technologies
 Over production (dynamic compression)
 This generally leads to homogeneity in the popular music genre
 To overcome these problems psychologists, musicologists, philosophers, and
engineers must work together
© Fraunhofer IDMT

13. A Brief History of MER
Why Interdisciplinary?
 Olighara 2003
 “One 39 year old male Chinese” annotator for a corpus of Western
contemporary popular music
 Wu 2006
 10 second music clips and no mention of music
 Schellenberg12
 Manual tempo calculation
 Yang07
 Expression/Induction distinction
© Fraunhofer IDMT

14. Current Experiments
 Based on 2 steps in MER
 Features
 Tone Objects
 Statistical properties of melody
 Modelling
 Predict tension gradients for use in syncronisation
 Includes creation of new features
 Feature fusion
© Fraunhofer IDMT

15. Current Experiments
Tone Objects
 Objective: Find novel ways of describing popular music by creating new
musical features
 Existing features
 Tempo, Mode, Key, Instrument Timbre
 New features
 Examine existing features based on tone objects
 Musical notes of the main melody
© Fraunhofer IDMT

16. Current Experiments
Tone Objects
 Extracting tone objects involves
many signal processing
techniques
 Source separation
 Computational Auditory
Scene Analysis (CASA)
 Identify the main
melody
 Results shows that tone objects
help identify particular types of
emotion
© Fraunhofer IDMT

17. Current Projects
Main Melody Statistics
 In linguistics, Zipf’s Law shows that: [CAREFUL! These figures might be
incorrect]
Given some corpus of natural language the frequency of any word is inversely
proportional to its rank in the frequency table
Word # of occurrences Word # of occurrences
the 69,971 Unison 69,971
of 36,411 Major 3rd 36,411
and 28,852 Perfect 5th 28,852
 Studies1 have shown that Zipf law statistics have a relationship with aesthetic
aspects of music – pleasant, beautiful
 Can Zipf‟s law statistics be applied in emotion classification?
1 http://sger.cs.cofc.edu/
© Fraunhofer IDMT

18. Current Projects
Tension Prediction
 Objective: Track time-continuous tension gradients in film music
 Applications in syncronization task
 Helps creators of films and adverts find music with specific
characteristics
 Approach:
 Step 1: Extract time-continuous features from a collection of film
music
 Step 2: Conduct a study which asks people to rate time-continuous
tension
 Step 3: Build models with the data which predicts tension gradients
in new music
 An example of supervised learning!
© Fraunhofer IDMT

19. Current Projects
Tension Prediction – Feature Extraction
 Step1: Extract time-continuous features
© Fraunhofer IDMT

20. Current Projects
Tension Prediction – Participant Testing
 Step 2: Asked participants to rate music
based on perceived tension
 General agreement
© Fraunhofer IDMT

21. Current Projects
Tension Prediction – Participant Testing
 Features most correlated with tension:
 Timbral Complexity: The rate of change of timbre (How many
„different sounding‟ instruments are present
 Spectral Dissonance: Perceived roughness
 Pure Tonalness: A measure of how „tone-like‟ a sound is
© Fraunhofer IDMT

22. Current Projects
Tension Prediction – Demonstration
© Fraunhofer IDMT

23. The Future of MER
 Automatic MER systems are only the beginning
 For MER systems to be truly effective it is necessary to adopt a user-centred
approach
 Emotions elicited in music are created through social factors and
environment
 Listening with friends
 Listening on the way to work
 Profile users to create bespoke emotion recommendation systems based on
 Geo-location, time of day, skipping behaviour
© Fraunhofer IDMT

24. Music Emotion Recognition
A State of the Art Review
Thank you !!
bevest@idmt.fraunhofer.de
http://tinyurl.com/bevestLinkedIn
© Fraunhofer IDMT