1. ShEMP
(AND FRIENDS)BRENNON BORTZ
DEPT OF COMPUTER SCIENCE
INSTITUTE FOR CREATIVITY, ARTS, AND TECHNOLOGY
VIRGINIA TECH

2. ShEMP AND FRIENDS
2 • 22VIRGINIA TECH
MUSE ABOUT US
The Music, Sensors and Emotion
research group is a multidisciplinary
team focused on both qualitatively and
quantitatively measuring the
relationship between music and emotion
and using this information to inform
musical composition and performance.

3. 3 • 22
MUSE RESEARCH AREAS
ShEMP AND FRIENDS
‣
Integral music control
‣
Measuring gesture and
emotion during performance
‣
Physiological monitoring
‣
Contagion of emotion
VIRGINIA TECH

4. 4 • 22VIRGINIA TECH
MUSE SIEMPRE
Developing novel theoretical and methodological
research frameworks, computational models, and
algorithms for the analysis of creative communication
within groups of people.
Research is focused on ensemble musical
performance and audience experience.
ShEMP AND FRIENDS

5. MUSE SHADOW MEDIA
5 • 22
‣ Inseparable
‣ Evocative of presence
‣ Expansion of self
‣ Moves in tandem with body
Ways through which space can be shared between
remote actual fields are necessary to support the
creation of a sense of co-existence and mutual trust
—a requirement for co-creation.
ShEMP AND FRIENDS
VIRGINIA TECH

6. 6 • 22VIRGINIA TECH
ShEMP AND FRIENDS
MUSE SHADOW MEDIA

7. 7 • 22VIRGINIA TECH
ShEMP AND FRIENDS
MUSE SHADOW MEDIA

8. MUSE PIEs
8 • 22
ShEMP AND FRIENDS
VIRGINIA TECH
Assessment of emotional state is
accomplished through a number of
physiological indicators of emotion
(PIEs).

9. 9 • 22
MUSE MOBILEMUSE
ShEMP AND FRIENDS
‣
Sensors
‣
Pulse oximetry
‣
EDA
‣
Temperature
‣
Triaxial accelerometer
‣
Audio signal output
VIRGINIA TECH

10. MUSE ShEMP
10 • 22
‣ Modular iOS Library for experimental
design and execution
‣ Tools for:
‣ Sensor data acquisition
‣ Various self-report mechanisms
‣ Visualization
‣ Shared media
‣ Data persistence
‣ Co-creation
ShEMP AND FRIENDS
VIRGINIA TECH

12. 12 • 22
ShEMP AND FRIENDS
VIRGINIA TECH
MUSE PERFORMER / AUDIENCE

13. 13 • 22
ShEMP AND FRIENDS
VIRGINIA TECH
MUSE PERFORMERS

14. 14 • 22
ShEMP AND FRIENDS
VIRGINIA TECH
MUSE PERFORMERS

15. 15 • 22
MUSE EMOTION IN MOTION
ShEMP AND FRIENDS
VIRGINIA TECH

16. ‣ Over 10,000 participants
‣ Average of 40 visitors per day
‣ Over 20,000 songs listened!
‣ Pool of 67 songs
‣ Happy/Sad, Tense/Relaxed
‣ From Handel to the Beach Boys to the
Black Eyed Peas and Slayer
MUSE EiM STATS
16 • 22VIRGINIA TECH
ShEMP AND FRIENDS

17. MUSE EiM
17 • 22
ShEMP AND FRIENDS
VIRGINIA TECH

18. KARBON PRESENTATION 8 • 24LEVEL 13, 82 SPENCER ST, MELBOURNE
IMMERSE
YOURSELF
MUSE EiM ANALYSIS
18 • 22
ShEMP AND FRIENDS
VIRGINIA TECH

19. 19 • 22
ShEMP AND FRIENDS
VIRGINIA TECH

20. 20 • 22
ShEMP AND FRIENDS
VIRGINIA TECH

21. MUSE THANK YOU
Questions?
ShEMP AND FRIENDS
21 • 22VIRGINIA TECH

22. ShEMP AND FRIENDS
22 • 22
MUSE CONTACT US
{brennon,ben,javier,niall,nick}
@musicsensorsemotion.com
VIRGINIA TECH

23. B. Bortz, I. Crandell, R. B. Knapp, Y. Miwa, and S. Itai, “Comparison of the Effect of Proximity on Emotional Change in Real and Virtual Shared
Spaces,” presented at the International Conference on Affective Computing and Intelligent Interaction 2013, Geneva, ShEMP: A Mobile Framework
for Shared Emotion, Music, and Physiology, ACM-ICMI SBM workshop, Santa Monica, California, USA.
Jaimovich, Javier, Niall Coghlan, and R. Benjamin Knapp (2012) Emotion in Motion: A Study of Music and Affective Response. In Proceedings of the
9th International Symposium on Computer Music Modeling and Retrieval (CMMR) Music and Emotions, pp.29-44. Queen Mary University of London.
Jaimovich, Javier, Miguel Ortiz, Niall Coghlan, and R. Benjamin Knapp (2012) The Emotion in Motion Experiment: Using an Interactive Installation as
a Means for Understanding Emotional Response to Music. In Proceedings of the 2012 Conference on New Interfaces for Musical Expression (NIME),
pp.457-8. University of Michigan, Ann Arbor: Division of Computer Science & Engineering, University of Michigan.
Jaimovich, J., Ortiz, M., et al., 2012. The Emotion in Motion Experiment: Using an Interactive Installation as a Means for Understanding Emotional
Response to Music. In Proceedings of the 2012 Conference on New Interfaces for Musical Expression (NIME 2012), Ann Arbor, Michigan. New
Interfaces for Musical Expression. Ann Arbor, Michigan, p. (In Press).
Jaimovich, J., Coghlan, N. & Knapp, R.B., 2012. Emotion in Motion: A Study of Music and Affective Response. In Proceedings of the 9th International
Symposium on Computer Music Modeling and Retrieval (CMMR) Music and Emotions. Symposium on Computer Music Modeling and Retrieval.
London, England, p. (In Press).
Switzerland.
R. B. Knapp and B. C. Bortz, “MobileMuse: Integral Music Control Goes Mobile,” presented at the 11th International Conference on New Interfaces
for Musical Expression, Oslo, Norway, 2011, pp. 203–206.B. Bortz, S. Salazar, J. Jaimovich, R. B. Knapp, G. Wang (2012)
ShEMP AND FRIENDS
VIRGINIA TECH
MUSE REFERENCES

24. ShEMP
(AND FRIENDS)BRENNON BORTZ
DEPT OF COMPUTER SCIENCE
INSTITUTE FOR CREATIVITY, ARTS, AND TECHNOLOGY
VIRGINIA TECH