Seminar for University of Manchester School of Computer Science, Wednesday 30th April 2014 at 14:00 in Lecture Theatre 1.4.

1. Digital Music Research: from
Music Objects to Social Machines
David De Roure
e-Research Centre, University of Oxford
@dder

2. The nature of multidisciplinary research
Structural Analysis of Music
Music as an exemplar of end-to-end digital
Social Objects and Social Machines

3. YES
https://xkcd.com/1289/

5. Richard Klavans and Kevin W. Boyack. 2009. Toward a consensus map of science. J. Am. Soc. Inf. Sci. Technol. 60, 3 (March
2009), 455-476. DOI=10.1002/asi.v60:3 http://sci.slis.indiana.edu/klavans_2009_JASIST_60_455.pdf

6. Pip WillcoxPip Willcox
From data to signal to understanding

8. INT. VERSE VERSE VERSE VERSEBRIDGEBRIDGE OUT.

The Problem
signal
understanding

9. Community
Software
Supercomputer
Digital Music
Collections
Student-sourced
ground truth
Community
Software
Linked Data
Repositories
Supercomputer
23,000 hours of
recorded music
Music Information
Retrieval Community
SALAMI

10. Ashley Burgoyne

11. salami.music.mcgill.ca
Jordan B. L. Smith, J. Ashley Burgoyne, Ichiro Fujinaga, David De Roure, and J. Stephen Downie.
2011. Design and creation of a large-scale database of structural annotations. In Proceedings of
the International Society for Music Information Retrieval Conference, Miami, FL, 555–60

12. class structure
Ontology models properties from musicological domain
• Independent of Music Information Retrieval research and signal
processing foundations
• Maintains an accurate and complete description of relationships
that link them
Segment Ontology
Ben Fields, Kevin Page, David De Roure and Tim Crawford (2011) "The Segment Ontology: Bridging Music-Generic and Domain-Specific" in 3rd
International Workshop on Advances in Music Information Research (AdMIRe 2011) held in conjunction with IEEE International Conference on
Multimedia and Expo (ICME), Barcelona, July 2011

13. MIREX TASKS
Audio Artist Identification Audio Onset Detection
Audio Beat Tracking Audio Tag Classification
Audio Chord Detection Audio Tempo Extraction
Audio Classical Composer ID Multiple F0 Estimation
Audio Cover Song Identification Multiple F0 Note Detection
Audio Drum Detection Query-by-Singing/Humming
Audio Genre Classification Query-by-Tapping
Audio Key Finding Score Following
Audio Melody Extraction Symbolic Genre Classification
Audio Mood Classification Symbolic Key Finding
Audio Music Similarity Symbolic Melodic Similarity
www.music-ir.org/mirex
Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). The Music Information
Retrieval Evaluation eXchange: Some Observations and Insights. Advances in Music Information Retrieval
Vol. 274, pp. 93-115
Music Information Retrieval Evaluation eXchange

15. seasr.org/meandreMeandre

17. chromogram
Representations
symbolic

18. Structural analysis

19. Autocorrelation

20. Bach

21. Hard Day’s Night: Self-Similarity Map

23. Stephen Downie

24. SALAMI results: a living experiment

25. DavidBainbridge
http://semanticmedia.org.uk/smam2013/

26. ABABCB… where A is bars 1-2, B is 3-4, C is 9-10
• This is like dictionary-based compression
• Or genetic programming (see also Schenkerian Analysis)
Symbolic algorithms

27. “Signal”
Digital Audio
“Ground Truth”
Community
It’s web-like!
Structural
Analysis
De Roure, D. Page, K.R., Fields, B., Crawford, T.,Downie, J.S. and Fujinaga, I. (2011) “An e-Research Approach to Web-Scale
Music Analysis”, Philosophical Transactions of the Royal Society Series A

28. Sean Bechhofer

29. How country is
my country?
Kevin Page

30. Sean Bechhofer, Kevin Page and David De Roure. Hello Cleveland! Linked Data Publication Of Live Music Archives. 14th
International Workshop on Image and Audio Analysis for Multimedia Interactive services
Sean Bechhofer

31. ElEPHãT from a distance
EEBO
-TCP
Hathi
Trust
• Smaller collection
• Well understood and
described
• Managed metadata
• Focussed corpus
• Manual transcriptions
• Extremely large collection
• Incomplete understanding
of content
• Variable metadata
• Broad corpus
• Variable quality OCR
Strengths of each informs
understanding of the other
Scholarly investigations through Worksets
bridging both collections
Technical challenges
• Necessary “anchors” at each “end”
• Tools for dynamic alignment
• Linked Data “bridging” between the collections
• Creation and viewing of Worksets using this linked data
Informing
future integration of
external collections
KevinPageandPipWillcox

33. • Transforming Musicology is funded under the AHRC Digital
Transformations in the Arts and Humanities scheme. It seeks to
explore how emerging technologies for working with music as
sound and score can transform musicology, both as an academic
discipline and as a practice outside the university.
• The work is being carried out collaboratively between Goldsmiths
College, Queen Mary College, Oxford University, the Oxford
e-Research Centre, and Lancaster University with an international
partner at Utrecht University.

34. • The world of music has changed for good in the digital age. This
revolution must be matched by a transformation of the means by
which music is studied.
• While preserving the best traditional values and practices of
musicology we must take advantage of the immense opportunities
offered by music information retrieval
• Three parallel musicological investigations
1. 16th-century vocal and lute music
2. Wagner's leitmotifs
3. Musicology of the social media
• Ensure sustainability and repeatability by embedding the above
research activities in a framework enabling data, methods and
results to be shared permanently as Linked Data
• Enhance Semantic Web workflow description methods for musicology

35. FUSING AUDIO AND SEMANTIC
TECHNOLOGIES for
INTELLIGENT MUSIC PRODUCTION AND
CONSUMPTION

36. “Gold Standard” Music Metadata
Enhancements for musical enjoyment
by home consumers
In-song browsing
• learn how songs and
symphonies are
structured
• e.g. find (and repeat)
the guitar solo
• e.g. find vocals and
enhance them
• e.g. create/locate
guitar tablature
In-collection browsing
• build great playlists easily: by mood
or emotion. e.g. for jogging, driving,
relaxing; containing only pieces in G
Major; containing Rock & Roll with
orchestral strings; with a synth sound
like Stevie Wonder
• discover and purchase new music,
whether using Spotify or iTunes
• discover shared musical tastes

37. “Gold Standard” Music Metadata
Enhancements for professionals
Content owners
• get instantaneous
information on trends, etc.,
from social media feeds
• enhance their product with
exclusive artist information,
locked to purchase
• distributers provide Digital
Music Objects with the right
bandwidth for the context
and ease congestion
Recording studio workflow
• engineers intelligently navigate
complex mixes
• producers can apply new
sound effects to isolated
elements of the music
Broadcast studio workflow
• producers select content for
the radio or TV show by mood,
by example or by intelligent
navigation

38. consume
produce
compose
perform
capture
distribute
Mark Sandler(plus curation, preservation, …)

39. Now
• No production or content metadata capture – c.f. still and video
cameras
• Clear audio standards (e.g. 192 kHz/24 bit) but incompatible
product-specific project files
• No intelligent, content-semantic automation or assistance
Goals
• Capture/ compute of GSMM to drive all down-stream processes
• Improved interoperability across system vendors
Challenges
• Develop equipment and instruments that capture metadata (e.g
mic with time-code and GPS)
• Standardised semantic, linked metadata
capture produce distribute consume

40. Now
• Convergence in function of pro- and consumer products
• No/little metadata kept
• No standards, particularly in describing processes (audio effects)
• Mostly PC/Mac software solutions for Digital Audio Workstation
Goals
• low cost equipment, including software and tablets
• assist/semi-automate (post) production
• capture post-production metadata for re-engineering content, user-
customisation.
Challenges
• Using cloud
• Standardised semantic, linked metadata
• Tools & kit for automated metadata processing, capture, logging
capture produce distribute consume

41. Now
• Different platforms & formats. Piracy.
• Increasing use of IP for distribution.
• Transcoding within channels, quality loss, managing multiple copies
Goals
• Simpler transcoding (e.g. embedded scalability
• Distribute content linked to metadata
• Encrypted metadata: supports consumer while defying piracy
• Digital Music Object
Challenges
• Encryption standards for metadata
• Linking semantic, standardised metadata.
• Aggregate metadata from up/down stream
capture produce distribute consume

42. Now
• No context awareness, no customisation.
• Some transcoding of bit-rates, #channels.
• Little immersion, both intellectual and audio.
• Unfulfilled desires to share, re-purpose, integrate with social media
Goals
• Modify experience to suit context
• Re-balance between instruments
• Seamlessly switch #channels as user context changes
• Navigate collections; songs
• Edutainment
Challenges
• Repurposing content to match device and context
capture produce distribute consume

43. NeilChueHong
An exemplar for software practice
• Global distributed system: software, data and
processor allocation by bandwidth but also
rights, copyright, …
• Realtime, streaming (cf big data)
• Digital Rights Management and provenance
• Algorithm IPR
• Heavily app based
• MIR open source community and MIREX
• Non-consumptive research

44. Digital Music Object
Mark Sandler, Geraint Wiggins

45. Edwards, P. N., et al. (2013) Knowledge Infrastructures: Intellectual Frameworks and
Research Challenges. Ann Arbor: Deep Blue. http://hdl.handle.net/2027.42/97552

46. Research Objects
Computational
Research Objects
The Evolution of Research Objects
Workflows
Packs
OAI
ORE
W3CPROV
Social Objects

47. Join the W3C Community Group www.w3.org/community/rosc
Jun Zhao
www.researchobject.org

48. The R Dimensions
Research Objects facilitate research that is
reproducible, repeatable, replicable, reusable,
referenceable, retrievable, reviewable, replayable,
re-interpretable, reprocessable, recomposable,
reconstructable, repurposable, reliable,
respectful, reputable, revealable, recoverable,
restorable, reparable, refreshable?”
@dder 14 April 2014
sci method
access
understand
new use
social
curation
Research
Object
Principles

49. The Big Picture
More people
Moremachines
Big Data
Big Compute
Conventional
Computation
“Big Social”
Social Networks
e-infrastructure
online
R&D
Social
Machines
deeply
about
society

50. Real life is and must be full of all kinds of social
constraint – the very processes from which society
arises. Computers can help if we use them to
create abstract social machines on the Web:
processes in which the people do the creative work
and the machine does the administration... The
stage is set for an evolutionary growth of new
social engines. The ability to create new forms of
social process would be given to the world at large,
and development would be rapid.
Berners-Lee, Weaving the Web, 1999 (pp. 172–175)
Social Machines

51. SOCIAM: The Theory and Practice of Social Machines is funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under
grant number EPJ017728/1 and comprises the Universities of Southampton, Oxford and Edinburgh. See sociam.org

52. Mark d’Inverno
http://goldsmiths.musiccircleproject.com/
PRAISE: Performance and pRactice Agents Inspiring Social Education

53. The Web
Observatory
Tiropanis, T., Hall, W., Shadbolt, N., De Roure, D., Contractor, N., and Hendler, J.
The web science observatory. IEEE Intelligent Systems 28, 2 (2013), 100–104.

54. Nigel Shadbolt et al

55. STORYTELLING AS A STETHOSCOPE
FOR SOCIAL MACHINES
1. Sociality through storytelling potential
and realization
2. Sustainability through reactivity and
interactivity
3. Emergence through collaborative
authorship and mixed authority
Zooniverse is a highly
storified Social Machine
Facebook doesn’t allow
for improvisation
Wikipedia assigns
authority rights rigidly
Tarte, S. M., De Roure, D., and Willcox, P. Working out the plot: the role of stories in social machines. In Proceedings of the
companion publication of the 23rd international conference on World wide web companion (2014), International World Wide Web
Conferences Steering Committee, pp. 909–914.

56. Big data elephant versus sense-making network?
The challenge is to foster the co-constituted socio-technical
system on the right i.e. a computationally-enabled sense-making
network of expertise, data, models, software, visualisations and
narratives
Iain Buchan

57. • Digital doesn’t respect disciplinary boundaries – don’t
just retrofit digital inside the barriers of historic academic
structures, think forward instead:
– End to end digital systems
– End to end semantics
• Try applying the lenses of
– Social Objects
– Social Machines
• Music as an exemplar for science, informing ICT strategy
and future of scholarly communications
• Always ask hard questions, especially given the
disruptions of increasing empowerment and automation
Take home messages

58. david.deroure@oerc.ox.ac.uk
www.oerc.ox.ac.uk/people/dder
@dder
SOCIAM: The Theory and Practice of Social Machines is funded by the UK Engineering
and Physical Sciences Research Council (EPSRC) under grant number EPJ017728/1 and
comprises the Universities of Southampton, Oxford and Edinburgh. See sociam.org
Slide and image credits: Sean Bechhofer, Iain
Buchan, Neil Chue Hong, Tim Crawford, Stephen
Downie, Ben Fields, Ichinaro Fujinaga, Carole
Goble, Mark d’Inverno, Kevin Page, Mark Sandler,
Pip Willcox, Jun Zhao.
Thanks to NEMA, SALAMI, Wf4Ever, Transforming
Musicology, FAST, SOCIAM, PRAISE and all our
colleagues in the ISMIR community.

59. Bechhofer, S., Page, K., and De Roure, D. Hello Cleveland! linked data publication of live music
archives. In Image Analysis for Multimedia Interactive Services (WIAMIS), 2013 14th
International Workshop on (2013), IEEE, pp. 1–4.
De Roure, D. Towards computational research objects. In Proceedings of the 1st International
Workshop on Digital Preservation of Research Methods and Artefacts (2013), ACM, pp. 16–19.
De Roure, D., Page, K. R., Fields, B., Crawford, T., Downie, J. S., and Fujinaga, I. An e-research
approach to web-scale music analysis. Philosophical Transactions of the Royal Society A:
Mathematical, Physical and Engineering Sciences 369, 1949 (2011), 3300–3317.
Fields, B., Page, K., De Roure, D., and Crawford, T. The segment ontology: Bridging music-
generic and domain-specific. In Multimedia and Expo (ICME), 2011 IEEE International
Conference on (2011), IEEE, pp. 1–6.
Page, K. R., Fields, B., De Roure, D., Crawford, T., and Downie, J. S. Capturing the workflows of
music information retrieval for repeatability and reuse. Journal of Intelligent Information Systems
41, 3 (2013), 435–459. (Also Reuse, remix, repeat: the workflows of mir. In ISMIR (2012), pp.
409–414.)
Page, K. R., Fields, B., Nagel, B. J., O’Neill, G., De Roure, D. C., and Crawford, T. Semantics for
music analysis through linked data: How country is my country? In e-Science (e-Science), 2010
IEEE Sixth International Conference on (2010), IEEE, pp. 41–48.
Tarte, S. M., De Roure, D., and Willcox, P. Working out the plot: the role of stories in social
machines. In Proceedings of the companion publication of the 23rd international conference on
World Wide Web companion (2014), pp. 909–914.
Tiropanis, T., Hall, W., Shadbolt, N., De Roure, D., Contractor, N., and Hendler, J. The web
science observatory. IEEE Intelligent Systems 28, 2 (2013), 100–104.
De Roure, D. Machines, methods and music: On the evolution of e-research. In High
Performance Computing and Simulation (HPCS), 2011 International Conference on (2011),
IEEE, pp. 8–13.

60. www.oerc.ox.ac.uk
david.deroure@oerc.ox.ac.uk
@dder