One of the biggest barriers for the uptake of a Web of Media is the availability of easy ways to reuse media fragments and manage their copyright. Existing proposals provide limited solutions or find it difficult to scale to the Web. MediaMixer contributes state of the art techniques for media fragment detection and semantic annotation. This is complemented with copyright management integrated into the Web fabric, using Linked Data principles and reasoning based on a Copyright Ontology. Altogether, it can make possible to navigate the Web retrieving the metadata describing a piece of content to be reused, linked to the agreement about its copyright, the parties that will share the revenue, etc. A typical MediaMixer demo involves: * Fragmenting media assets * Annotating them using semantic descriptions * Modeling licenses, policies,... using the Copyright Ontology * Exposing them for fragment level retrieval and re-use, including copyright reasoning

1. Linked Data: the Entry Point for Worldwide
Media Fragments Re-use and Copyright
Management?
Roberto García
Universitat de Lleida, Spain
SemTechBizNYC’13 - October 2-3, 2013, New York City

2. 02.10.2013 Slide 2 of 26
MediaMixer
Community set-up and networking for the
reMIXing of online MEDIA fragments
• FP7 Coordination Action
– Started November 2012
– Facilitate a Web-wide market for media
fragments re-purposing and re-selling
A history of G8 violence (© Reuters) http://slideshare.net/troncy/a-semantic-multimedia-web-create-annotate-present-and-share-your-media

3. 02.10.2013 Slide 3 of 26
Introduction
• MediaMixer promotes the use of
semantic technologies for media mixing
• Apply them in real use cases and demos
– Fragmenting media assets
– Annotating them using semantic descriptions
– Exposing descriptions for fragment level
search
– Representing, integrating and reasoning
about rights information

4. 02.10.2013 Slide 4 of 26
Use Case
• User Generated Content

5. 02.10.2013 Slide 5 of 26
Use Case
• UGC from reused media

6. 02.10.2013 Slide 6 of 26
Use Case – ContentID
ContentID ©
©ContentID
match
match
match

7. 02.10.2013 Slide 7 of 26
Use Case
©
Block
Choices:
Track
Monetize
“Gangnam Style”, plus +33,000 videos using it, more than
$870,000 ad revenue - New York Times, Dec 7th 2012

8. 02.10.2013 Slide 8 of 26
Use Case - Disputes
• What if more than one ownership claim?
• If disputed, no one can monetize
©©

9. 02.10.2013 Slide 9 of 26
Use Case - Claim?
• Are we sure we can claim?
• Do we own that particular copy?
• In that territory?
• Also in relation with streaming and
YouTube?
• Does the artist like being on YouTube?
• …

10. 02.10.2013 Slide 10 of 26
Use Case - Claim?
Thousands of pieces of registered content…
Thousands of videos on YouTube…
• Decision Support System (DSS)
– You can claim on content-id X
– You can’t really claim on content-id Y
– Trade X for Y with C (who owns Y and
wrongly claims X)

11. 02.10.2013 Slide 11 of 26
Use Case - Claim?
• Rights DSS requirements:
– Fine grained
– Scalable (largely automatized)
– Takes into account:
• Contracts
“…all rights on the live version but studio version just in the
US.”
• Policies
“…artist does not want his music together with violent
images”.
• Rights Expression Languages
DDEX metadata:<UseType>OnDemandStream</UseType>
<TerritoryCode>Worldwide</TerritoryCode>

12. 02.10.2013 Slide 12 of 26
MediaMixer
Technologies
Proposed Architecture
RDF Store
Rhizomer
Media
Explorer
UI generation
and user
assistance
UI generation
and navigation
Store license
or policy
Semantic
Media
Annotation
Rights Builder
User Interface
SPARQL
Rights Language
Mapping
DDEX
to RDF
DDEX
to RDF
DDEX
to RDF
Load licenses Load media
fragments annotations

13. 02.10.2013 Slide 13 of 26
Media Fragments
13
• Goal: transform whole videos to sets of
meaningful, indexable and re-usable video
fragments
Person, Snow, Trees,
Building, Ski, …
Baseball, Throwing, Sports,
Plant, Running, …
Kitchen, Indoor, Cake, …
…
Concept
and Event
Detection
Semantic
Media
Annotation

14. 02.10.2013 Slide 14 of 26
Linked Data
• Globally unambiguous
identifiers for any
concept needed
– Ideally more information
about the concept itself
is available via the
identifier
– Use of Web URIs →
Linked Data concept
space
– Would allow for
inference of concept
type and relationship to
other concepts
Semantic
Media
Annotation

15. 02.10.2013 Slide 15 of 26
http://rhizomik.net/mediamixer/rightsbuilderui/
Assisted Policy and Agreement
Modelling User Interface
Rights Builder
User Interface

16. 02.10.2013 Slide 16 of 26
Semantic Data Exploration
http://rhizomik.net/mediamixer/
Rhizomer
Media Explorer

17. 02.10.2013 Slide 17 of 26
DDEX Sample-08.04.xml
<Deal>
<DealTerms>
<ValidityPeriod>
<StartDate>2013-01-
01</StartDate>
</ValidityPeriod>
<Usage>
<UseType>
OnDemandStream
</UseType>
<DistributionChannelType>
Internet
</DistributionChannelType>
</Usage>
<TerritoryCode>ES</TerritoryCode>
<TerritoryCode>US</TerritoryCode>
</DealTerms>
</Deal>
Copyright Ontology Model
<http://media.com/agreement#1> owl:Class;
co:start "2013-01-01" ;
owl:intersectionOf (
ddex:OnDemandStream
[ a owl:Restriction ;
owl:onProperty co:medium ;
owl:someValuesFrom ddex:Internet ]
[ a owl:Restriction ;
owl:onProperty co:location ;
owl:someValuesFrom
[ a owl:Class ;
owl:oneOf (territory:ES territory:US) ]
[ a owl:Restriction ;
owl:onProperty co:theme ;
owl:hasValue
<http://my.tv/video.ogv#t=60,100>
]
]
) .
DDEX XML to
Semantic Web Technologies
Rights Language
Mapping

18. 02.10.2013 Slide 18 of 26
ODRL to
Semantic Web Technologies
@prefix co: <http://rhizomik.net/ontologies/2013/05/copyrightonto.owl#> .
@prefix odrlv: <http://w3.org/ns/odrl/vocab#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
<http://example.com/policy> a co:Agree ;
co:agent <http://example.com/billie>, <http://example.com/sony> ;
co:theme [ a owl:Class ;
co:end "2012-12-31" ;
co:start "0001-01-01" ;
owl:intersectionOf (
[ owl:intersectionOf (
odrlv:play [ owl:complementOf odrlv:mobile_ringtone ] ) ]
[ a owl:Restriction ;
owl:hasValue <http://example.com/billie> ;
owl:onProperty co:agent ]
[ a owl:Restriction ;
owl:hasValue <http://example.com/music> ;
owl:onProperty co:theme ] ) ] .
<http://example.com/sony:10> a schema:Organization.
<http://example.com/music> a co:Creation.
<http://example.com/billie> a schema:Person.
Copyright Ontology license to text:
“Sony and billie Agree that billie play
and not mobile ringtone music”.
Rights Language
Mapping

19. 02.10.2013 Slide 19 of 26
Creative Commons BY-SA to
Semantic Web Technologies
Rights Language
Mapping

20. 02.10.2013 Slide 20 of 26
Copyright Ontology
Lifecycle, Actions, Rights,…
Creator
Actor
Producer
Broadcaster
User
Motion Picture
Script
Adaptation
Performance
manifest
perform
record
Communication
broadcast
transform
Literary Work
tune
Copyright
EconomicRights
RelatedRights
MoralRights
DistributionRight
ReproductionRight
PublicPerformanceRight
FixationRight
CommunicationRight
AttributionRight
TransformationRight
IntegrityRight
DisclosureRight
WithdrawalRight
PermorfersRights
ProducersRights
BroadcastersRights
RentalRight
ImportationRight
SoundRecordRight
MotionPictureRight
BroadcastingRight
MakingAvailableRight
AdaptationRight
TranslationRight
RDF Store &
Reasoner

21. 02.10.2013 Slide 21 of 26
• Ontology provides building blocks to model and reason
about contracts, policies, rights expression languages…
No, because it is matched
to pattern
which prohibits it:
“Artists does not want her
music with violent images”
Can we claim asset X
Copyright Ontology
Reasoning
RDF Store &
Reasoner

22. 02.10.2013 Slide 22 of 26
• Following Linked Data principles,
starting from just content ID (URI)…
Use Case-focused solution…
but Web-wide applicability
Content
URI
URI a ma:Image
dct:title “EBU HQ”
dct:copyright URI
…
HTTP GET
image/jpeg

23. 02.10.2013 Slide 23 of 26
Linked Data for Worldwide
Copyright Management?
Content
URI
URI a ebucore:Image
dct:title “EBU HQ”
dct:copyright URI
…
Agreement
URI
URI a co:Agreement
co:agent URI
co:theme …
…
Person
URI
URI a foaf:Person
foaf:name “…”
vcard:address …
vcard:country URI

24. 02.10.2013 Slide 24 of 26
Semantic Technologies for
Rights Data Integration & Intelligence
MediaMixer
Technologies
(media fragments,
semantic data,
annotation, ontologies,
reasoning,…)
Policy 1:
The licensor
disagrees the
licensee
adaptsmedia
fragments
containing…
<xml>
<right>
<play/>
</right>
<content
id=“frg1”/>
</xml>
Semantic
Media
Annotation
Rights Builder
User Interface
Rights Language
Mapping

25. 02.10.2013 Slide 25 of 26
Interested? Get involved
• Join MediaMixer community to know more
(http://community.mediamixer.eu):
– discussions, use cases, demonstrators, tutorials,
presentations, software,…

26. More details:
http://community.mediamixer.eu
@project_mmixer
Roberto García (@rogargon)
Universitat de Lleida, Spain
http://rhizomik.net/~roberto
Thank you for your attention

27. Annex A
Copyright Reasoning Explained
More details: http://rhizomik.net/copyrightonto/

28. 02.10.2013 Slide 28 of 26
Copyright Reasoner
Reasoning Explained
• Vocabularies and ontologies provide the building blocks:
– Classes
– Properties
– Instances
– Restrictions
– …
Copyright Ontology Schema.org
Media Ontology

29. 02.10.2013 Slide 29 of 26
Copyright Reasoner
Reasoning Explained
• These blocks can be combined to
model facts like:
– A person is the creator of an asset
– An asset is part of a compilation
– …or an employee wants to reuse a
piece of content.
Copyright Ontology Schema.org Media Ontology
Event 1: A
XYZ Media
employee
Adapt Video
result
Creation…
Uses:

30. 02.10.2013 Slide 30 of 26
Copyright Reasoner
Reasoning Explained
• Blocks can also model patterns
to be matched by facts:
– For instance a rights expression
authorising to reuse a media
fragment…
Copyright Ontology Schema.org Media Ontology
<odrl><reso
urce>…</re
source>
<adapt>…<c
ontent id>
</odrl>
ODRL file:

31. 02.10.2013 Slide 31 of 26
Copyright Reasoner
Reasoning Explained
• Blocks can also model patterns
to be matched by facts:
– …or prohibitions, like a policy
forbidding to reuse violent media
for educational content
Copyright Ontology RRM Ontology Media Ontology
Policy 1:
Disagree
Adapt
Creation
has subject
Violence…
Policy:

32. 02.10.2013 Slide 32 of 26
Copyright Reasoner
Reasoning Explained
• All these models are then placed into a reasoner
– A tool based on semantic technologies capable of mixing them
taking into account ontologies (constraints, functions, rules,…)

33. 02.10.2013 Slide 33 of 26
Copyright Reasoner
Reasoning Explained
• Ontologies teach the reasoner how to:
– Detect when some facts match a pattern

34. 02.10.2013 Slide 34 of 26
Copyright Reasoner
Reasoning Explained
• Ontologies teach the reasoner how to:
– Give precedence to patterns so others cannot match

35. 02.10.2013 Slide 35 of 26
Copyright Reasoner
Reasoning Explained
• Ontologies teach the reasoner how to:
– …and interpret some patterns as allowed and others
as prohibited Allowed
Prohibited

36. 02.10.2013 Slide 36 of 26
Copyright Reasoner
Reasoning Explained
• So they can respond to queries based on all this:
No, because it is matched
to pattern
which prohibits it.
Can we reuse asset A