BBC Archive Exploration with Video Hyperlinking
•
1 j'aime•604 vues
Presentation on the set-up of the MediaEval Search & Linking task 2013
Recommandé
Recommandé
Contenu connexe
Similaire à BBC Archive Exploration with Video Hyperlinking
Similaire à BBC Archive Exploration with Video Hyperlinking (20)
Plus de roelandordelman.nl
Plus de roelandordelman.nl (12)
Dernier
Dernier (20)
BBC Archive Exploration with Video Hyperlinking
- 1. Set-‐up of the Search & Hyperlinking Task in the Series Evalua'on of Video Hyperlinking Roeland Ordelman -‐ Audiovisual Content Access University of Twente, The Netherlands Netherlands Ins@tute for Sound and Vision Cross-‐Media Interac@on
- 2. Video hyperlinking • Topic with increasing aEen@on • Relate video (fragments) to other mul@media sources • Various perspec@ves and approaches, e.g.,: – Recommenda@on/Detail-‐on-‐demand – Seman@c/Event Web – Storytelling/narra@ves – Summariza@on/Clustering • Link up communi@es by means of special sessions, workshops and benchmarks – Mul@media Modeling (MMM 2014) – Dublin. Special session on Mul@media Hyperlinking and Retrieval – CfP due 01 August 2013 hEp://mmm2014.org/special-‐sessions/ 7/2/13 DGA workshop -‐ July 2013, Paris
- 3. MediaEval • a mul$media benchmarking ini@a@ve running several tasks • evaluates new algorithms for mul@media access and retrieval linked to a use case • emphasis on the "mul$" in mul@media: audio, speech, visual content, tags, users, context. • new evalua@on paradigms involving users h3p://www.mul:mediaeval.org
- 4. 7/2/13 DGA workshop -‐ July 2013, Paris
- 5. Video linking scenario • Audiovisual archive explora$on scenario • Video-‐to-‐video linking • Start with archive search: assuming an “informa$on need” • Poten@ally search may go into: serendipitous or exploratory type of behavior • Create meaningful cross-‐sec:ons through archive • Audiovisual archive interlinked on the fragment level • Automated “Storytelling” type of approaches 7/2/13 DGA workshop -‐ July 2013, Paris
- 6. Terminology • Video (e.g, 2 hours) • Interes@ng segment (e.g. 10 min) • Anchor: segment for which a user requests a link (e.g., 1 min) “I want to know more about this” • Hyperlink • Target: relevant segment for given anchor (e.g., 5 min) 7/2/13 DGA workshop -‐ July 2013, Paris
- 7. ME 2013 Search & Hyperlinking • ME 2012 S&HL “brave new” task: – Search: retrieve known-‐item video segment given a natural language descrip@on – Linking: link known-‐item video segment to similar segments in the collec@on (blib.tv) • ME 2013 S&HL “regular” task – Search: retrieve known-‐item video segment given a natural language descrip@on – Linking: link user-‐defined anchor within the known-‐ item to relevant target video segments in the collec@on (bbc archive) 7/2/13 DGA workshop -‐ July 2013, Paris
- 8. ME 2013 Search & Hyperlinking 7/2/13 DGA workshop -‐ July 2013, Paris Anchor Jump-‐in point
- 9. Task set-‐up 1. Organizers: provide anchors to par@cipants 2. Par@cipants: find relevant targets 3. Organizers: evaluate relevance targets 4. Organizers+Par@cipants: discuss at workshop 7/2/13 DGA workshop -‐ July 2013, Paris
- 10. 1. Provide anchors • Requirements: – Large video data set – Real ‘users’ selec@ng anchors • Data: – 270h of BBC archival data • One complete week • Excluded: material not owned by BBC (e.g., Dr. Who) • Users: … 7/2/13 DGA workshop -‐ July 2013, Paris
- 11. Researchers & Educators AXES Research system Journalists Research Academic researchers & students Inves:gate Academic educators Educate Home users AXES Home system Home users Entertainment, Edutainment Main group User Target Media Professionals AXES Pro system Broadcast Professionals Reuse Media Archivists Annotate User requirements study 7/2/13 DGA workshop -‐ July 2013, Paris ! !!
- 12. Users • Recruitment Agency • In an@cipa@on on a rather complex task: public users with computer & online experience • 30 Public Users – Na@onality: UK – Age group: 16-‐30 – Computer familiarity: high – Online ac@vity: high 7/2/13 DGA workshop -‐ July 2013, Paris
- 13. ANCHOR SELECTION BBC User Trials 7/2/13 DGA workshop -‐ July 2013, Paris
- 14. Search for an interes@ng video 7/2/13 DGA workshop -‐ July 2013, Paris potentially interesting videoSEARCH BBC ARCHIVE
- 15. 7/2/13 DGA workshop -‐ July 2013, Paris Formulate Informa@on need Text search Visual search AXES PRO system
- 16. Select an interes@ng clip 7/2/13 DGA workshop -‐ July 2013, Paris BBC ARCHIVE potentially interesting video interesting clip SEARCH
- 17. Select an anchor 7/2/13 DGA workshop -‐ July 2013, Paris BBC ARCHIVE potentially interesting video interesting clip interesting segment for linking SEARCH
- 18. 7/2/13 DGA workshop -‐ July 2013, Paris Describe target Describe anchor Select anchor using sliders
- 19. Trial evalua@on • Most par@cipants understood the task • Not difficult • Technology would be a very nice-‐to-‐have men@oning the “serendipitous” type of browsing behavior • Par@cipants want music as link anchors (which we did not an@cipate on!) 7/2/13 DGA workshop -‐ July 2013, Paris
- 20. Classifica@on of selected clips 7/2/13 DGA workshop -‐ July 2013, Paris 0 5 10 15 20 25
- 21. Classifica@on of anchors 7/2/13 DGA workshop -‐ July 2013, Paris 0 10 20 30 40 50 60 70 80 90 Whole Scene Speech Other Object, moving Object, sta:c First FA cup Final at Wembley Mayor Bloomberg of New York talking about Ken Livingstone
- 22. 2. Automa@c target selec@on • ~ 15 Par@cipants • Data – 50 NL queries, 50 anchors – 270h of BBC archival data (used for anchor selec@on) – + One month before ~1000h • Metadata – Archival metadata – Sub@tles – Speech transcripts (provided by LIMSI & LIUM) – Shot segmenta@on – (Self-‐derived features) 7/2/13 DGA workshop -‐ July 2013, Paris
- 23. 7/2/13 DGA workshop -‐ July 2013, Paris BBC ARCHIVE potentially interesting video interesting clip interesting segment for linking related video clip video hyperlink SEARCH LINK
- 24. 3. Evaluate target relevance 7/2/13 DGA workshop -‐ July 2013, Paris BBC ARCHIVE potentially interesting video interesting clip interesting segment for linking related video clip video hyperlink SEARCH LINK Relevance Assessment
- 25. Relevance assessment • Default procedure: – Par@cipants of 1st user trials – Evaluate for their own anchors the automa@cally generated targets from ME par@cipants • Risks: – No shows – Par@cipants can only evaluate top-‐N (N-‐max=5?) targets for a given anchor • Fall-‐back op@on: – Use Amazon Mechanical Turk approach – (same as we did in ME-‐2012 task) – Drawback • relevance assessment not based on personal trail “informa@on need” >> “clip selec@on” >> “anchor selec@on” >> “target evalua@on” 7/2/13 DGA workshop -‐ July 2013, Paris
- 26. Assessment interface 7/2/13 DGA workshop -‐ July 2013, Paris Original anchor & target descrip@on ANCHOR SUGGESTED TARGETS Relevance assessment & comments
- 27. Evalua@on metrics • MAP evalua@on for both searching and linking • Issues: – MAP does not provide means to evaluate segmenta@on (e.g., correct video but jump-‐in point is incorrect) – limit relevant results to a window of N (N=60 seconds) before and awer the actual beginning of the relevant segment • Addi@onal metric: – Mean Generalized Average Precision (mGAP) • awards runs that are closer to the jump-‐in point of the relevant content 7/2/13 DGA workshop -‐ July 2013, Paris
- 28. 4. Discuss at workshop 7/2/13 DGA workshop -‐ July 2013, Paris h3p://www.mul:mediaeval.org 18-‐19 October 2013 Barcelona, Spain
- 29. 7/2/13 DGA workshop -‐ July 2013, Paris Work funded by: Task Organizers: Robin Aly Maria Eskevitch Gareth Jones Roeland Ordelman h3p://www.mul:mediaeval.org