A conceptual model for the annotation of audiovisual heritage in a media studies context
•Télécharger en tant que PPTX, PDF•
1 j'aime•261 vues
See summary here: https://avindhsig.wordpress.com/workshop-2016-krakow/accepted-abstracts/liliana-melgar-jaap-blom-eva-baaren-marijn-koolen-roeland-ordelman/
Recommandé
Recommandé
Contenu connexe
Similaire à A conceptual model for the annotation of audiovisual heritage in a media studies context
Similaire à A conceptual model for the annotation of audiovisual heritage in a media studies context (20)
Dernier
Dernier (20)
A conceptual model for the annotation of audiovisual heritage in a media studies context
- 1. A conceptual model for the annotation of audiovisual heritage in a media studies context Liliana Melgar, Marijn Koolen, Jaap Blom, Eva Baaren, Roeland Ordelman Workshop “Audiovisual Data And Digital Scholarship: Towards Multimodal Literacy” Digital Humanities Krakow, 2016
- 2. What do “wish lists” and research have in common?
- 3. “Digital humanists are motivated annotators” (Walkowsky & Barker, 2014) Annotating is one of the so-called “scholarly primitives” (Unsworth, 2000)
- 4. A (oral) historian’s case How cultural activism is used in the claim for identity of “x” ethnic group? © AAC Library © southernfoodways.org ...“My focus is on the representations of the past as acts of storytelling (or personal “narratives”)”
- 5. The “Bricolage” of qualitative data analysis
- 6. Buster Keaton. 'Sherlock Jr.' (1924) © Tumblr.com “Moving images are a blind medium” (Sandom & Enser, 2001) Added challenges of AV media
- 7. Analysis of our research use cases Media aesthetics Media representations Cross-media analysis Social history of the media Social and cultural history Fan culture studies Genre studies Discursive representations (media linguistics) Visual representations Public debates War and conflict Journalism (data-driven, narrative) Postcolonial studies Migration Desmet case Reality tv Memories from the Moroccan- Berber activists in NL/FL Muslims/islam representations Amateur film/video Youtube broadcasts World War IIIndonesian decolonisation Public contention Refugee crisis
- 8. RQ: How to provide system support for manual annotation tasks of AV media in the context of Digital Humanities infrastructures? Requirements analysis Research use cases
- 9. Method Use case / requirements analysis: one to one interviews with media scholars Literature review Analysis of the data models, functionalities and interface features of current tools that support video annotation CAQDAS VIDEO ANALYSIS PROFESSIONAL VIDEO EDITORS DOMAIN SPECIFIC APPLICATIONS GENERIC WEB AV ANNOTATION ➔ Transana ➔ Nvivo ➔ Atlas.ti ➔ Anvil ➔ Advene ➔ Elan ➔ Lignes du temp ➔ Final Cut Pro ➔ Adobe’s Premiere ProCC ➔ Cinemetrics ➔ Linked TV ➔ ArtTube ➔ Youtube ➔ Synote
- 10. Preliminary Outcomes Study of the information annotating behavior of media scholars Analysis of existing tools for video annotation A process model (use case-driven) AV media-centered data model A concept model
- 11. Study of the information annotating behavior of media scholars Analysis of existing tools for video annotation A process model of annotation in a research workflow (use case- driven) A media-centered model of AV annotation (data-oriented) A concept -integrative model of annotation Preliminary Outcomes
- 12. Conceptual model
- 15. Scholar/ (oral) historian Thematization ManualInterpreting Themes / Narratives Oral history interviews Academic research
- 16. Current Open Annotation Data Model Current Open Annotation Data Model for scholarly research The Open Annotation Collaboration (OAC) Project (Hunter et al., 2010) Adds context and agent = Actor
- 17. Method, task, and purpose (in stages)
- 18. Previous work has identified stages in the scholarly research process Bron et al., 2012. “Overview of the phases in the media studies research cycle with associated search processes and changes in the research question (RQ). Arrows indicate possible sequences.” Traditional scholarly research process in the previous use cases include an “analysis” phase, which is mostly done through manual annotations, named as: -Coding (Grounded theory analysis) -Thematization (narratives) -Content analysis Previous work has also identified stages in search process (Huurdeman & Kamps, 2014; Koolen…)
- 19. ? RQ -Storytelling -Reporting Exploratory data analysis Data preparation Data enrichment Explore (other) datasets Research phases & annotation process Pre-focused annotation ➔ Bookmarking ➔ Open coding (initial coding)- -tagging ➔ Commenting (memos, tasks) Focused annotation ➔ Focused coding ➔ Classification/ Thematization / Narrative ➔ Codebook ➔ Commenting (analytical memos) Annotation preparation ➔ Defining coding schemes (layers) ➔ Select data enrichment services per layer Based on studies about research stages (Bron et al., 2015), search stages (Huurdeman & Kamps, 2014, Koolen et al., 2015); Qualitative data analysis theory (Charmaz, 2006); Concept of annotation (Agosti et al., 2012; Melgar, 2016); CLARIAH use cases (interviews with media scholars) and discussions during CLARIAH WP5 requirement analysis. Corpus selection Corpus analysis & enrichment Exploration Contextualization (Assembling) PresentationAnalysis
- 20. A (oral) historian’s case How cultural activism is used in the claim for identity of “x” ethnic group? Data preparation Explore (other) datasets Data enrichment Exploratory data analysis Focused annotation Corpus selection Pre-Focused annotation
- 21. Conclusion and future work • We need to approach “annotation” in a broader scope • We have come up with three models for annotation: • A conceptual model of annotation • A process model(s) • A media-centered, data-oriented model • Future work includes • Implementation of our model in the CLARIAH services • Evaluation with media scholars • New requirements and further development for each research use case
- 22. References Bron, Marc, Jasmijn van Gorp, and Maarten de Rijke. “Media Studies Research in the Data-Driven Age: How Research Questions Evolve.” Journal of the Association for Information Science and Technology, 2015, n/a-n/a. doi:10.1002/asi.23458. Huurdeman, Hugo C., and Jaap Kamps. “From Multistage Information-Seeking Models to Multistage Search Systems.” In Proceedings of the 5th Information Interaction in Context Symposium, 145–154. IIiX ’14. New York, NY, USA: ACM, 2014. doi:10.1145/2637002.2637020. Koolen, Marijn, Toine Bogers, Antal van den Bosch, and Jaap Kamps. “Looking for Books in Social Media: An Analysis of Complex Search Requests.” In Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Vienna, Austria, March 29 - April 2, 2015. Proceedings, edited by Allan Hanbury, Gabriella Kazai, Andreas Rauber, and Norbert Fuhr, 9022:184– 196. Lecture Notes in Computer Science, 2015. doi:10.1007/978-3-319-16354-3_19. Sandom, Christine, and P.G.B. Enser. “VIRAMI: Visual Information Retrieval for Archival Moving Imagery.” Milano, Italy: Archives & Museum Informatics, 2001. http://www.archimuse.com/publishing/ichim01_vol1/sandom.pdf. Unsworth, John. “Scholarly Primitives: What Methods Do Humanities Researchers Have in Common, and How Might Our Tools Reflect This?” London: King’s Collegue, 2000. http://people.brandeis.edu/~unsworth/Kings.5-00/primitives.html. Walkowski, Niels-Oliver, and Elton T.E. Barker. “Digital Humanists Are Motivated Annotators.” Laussane, Switzerland, 2014. http://dharchive.org/paper/DH2014/Paper-296.xml
Notes de l'éditeur
- Welcome to this presentation of a conceptual model for the annotation of audiovisual heritage in a media studies context. The model is part of the preliminary results of our work for the Dutch research infrastructure CLARIAH, which we will tell a bit more about later in this presentation. I am ? (my background is ?), I will present our model, also on behalf of ?, Jaap Blom, Eva Baaren, and Roeland Ordelman. LEUREN, TUNING
- Do these images and icons look familiar to you? These are examples of interactions that occur in current online web services where users can save favourite items on a wish list or basket, add comments, rankings or other types of “annotations” for their personal use. You may have thought already that these annotation tasks during item selection, in the Web or in other information systems, are somehow equivalent to manual practices in scholarly domains.
- Indeed, several authors and projects in the digital humanities have shown that scholars base their research on selecting sets of sources, and that they are active and motivated annotators of the sources that they select for their research, and also intensive note-takers during their research process.
- For example, one of the scholars in our team, Norah, is a historian who investigates cultural activism and collective memory of an ethnic group. She uses oral history interviews as her primary materials, and also combines them with other sources such as: Archival materials Private records of activists Reports of meetings kept by these activists, or History books
- During and after the process of collecting her sources, she analyzes them closely. For doing this, she transcribes all her audio interviews and then performs a close reading of all the transcripts in order to identify stories or themes of events, persons and places This is combined with the analysis of the other sources that she gathered, This analysis work becomes a kind of “bricolage”, as she described it herself
- In the context of media studies, or historical research which makes intensive use of audiovisual media, these annotation tasks are even more important! This is so because audiovisual materials are a kind of “blind medium”. Even though the current status of automatic content indexing is quite advanced and used in broadcast contexts, those techniques are not good enough yet to support academic researchers*. -- *This is a conclusion presented in Liliana’s thesis, if someone asks…
- Besides Nora’s case, there are other research use cases in media scholarship. We have grouped them into five categories or perspectives: (1)media aesthetics (2)The study of media representations (3)Cross-media analysis (4)Analysis of social history of the media. (5)And social and cultural history. This later perspective is not media scholarship per se, but similarly to Norah, several groups of historians (e.g., oral historians) make intensive use of audiovisual media, and are in consequence part of our use cases In all of these cases we have found a variety of annotation practices. For example, scholars studying the current refugee crisis in a cross media perspective, need to code, as they call it, enormous amounts of social media data, and would like to link those annotations to the content of other AV mass media, as well as to newspapers.
- In the CLARIAH projcet a service will be offered to media scholars to support their annotation process of AV media. Because of access problems related to copyright, and the issues we commented before in relation to AV indexing, one important part of the media suite will be a user space, in which researchers can individually or collaboratively annotate their sources. It is a kind of “annotation studio” for media scholars, In order to implement the media suite, we have have conducted a detailed requirements analysis
- START AT MIN. 6 This has been based on the study of the research and annotating behavior of media scholars, through one to one interviews, which gave as an output the use cases that we just described. In all these cases, and in the literature, we have also found that scholars either use or refer to different tools that support their annotation process. We have grouped them into 5 categories: -Qualitative assisted data analysis software, or caqdas, -Video analysis tools -Professional tools for video editing -Domain specific applications, the most important for us being Linked TV and ArtTube, which have been implemented by one of the authors of this paper -And generic web AV annotation systems We are currently analysing their data models and functionalities.
- As a result of the requirements analysis we have created three models: a conceptual model, a process model, and a data-oriented model of a great range of annotating activities.
- In this presentation our focus is on the conceptual model and on the process model.
- The bottom up analysis of the scholarly and curatorial annotation phenomena resulted in this complex model :-). You may not be able to see the details of this image, but what it includes is the graphical representation of our conceptual model :-) We will make it available as an ontology during the course of our project. At this stage, it is presented as a relational graph, a highly informal semantic network, technology independent, which provides a high-level view of the main conceptual dimensions and behavioral aspects involved in media annotation in the context of media scholarship.
- By grouping the concepts, we identified main areas in our diagram which actually correspond to the main dimensions: -actor -context -task -the actors’ motivation for the annotation, -the media documents, -the method -and the characteristics of the annotation output.
- These dimensions were abstracted in this simplified version of the model. Here we observe the “actor” as the center. This coincides with the tradition of Interactive information retrieval and information behavior, which places the actor (also known as the user) in the center.
- For example, our previous researcher’s case, Norah’s in this model can be summarized as follows: a scholar (historian), performs an annotating task in the form of thematization, using a manual method, on a group of oral history interviews with a purpose of interpreting in an academic organizational context. --Several other examples can be added depending on the type of researcher, the context, method and the annotating task.
- START AT MIN.8 Previous work on modeling annotation phenomena has produced at least two significant models: -The open annotation data model, which is the most abstract view on annotation, in which you have a target (meaning the object being annotated), and a body (meaning the “content” of the annotation). As you can see, the actor, and the context are missing in this model. -The Open Annotation data model for scholarly research adds to this simple model those missing dimensions: the context, and the actor. Which our model also identified.
- Our conceptual model adds to these two pre-existing models four new elements: the method (manual, automatic, semi-automatic), the task, the purpose of the task, and most importantly, the process dimension inherent to the method. Indeed, one important element identified in our model, zoomed in in this image, was that annotation in a scholarly context often occurs in the form of a process, labeled here as “research stage”.
- START AT MIN. 9 This conclusion agrees with previous research which has identified the characteristics of different research stages. Recent work includes Marc Bron’s thesis (2014), and Bron, van Gorp, and De Rijke’s paper, who identified three main phases in the research cycle of media studies researchers: exploration, contextualization and presentation. Similar work on information behavior and interactive information retrieval has identified the different stages of search in academic context and in book search.
- START AT MIN. 10 We departed from that research to develop the “process” element of our conceptual annotation model. This is the resulting diagram, which shows three phases of the annotation process: -A pre-focused phase, which occurs during the exploratory search and exploratory data analysis phases of the research process. In this stage, it is common that scholars use online systems as other general web users, making use of “baskets” and wish lists. -After refining their research question and the criteria for the selection of their sources (also called corpus selection, or contextualization in Bron’s work), the scholar determines which categories are necessary to conduct her work (e.g., locations, persons). We called this phase: annotation preparation. -Finally, a third stage of “focused annotation” is conducted during the analysis phase of the research process, in which codes or themes are grouped into categories and narratives. Obviously, these phases don’t necessarily occur linearly. The cycles are actually different depending also on the type of research question.
- MIN. 11 TO 14 Coming back to our researcher, Norah, once this model is implemented into the user space her research process and annotation needs will be hopefully supported. For example: --She enters the media suite and creates a project in the user space --Uploads her source materials (since she already had recorded some interviews and gathered archival collections) --She can here prepare her audio interviews for data analysis, by requesting the automatic speech recognition service --Because this automatic process may not be perfect, she can decide then to start a correction task herself, or postpone it for the data analysis phase, or send it to a crowdsourcing service. --Next, she can start coding or identifying themes in these interviews text bottom up, or she can also pre-define her categories via a coding scheme --In this way, she can determine which entities could be identified automatically or semi-automatically, or also in a collaborative way, and enrich her materials via named entity recognition or topic modeling services --The focus coding phase may be accompanied by parallel exploratory data analysis, in which she visualizes patterns in the data they are being annotated --Simultaneously, she could also search other datasets or collections, such as newspapers, as a way to complement her analysis, or as a way to create a new corpus for her cross-media analysis --In a cycle that ends when the researcher determines it, she could also pre-code those additional sources during the exploratory phase bringing them to the user space. These selections and annotations would be supported at different granularity levels. There are other scenarios depending on each research use case, and also on the departing point of the scholar: a pre-defined topic or research question, or an open curious mind to observe what new insights other datasets may bring. -- N. can via the User space upload her corpus dataset She can do manual annotation of her different media via fragment selection and bottom-up “coding” by also being able to pre-define her coding scheme (annotation layers) She can use speech recognition (speech to text transcription) for her audio records N. can decide whether she wants to use use semi-automatic support: N. can select the option to manually correct the automatic transcription, or send it to a crowdsourcing service N. can decide which “entities” need to be automatically extracted from the text: places, persons, She can perform topic modeling to identify recurrent themes
- MIN.15