SlideShare une entreprise Scribd logo
1  sur  31
Télécharger pour lire hors ligne
Superimposed Image Description
and Retrieval for Fish Species
Identification
29 September, 2009
ECDL 2009, Corfu, Greece


Uma Murthy, Edward Fox,
Yinlin Chen, & Eric Hallerman*
Computer Science
*Fisheries and Wildlife Sciences


Ricardo Torres, Evandro Ramos,
& Tiago Falcão
Institute of Computing
Context of this work
                                          digital
                                         libraries




         content              services                users            scenarios



documents       annotations    search      annotation     scholars        biodiversity



                       text-based                                          fisheries
                       retrieval                     students   faculty
images   text
                sub-            content-based            researchers        species
                documents       image retrieval                             identification

                                                                                   2
Motivation
•  Many scholarly tasks involve working with
   images with significant number of details
  –  Species identification, analyzing paintings,
     studying architecture styles, analyzing medical
     images, etc.

•  Scholars combine paper-based and electronic
   methods/tools
  –  Tools are not well-integrated, information is
     fragmented and tedious to access  ineffective
     and inefficient task execution
                                                       3
Paper-based methods for fish
identification
Dichotomous keys                                                              Personal Notes
1a Paired fins absent; jaws absent, mouth in an oral
  disk (the disk mostly surrounded by a fleshy hood
  in larvae); 7 external gill openings present in row
  behind eye .......... Lampreys - Petromyzontidae p. ooo
1b Paired fins present (at least 1 set); jaws present;
  1 external gill opening per side ................................... 2
2a Caudal fin heterocercal or abbreviate heterocercal
  (Figure 5) ............................................................ 3
2b Caudal fin protocercal (Figure 13, Part 2, upper left)
  or homocercal (Figure 5) ............................................ 6




                                                                                               4
Popular electronic methods
                  FISHBASE: Image collection
                  with browsing, field-wise
                  searching and use of forums

                                     EFISH: Organization and
                                     browsing based on taxonomy




                     EKEY: Text search and shape-
                     based image retrieval on
                     predefined shapes                    5
Requirements of a new system
•  Incorporate more visual and descriptive
   information
  –  Ability to add user content
•  Improve information management access
   to heterogeneous information
  –  Images, textual descriptions, notes, image
     markings, etc.
•  Provide well-integrated functionalities to
   support task execution
•  Provide capability to share content
                                                  6
The Superimposed Image Description
  and Retrieval Tool (SuperIDR)

                Content-based
                image retrieval
                (CBIR)
                description and       Digital library
                retrieval of images   services
Superimposed    using image           •  searching,
information     features such as         browsing,
subdocuments    shape, color, etc.       indexing, etc.
annotations                           •  Managing
new structure                            documents,
                                         collections,
                Text-based               metadata, etc.
                image retrieval
                description and
                retrieval of images
                using text

                                                          7
SuperIDR: species organization




                                 8
SuperIDR: species description (1/2)




                                      9
SuperIDR: species image annotation




                                 10
SuperIDR: species description




                                11
SuperIDR: text-based image retrieval
query




                                       12
SuperIDR: text-based image retrieval
results




                                       13
SuperIDR: text- and content-based image
retrieval (1/2)




                                      14
SuperIDR: text- and content-based image
retrieval (2/2)




                                      15
User study in an UG Ichthyology class
(Spring 2008) – 1/2

•  Goal: to assess the usefulness of
   SuperIDR in fish species identification
•  Participants: 28 undergraduates,
   working in teams of two with a tablet
   PC


                                        16
Fill entry
     questionnaire


   Get assigned tablet
                            Study procedure
   PCs with SuperIDR


        Tutorial


  Use tool for a week,            Compare methods
   make annotations
                               Session 1: half the teams
                                use SuperIDR and half
Task: identify 20 unknown
Task: identify 20 unknown      used traditional methods
 specimens in 2 sessions
 specimens in 2 sessions

                                  Session 2: teams
        Fill exit
        Fill exit                 swapped methods
     questionnaire
     questionnaire


 Keep/return SuperIDR
 Keep/return SuperIDR                                  17
Summary of results (1/2)
•  Method had a significant impact on task
   outcome (p-value=0.015), with higher
   likelihood of success in using SuperIDR
   than traditional methods
•  In general, students were interested in
   using SuperIDR for species
   identification
  –  Positive comments by students
  –  Six teams chose to keep the tool

                                         18
Comments

“it was very helpful”

“very helpful for taxonomy but need better
photos”

“very neat but might take a while to master all
the key concepts”



                                                  19
Summary of results (2/2)

•  No significant evidence to show that
   annotation or search on parts of images
   is useful
  –  Possibly due to timing and duration of the
     study




                                                  20
Conclusion
•  Developed SuperIDR that combines text- and content-
   based image description, retrieval, and browsing,
   including for parts of images
•  Ichthyology students performed better with SuperIDR
   than with traditional methods in fish species
   identification
•  Current/future work
   –  Conduct further experiments to test retrieval effectiveness
      and usefulness of combined search (text + CBIR) on parts
      of images
   –  Make SuperIDR available for download
   –  Connect with Flickr to enable information sharing and to
      leverage existing Flickr collection                      21
Thank you


                           ?
     ?
      http://si.dlib.vt.edu/superidr
                                       22
Back-up slides




                 23
Acknowledgements

•  Dr. Orth, Ryan McManamay,
   Lindsey Pierce, and others at the
   Fisheries and Wildlife Sciences
   department
•  Microsoft (tablet PC grant)
•  Jason Lockhart for loaning tablets
•  NSF (DUE-0435059)
                                        24
SuperIDR architecture
      Annotation              Search             Browse




                          Annotation

                           Search
CBISC                                                   Lucene

                           Browse




Feature      Taxonomy                  Image     Species
                          Images               descriptions
                                                              Annotations
Vectors      & key data                marks
                                                                  25
Content Based
Information
Retrieval




                26
Goal
Provide better support to work with
images, parts of images, providing
description through personal notes,
linking related information, providing
access to existing and new information
easily, and being able to share all this
information



                                           27
Species and image data

•  207 species of Virginia freshwater
   fishes
  – 25 families and 124 genera
•  213 images, mostly from Jenkins
   and Burkhead’s Freshwater Fishes
   of Virginia


                                        28
Data collected in experiment

•  Entry and exit questionnaire responses
•  Species identification responses and
   time to identify
•  Logs of user interaction with the tool
•  Data from 9 teams (18 students) was
   considered in analysis
  –  Others not present for both sessions
  –  Incomplete species id. responses
                                            29
Species identification response summary
Traditional methods                 SuperIdR
Team                 # Correct        Team                # Correct
  ID         Session (out of 20)         ID     Session   (out of 20)
       2           1          15        2            2        18
       4           1          16        4            2        17
       6           1          13        6            2        17
      11           1          12        11           2        16
       3           2           8        3            1        14
       5           2          13        5            1        15
       9           2          12        9            1        10
     10            2          10        10           1        14
       13           2          11       13           1        11
                Mean         12.2             Mean           14.67
            Method had a significant impact on outcome and           30
                  students did better with SuperIDR
Next steps
•  Further evaluation to show that
   combining text- and content-based
   image retrieval on parts of images is
   effective and useful
•  Explore use in other fields - art history,
   architecture, and other biology-based
   fields
  –  Involving images with significant number of
     details

                                                31

Contenu connexe

Similaire à SuperIDR - ECDL 2009

DataOne - Suzie Allard - RDAP12
DataOne - Suzie Allard - RDAP12DataOne - Suzie Allard - RDAP12
DataOne - Suzie Allard - RDAP12ASIS&T
 
Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...
Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...
Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...GigaScience, BGI Hong Kong
 
One-shot Wonder: Integrating Chemical Information Literacy into the Curriculum
One-shot Wonder: Integrating Chemical Information Literacy into the CurriculumOne-shot Wonder: Integrating Chemical Information Literacy into the Curriculum
One-shot Wonder: Integrating Chemical Information Literacy into the CurriculumLinda Galloway
 
Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011Rudy Potenzone
 
Introduction to Research Data Management for postgraduate students
Introduction to Research Data Management for postgraduate studentsIntroduction to Research Data Management for postgraduate students
Introduction to Research Data Management for postgraduate studentsMarieke Guy
 
Syllabus ed sci 198 methods of research
Syllabus ed sci 198 methods of researchSyllabus ed sci 198 methods of research
Syllabus ed sci 198 methods of researchMaria Theresa
 
World War II Poster Project & ACRL Information Literacy
World War II Poster Project & ACRL Information LiteracyWorld War II Poster Project & ACRL Information Literacy
World War II Poster Project & ACRL Information LiteracyAbby Clobridge
 
Functional and Architectural Requirements for Metadata: Supporting Discovery...
Functional and Architectural Requirements for Metadata: Supporting Discovery...Functional and Architectural Requirements for Metadata: Supporting Discovery...
Functional and Architectural Requirements for Metadata: Supporting Discovery...Jian Qin
 
Triplifier talk
Triplifier talkTriplifier talk
Triplifier talkJohn Deck
 
Research Data Sharing LERU
Research Data Sharing LERU Research Data Sharing LERU
Research Data Sharing LERU LIBER Europe
 
RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...
RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...
RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...ASIS&T
 
Biotracker: Melding Human and Machine Capabilities to Document the World's L...
 Biotracker: Melding Human and Machine Capabilities to Document the World's L... Biotracker: Melding Human and Machine Capabilities to Document the World's L...
Biotracker: Melding Human and Machine Capabilities to Document the World's L...Harish Vaidyanathan
 
Databases and Ontologies: Where do we go from here?
Databases and Ontologies:  Where do we go from here?Databases and Ontologies:  Where do we go from here?
Databases and Ontologies: Where do we go from here?Maryann Martone
 
Scientific data management from the lab to the web
Scientific data management   from the lab to the webScientific data management   from the lab to the web
Scientific data management from the lab to the webJose Manuel Gómez-Pérez
 
Bacterial Counting: Quick, easy and accurate?
Bacterial Counting: Quick, easy and accurate?Bacterial Counting: Quick, easy and accurate?
Bacterial Counting: Quick, easy and accurate?MACE Lab
 
Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012
Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012
Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012SEAD
 

Similaire à SuperIDR - ECDL 2009 (20)

Nutr8621 day1
Nutr8621 day1Nutr8621 day1
Nutr8621 day1
 
DataOne - Suzie Allard - RDAP12
DataOne - Suzie Allard - RDAP12DataOne - Suzie Allard - RDAP12
DataOne - Suzie Allard - RDAP12
 
Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...
Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...
Alexandra Basford, InCoB 2011: A Journal’s Perspective on Data Standards and ...
 
Michener Plenary PPSR2012
Michener Plenary PPSR2012Michener Plenary PPSR2012
Michener Plenary PPSR2012
 
One-shot Wonder: Integrating Chemical Information Literacy into the Curriculum
One-shot Wonder: Integrating Chemical Information Literacy into the CurriculumOne-shot Wonder: Integrating Chemical Information Literacy into the Curriculum
One-shot Wonder: Integrating Chemical Information Literacy into the Curriculum
 
Engaging the Researcher in RDM
Engaging the Researcher in RDMEngaging the Researcher in RDM
Engaging the Researcher in RDM
 
Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011
 
Introduction to Research Data Management for postgraduate students
Introduction to Research Data Management for postgraduate studentsIntroduction to Research Data Management for postgraduate students
Introduction to Research Data Management for postgraduate students
 
Syllabus ed sci 198 methods of research
Syllabus ed sci 198 methods of researchSyllabus ed sci 198 methods of research
Syllabus ed sci 198 methods of research
 
World War II Poster Project & ACRL Information Literacy
World War II Poster Project & ACRL Information LiteracyWorld War II Poster Project & ACRL Information Literacy
World War II Poster Project & ACRL Information Literacy
 
Functional and Architectural Requirements for Metadata: Supporting Discovery...
Functional and Architectural Requirements for Metadata: Supporting Discovery...Functional and Architectural Requirements for Metadata: Supporting Discovery...
Functional and Architectural Requirements for Metadata: Supporting Discovery...
 
Maeder nov2011
Maeder nov2011Maeder nov2011
Maeder nov2011
 
Triplifier talk
Triplifier talkTriplifier talk
Triplifier talk
 
Research Data Sharing LERU
Research Data Sharing LERU Research Data Sharing LERU
Research Data Sharing LERU
 
RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...
RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...
RDAP13 Ixchel Faniel: Can Quantitative Social Scientists Get Data Reuse Satis...
 
Biotracker: Melding Human and Machine Capabilities to Document the World's L...
 Biotracker: Melding Human and Machine Capabilities to Document the World's L... Biotracker: Melding Human and Machine Capabilities to Document the World's L...
Biotracker: Melding Human and Machine Capabilities to Document the World's L...
 
Databases and Ontologies: Where do we go from here?
Databases and Ontologies:  Where do we go from here?Databases and Ontologies:  Where do we go from here?
Databases and Ontologies: Where do we go from here?
 
Scientific data management from the lab to the web
Scientific data management   from the lab to the webScientific data management   from the lab to the web
Scientific data management from the lab to the web
 
Bacterial Counting: Quick, easy and accurate?
Bacterial Counting: Quick, easy and accurate?Bacterial Counting: Quick, easy and accurate?
Bacterial Counting: Quick, easy and accurate?
 
Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012
Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012
Data 2012 -- Presentation by Margaret Hedstrom (Jan 2012
 

SuperIDR - ECDL 2009

  • 1. Superimposed Image Description and Retrieval for Fish Species Identification 29 September, 2009 ECDL 2009, Corfu, Greece Uma Murthy, Edward Fox, Yinlin Chen, & Eric Hallerman* Computer Science *Fisheries and Wildlife Sciences Ricardo Torres, Evandro Ramos, & Tiago Falcão Institute of Computing
  • 2. Context of this work digital libraries content services users scenarios documents annotations search annotation scholars biodiversity text-based fisheries retrieval students faculty images text sub- content-based researchers species documents image retrieval identification 2
  • 3. Motivation •  Many scholarly tasks involve working with images with significant number of details –  Species identification, analyzing paintings, studying architecture styles, analyzing medical images, etc. •  Scholars combine paper-based and electronic methods/tools –  Tools are not well-integrated, information is fragmented and tedious to access  ineffective and inefficient task execution 3
  • 4. Paper-based methods for fish identification Dichotomous keys Personal Notes 1a Paired fins absent; jaws absent, mouth in an oral disk (the disk mostly surrounded by a fleshy hood in larvae); 7 external gill openings present in row behind eye .......... Lampreys - Petromyzontidae p. ooo 1b Paired fins present (at least 1 set); jaws present; 1 external gill opening per side ................................... 2 2a Caudal fin heterocercal or abbreviate heterocercal (Figure 5) ............................................................ 3 2b Caudal fin protocercal (Figure 13, Part 2, upper left) or homocercal (Figure 5) ............................................ 6 4
  • 5. Popular electronic methods FISHBASE: Image collection with browsing, field-wise searching and use of forums EFISH: Organization and browsing based on taxonomy EKEY: Text search and shape- based image retrieval on predefined shapes 5
  • 6. Requirements of a new system •  Incorporate more visual and descriptive information –  Ability to add user content •  Improve information management access to heterogeneous information –  Images, textual descriptions, notes, image markings, etc. •  Provide well-integrated functionalities to support task execution •  Provide capability to share content 6
  • 7. The Superimposed Image Description and Retrieval Tool (SuperIDR) Content-based image retrieval (CBIR) description and Digital library retrieval of images services Superimposed using image •  searching, information features such as browsing, subdocuments shape, color, etc. indexing, etc. annotations •  Managing new structure documents, collections, Text-based metadata, etc. image retrieval description and retrieval of images using text 7
  • 10. SuperIDR: species image annotation 10
  • 12. SuperIDR: text-based image retrieval query 12
  • 13. SuperIDR: text-based image retrieval results 13
  • 14. SuperIDR: text- and content-based image retrieval (1/2) 14
  • 15. SuperIDR: text- and content-based image retrieval (2/2) 15
  • 16. User study in an UG Ichthyology class (Spring 2008) – 1/2 •  Goal: to assess the usefulness of SuperIDR in fish species identification •  Participants: 28 undergraduates, working in teams of two with a tablet PC 16
  • 17. Fill entry questionnaire Get assigned tablet Study procedure PCs with SuperIDR Tutorial Use tool for a week, Compare methods make annotations Session 1: half the teams use SuperIDR and half Task: identify 20 unknown Task: identify 20 unknown used traditional methods specimens in 2 sessions specimens in 2 sessions Session 2: teams Fill exit Fill exit swapped methods questionnaire questionnaire Keep/return SuperIDR Keep/return SuperIDR 17
  • 18. Summary of results (1/2) •  Method had a significant impact on task outcome (p-value=0.015), with higher likelihood of success in using SuperIDR than traditional methods •  In general, students were interested in using SuperIDR for species identification –  Positive comments by students –  Six teams chose to keep the tool 18
  • 19. Comments “it was very helpful” “very helpful for taxonomy but need better photos” “very neat but might take a while to master all the key concepts” 19
  • 20. Summary of results (2/2) •  No significant evidence to show that annotation or search on parts of images is useful –  Possibly due to timing and duration of the study 20
  • 21. Conclusion •  Developed SuperIDR that combines text- and content- based image description, retrieval, and browsing, including for parts of images •  Ichthyology students performed better with SuperIDR than with traditional methods in fish species identification •  Current/future work –  Conduct further experiments to test retrieval effectiveness and usefulness of combined search (text + CBIR) on parts of images –  Make SuperIDR available for download –  Connect with Flickr to enable information sharing and to leverage existing Flickr collection 21
  • 22. Thank you ? ? http://si.dlib.vt.edu/superidr 22
  • 24. Acknowledgements •  Dr. Orth, Ryan McManamay, Lindsey Pierce, and others at the Fisheries and Wildlife Sciences department •  Microsoft (tablet PC grant) •  Jason Lockhart for loaning tablets •  NSF (DUE-0435059) 24
  • 25. SuperIDR architecture Annotation Search Browse Annotation Search CBISC Lucene Browse Feature Taxonomy Image Species Images descriptions Annotations Vectors & key data marks 25
  • 27. Goal Provide better support to work with images, parts of images, providing description through personal notes, linking related information, providing access to existing and new information easily, and being able to share all this information 27
  • 28. Species and image data •  207 species of Virginia freshwater fishes – 25 families and 124 genera •  213 images, mostly from Jenkins and Burkhead’s Freshwater Fishes of Virginia 28
  • 29. Data collected in experiment •  Entry and exit questionnaire responses •  Species identification responses and time to identify •  Logs of user interaction with the tool •  Data from 9 teams (18 students) was considered in analysis –  Others not present for both sessions –  Incomplete species id. responses 29
  • 30. Species identification response summary Traditional methods SuperIdR Team # Correct Team # Correct ID Session (out of 20) ID Session (out of 20) 2 1 15 2 2 18 4 1 16 4 2 17 6 1 13 6 2 17 11 1 12 11 2 16 3 2 8 3 1 14 5 2 13 5 1 15 9 2 12 9 1 10 10 2 10 10 1 14 13 2 11 13 1 11 Mean 12.2 Mean 14.67 Method had a significant impact on outcome and 30 students did better with SuperIDR
  • 31. Next steps •  Further evaluation to show that combining text- and content-based image retrieval on parts of images is effective and useful •  Explore use in other fields - art history, architecture, and other biology-based fields –  Involving images with significant number of details 31