Talk given at the University of Middlesex, London to the Interaction Design Group, 2011-11-01.

1. Off the beaten track Using content-based multimedia similarity search for learning Suzanne Little Knowledge Media Institute The Open University 1 st November 2011

3. What can CBMS be used for?

4. How does CBMS work?

5. CBMS for interactive learning

7. Known object identification

8. General Search ?

10. Similarity

12. Calculate the “similarity” score between your query and the images you are searching

13. Return a ranked list of results

15. Find instances of re-use

16. Discover and/or exposit (undirected search)

17. Spot&Search

18. Spot&Search Scott Forrest : E=MC squared "Between finished surface texture and raw quarried stone. Between hard materials and soft concepts. Between text and context." More information

22. What could I use? I don't understand? How are other people using the concept in different contexts?

23. SocialLearn+Media

24. SocialLearn+Media

26. What can CBMS be used for?

27. How does CBMS work?

28. CBMS for interactive learning

30. Speed

31. Memory

32. Precision Q

35. Local features – invariant to changes in scale and transformation (e.g., colour, rotation) Lowe2004 - http://www.cs.ubc.ca/~lowe/keypoints/

36. Feature Space X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X SIFT has 128 dimensions X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X

37. Results Rank images by frequency of occurrence in the neighbourhoods of a query's keypoints Each nn is a keypoint of an image in the index, I. Q {x 1 , x 2 ,…, x n } SIFT Nearest Neighbour Index (FLANN) x 1 ->{nn 1 , nn 2 , ..., nn K } x 2 ->{nn 1 , nn 2 , ..., nn K } x n ->{nn 1 , nn 2 , ..., nn K } … I

42. Precision at 4: 0.77

44. Users like to know why

45. Confidence

47. I've got a screenshot – where does it come from? What is it's context?

48. I've got to prepare a course on Rome. I need some illustrations.

49. I don't understand this slide – find alternatives or discussions How can multimedia content-based search help?

51. Users like to know why (they're more forgiving!)

52. Tradeoff – speed vs accuracy

54. More sophisticated nearest neighbour analysis

59. to interact with material

60. to identify interesting stuff How can multimedia content-based search help?

62. Paul Hogan for iPhone development