2. Searches originating inside
and outside of your head
Hartwig Adam, Ulrich Buddemeier, David Petrou, Ted Power, Andrew Harp,
Mihai Badoiu, Avi Flamholz, Andrew Harp, Anthony Sciola, Gabriel Taubman,
Matt Bridges, Matt Casey, Sharvil Nanavati, Max Braun Johannes Steffens,
Jiayong Zhang, Lijia Jin, Alessandro Bissacco, Fernando Brucher, John Flynn,
Anand Pillai, Yuan Li, Andrew Rabinovich, Henry Rowley, Rafael Spring,
Andrew Hogue, Shailesh Nalawadi, Hartmut Neven

3. Sananga, eye drop entheogen
used by the Yawanawa tribe

4. Enables hunters to better see monkeys

5. Enables hunters to better see monkeys

6. Definition
Augmented Reality
View of a physical real-world environment augmented by computer-
generated imagery R. Azuma 1997, P. Milgram and A. F. Kishino 1994
Generalization
Presentation of information as a function of the environmental context
Not: Visual Search
Using an image as a search query
Not: Wearing a head mounted display

7. Will AR cause a paradigm shift comparable to the GUI?
Some deconstruction …
Pixels are from men, photons are from god.

8. If this is our future then maybe AR
is not all that important

9. Augmented Reality requires a physical reality
one is interested in
Kaiser Family Foundation: Today, 8-18 year-olds devote an
average of 7 hours and 38 minutes (7:38) to using
entertainment media across a typical day (more than 53 hours a
week). And because they spend so much of that time 'media
multitasking' (using more than one medium at a time), they
actually manage to pack a total of 10 hours and 45 minutes
(10:45) worth of media content into those 7½ hours.

10. Augmented Reality requires a physical reality
one is interested in
• Prerequisite is a novel object in my environment or the availability of
novel information about an object nearby.
• Hence AR may not meet the bar of daily engagement.
• Same holds for visual search.

11. At least a 10:1 ratio between externally
and internally triggered searches
• Less opportunity for externally triggered searches
• But if performed it feels magical and is often of high utility
• Most valuable when there is no voice or text substitute:
o Faces, do a background check on new nanny
Disclaimer: Face recognition will only be offered once accepatable privacy
models have been established
o Restaurant in Tokyo for person who does not speak Japanese
• High convenience
o Barcode
o Text for translation

12. Sources of Augmentation
From Memory
• Conventional
• Examples:
• Navigation information
• User generated updates and themes: Layar, Tonchidot

13. Sources of Augmentation
From Memory
• More examples
• AR ads
• AR in complex visualizations

14. Sources of Augmentation
From Sensors
• Unusual
• Multispectral camera
o infrared: notice a person lying, find your pet
o ultraviolet: determine quality of food
• Far out: Lockheed's recent patent on quantum radar

15. Google Goggles
Design principles behind Goggles
1. Universal
1. Makes it more difficult compared to vertical solutions
2. Needs very low false positive rate
2. To the degree possible do not force the user to select modes
3. Specificity
1. Object instance more important than object class recognition
4. Put best foot forward
5. Recall, precision, scale and speed

16. Recognition disciplines that work and do not work

17. Summary of the state of the art
(for product managers:)
Product Faces Unpackaged
Rigid Packaging Products
Articulate
Textured Logos Contour Defined
Landmarks Cars Pets
Easy Hard

18. Accuracy of place recognition
130K Place Models automatically
mined from Photo Collections

19. TOTO The Other Textured Objects
100
10-1
> 50M images
in database
10-2
10-3

20. Current accuracy in face recognition

22. Word level recall/precision of OCR
Dense Text Sparse Text

23. Grand challenge vision problems
1. Massive Class Recognition
2. Recognition of weakly textured,
contour defined objects

24. Demo: Recognition triggered virtual objects

25. Roadmap
• Combine external and internal searches
within one framework
• Audio-visual search
• Audio feedback to guide the user
• Make the search engine your personal companion
• Offer AR when it is the superior UI paradigm

26. Thank you!
Hartwig Adam, Ulrich Buddemeier, David Petrou, Ted Power, Andrew Harp,
Mihai Badoiu, Avi Flamholz, Andrew Harp, Anthony Sciola, Gabriel Taubman,
Matt Bridges, Matt Casey, Sharvil Nanavati, Max Braun Johannes Steffens,
Jiayong Zhang, Lijia Jin, Alessandro Bissacco, Fernando Brucher, John Flynn,
Anand Pillai, Yuan Li, Andrew Rabinovich, Henry Rowley, Rafael Spring,
Andrew Hogue, Shailesh Nalawadi, Hartmut Neven