This document discusses using visual approaches and screenshots to automate graphical user interfaces. It demonstrates a technique called template matching that allows finding small patterns in images despite changes in scale, rotation, color palette, or texture. The document also introduces various functions like find(), click(), type(), and wait() that can perform actions and interactions on the user interface based on visual matching of screenshots. Finally, it encourages the reader to experiment with building their own automation tools using these approaches.

2. What you’ll learn today

3. Visual approach to search and
automation of graphical user interfaces
using screenshots.

4. Just to see if it can be done
 Usually needed to automate GUIs:
 support from developers
 API access
 language/OS dependency
 position/naming dependencies

5. Demo

7. Template Matching for small patterns
 Invariance
 Resized versions
 Texturally similar, different color pallets

8.  Learn
 Extract model from training pattern
▪ Invariant to scale & rotation
 Encode in the model the relative position of the center
 Search
 Extract invariant features
 Infer possible model center position
 Cluster consistent features
 Validate supposition

9. find( )

10. f = Finder(path-to-imagefile)
f.find(path-to-imagefile, [similarity])
// iterates through Match objects
while(f.hasNext):
print “found match: “ + f.next().getScore()

11. Pattern(string)
Pattern(string).similar(0.9).targetOffset(10,30)
 Abstraction for visual patterns
 Are used by finding operations
 Methods can be chained to refine the pattern
 Can define click points

12. Region(x, y, w, h)
Region(region)
Region(Rectangle)
 Search in a given region
 Observe a region in background for changes
 Retrieve matches
 Optimize the search by chaining regions
 No need to be aware of the content

13. click( )

14. click(PSMLR),doubleClick(PSMLR)
dragDrop(PSMLRtarget, PSMLR destination)

15. type(PSMLR, text)
text()
 Uses the Tesseract OCR engine

16. mouseDown(button), mouseUp(button)
keyDown(keys), keyUp(keys)
 They can be used along with KeyModifiers

17. wait(PS), waitVanish(PS)

18. onAppear(PS, handler)
onVanish(PS, handler)
onChange([minChengedSize], handler)

19. How performant do you want it to be?
 A typical call to find() for a 100x100 target on
a 1600x1200 screen takes less than 200 msec
 Improve your searches

20. Why not give it a try and make your own?
 Record-playback
 Sikuli Guide
 …

21. Ok, the moment of truth!
 Screenshots – unstable interfaces
 Visibility constraints
A paradigm shift requires a thinking shift

22. Demo
What we’ve made of it