Facial recognition in production is difficult because neural networks are slow and expensive to train, and must be retrained to recognize new faces added to the set. Older approaches which address these issues such as eigenfaces exist don’t scale as they require a matrix decomposition. Apache Mahout offers a distributed singular value decomposition method, which scales to matrices of arbitrary sizes on Apache Spark, making it possible to use the older yet still powerful Eigenfaces approach to recognize and add new faces in near real time (with the help of Apache Solr). In this talk we present a full stack lambda-style facial recognition system. The offline component uses Mahout/Spark to compute the eigenfaces. The online component leverages Apache Flink to identify faces in an image, decompose the face into a linear combination of the eigenfaces, search for a matching face using SOLR, and if no match is found add the face as a “new face”.

1. Twitter: @rawkintrevo

2. Twitter: @rawkintrevo
About me-
• Trevor Grant
• PMC Apache Mahout/Apache Streams
• Open Source Evangelist and AI Engineer @IBM

3. Twitter: @rawkintrevo
Simple Cylon
• OpenCV for Facial Detection
• Mahout / Eigenfaces for Facial Recognition
• Problems
Image Credit: Electric Alivia
https://i.pinimg.com/736x/6a/be/0d/6abe0da5907a5f4923b0c94389a02ddc--pretty-tattoos-pin-up-tattoos.jpg

4. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

5. Twitter: @rawkintrevo
Mahout Decompose FaceRect into Linear Comb Vec
321
4 5 6
7 8 9
1
2
3
4
5
6
7
8
9
Eigenfaces
Ordinary Least Squares
Linear Combination of Eigenfaces

6. Twitter: @rawkintrevo
Search k-Length Vector
(k is number of Eigenfaces ~ 130)

7. Twitter: @rawkintrevo
That’s Apollo
Solr Returns Closest Match
Drone says “hi” or shoots to kill

8. Twitter: @rawkintrevo
Cascade Filter Overview
• Not a Neural Net (NNets are too slow)
• OpenCV -> Haar Cascades are much more efficient
• Scans for areas that match certain patterns.

9. Twitter: @rawkintrevo
Cascade Filter Overview

10. Twitter: @rawkintrevo
Cascade Filter

11. Twitter: @rawkintrevo
Cascade Filter (Areawise)

12. Twitter: @rawkintrevo
Eigenfaces

13. Twitter: @rawkintrevo
Eigenfaces

14. Twitter: @rawkintrevo
Eigenfaces (Pixelwise)

15. Twitter: @rawkintrevo
Eigenfaces (Pixelwise)

16. Twitter: @rawkintrevo
Eigenfaces (Pixelwise)

17. Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
22 85 54 123
56 187 92 91
111 204 103 245
8 247 155 212
239 87 99 84

18. Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
22 85 54 123
56 187 92 91
111 204 103 245
8 247 155 212
239 87 99 84

19. Twitter: @rawkintrevo
Eigenfaces

20. Twitter: @rawkintrevo
Eigenfaces

21. Twitter: @rawkintrevo
Eigenfaces
Matrix of Faces
ith Image
jth Pixel Position

22. Twitter: @rawkintrevo
Eigenfaces: Singular Value Decomposition
Matrix of FacesU Vx =

23. Twitter: @rawkintrevo
Eigenfaces: Matrix V
Matrix of FacesU V (Eigenfaces)x =

24. Twitter: @rawkintrevo
Eigenfaces: Matrix U
Matrix of FacesU Vx =
Linear combinations of Eigenfaces required to form the Nth Face
= 2.456 x - 7.2345 x + 0.4125 x

25. Twitter: @rawkintrevo
New Faces
y
V Transpose
(each column is
eigenface)

26. Twitter: @rawkintrevo
New Faces
y X
Simple Regression (OLS)
Ordinary Least Squares
β

27. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

28. Twitter: @rawkintrevo
Mahout Decompose FaceRect into Linear Comb Vec
321
4 5 6
7 8 9
1
2
3
4
5
6
7
8
9
Eigenfaces
Ordinary Least Squares
Linear Combination of Eigenfaces

29. Twitter: @rawkintrevo
Search k-Length Vector
(k is number of Eigenfaces ~ 130)

30. Twitter: @rawkintrevo
That’s Apollo
Solr Returns Closest Match
Drone says “hi” or shoots to kill

31. Twitter: @rawkintrevo
Problems with Simple Cylon
• OpenCV/Haar Cascades inconsistently boxes face
• (Eigenfaces requires images be centered and same size as training)
• Adds face every time angle changes (ideally we want multiple
combinations assosciated with each person)
• Will insert “bad” faces, and “not” faces.
• To much ”radio chatter” with Kafka (esp with “who else is around”
context clues)

32. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

33. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

34. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

35. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

36. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame

37. Twitter: @rawkintrevo
OpenCV/ Haar Cascade Filter inconsistent

38. Twitter: @rawkintrevo
Canopy Clustering
• Create N Second Window
• Cluster Faces in Window
• Quick dirty clustering- but effective.
• First point is “center”
• All points within distance t2 are “in that cluster.
• If a point is not within t2 of any cluster- it becomes a new cluster center.

39. Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
t2= max square width

40. Twitter: @rawkintrevo
Canopy Clustering To Remove “Ghost” Faces
t2= max square widthFirst rect – new cluster
Second Rect- within one width of first rect (same cluster)
Third Rect- within one width of first rect (same cluster)
Forth Rect- NOT within one width of first rect (new cluster)
Fifth Rect- within one width of first rect (same cluster)
Finally- any cluster with less than two entities in windows gets filtered out.

41. Twitter: @rawkintrevo
Canopy Clustering To Remove “Ghost” Faces

42. Twitter: @rawkintrevo
Remaining Faces (Stupid Way)
HUMAN-0001
HUMAN-0002
HUMAN-0003
HUMAN-0004

43. Twitter: @rawkintrevo
Remaining Faces (Smarter Way)
HUMAN-0001
HUMAN-0001
HUMAN-0001
HUMAN-0001
All faces from a given cluster are likely from the same entity

44. Twitter: @rawkintrevo
Flink Enabled
• Windowing lets us use “Intelligence”
• Removing Ghost Faces
• Using Context Clues (Person in one part of frame doesn’t magically become
someone else)

45. Twitter: @rawkintrevo
Flink Plan

46. Twitter: @rawkintrevo
Definition of Intelligence
• Merriam Webster:
• a (1) : the ability to learn or understand or to deal with new or trying
situations : reason; also : the skilled use of reason
• (2) : the ability to apply knowledge to manipulate one's environment or to
think abstractly as measured by objective criteria (such as tests)
• Oxford
• The ability to acquire and apply knowledge and skills.
• Contested subject among researches…

47. Twitter: @rawkintrevo
Artificial Intelligence
• Collect information from environment
• Adapt to changing environment
• Learn and understand / gain knowledge / improve skills
• Apply knowledge

48. Twitter: @rawkintrevo
IoT AI - A Diagram
Edge
Device
Edge
Device
Edge
Device
…
Edge devices produce data (sensors, drones, cars, etc.)

49. Twitter: @rawkintrevo
A Diagram: Collecting from environment
Edge
Device
Edge
Device
Edge
Device
…
“Something” needs to collect and process the data

50. Twitter: @rawkintrevo
A Diagram: Learn, understand, adapt, affect
Edge
Device
Edge
Device
Edge
Device
…
That “Something” needs to support learning and decision frame works
Thoughts
Actions

51. Twitter: @rawkintrevo
A Diagram: Bonuses
Edge
Device
Edge
Device
Edge
Device
…
Perhaps it can learns offline and remember as well.
Thoughts
Actions
Memories
learning

52. Twitter: @rawkintrevo
A Diagram: Apache AF
Edge
Device
Edge
Device
Edge
Device
…
Perhaps it can learns offline and remember as well.
Thoughts
Actions
Memories
learning

53. Twitter: @rawkintrevo
A Diagram: Apache AF (Lambda)
Thoughts Memories
learning
”Stream of Conscious”
”Short Term Memory”
* Windowing and CEP *

54. Twitter: @rawkintrevo
A Diagram: Apache AF (Kappa)
Thoughts
Memories
”Stream of Conscious”
”Short Term Memory”
* Windowing and CEP *
learning

55. Twitter: @rawkintrevo
Technically “Borg-style” AI, not Cylons
• A finer technical point for those familiar with the Cylons and the Borg
• “Hive Mind” Architecture

56. Twitter: @rawkintrevo
NEW HUMAN-0001
Solr Returns Closest Match
Drone says “hi” or shoots to kill
OH, hai HUMAN-0001

57. Twitter: @rawkintrevo
Shape of things to come.
”Science Fiction” of 10 years ago, today is domain of
hobbyists
Demo presented here is “Science Fair” grade AI.
Vlad Putin’s recently talking about “it is undesirable for
anyone to monopolize AI”. (Yay Apache!)