Marcus gave a talk called "Recognizing Locations on Objects" during the HTM Meetup on 11/03/2017. The brain learns and recognizes objects with independent moving sensors. It’s not obvious how a network of neurons would do this. Numenta has suggested that the brain solves this by computing each sensor’s location relative to the object, and learning the object as a set of features-at-locations. Marcus showed how the brain might determine this “location relative to the object.” He extended the model from Numenta’s recent paper, "A Theory of How Columns in the Neocortex Enable Learning the Structure of the World," so that it computes this location. This extended model takes two inputs: each sensor’s input, and each sensor’s “location relative to the body.” The model connects the columns in such a way that a column can compute its “location relative to the object” from another column’s “location relative to object.” When a column senses a feature, it recalls a union of all locations where it has sensed this feature, then the columns work together to narrow their unions. This extended model essentially takes its sensory input and asks, “Do I know any objects that contain this spatial arrangement of features?”

1. Recognizing locations
on objects
Marcus Lewis — November 2017
mlewis@numenta.com

2. Agenda
• The problem, in context:
Detecting locations relative to objects
• Inspiration:
Principles of grid cells
• A solution:
Recall, Normalize, Vote

3. Part 1
The problem, in context:
Detecting locations relative to objects

4. Consider the inputs to the brain.
Sensors move.
The brain is very good at making
sense of sensorimotor sequences.
Sensors are often independent.
Familiar objects often cause novel
combinations of sensory features.
The body is an array of sensors.
With both vision and touch, the brain
receives an array of sensory features.

5. ?
Sensorimotor
sequences
Model of
objects

6. ?
Features +
Locations relative to object
solvable
Sensorimotor
sequences
Model of
objects

11. These
locations
aren’t relative
to the body.

12. These
locations
aren’t relative
to the body.
They’re relative
to the object

13. They’re relative
to the object
These
locations
aren’t relative
to the body.

14. ?
Features +
Locations relative to object
solvable
Sensorimotor
sequences
Model of
objects

15. solvable?
Features +
Locations relative to body
solvable
This Talk
Sensorimotor
sequences
Model of
objects
Features +
Locations relative to object

16. Given the sensor’s
location relative to
the body…
…calculate its
location relative
to the object.

18. Part 2
Inspiration:
Principles of grid cells

19. A rat
An environment
Electrode
Entorhinal
cortex
Cell

20. Electrode
Entorhinal
cortex
Cell

21. Each grid cell fires at multiple
equally spaced locations.
Electrode
Cell
Entorhinal
cortex
A Room

22. Firing fields of 16 different cells.
Electrode
Cell 1
Cell 2
Cell 3
1 2 3 4
8765
1211109
16151413
1 2 3
8765
1211109
16151413
1 2 3 4
8765
1211109
16151413
1
65
109
151413
1 2 3 4
8765
1 2 3 4
8765
1211109
161514
2 3 4
87
1211
16
1 2 3 4
8765
4 1
5
4
13 Entorhinal
cortex
A Room

23. Room 1
After the rat learns a room,
grid cells anchor to the
room consistently.
Room 2
They anchor to other
rooms differently.
“Purely Translational Realignment in Grid Cell Firing Patterns Following
Nonmetric Context Change”
Marozzi, Ginzberg, Alenda, Jeffery (2015)

24. A cool grid cell trick: Modules
Module 2Module 1
+ +
Module 3
Entorhinal
cortex
=
Big space of
unique locations

25. Principles of grid cells
The cell activity moves between a fixed
set of locations.
These locations are mapped onto the
environment.
=+
Body
Motion2. Updating it from motion cues
The brain determines the location by…
1. Recalling it from sensory cues +
Feature
=

27. Part 3
A solution:
Recall, Normalize, Vote

28. Sensory 1
Object
Input
Object
Input
Location Sensory 2Location
Column 1 Column 2
Layer 2
Layer 4

29. Column 1 Column 2
Object
Input
Feature
relative to
Object
Object
Input
Feature
relative to
Object
Body
relative to
Object
Feature
relative to
Body
Feature
relative to
Body
New:
Recall past
locations of
sensed
features.
New:
Vote on the body’s location
relative to the object.
Layer 2
Layer 4
Layer 6A
Sensory 1 Sensory 2

30. BB
A
B
A
BB
A
B
A A A
B
A
A B
A
A
B
A B
B
A
A
A
Object 1
Object 3
Object 2
Object 4
B
A B
B
A
A
A
Demo:

31. BB
A
B
A
Visualizing the solution
All objects are learned at points
in a large “location” space.

32. A
BB
BB
A
B
A
All objects are learned at points
in a large “location” space.
Visualizing the solution

33. B
A
A
BB
BB
A
B
A
All objects are learned at points
in a large “location” space.
Visualizing the solution

34. B
A
A
BB
All objects are learned at points
in a large “location” space.
B
A B
B
A
A
A
Visualizing the solution

35. B
A B
B
A
A
A
B
A
A
BB
All objects are learned at points
in a large “location” space.
A
B
A
Visualizing the solution

36. B
A B
B
A
A
A
B
A
A
BB
All objects are learned at points
in a large “location” space.
A
A
B
B
A
Visualizing the solution

37. B
A B
B
A
A
A
B
A
A
BB
All objects are learned at points
in a large “location” space.
A
A
B
BB
A
A
Visualizing the solution

38. B
A
A
A
A
B
B
BB
B
A
A
This “location” space now contains
maps of multiple objects.
Visualizing the solution
Map of
Object 1
Map of
Object 2
Distance between
isn’t meaningful

39. B
A
A
A
A
B
B
BB
B
A
A
BB
A
B
A
To recognize a location, each sensor
votes on the body’s location.
Visualizing the solution

40. B
A
A
A
A
B
B
BB
B
A
A
BB
A
B
A
To recognize a location, each sensor
votes on the body’s location.
Visualizing the solution

41. B
A
A
A
A
B
B
BB
B
A
A
BB
A
B
A
To recognize a location, each sensor
votes on the body’s location.
Visualizing the solution

42. B
A
A
A
A
B
B
BB
B
A
A
BB
A
B
A
2
To recognize a location, each sensor
votes on the body’s location.
Visualizing the solution

43. B
A
A
A
A
B
B
BB
B
A
A
BB
A
B
A
3 2
2
2 2
To recognize a location, each sensor
votes on the body’s location.
Visualizing the solution

44. B
A
A
A
A
B
B
BB
B
A
A
BB
A
B
A
3
To recognize a location, each sensor
votes on the body’s location.
Visualizing the solution

45. solvable?
Features +
Locations relative to body
solvable
This Talk
Sensorimotor
sequences
Model of
objects
Features +
Locations relative to object

46. “Do I remember any
objects that contain this
arrangement of features?”
The holistic system does this:

47. Summary
Location is probably one of the brain’s main primitives.
Grid cells suggest an approach to location:
• Cortex has a “space” of locations that it can process.
• Cortex maps objects into this space.
Cortex could recognize these locations in 3 steps:
Recall, Normalize, Vote
B
A
A
BB

48. Thank you.