The document describes an autonomous coffee run performed by a mobile robot. It summarizes the key steps taken which included navigating doors and elevators, passing objects to humans, and ordering and obtaining coffee. Navigation used laser scanning and SLAM mapping to build maps and avoid obstacles. Door pushing and pulling used mechanical alignment. Elevator operation involved finding buttons and exiting on the correct floor. Object passing was done by holding out hands rather than finding objects or hands. Lessons learned were that simple heuristics often outperform complex algorithms, failures should be detected and retried, and software reuse is important for personal robotics.

1. Replacing the Office Intern:
An Autonomous Coffee Run with a Mobile Robot
Tony Pratkanis

2. Outline of the Talk
● General Background
● Coffee Grasping Steps
– Navigation
– Doors
– Elevators
– Object Passing
● Lessons Learned

3. About the Salisbury Robotics Lab
● sr.stanford.edu

4. Personal Robotics: The PR2
● Based on the PR1 at Salisbury Lab
● Spun out to Willow Garage to become PR2

5. Personal Robotics: The PR2
● Costs $400,000+, weighs 400 pounds
– More battery capacity than a Prius
– Two laser scanners, many color cameras, Kinect-like depth cameras, two
arms, etc.
– Despite this, it is still human-safe
● The PR2 is a “Kitchen Sink” robot
– Designed exclusively for research purposes
– It has a vast (likely excessive) number of sensors and features
● Ships integrated with ROS
– An open-source robotics middleware developed by Willow Garage
– Vast amounts of useful software including motion planning, navigation,
SLAM, computer vision, 3D object recognition, linear algebra, etc.

6. The Task

7. Personal Robotics
● Personal robotics is the creation of robots that live and play safely
and effectively in human-centric environments
– The ideal is “Rosie” from The Jetsons
● Faces many challenges not present in other forms of robotics
– Extremely diverse obstacles and objectives
– Highly unpredictable and unstructured environment
– Safety issues
● My solution to these challenges:
– Analyze the nature of the task and the available information
– Develop simple procedures that exploit that information
● The “coffee bot” allows us to demonstrate this approach

8. Important Qualifications
● The robot must be fully autonomous
– No human intervention except for interaction with
coffee shop employees
● The environment must be unmodified
– Modification of human environments is often
socially and politically intractable
– Defeats the purpose of building such a robot

9. Navigation Video
● Navigation basic demo
● Note how the robot intelligently avoids both
static obstacles and people

10. How Mapping and Navigation Work
●
Two sensors are used
– Wheel odometry
● Very accurate over short distances
● Error builds up
– Laser scanners
● Accurately (to approximately 1cm)
measure distance to objects
●
Integrated by software to create a
detailed map of the environment
– “SLAM” - Simultaneous localization
and mapping
●
Then the map is used for navigation

11. Navigation and Obstacle Avoidance
●
Laser data, the map, and odometry are
fused for localization
– Particle-filter based approach
– Obtains the position of the robot
● A cost-map grid is built of all obstacles
– Real-time updates of the obstacle grid
– Fed to path-finding algorithms
●
The navigation software was
modified to handle multiple floors
– Leading to “multi_map_navigation”

12. Door Pushing Video
● Open the door by pushing
● Note how the robot lines up in all three axes
then rotates to open the door

13. Pushing Doors
● These specific doors are challenging from
two perspectives
– Transparent and thus hard to detect
– Heavy and thus physically hard to open
● The PR2 uses mechanical approaches to
detection instead of vision
● Uses the entire body and strength of the
robot to overcome the doors

14. ●
The robot uses the mapping and navigation software to
approximately (<30 cm) locate the door
● Next, it uses the tilting laser scanner to line up with the door:
– Travels to the correct distance from the door
– Aims laser at the base of the door
● Lines up rotationally using the base of the door
– Aims laser at middle of the door
● The central window of the door leaves a gap in the laser data compared to the
metal sides – the robot centers this gap to align horizontally with the door
●
Then, it spins around and backs through the door
– Backing through the door is important to hit the door metal bar
Pushing Doors

15. Driving and Door Pulling Video
● Drive to the next door
● The lasers were negatively impacted by the
sun, requiring adjustments to software filters
● Note how the door is pulled open by the robot

16. Pulling Doors
● Detecting the handles of transparent doors is
difficult
– The background is unpredictable because of the
window
– The window reflects the handle, leading to multiple
images
– The handle itself is shiny, leading to unpredictable
coloration and edge structure

17. Pulling Doors
● Solution: Purely mechanical approach to handle
detection and door opening
– Once again, the robot uses the map to know the approximate
location of the door
– Drives up to the door and does a “waggle dance” to align with
it mechanically
– Backs up and slides the hand across the door to find the
handle
– Grabs the handle and moves so that the handle is at a fixed
position with respect to the robot
– Dances the door opening dance to open the door

18. Elevator Video
● Note:
– Avoidance of people on the way to and inside the
elevator
– Elevator operation by finding call and control panel
buttons
– Exiting the elevator on the correct floor

19. Elevator Overview
● Call the elevator by pushing the call button
– Find the call button and press it
● Wait for the elevator to arrive
– Identify the correct elevator (up or down)
● Enter the elevator
– Avoid humans and obstacles in the elevator
● Push the buttons in the elevator control panel
– Elevator interior poses challenging computer vision problem
● Exit the elevator
– Check to ensure the correct floor

20. Pushing the Elevator Call Buttons
● PR2 knows the approximate
location of elevator buttons due
to navigation and map
● Lines up with the wall using the
laser scanner
● Finds the button using vision
● Repeats this process if the
elevator does not arrive

21. Waiting for the Elevator to Arrive
● Scans the indicator lights for
elevator arrival
– Identifies correct direction (up or down)
and elevator (left or right)
● Moves quickly to the elevator before
door closes
– Avoids humans using the laser
– Rule: Only rides in empty elevators

22. Pushing the Elevator Control
Buttons
● Uses mechanical procedure to
align with buttons
– Similar to door pulling
● Once alignment is achieved, the
buttons are at a fixed position
with respect to the robot
– With the known position and
height, it is easy to press the
correct floor
● If the elevator door fails to open,
then the robot will repeat this
process

23. Exiting the Elevator
● Waits and uses laser
to detect when the
door opens
● Checks for correct
floor at exit
– Important because
humans may have
ordered the elevator to
stop at additional floors

24. Exiting the Elevator
● Determining the correct floor
was very challenging
– First approach using vision to
see the floor number sign at
exit too prone to failure
● Not robust to lighting changes
– Second approach using robot's
accelerometer much more
successful
● Detecting time interval between
elevator start and stop accurately
predicts number of floors traveled
● Much more robust

25. Driving to the Coffee Shop
● More door pulling and pushing
● Note the avoidance of the tables and people

26. Waiting in the Coffee Shop Line
● The robot drives along a predefined course
using its map
– If the laser detects a person in line, the robot stops
and only advances as the person moves forward
● While this approach works well for many
stores, it would not work in larger stores that
have multiple cashiers

27. Waiting in Line Video
● Note how the robot drives this course

28. Ordering and Obtaining the Coffee
● First, give the barista the written coffee order
and cash payment
● Second, take the coffee and place it in the cup
holder
● This requires an intuitive approach to enabling
humans and robots to pass objects to each
other

29. How Do Humans Pass Objects?
● Humans pass objects by two main approaches:
1. Receiver holds out his/her hand, giver places object into hand
2. Giver holds out object, receiver grabs object
● Thus, by using Case 1 to receive objects and Case 2 to give objects,
the robot never needs to find the object or the human's hand: it can
just hold out its hand or the object
● Humans are also very good at knowing when to let go of objects
– Humans hold onto an object until they feel the other person pull the object
back from the hand, ensuring a good grip

30. Giving an Object to a Human
● Object giving sequence used by PR2:
1. Holds out the robot's hand with the object
2. Uses text-to-speech to tell the person to take
the object
3. Releases the object when either of two
conditions is met:
● There is significant hand acceleration
● The human has forced the robot to move its hand
4. Folds back the arm

31. Receiving an Object from a Human
● Initial simple process is similar to giving:
1. Holds out the robot's empty hand with an open
gripper
2. Uses text-to-speech to ask for the object
3. Grasps the object when the previous conditions
are meet (accelerometer or hand being forced
back)
● This worked relatively well, but failed in some
common cases

32. Receiving an Object from a Human
● Most common failure: people did not actually push the object
into the hand, just into the gripper
– The solution was to use the forearm camera to detect when
significant motion occurred in front of the hand
● Despite this, sometimes the gripper closes when the object
is not present or the gripper slips off the object
– The solution to this problem was to ensure that the gripper did not
fully close (indicating no object), and if it did, try the whole process
again

33. Coffee Grasping and Return Video
● Observe money passing and coffee grasping
process
● Returning to our lab with the coffee by doing
all the previous navigation steps backwards
● Giving the coffee to the faculty advisor

34. Lessons Learned
●
It is extremely fun and exciting to do this sort of work
●
It is impractical in the current state – the robot is too slow, requires too
much attention and is too expensive
– However, it could still be helpful for disabled people or for artistic purposes
● Computer vision can be useful but is often unreliable
● Simple heuristic approaches often exceed the performance and especially
the reliability of complex math-heavy algorithms
– In addition, such approaches are more predictable and easier to understand and
thus far easier to maintain
● Don't be afraid to fail, don't be afraid to retry
– Your robot is not going to work every time – it gets exponentially harder to increase
reliability as you go up from 90% to 99%, 99.9% etc.
●
A better approach is to detect when the robot fails and simply try again

35. Future Work
● Robotics software keeps being reinvented to perform the
same tasks with different robots
– Need robust development model that allows sharing of code
and building off each coder's work
– Critical because of the large amount of software needed for
personal robotics
● I want to develop a next-generation approach to handling
this problem
– Develop an easy-to-use framework for specifying what each
robotic software application does so multiple applications can
be automatically integrated

36. Questions
Tony Pratkanis
sr.stanford.edu