1. Autonomous Exploration for
Gathering Increased Science
AEGIS
Tara Estlin
Benjamin Bornstein, Daniel Gaines, David R. Thompson, Rebecca Castano,
Robert C. Anderson, Michael Burl, Charles de Granville and Michele Judd
2011 NASA Software of the Year
2. Consider the following problem…
• You are a robotic explorer
on another planet
• You only talk to Earth
once a day
• You are in a hostile
environment
• You have limited power and computing abilities
• You are constantly on the move exploring different
terrains
• As you move, you need to quickly determine if you
see objects that are interesting to scientists
• If you do, you want to acquire data on these
objects, before the rover moves past 2
3. This is why we developed AEGIS
AEGIS:
• Is a new paradigm for in‐situ
science using onboard autonomy
• Provides intelligent targeting and
data acquisition by
– analyzing images of the rover scene
– identifying high‐priority science targets
(e.g., rocks), and
– taking high quality data of these targets
completely autonomously with no
ground interaction required
3
5. How is AEGIS being used?
AEGIS:
• Is in regular operational use onboard
the Mars Exploration Rover (MER) Mission
Opportunity rover for the past two years
• Excels in automated targeting with narrow field‐of‐
view (FOV) remote sensing instruments, such as:
– MER Panoramic Cameras (in current use)
– MER Mini‐Thermal Emission Spectrometer
– Mars Science Laboratory (MSL) Rover
ChemCam Spectrometer
Mini-TES Mosaic
– Before AEGIS, had to manually select
targets, based on ground analysis
5
6. AEGIS Process for MER
Process fully automated!
Advanced image
Navcam acquisition processing technique
enables reliable, rapid
identification of candidate
targets.
Target detection
Algorithms quantify key Target feature Scientists can
intuitive target extraction prioritize
properties such as important
brightness, size, and properties
shape. for each run
Target prioritization
Target pointing
determination
Robust approach to Top score
for large size
pointing selection
maximizes data of target.
Pancam pointing
High-quality, 13 color
filter, quarter-frame
Pancam acquisition
Panoramic camera
image
7. Benefit of AEGIS for Rover Drive
Autonomously‐targeted remote sensing
taken end of drive by AEGIS
X
X
Autonomously‐targeted remote
sensing taken mid‐drive by AEGIS
Manually‐targeted remote sensing
as specified by science team
(taken before drive)
Targeted data Targeted data
selected manually with AEGIS
7
12. MER before AEGIS
Sol 1 Sol 2 Sol 3
Perform
MANUALLY
TARGETED remote Pancam Sol 2
sensing of current Perform Perform
rover area untargeted untargeted
Drive rover 100m remote remote
to new stopping sensing of sensing of
point local area local area
Acquire wide‐
angle images of
new terrain area
Pancam Sol 3
Multi-Sol Plan 12
13. MER after AEGIS
Sol 1 Sol 2 Sol 3
Perform
MANUALLY
TARGETED remote AEGIS Pancam 1
sensing of current Perform Perform
rover area autonomously autonomously
targeted targeted
Drive rover 100m remote remote
to new stopping sensing of sensing of
point local area local area
Acquire wide‐
angle images of
new terrain area
AEGIS Pancam 2
Multi-Sol Plan 13
15. Significance: Aerospace
• Fully operational and used Opportunity Today
regularly on MER mission
– Saves valuable time every targeted
data collection
– Has been used more than any other
new technology on the Opportunity rover
• Currently infusing into Mars Science
Laboratory (MSL) Mission
• Attracting strong interest for
– 2018 Mars rover and other future in‐situ
missions (e.g., Titan, Venus, Europa)
– Military applications (e.g., UAVs)
15
16. Significance: Science/Technology
• Enables the collection of science data that would
otherwise not be possible
– During or right after drives
– Different times of day and temps
• Saves significant time and cost for
targeted data collection
– Gets data into the hands of scientists twice as fast
(or more) than standard operations
• Enables scientists to easily use and
interact with autonomy software
– Parameters chosen after significant
consultation with scientists
• 25+ science and technology publications
• Application to large number of problems in
industry and academia (e.g., underwater robotics) 16
17. Significance: Humanitarian
• Directly contributing to humanity goal of
finding life on other planets
– Mars program theme of “Follow the water…”
• Significant outreach vehicle; over 35 media
articles since release in 2010
– “Mars Rover Getting Smarter As It Gets Older”
– “NASA upgrades Mars rover brain”
17
18. Significance: Humanitarian
Inspiring the next generation in STEM:
Science, Technology, Engineering and Mathematics
Over 46 Amateur Astronomy Clubs,
Australia Schools and Teacher Organizations around
United Kingdom the world featured our “rock hound”
software in their newsletters
18
20. Development Status
• TRL Level 9: Flight Proven
– Software fully operational
– In regular use on MER Opportunity rover
• MER and MSL Scientists have already asked for
extensions, which are in progress:
– Enabling multiple targeted observations
– Triggering on single filter color images
– Identifying novel targets
– Identifying representative targets
– “Soil only” detector
“AEGIS is a true success story for the Mars Technology Program”
Dr. Samad Hayati
Manager of Mars Technology Program
20
21. ASSESSMENT
of USE
AEGIS technology is being applied in a wide range
of applications.
22. NASA Use – MER
AEGIS is considered every multi‐sol plan.
Sol 2138 Sol 2172 Sol 2313 Sol 2221 Sol 2247
Sol 2278 Sol 2290
Sol 2304 Sol 2312
“AEGIS is a significant enhancement for the mission and the scientific
community. MER is the first mission to implement the capability that AEGIS
provides – and it has really paid off.”
Dr. John Callas
22
Mars Exploration Rover Mission Project Manager
23. NASA Use – MSL Rover
• The MSL Rover ChemCam Team has requested
AEGIS (PI: Roger Wiens)
• AEGIS is ideal for ChemCam’s narrow field‐of‐view
Laser‐Induced Breakdown Spectrometer (LIBS)
– Samples rocks from a distance of 1 to 7 meters
– Able to rapidly identify rock elemental composition
• AEGIS enables multiple autonomously targeted
ChemCam measurements throughout the day
• MSL flight software integration in progress
23
26. NASA Use – Mars 2018 Rover
• 2018 Rover Mission will have
limited time to core and store
up to 30 rock samples
– Will need to drive up to
20 kilometers
– Will need to consider targets
from distinct areas
• Strong interest from Mars 2018 Mission Program
Office (Charles Whetsel, Chris Salvo) in using
AEGIS to collect data on potential targets
– Get data to science team faster
– More targets could be considered
26
27. Future Use – Other Missions
• AEGIS system can enable a wide spectrum of
missions:
– Collect valuable science more often
– Enhance onboard autonomy capabilities
• Strong application to in‐situ missions to Titan,
Europa, Venus, Mars, the Moon, and small
bodies
• Science autonomy listed as critical capability
in Titan Prebiotic Explorer Mission Study
– Helps address challenges such as extremely limited
communication, high platform mobility, etc.
“Onboard science algorithms will analyze the image
data to detect trigger conditions such as science
events, interesting features, changes relative to
previous observations, …”
TiPEx mission study team 27
28. Industry, Government, Research Use
Autonomous
Underwater
Vehicles (AUVs)
Unmanned
Lunar
Aerial
Exploration
Vehicles
Unmanned
Sea Surface
AEGIS
AEGIS is transferable to
Multi‐core
Processor
Vehicles a wide range of Benchmarking
(USSVs)
application domains
Search and
Commercial
Rescue
Spectroscopy
Robotics
28
29. Industry, Government, Research Use
Autonomous
Underwater
Vehicles (AUVs)
Unmanned Aerial Vehicles (UAVs)
Unmanned
• Developing Lunar
automated Aerial
Exploration
cueing capability for UAV Vehicles
surveillance platforms.
• Lower-resolution wide area
imagery used to trigger
on selected areas.
Sea Surface
AEGIS
higher-resolution follow-up
Unmanned
Multi‐core
Processor
• Proof-of-concept completed for ships using satellite imagery
Vehicles
Benchmarking
(USSVs)
• Evaluating for use identifying ground vehicles on imagery from
AFRL "Angel Fire" aerial asset
Search and
Commercial
Rescue
Spectroscopy
Robotics
29
30. Industry, Government, Research Use
Autonomous
Underwater
Vehicles (AUVs)
Unmanned
Lunar
Aerial
Exploration
“Moon Express is developing a lunar lander and
Vehicles
mobility system for exploration of platinum
group metals on the surface of the moon as
well as compete for the Google Lunar X-Prize.
Unmanned
AEGIS
AEGIS could be a great asset to
this quest by autonomously
recognizing rocks from iron-rich
Sea Surface
Multi‐core
Processor
Vehicles
asteroids that might contain Benchmarking
(USSVs)
platinum.”
-- Moon Express, Inc.
Search and
Commercial
Rescue
Spectroscopy
Robotics
30
31. Industry, Government, Research Use
Autonomous
Underwater
Robotic Underwater Vehicles
Vehicles (AUVs)
• WHOI Nereus vehicle
Lunar
Unmanned
Aerial
Exploration
– Performs deep ocean scientific
Vehicles
survey and sampling
– Used to locate hydrothermal
systems, volcanic processes, etc.
Unmanned
AEGIS
• CMU/Pittsburgh Aquarium Reefbot
– Automatically detect,
classify, and count fish
Sea Surface
Multi‐core
Processor
Vehicles in their natural habitat
Benchmarking
(USSVs)
• AEGIS could save days/weeks
of exploration time through
autonomous data collection
Search and
Commercial
Rescue
Spectroscopy
Robotics
31
32. Industry, Government, Research Use
Autonomous
Underwater
Robotic Underwater Vehicles
Vehicles (AUVs)
• WHOI Nereus vehicle important advances in automatic
Lunar
“AEGIS makes Unmanned
Aerial
– Performs deep ocean.. It has direct relevance to work
Explorationdata analysis. scientific
Vehicles
at CMU in underwater vehicles for detecting
survey and sampling
– Usedand cataloging fish in deep water reefs.”
to locate hydrothermal
systems, volcanic processes, etc.
D. Wettergreen
Unmanned
AEGIS
• CMU/Pittsburgh Aquarium Reefbot
– Automatically detect,
CMU Robotics Institute
classify, and count fish
Sea Surface
Multi‐core
Processor
Vehicles in their natural habitat
Benchmarking
(USSVs)
• AEGIS could save days/weeks
of exploration time through
autonomous data collection
Search and
Commercial
Rescue
Spectroscopy
Robotics
32
34. MER Impact: Increased Science
• Before AEGIS, all targeted data required:
– Manual evaluation of images
– One to several communication cycles
– The rover to remain stationary and sometimes backtrack
By the time the “Block Island”
meteorite was noticed in an image,
the Opportunity rover was already
200 meters past. The rover had to
turn around and backtrack (costing
25 additional sols).
• After AEGIS, targeted data can be collected:
– Without ground analysis of context images
– Without communication cycles
– Any time during a rover drive
– Any time of day 34
40. Creativity: Innovation
• AEGIS provides new paradigm for
Original Optimized
surface data acquisition 62
Memory Usage (megabytes)
– Scientist provides description of target
– System can collect data whenever target
detected
• Flight challenges 3.75
– Image processing performed on RAD6000
(orders of magnitude slower than standard Memory reduced 16x
desktop machine) 62 MB to 3.75 MB
– AEGIS limited to < 4 MB of memory
– Large performance optimizations made!
Runtime (seconds)
• Inventive approach to flight
software change
– Full flight software upload not possible
Benchmark Images
– AEGIS uploaded as standalone module
– Loaded into memory whenever want to use Performance improved 7X
(< 30 secs)
40
41. Creativity: Usability
• Parameters defined through collaboration with scientists
– Describe attributes of candidate targets
– Express diverse and evolving science goals
• System did not require extensions to
Albedo, shape, and size
MER command dictionary or telemetry
• Training materials
– Web interface for creating commands
– User’s guide for software usage and
sequencing
– Standard terrain profiles available
– Result message (EVR) interpreter
“One of the key aspects that has made the AEGIS team successful is their long
track record of working with the scientists.”
Dr. Jack Stocky
New Millennium Program Manager 41
42. Creativity: Quality Factors
• Reliable target detection
– Find rock targets in diverse terrain
– Is resilient to dust‐covered or shadowed rocks
– Works under strict computation constraints
• Risk control through resource limits and time
deadlines
• Validation and Verification
– Extensive MER testing procedure and code reviews
– Nightly build, static analysis, unit and regression tests
42
44. Summary
• Significance: Far Reaching
– Aerospace: Routinely used on MER
– Science/Technology:
• Enables science that could not be previously collected
• Applications in military, commercial and research fields
– Humanitarian:
• Contributing to goal of finding life on other planets
• Inspiring next generation in STEM areas
• Development: Flight Proven
• Assessment of use:
– Planned for Infusion into New Missions and Applications
• Creativity: Pioneering/Deeply Innovative
– Innovation: New paradigm for in‐situ data acquisition
– Usability: System parameters designed through direct
collaboration with scientists
– Quality: Reliable target detection under strict computation
44
constraints
46. For more information…
Questions?
Tara.Estlin@jpl.nasa.gov
We’d like to acknowledge our sponsors:
New Millennium Program, Mars Technology Program,
JPL Research and Technology Development Program,
and the IND Technology Program
and thank you to the:
The Mars Exploration Rover Mission
50. Target Feature Extraction
Reflectance
– Mean
– Variance
– Skew
– Kurtosis
Size Light Dark
– Inscribed circle
– Pixel area
Shape
– Eccentricity
– Ellipse fit error
– Roundness
– Ruggedness
– Angularity
Rounded Angular
51. Target Prioritization / Top Target Selection
Images from MER field trial
• Scientists can
Near the
prioritize different top of the
feature values and list of
combinations of two “round”
– e.g., prefer large, high rocks
albedo rocks
– Can also support MER
cobble campaign, outcrop
finder, soil finder, etc.
• Priority specification is Near the
part of command bottom of
the list of
sequencing “round”
rocks
• Can be easily changed
as rover enters
different terrain areas
52. AEGIS Code Details
• AEGIS is 7968 SLOC (C)
• Limited to less than 4 MB of memory
• Requires only 232 KB of disk space
• Regular static analysis using Coverity
PreventTM
• Formal code reviews
– Internal AEGIS Team
– Other JPL AI/machine-learning developers not
members of AEGIS Team
– MER Team
52
55. AEGIS Target Detections
• Target detections are consistent with AEGIS selection
profiles
– 90% of top targets meet the selection profile
– Confirmed by evaluation of context Navcam imagery
– All results reviewed with MER Science Team
• The MER Science Team is very happy with AEGIS and
continues to request it regularly
55
56. OASIS Framework
• OASIS: Onboard Autonomous Science
Investigation System
• Objective: Maximize science returned on surface mission
– Identify and respond to science opportunities
– Data prioritization for downlink
– Maximize utilization of onboard resources
• Approach
– Data segmentation and feature
extraction for multiple instruments
– Science Data Analysis
• Prioritize targets and/or data
• Summarize data
– Automated Planning and Scheduling
• Adjust rover activities to collect new
data
• Ensure operation within rover
resource and operation constraints
57. AEGIS in the OASIS Framework
• AEGIS is a flight software system derived from the
larger OASIS framework
• Developed by same team of people
• AEGIS includes a subset of OASIS capabilities
selected for MER
Relevant - Instruments available on MER
Desired - Requested by scientists
Feasible - Fit within memory and time limits
