business model, customer development, hacking for defense, H4D, lean launchpad, lean startup, stanford, startup, steve blank, pete newell, bmnt, video

1. Identifying objects in U2 high-resolution
imagery
Team Theia
Minjia Zhong
MCS / CISAC
Christopher Yeh
M.S. CS (AI track)
David Kingbo
MSx 18 (Consulting)
Joseph Lee
M.S. MS&E / AI
Original Problem
Statement
Develop the capability for analysts to
automate object identification in
order to focus on the highest priority
geographic areas most relevant to
their task at hand.
Final Problem
Statement
Develop machine learning software for
imagery analysts to detect anomalous
activity in aerial and satellite images in
order to prioritize relevant geographic
areas for further analysis.
100 Interviews
US Air Force,
9th Intelligence
Squadron
Sponsor

2. Know Your Customer
Problem Finding
Pivot 1
Week 1 Week 3 Week 5 Week 7 Week 9
Pivot 2
The Evolution of Our Hypotheses/MVPs
Go Deep
Product Development
Week 2 Week 4 Week 6 Week 8 Week 10
Get Early Wins
Deployment

3. Know Your Customer
Problem Finding
Week 1 Week 3 Week 5 Week 7 Week 9
Week 2 Week 4 Week 6 Week 8 Week 10
“Manual labor is the biggest
problem. … Sometimes we
have to turn down a
request, but then [our
clients] won’t come back.”
Imagery Analyst at the 9th
Intelligence Squadron
The Evolution of Our Hypotheses/MVPs
Go Deep
Product Development
Get Early Wins
Deployment
Pivot 1 Pivot 2

4. Why is automation relevant to the
film camera? Why does it matter
to improve film camera
processes?
Is the film camera even
necessary? Is there demand
for their products?
What is the mission of the film
camera and how do they serve
customers?
Know Your Customer (Week 1-2)

5. Initial Mission Model Canvas (Week 1)
Value Proposition
Increase detection
efficiency
Reduce operational
overhead
Beneficiaries
9th IS
DoD/NGA
Satellite companies

6. Getting Out of the Building (Week 3)

7. Mapping Out the 9th IS Imagery Analysis
Workflow (Week 1-3)
Image Collection (9th IS)
❏ How does the 9th IS
work?
❏ How does it serve its
customers?
Image Analysis (IC) Dissemination
Use

8. Identifying Bottlenecks in 9th IS Workflow (Week
1-3)
Image Collection (9th IS)
❏ How does the 9th IS
work?
❏ How does it serve its
customers?
Image Analysis (IC) Dissemination
Use

9. Identifying Bottlenecks in 9th IS Workflow (Week
1-3)
● Inefficient manual film processing
● Time consuming manual imagery analysis
● Lack of awareness of film camera
capabilities

10. Know Your Customer
Problem Finding
“I don’t care how many
cars or trucks. I care
about what sorts of
activities are going on.”
NGA Manager
Week 1 Week 3 Week 5 Week 7 Week 9
Week 2 Week 4 Week 6 Week 8 Week 10
The Evolution of Our Hypotheses/MVPs
Go Deep
Product Development
Get Early Wins
Deployment
“Manual labor is the biggest
problem. … Sometimes we
have to turn down a
request, but then [our
clients] won’t come back.”
Imagery Analyst at the 9th
Intelligence Squadron
Pivot 1 Pivot 2

11. Hypothesis
Speeding up the film image developing
and scanning process would enable
the 9th IS produce more relevant
intelligence.
Feedback
“For me it doesn’t matter if [the film
scanning and digitization process]
takes 30 days or 10 days”
- Intelligence officer
Way Too Many MVPs… (Week 4-5)
Film developer with built-in digital scanner and object identification

12. Way Too Many MVPs… (Week 4-5)
Hypothesis
An automated object
identification software would
mitigate imagery
underutilization.
Feedback
“DigitalGlobe works pretty well,
but as computer vision gets
better, it needs to marry with
how an analyst does its job.”
- NGA manager
Automated Object Identification Software

13. Way Too Many MVPs… (Week 4-5)
Hypothesis
Improving imagery search
at the NGA would help
analysts find relevant
images faster, including
images from the 9th IS.
Feedback
“I don’t use the NGA
search engine. Talking to
people is the best way to
find intel.”
- Intel Analyst
NGA Google Search Engine

14. Three Layers of Confusion??? (Week 6)
National Geospatial
Intelligence Agency (NGA)
vs
9th IS Beale Air Force Base
BENEFICIARY?
vs
Automated Object
Identification
NGA Search Engine
PROBLEM?
Off the Shelf Solution
vs
New Product
PRODUCT?

15. Who’s Your Beneficiary? (Week 6)
vs
9th IS Beale Air Force Base National Geospatial
Intelligence Agency (NGA)

16. Pivot 1: Broadening Our Scope (Week 6)
Sponsor
9th IS w/ film
images taken by
the OBC (optical
bar camera)
All beneficiaries
of AI-based tools
● all aerial image providers,
including 9th IS (sponsor)
● military analysts
● intelligence analysts
● mission planners
We believe we can have broader impact on
aerial imagery intelligence.

17. A Common Thread (Week 7)
A Recurring Need for Automated Image Analysis
Potential
Beneficiaries
Validating Feedback
USAF Intel Analyst “Change detection is a big deal for analysts”
Google Imagery
Analyst
“I think you all have an interesting idea and even with competition there is
really a lot of room in this nascent industry.”
DIUx Project Manager “There is a huge need for innovation in this space, only 2 key players right
now - Orbital Insights + Progeny”
NGA Manager “It’s really about object identification and being able to search through that”
USAF Lt. Col. “The issue is that this is hoovering up a lot of data and the analysts cannot
process all of it… This is why I need AI.”

18. Getting Validation (Week 8)
A Recurring Need for Automated Image Analysis
Continuous Checking:
“One of the top tech needs for NGA is
Change Detection Alert Service,
specifically from multiple sources.
Being able to layer different data
together, and the AI would say’“Hey,
something’s changed here!’”
- Senior Officer at NGA
Image adopted from a screenshot taken from Spaceknow (https://www.spaceknow.com/)

19. Know Your Customer
Problem Finding
“I can dedicate 15 analysts
for 2 to 3 hours a week [to
label and curate an imagery
dataset]”
Officer at 9th Intelligence
Squadron
Week 1 Week 3 Week 5 Week 7 Week 9
Week 2 Week 4 Week 6 Week 8 Week 10
The Evolution of Our Hypotheses/MVPs
Go Deep
Product Development
Get Early Wins
Deployment
“Manual labor is the biggest
problem. … Sometimes we
have to turn down a
request, but then [our
clients] won’t come back.”
Imagery Analyst at the 9th
Intelligence Squadron
“I don’t care how many
cars or trucks. I care
about what sorts of
activities are going on.”
NGA Manager
Pivot 1 Pivot 2

20. Mission Model Canvas (Week 9)
Key Partners
9th Intelligence
Squadron
Key Activities
Training a Machine Learning
Model to detect meaningful
activity
Key Resources
Labels of meaningful activity in
images relevant to the Intelligence
Community

21. Pivot 2: Leveraging Our Partnership with the 9th
Intelligence Squadron (Week 9)
“What I can possibly dedicate is 15
analysts [labelling and curating
images] for 2 to 3 hours a week... If
this is something I can get approved
from my squadron leader, we can get
150 analysts.”
- Officer at 9th Intelligence Squadron9th IS Beale Air Force
Base

22. Iterating on our MVP (Week 9)
Allowing for new datasets:
“There are likely to be some
future use cases that we
might not even know now.”
- Officer at the 9th
Intelligence Squadron

23. Iterating on our MVP (Week 9)
Allowing for new datasets:
Analysts can then manually
label the images that
correspond to the type of
activity that they were
interested in searching for.

24. Where We Are Now

25. Our Current MVP
Change Detection
“Definitely something like this is
something I hope we have going
forward in the future.”
- Image Analyst at 9th IS
Image adopted from a screenshot taken from Spaceknow (https://www.spaceknow.com/)

26. Our Current MVP
Integrating Insights / Other Reports
to pull up
“Making sure that we pull from
different intelligent services... If we
can pull everything on one user-
friendly interface.”
- Image Analyst at 9th IS
Image adopted from a screenshot taken from Spaceknow (https://www.spaceknow.com/)

27. Funding Strategy
Multiple funding opportunities
We can look at several funding
opportunities in parallel to build
our product.
SBIR, In-Q-Tel and DIUx are the
most promising funding sources
available for our MVP.

28. What’s Next: Handoff
Theia

29. Acknowledgements
Our work would not have been possible without our sponsors at the US Air Force
9th IS (especially CMSgt Ian Eishen, Tsgt Nathaniel Maidel, SMSgt Lisa Payne,
and Capt. Timothy Wilde), H4D military liaisons, our mentors (Samir Patel and
Michael Chai), countless military and commercial partners, and the teaching staff
and TAs (especially Will Papper).