Wharton Aerospace Conference Innovation @ 50x 042116
•Télécharger en tant que PPTX, PDF•
4 j'aime•12,107 vues
,wharton ,steve blank ,lean ,agile ,dod ,hacking for defense
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Wharton Aerospace Conference Innovation @ 50x 042116
Similaire à Wharton Aerospace Conference Innovation @ 50x 042116 (20)
Plus de Stanford University
Plus de Stanford University (20)
Dernier
Dernier (20)
Wharton Aerospace Conference Innovation @ 50x 042116
- 1. Innovation @ 50x Steve Blank @sgblank www.steveblank.com
- 2. Disruption In the Corporate America
- 3. 20th Century Corporate Lifecycle
- 4. 21st Century Corporate Lifecycle
- 5. Disruption In the Aerospace Business
- 9. Disruption In the Space Launch Business
- 12. 20th Century DOD/IC Dominance • We owned & deployed the most advanced technology • R&D & contractors had the smartest domain experts • We could design and manufacture the best systems
- 13. Decades-long acquisition process Off-the-shelf Tech Nation-state asymmetric disruption Non-state asymmetric disruption
- 14. Organizational Capabilities Dept of Defense / Intelligence Community
- 16. Technical Capabilities Dept of Defense / Intelligence Community
- 18. The Offset Strategy • 1950’s Eisenhower – mass, mechanized warfare to nuclear deterrence • 1970’s Perry: stealth, precision strike, – Technology as a “force multiplier” – battlefield information and communications systems, intelligence systems, positioning and navigation capabilities • 2014 Hagel – robotics and system autonomy, miniaturization, big data, and advanced manufacturing and innovation
- 19. The Offset Strategy • 1950’s Eisenhower – mass, mechanized warfare to nuclear deterrence • 1970’s Perry: stealth, precision strike, – Technology as a “force multiplier” – battlefield information and communications systems, intelligence systems, positioning and navigation capabilities • 2014 Hagel – robotics and system autonomy, miniaturization, big data, and advanced manufacturing and innovation Adversaries aren’t waiting for our 3rd offset strategy
- 20. U.S. China DF-21D ICBM Carriers Project power in the western pacific Air superiority in the western pacific Guam DF-31 IRBM Island basesCarriers Logistics Transcom Cyber attacks Overhead Intelligence NRO Cyber Attacks ASATs External Disruption – China’s Offset Strategy YJ-18/DH-10/CJ-10 A2/AD OTH Radar Taiwan/Operating bases in pacific Taiwan, Diego © sblank
- 21. Our Adversaries Speed is a Threat technology, deployment, etc. Our Enemy is Time
- 22. Our Adversaries Speed is a Threat Our Speed is the Mitigation Continuous innovation, adaptability, lean, etc
- 23. Silicon Valley Has Been Operating at Speed for 50 Years
- 24. Silicon Valley Has Been Operating at Speed for 50 Years Not Just Technology But Speed of Innovation
- 25. Here’s How
- 26. Types of Innovation In Large Companies Steve Blank
- 27. Three Horizons of Innovation Source: Baghai, Coley, White Mature Business our established capabilities Rapidly Growing Business Emerging Business
- 28. Three Horizons of Innovation Source: modified Baghai, Coley, White programs capabilities
- 29. Three Horizons of Innovation Known Unknown Partially Known Level of innovation is defined by whether the Business model is being executed, extended or explored! Execute Explore Extend
- 30. Three Horizons of Innovation Existing Mission Model: Process Innovation Execute Core Mission Known
- 31. Three Horizons of Innovation Existing Business Model: Process Innovation Execute Known Partially Known New Opportunities via Mission Model Innovation Extends Core Business
- 32. Three Horizons of Innovation Existing Business Model: Process Innovation Execute New Opportunities via Business Model Innovation Execute/Search Known Unknown Partially known New/Disruptive Mission Model Explores
- 33. Capabilities/Risk Assessment Existing Capabilities Low Risk Need New Capabilities High Risk Some Capabilities Moderate Risk Known Unknown Partially known
- 34. Innovation Allocation Across the Horizons Known Unknown Partially known 60-70% 20-30% 5-10%
- 35. Return on Investment by Horizon Known Unknown Partially known ROI 1-3 years • Improve • Partner • Acquire ROI 4-6 years • Extend • Invest • Partner • Acquire ROI 4-10 years • Incubate • Invent • Invest • Acquire Evangelos Simoudis/Steve Blank
- 36. Horizon 1: Process and Procedure = Profit KPIs HR Legal Finance Bonus Known Unknown Partially known Process and Procedures
- 37. Horizon 3: Process and Procedure = Death KPIs HR Legal Finance Bonus Product/Market Fit Pivots Agile Risk Large Reward Known Unknown Partially known Risk Process and Procedures
- 38. New Ventures/Startups are Horizon 3 Known Unknown Partially known • Improve • Partner • Acquire • Extend • Invest • Partner • Acquire ROI 1-7 yearsEvangelos Simoudis/Steve Blank
- 39. Startups are Horizon 3 Not burdened by: • Existing Customers, Channels, Inventory • Process, Procedures, KPI’s Everyone focused on a single mission and survival
- 40. Four Startup Principles • Money matters • Performance counts • Time is the enemy • Your investors eventually want liquidity
- 43. Lean = 3 parts Business Model Canvas Part 1 Customers Channels Customer Relationships Revenue Model Value Proposition Activities Resources Partners Costs Source: Alexander Osterwalder- Business Model Generation
- 44. Business Model Canvas = hypotheses of how you create and deliver value for the company and its customers Part 1 Source: Alexander Osterwalder- Business Model Generation Beneficiaries Deployment Buy- in/Support Mission Achievement Value Proposition Activities Resources Partners Costs
- 45. 1. Frame Hypotheses • Frame Hypotheses
- 46. 1. Frame Hypotheses • Frame Hypotheses Business Model Canvas
- 47. Mission Model Canvas Source: Alexander Osterwalder- Business Model Generation Customer Segments Channel Customer Relationships Revenue Streams Value Proposition Activities Resources Partners Costs how does the team get “Buy- In” from all the beneficiaries? How will we deploy the product to widespread use? What constitutes a successful deployment? Who are our most important customers? Stakeholders? What are their pains/gains? What job do they want us to get done for them> How are we solving each customers pains/gains? How? What product/servic e features match their needs? What key activities do we need to be expert in? What key resources do we need to own or acquire? Financial? Human? Who are our key partners? Suppliers? What are we getting from them? Giving them? What is the Budget/Cost? What is the revenue model? What are the pricing tactics?
- 48. 2. Test Hypotheses • Frame Hypotheses • Test Hypotheses Business Model Customer Development Customer Development is how you search for the model
- 49. Customer Development Turning the Business Model Canvas Into Facts
- 50. 9 Guesses Guess Guess Guess Guess Guess Guess Guess GuessGuess Customers Channel Customer Relationships Revenue Model Source: Alexander Osterwalder- Business Model Generation
- 51. Customer Development is Hypothesis Testing
- 52. 3. Build Incrementally & Iteratively • Frame Hypotheses • Test Hypotheses • Build the product incrementally & Iteratively Business Model Customer Development Agile Engineering
- 53. The Minimum Viable Product (MVP) • Smallest feature set that gets you the most … - learning, feedback, failure, orders, … - incremental and iterative • It is not a prototype • It is not a deployable version with the fewest features • It is what enables a test of a hypothesis • It may be a drawing, a slide, a wireframe, clickable workflow, etc…
- 54. The Pivot • Definition: A substantive change to one or more of the business model canvas components • Iteration without crisis • Fast, agile and opportunistic • Weeks and $100K
- 55. Pivot Cycle Time Matters • Speed of cycle minimizes cash needs • Minimum feature set speeds up cycle time • Near instantaneous customer feedback drives feature set Customer Discovery Customer Validation Company Building Customer Creation ExecutionSearch Pivot
- 56. Part 1 Agile Engineering + Part 2 Part 3 Elements of Lean Startup
- 57. Lean Gets Theory Customer Development 2003 Blank Agile Engineering 2011 Ries Business Model Canvas 2010 Osterwalder HBR Cover 2013 Blank
- 58. Lean Gets Practice MS&E 297: “Hacking for Defense”: Solving National Security issues with the Lean Launchpad In a crisis, national security initiatives move at the speed of a startup yet in peacetime they default to decades-long acquisition and procurement cycles. Startups operate with continual speed and urgency 24/7. Over the last few years they’ve learned how to be not only fast, but extremely efficient with resources and time using lean startup methodologies. In this class student teams will take actual national security problems and learn how to apply “Lean Startup” principles, ("business model canvas," "customer development," and "agile engineering”) to discover and validate customer needs and to continually build iterative prototypes to test whether they understood the problem and solution. Teams take a hands-on approach requiring close engagement with actual military, Department of Defense and other government agency end-users. Team applications required in February. Limited enrollment. Course builds on concepts introduced in MS&E 477. Terms: Spr | Units: 3-4 | Grading: Letter (ABCD/NP) Instructors: Blank, S. (PI) ; Byers, T. (PI) ; Felter, J. (PI) 2015-2016 Spring • MS&E 297 | 4 units | Class # 47395 | Section 01 | Grading: Letter (ABCD/NP) | LEC • 03/28/2016 - 06/01/2016 - with Blank, S. (PI); Byers, T. (PI); Felter, J. (PI) Lean LaunchPad For Students 2011 1250+ teams Taught in 75 Universities 800+ teams Taught by 53 Universities I-Corps For SBIR/STTR 2012 I-Corps For Life Sciences 2014 I-Corps For NSA 2015 ~250,000 on-line students Udacity.com Hacking for Defense 2016
- 59. How Does This Really Work?
- 60. Examples: Hacking for Defense @ Stanford
- 61. Week 1
- 62. Live Tactical Threat Toolkit (LTTT) Surgical-grade EOD, remotely We seek to create a stream of accurate information and insights for real-time defeat of improvised explosive devices by partner and coalition forces. The device we are building will be network- agnostic and designed for adverse and low-connectivity environments, and will be scalable across levels of information fidelity. Alex Zaheer CS Cyberpolicy Andreas Pavlou Physics/International Security Nitish Kulkarni Hardware Engineering Alex Richard CS Gaurav Sharma GSB Main Sponsor: OSD // JIDA Customers spoken to this week: 14 Support team/Sponsors: Booz Allen Hamilton US Army AWG AGUA Holdings Wolters Kluwer Crossdeck, Inc.
- 63. Live Tactical Threat Toolkit (LTTT) surgical-grade EOD, remotely Team Member 1 Nitish Kulkarni Alex Zaheer Alex Richard Gaurav Sharma Academic Program MA IPS 2016 ME 2016 CS 2019 CS 2016 GSB MSx LinkedIn https://www.linkedin.c om/in/andreas-pavlou- 02676753 linkedin.com/in/nitishk 2 linkedin.com/in/alex- zaheer-ba6548bb https://www.linkedin.co m/in/arichard4 https://www.linkedin.co m/in/gauravsharmastanf ord Subject Matter Expert? No (Some Software/Proj Mgmt) No (Sensor Fusion, Systems Integration) Yes (Software Engineering) Yes (Software Engineering) Yes (Military Information Logistics) Role Jack of All Trades Hardware Engineering SWE SWE PM Experience Solving a Problem that Seemed Impossible Developing a curriculum for a course on Cybersecurity & International Relations in less than two days Developed and designed early-stage concepts for multi-million dollar DARPA aerospace and ISR vehicle programs, hypersonics. Educating mostly elderly workforce as a youth regarding “good” security practices -- many blank stares Senior in CS at Stanford, specializing in Logic Led a team that built a system to ensure seamless flow of equipment to build rockets for SpaceX in 3 months (when everyone said it would take atleast twice as long)
- 64. Current MVP Key Pain Points: - Real-time translation (text, voice, video) - Persistent Communication Links - Operation in RF-denied or RF-unavailable environments - Language/Culture-agnostic UI/UX
- 65. Customer Discovery Hypothesis: • Build a Live Tactical Toolkit digital notebook for differently located coalition & US forces to address diffusing remote IED (Improvised Explosive Device) Experiments: • Interviewed 14 stakeholders including end users, planners, solution providers, and procurement teams • Ascertained organization structure and its relation to JIDA Results: Challenges include • Information deficit, data reliability • Coalition forces are illiterate, semi-literate • Language barrier, no minimum training standards for coalition forces • Incentivizing coalition forces to use technology • Bandwidth and network capability • Possibility of RF-denied environments • Solution should be as easy as for kindergarten children • Hardware is not an issue, software is • Data encryption • Limited subject matter experts on IED & language translators available Action: • Study existing solutions within JIDA & private solution providers • Interview more end users and private solution providers • Possibly interview coalition forces
- 66. DARPA PCAS Team - modifying existing systems Joint Improvised Threat Defeat Agency - work with technicians who deal with problem day-in day-out General Atomics, Raytheon - contractors already working in this space may be able to provide key insight ARMY Research Lab NAVEODTECHDIV ARMY Asymmetric Warfare Group Localization - ensuring system is applicable to every forward deployment Minimization - ensuring product fits every tactical situation Bridge - bridging the gap between forward deployed technicians and US remote technicians Personnel-centered - creating a system designed around those who need it most Integrated - creating a tool that is an aid, not a hinderance, to time- critical operations Engineers - contractor/DARPA employees may provide technical insight Technicians - connections to those who need this product most Embedded operations need support from those who need product Translation experts - those who do job already Iterate - Multiple simulated field test scenarios with technicians to fit product to need Incorporation - product has many possible futures - can potentially be incorporated into other DoD product needs Primary - JIDA, EOD specialists in recurring forward deployment operations across systems Secondary - Military remote technical communications wings. Software can be co-opted for many different communications missions Fixed: Variable: Software development optimization IT Management/maintenance Deployment training Mission Achievement Criteria Enable clear, safe, common-sense communication of time-sensitive high- stress technical information Real-time IED Defeat Assistance for Coalition and Remote Forces
- 67. Week 2
- 68. Live Tactical Threat Toolkit (LTTT) Surgical-grade EOD, remotely We seek to advance JIDA’s mission of countering improvised threats by developing systems and workflows for real-time remote advise and assist for allied foreign militaries, with the short-term product goal of real-time defeat of IEDs. This is of critical importance to JIDA in the context of reduced presence of US forces and the consequently mitigated ability of allied militaries to leverage US EOD expertise. Alex Zaheer CS Cyberpolicy Andreas Pavlou Physics/International Security Nitish Kulkarni Hardware Engineering Alex Richard CS Gaurav Sharma GSB Main Sponsor: OSD // JIDA Total Partners spoken to: 31 Partners spoken to this week: 17 Support team/Sponsors: Booz Allen Hamilton US Army AWG DIUX US Army Explosive Ordnance Disposal AGUA Holdings Wolters Kluwer Crossdeck, Inc. CENTCOM
- 69. Customer Discovery Hypothesis: • Build a Live Tactical Toolkit for coalition forces to do and gain assistance with: • Initial situation assessment • Blast damage mitigation procedures • Post-incident reporting and information exploitation Experiments: • Interviewed 17 stakeholders including end users, planners, solution providers, and procurement teams • Ascertained organization structure and its relation to JIDA Results: Challenges include • Incentivization to cover 50 km radius • Information deficit, data reliability • Coalition forces are illiterate, semi-literate • Language barrier, no minimum training standards • Security is important yet solution should be simple • US Forces also not incentivized to act as R&D testbeds • Bandwidth and network capability • No awareness of current information environment • Possibility of RF-denied environments • US-side EOD users adamant on not using toolkit during EOD operations • Data encryption for report dissemination on JIDA end • Too many solutions already available • Limited subject matter experts on IED & language translators available Action: • Continue prototype development and testing at EOD centers • Working on a connection to local training centers for iterative product development • Continue to study existing solutions and developers • Work with JIDA to explain white/black box approach • Possibly interview partner nation forces (?)
- 70. DARPA PCAS Team Joint Improvised Threat Defeat Agency General Atomics, Raytheon - contractors ARMY Research Lab NAVEODTECHDIV ARMY Asymmetric Warfare Group US Central Command US Army EOD Alaska Airlines/American Airlines Localization - ensuring system is applicable to every forward deployment Minimization - ensuring product fits every tactical situation even in A2/AD areas Bridge - bridging the gap between forward deployed technicians and US remote technicians Personnel-centered - creating a system designed around those who need it most Integrated - creating a tool that is an aid, not a hindrance, to time- critical operations Engineers - contractor/DARPA employees may provide technical insight EOD Technicians – iterative design process needs more user input Comm Technicians Civilian Transportation Embedded operations need support from those who need product Translation experts - those who do job already EOD Operators Partner Militaries (?) Iterate - Multiple simulated field test scenarios with technicians to fit product to need Incorporation - product has many possible futures - can potentially be incorporated into other DoD product needs Primary - JIDA, EOD specialists in recurring forward deployment operations across systems Secondary - Military remote technical communications wings. Software can be co-opted for many different communications missions Transportation Industry – remote medical care (airlines) Fixed: Variable: Software development Device Cost IT Management/maintenance Deployment training Mission Achievement Criteria Enable clear, safe, common-sense communication of time-sensitive high-stress technical information and post-incident reporting and analysis dissemination Real-time IED Defeat Assistance for Coalition and Remote Forces
- 71. Virtual Threat Toolkit: Value Proposition Canvas for Public Sector Products & Services Virtual Toolkit Linking Iraqi and American Forces - Real time consultation with Iraqi counterparts on how to disarm IED’s - Gain and document intelligence from Iraqi security forces as they encounter IED’s Customer Jobs Assess and defeat IEDs Restart report workflow Lack of information on evolution of IED’s utilized by insurgents and their tactics Difficulty advising operators in the field on how to defuse IEDs Explosive Ordnance Disposal Teams Gains Pains Gain Creators Pain Relievers Ability to advise coalition counterparts on how to dismantle IEDs Development of predictive models for insurgent use of IEDs Accurate reporting data on IED usage - Reduction of communication barrier between base-deployed US forces and Iraqi security forces in the field
- 72. Virtual Threat Toolkit: Value Proposition Canvas for Private Sector Products & Services Virtual Toolkit linking information sources (emergency services) and remote semi- trained operators - Real time access to expert-level lifesaving medical knowledge in situations where it is otherwise unavailable - Dynamic provision of necessary information in situations with communications obstacles Customer Jobs Assess and solve EMS problems remotely Lack of information on the specific problem set that commercial partners want to address Current needs are vague and need client-side definition Commercial Partners interested in VAA (e.g. Airlines, Transport Providers) Gains Pains Gain Creators Pain Relievers Ability to provide expert-level care without needing expensive resources tasking Remote assistance in areas where EMS expertise is sparse or temporally difficult Reduction of the communications barrier between semi-trained EMS operators and knowledge base Development of non-connectivity solutions to access a dynamic knowledge base Reducing communication latency and improving information flow bandwidth
- 73. Customer Workflow Partner military unit encounters IED Link to EOD expert using LTTT initiated: video feed/or minimal workflow depending on connectivity IED data logged Report sent to JIDA IED diagnosis and EOD process leveraging US expertise Resolution JIDA analyzes data and distributes intelligence as incentive for use
- 74. Current MVP Key Pain Points: - Real-time translation (text, voice, video) - Persistent Communication Links - Product adoption - Non-disruptive UI/UX during ops JIDA Database + Analysis
- 75. Week 3
- 76. Right of Boom Surgical-grade EOD, before and after Alex Zaheer CS Cyberpolicy Andreas Pavlou Physics/International Security Nitish Kulkarni Hardware Engineering Alex Richard CS Gaurav Sharma GSB Main Sponsor: OSD // JIDA US Army Explosive Ordnance Disposal US Air Force Explosive Ordnance Disposal FBI Bomb Defusal – San Francisco US Army Asymmetric Warfare Group US Central Command DIUx Wolters Kluwer Romania Booz Allen Hamilton Crossdeck, Inc. Iraqi Embassy We seek to advance JIDA’s mission of countering improvised threats by developing systems, workflows, and incentives for real-time remote advise and assist for allied foreign militaries with the goal of improved intelligence fidelity. 12 Interviews 1 Buyer 6 Experts 5 Users
- 78. Customer Discovery Hypothesis: JIDA’s previous unwillingness to share finished intel reduces Iraqi participation incentives Hypothesis: JIDA is willing grant partner forces limited access to IED reporting information via PiX Hypothesis: EOD units will want a LTTT to facilitate their ability to defuse IEDs. 12 Interviews 1 Bomb Suit 3 Prototypes 4 EOD Techs Solution: Incentives for EOD squadron commanders to generate data Next Week: Contacting Iraqi Military via Ambassador Field test of form+function(?) prototype of VAA Contacting JET
- 79. Joint Improvised Threat Defeat Agency ARMY Research Lab NAVEODTECHDIV ARMY Asymmetric Warfare Group US Central Command US Army EOD US Air Force EOD FBI Bomb Squad – SF Alaska Airlines/American Airlines Post-Incident Reporting - Ensuring high volumes of detailed IED reporting data Saving lives - Using VAA/PAA to mitigate allied casualties Bridge - bridging the gap between forward deployed technicians and US remote technicians Incentivization – gamification to increase likelihood of use by partner militaries Integrated - creating a tool that is an aid, not a hindrance, to time- critical operations Reporting – a richer data set for JIDA’s mission Engineers - contractor/DARPA employees may provide technical insight EOD Technicians – iterative design process needs more user input JIDA SETAs Coalition Forces - EOD + teams - Coalition Intel JIDA - Accurate reporting - Incentivized Intel Collection Iterate - Multiple simulated field test scenarios with technicians to fit product to need Incorporation - product has many possible futures - can potentially be incorporated into other DoD product needs Primary - JIDA - Coalition EOD - People in IED- impacted zones Secondary - Military remote technical communications. - Software can be co- opted for many different communications missions Transportation Industry – remote medical care Fixed: Software development IT Management/maintenance Deployment training Mission Achievement Criteria Enable clear, safe, common-sense communication of time-sensitive high-stress technical information and post-incident reporting and analysis dissemination Accurate high-volume post-incident IED reporting Variable: Maintenance Equipment retrofit Equipment refurb
- 80. Virtual Threat Toolkit: Value Proposition Canvas for Coalition EOD Products & Services -Virtual Toolkit Linking Iraqi and American Forces -Gamified EOD analytics and reporting - - Real time consultation with US EODs on how to dismantle IEDs - - EOD commander analytics and transparency Customer Jobs -Assess and defeat IEDs -Restart report workflow -Lack of information on evolution of IEDs utilized by insurgents and their tactics -Insufficient EOD knowledge Explosive Ordnance Disposal Teams Gains Pains Gain Creators Pain Relievers -Less Iraqis dying -Advice on how to better defuse IEDs -Accurate reporting data on IED usage (?) - - Reduction of communication barrier with base-deployed US forces - - Less likely to catastrophically fail while defusing an IED
- 81. Virtual Threat Toolkit: Value Proposition Canvas for JIDA/Intelligence Community Products & Services -Higher-fidelity post-incident EOD analytics and reporting -Higher- accuracy device classification and intelligence - Gain and document intelligence from Iraqi security forces as they encounter IEDs - Alignment of incentives between foreign EODs and JIDA such that blast data is generated - Real time consultation provides environmental and circumstantial intelligence (?) Customer Jobs -Analyze IED patterns and provide actionable intel -Lack of information on evolution of IED’s utilized by insurgents and their tactics -Wasting money on apps that nobody uses Explosive Ordnance Disposal Teams Gains Pains Gain Creators Pain Relievers -Accurate reporting data on IED usage -Development of predictive models for insurgent use of IEDs - Reduction of communication barrier between base-deployed US forces and Iraqi security forces in the field
- 82. Current MVP
- 83. Coalition EOD Technicians (US Point of Contact) JET Coalition EOD Commanders US FOB [EOD]Post-incident reporting Some Intel Assistance (?) Some Incentive/Assistance (?) JIDA [CONUS] Workflow
- 84. Week 4
- 85. Right of Boom Surgical-grade EOD, after Alex Zaheer CS Cyberpolicy Andreas Pavlou Physics/International Security Nitish Kulkarni Hardware Engineering Alex Richard CS Gaurav Sharma GSB Main Sponsor: OSD // JIDA US Army Explosive Ordnance Disposal US Air Force Explosive Ordnance Disposal FBI Bomb Squad – San Francisco US Army Asymmetric Warfare Group US Central Command DIUx Wolters Kluwer Romania Booz Allen Hamilton Crossdeck, Inc. We seek to advance JIDA’s mission of countering improvised threats by developing systems, workflows, and incentives for allied foreign militaries with the goal of improved intelligence fidelity. 19 Interviews 3 In MENA 61Total Interviews 10 EOD Techs
- 86. Customer Discovery Product Managers • J8 Program Co- ordinator, JIDA/OSD • J6 Operations, JIDA/OSD Concept Developers • [Stanford] • PiX SETA, CENTCOM-OM Users • Iraqi / ANA acq chiefs • J2 Analysist, JIDA/OSD Recommenders • Knowledge Manager, CENTCOM • EOD Team Lead Sgt., 65th Ordnance Company, US Army • JET • Coalition EOD Technician [PLS HELP] Saboteurs • Organizational Saboteurs • JIDA Classification Office Buyers • Mr. James P. Craft, Deputy Director, Information Enterprise Management, JIDA [Full Names Redacted For Privacy]
- 87. Hypotheses + Future Plans Hypothesis: EOD units will want a LTTT to facilitate their ability to defuse IEDs. Hypothesis: No infrastructure for a unified data system for IED incident reporting [PiX + JIDA SQL] Hypothesis: Material benefits are sustainable long- term for an information exchange relationship with coalition forces 19 Interviews 3 In MENA 1 Prototypes 6 EOD Techs Next Week: - Obtaining test/demo devices - Deep dive into PiX + API access - Accessing + using exisiting apps (HYBRID, PiX MENA, INDURE) - Talk to Iraqi liaison at Defense Language Institute - Speak with FBI, JET, HR/softer skills experts - Understand JIDA/Coalition workflows better
- 88. Coalition EOD Technicians (US Point of Contact) JET Coalition EOD Commanders US FOB [EOD] Post-incident reporting Some Intel Assistance (?) Some Incentive/Assistance (?) JIDA [CONUS]
- 89. Coalition EOD Technicians US POC [JET] [ONSITE] FBI [CONUS] DOD EOD DATABASE [CONUS] JIDA DATABASE [CONUS] US AIR FORCE EOD [CONUS] US ARMY EOD [CONUS] US NAVY EOD [CONUS] [UNCLASS] [UNCLASS] [UNCLASS FOUO S/TS] Information flow, in an ideal world [UNCLASS FOUO S/TS] [UNCLASS FOUO S/TS] [UNCLASS FOUO S/TS]
- 90. INTERNET AFGHAN NETS SIPRnet NIPRnet DOD (DISA) APAN DOD (NGA) PiX DOD (DISA) APAN AF RONNA SharePoint 2010 BASEER NIMS CIDNE RED DOT DAIIS PORTALS HERMES Open Source Program CDC Dissemination Program Dept. of Army Intelligence Information System Smartphone Web Browser PiX Android/iOS Apps INDURE Android View/Report App INDURE Android View/Report App JIDA Hybrid Reporting App SOCCENT ATAK Tactical Ops App PiX SMS PiX WIKI Mediated WIKI INDURE API IRAQ/Other Nets Collaboration Portal Zimbra/Sharepoint 2007 FOUO/SBU INDURE Reporting/DB GPS Mapping Current Information Map
- 91. Joint Improvised Threat Defeat Agency ARMY Research Lab NAVEODTECHDIV ARMY Asymmetric Warfare Group US Central Command US Army EOD US Air Force EOD FBI Bomb Squad – SF Alaska Airlines/American Airlines Post-Incident Reporting - Ensuring high volumes of detailed IED reporting data Saving lives - Using VAA/PAA to mitigate allied casualties Bridge - bridging the gap between forward deployed technicians and remote technicians IED collection capabilities and analysis Alignment of Incentives High-speed intelligence allows for same-day incident analysis Improved Reporting – a richer data set for JIDA’s analysis Engineers - contractor/DARPA may provide technical insight EOD Technicians – iterative design process needs more user input JIDA SETAs – provide details on existing solutions and flaws Coalition Forces - NEED: intel, supplies - HOW: Incentivize - SABOTEURS: Bureaucracy JIDA - NEED: increased reporting - HOW: incentivizing coalition forces - SABOTEURS: Over- classification Iterate - Multiple simulated field test scenarios with technicians to fit product to need Incorporation - product has many possible futures - can potentially be incorporated into other DoD product needs Primary - JIDA (Matt, Carie, James) - EoD (Ike, Ethan, Christine, Whitner, especially J2 (Intelligence) and J5 (Strategic Plans and Policy) - Coalition EOD (Sgt. Bashir) - Other coalition forces (Romania, Georgia, Bulgaria: no contacts) - Other entities such as FBI (Sean) Fixed: Software development IT Management/maintenance Deployment training Mission Achievement Criteria Enable clear, safe, common-sense communication of time-sensitive high-stress technical information accurate actionable intel and post-incident reporting and analysis dissemination. Accurate high-volume post-incident IED reporting Variable: Maintenance Equipment retrofit Equipment refurb [Full Names Redacted For Privacy]
- 92. Products & Services -Higher-fidelity post-incident EOD analytics and reporting -Higher- accuracy device classification and intelligence - Gain and document intelligence from Iraqi security forces as they encounter IEDs -Alignment of incentives between foreign EODs and JIDA such that blast data is generated -Bringing coalition forces into intelligence umbrella Customer Jobs -Analyze IED patterns and provide actionable intel -Lack of information on evolution of IED’s utilized by insurgents and their tactics -Reliance on material incentives to coerce Iraqis into giving intel Gains Pains Gain Creators Pain Relievers -Accurate reporting data on IED usage -Development of predictive models for insurgent use of IEDs - -Reduction of communication barrier between base-deployed US forces and Iraqi security forces in the field - -Transition from material incentives to informational incentives, allowing for more sustainable cooperation Virtual Threat Toolkit: Value Proposition Canvas for JIDA J2 and J5 Explosive Ordnance Disposal Teams
- 93. Virtual Threat Toolkit: Value Proposition Canvas for Coalition EOD Products & Services -Integrated platform for both reporting raw intel and receiving analysis -Gamified EOD analytics and reporting - -Timely and refined intelligence on IEDs and the tactics of insurgents using them - - EOD commander analytics and transparency Customer Jobs -Assess and defeat IEDs -Restart report workflow -Lack of information on evolution of IEDs utilized by insurgents and their tactics Explosive Ordnance Disposal Teams Gains Pains Gain Creators Pain Relievers -Less Iraqis dying -Improved counter-IED capability - - Reduction of communication barrier with base-deployed US forces - -Less likely to encounter IEDs without being forewarned
- 94. PiX JIDA SQL DB PiX MENA US DOD Coalition end-user App New App OpenAPI Open API (Confusing) API (Confusing) API MVP
- 95. Week 1
- 96. Narrative Mind Michael May, Zach Maurer, Shazad Mohamed, Will Kim Customers spoken to this week: 14 Total customers spoken to: 14 Sponsor: Army Cyber Command (ARCYBER) - MAJ Kirk Kudrna, LTC Matthew Zais Liaisons: DIUx - Lt. Col. Ernie Bio The Narrative Mind team contains experts in software engineering, social media design, and web-based information operations (IO). We seek to develop tools that will optimize discovery and investigation of adversary communication trends on social media, allowing ARCYBER and others to more efficiently respond and mitigate threats posed by enemy messaging.
- 97. Michael May M.A. IPS 2017 Former PM of counterterrorism communications at USCENTCOM Zach Maurer B.S. CS 2016 Software engineer, int’l development, UI research William Kim M.S. CS 2017 Software engineer, private equity and community finance Shazad Mohamed B.A. Political Science 2016 Former PM Facebook, co-founder of security startup. Team Michael May M.A. IPS 2017 Former PM of counterterrorism Zach Maurer B.S. CS 2016 Software engineer, int’l development, UI
- 101. Value Proposition Canvas Products & Services Desktop tracking and analysis platform. Customer Jobs Develop potential counter narratives to propaganda No way to easily track all the known extremists and real-time action of tweets/hashtags Gains Pains Gain Creators Pain Relievers Platform for automatically categorizing tweets, escalating potentially viral threats early on - Better detection ability and improve response time to potentially viral narratives, shut down or respond before it gains momentum - Semi-automation of tweet categorization and virality detection - Filing tools for user & hashtag histories
- 102. Week 4
- 103. Team Narrative Mind Michael May, Zach Maurer, Shazad Mohamed, Will Kim Stanford University
- 104. Team Info Sponsor: US Army Cyber Command (ARCYBER) Liaisons: DIUx - Lt. Col. Ernie Bio The Narrative Mind team contains experts in software engineering, social media design, and web-based information operations (IO). We seek to develop tools that will optimize discovery and investigation of adversary communication trends on social media, allowing ARCYBER and others to more efficiently respond and mitigate threats posed by enemy messaging. Weekly Total: 10 interviews Users: 11 Experts: 31 Buyers: 4 Cumulative Total: 46 interviews
- 105. All the Hypotheses Global Tweet-level Awareness Response Adversary IO Org Chart No baseline for monitoring/aggregating adversary use of tech; cyber targeting Automatic Narrative Detection Language/culture experts aren’t able to work at scale Virality Predictor Understand which narratives are the most salient Important Event Predictor Preempt terrorist attacks Counter-Narrative Generator DoD/Gvt. social media presence is weak Persistent ID-Alias tracker Account bans and multiple aliases across different networks make IDs hard to track Poison Link Generator Monitor who views adversary dark-web content File sharing Site Scraper Need more cached information access Expedited Content Categorization Scale of social media makes manual efforts painful
- 106. Organizational Chart (updated) Analyst “Fish” (CW3 or NCO) MAJ “Kramer” Reports info to... COL “Tom” LTC “Zeke” Director & Deputy Assesses threats 1-3yrs out Proposes CONOP? Gets approval? BG Frost Deputy Commander for Ops ARCYBER (service component) NSA ADM Rogers Internal ARCYBER office - Uses OTA option LTG Cardon BG (O8-level) JOINT FORCES HQ-CYBER USCYBERCOM ADM Rogers USSTRATCOM COCOMs CENTCOM, AFRICOM, NORTHCOM Proposes concepts Blurred lines between the two Secretary Carter CYBER-only Acquisition process NSA
- 107. Hypotheses ❏ Last week’s hypotheses were not relevant to updated problem definition. ❏ Too MVP specific. Experiments ❏ Weren't able to consistently test any previous hypotheses. Results ❏ ARCYBER confidently re-focused the problem on “identifying narratives in real time”. ❏ Full-circle evolution. ❏ The problem is not content categorization nor org chart-level awareness nor persistent ID tracking nor event prediction. Actions 1. Increased frequency of communication with sponsor. 2. Help ARCYBER define what a “narrative” is. 3. Research how commercial companies are visualizing “narratives”. 4. Continue expanding network to beneficiaries who can answer the above questions. - PACOM, CENTCOM, NPS CORE Lab Customer Discovery
- 108. Mission Model Canvas- Categorize social media posts and users by content for monitoring and tactical purposes. - Understand viral potential of extremist social media posts in real-time. - Track how extremist groups use technology over time. - Gnip/Twitter/Facebook - CrowdFlower, Samasource, or Mechanical Turk - Pre-existing social media service and micro- labor aggregators - Third-party access platforms for social media, e.g. Hootsuite which allow users to circumvent barriers, but allows us to track IP addresses. -Data visualization (Zoic) -Content analysis platforms (Sens.ai, Leidos) ARCYBER -Bg. General (decision maker) -MAJ/LTC/COL (operational plan) -Analysts/Operators (actionable insights) COCOMs -Generals/Analysts -Tactical decision makers - Optimize workflow for social media analysts - Effectively visualize how extremist groups use technology as they grow. - Aggregate known research? - Expedite categorization of social media content. - Use MechanicalTurk to crowdsource categorization of content and users. - Algorithmic virality predictor to create alerts for important, time- sensitive threats. - Use design of now-defunct Palantir Torch as inspiration for how to present content in a streamlined manner. - Force multiplier for intelligence analysts: receive cleaner, pre-categorized data, target the most urgent threats to national security in real-time. - Increase throughput to quantify and flag viral extremist content. - Improve the categorization of unstructured social media data points using crowdsourced micro-task labor. ?????????? - Architecture that can support massive concurrent data aggregation and analysis. - Customized UI - Testing with analysts - MechanicalTurk or crowdsourcing labor (microtasks) - UI Development/Testing with ARCYBER analysts. - Software Development - Research aggregation? (e.g. Import and reformat existing knowledge about extremist organizational structure) - Access to Twitter firehose or API - Local language speaking crowdsourcing staff. - Accurate testing for intercoder reliability - Individual Analysts/Operators - ARCYBER (Bg. General, LTC, Strategic Intiatives Group) - Continued partnership with crowdsourcing firms, Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners
- 109. MVP - 3 Types of “Narratives” Outputted by Our System E.g. Madison, AL: Pro-Trump (Positive: 61% ± 5%) Millbrae, CA: Anti-Trump (Positive: 11% ± 5%) A ranked list of extremists’ most popular content over time A list of hashtags that appear with the same message with extreme frequency. Positive or negative opinion of adversaries among geographic or demographic population. Narrative Type #1 Narrative Type #2 Narrative Type #3 E.g. Adversary: Trump Campaign #NY4Trump #Trump2016 Could be classified by an analyst as “New York Primary Push” or similar. E.g Could be classified by an analyst as “Political Outsider” or similar.
- 110. Value Proposition Canvas - ARCYBER Products & Services Web/Desktop Application Provide high-level overview of terrorists and groups,update research Customer Jobs - New mission priorities. - Little familiarity with overall environment. - No repository for high-level learnings Gains Pains Gain Creators Pain Relievers - New opportunities for cyber- ops against adversary groups - Social network graph of all terrorist groups? -Illustrates evolution of enemy info dissemination networks - Provides insight for mapping enemy “online terrain” Visualize information about how adversary organizations are using social media ARCYBER - Operator/Analyst (Actionable Insights)
- 111. Value Proposition Canvas - ARCYBER Products & Services Web/Desktop Application Aggregate insights from analysts, compile into plan passed to decision maker - New mission priorities. - Proliferating/viral social media platforms - Urgency of threat clouds ability to prioritize overabundance of threats Customer Jobs Gains Pains Gain Creators Pain Relievers -Faster turnaround time with better data gained from information awareness software and technology - Clear directive for neutralizing cyber threats - Clear hierarchy of cyber threat levels ARCYBER - MAJ/LTC/COL (Operational Plans) -Better data in informing operational plans provided to decision makers -Reducing uncertainty in operational planning
- 112. Value Proposition Canvas - ARCYBER Products & Services Web/Desktop Application Act on reports and plans generated about various cyber threats across globe - Little understanding of ground-level nuances of threats - Little understanding of proliferating platforms Customer Jobs Gains Pains Gain Creators Pain Relievers - Organizational chart of technology broken down by functional purpose - Timeline-based viewer - Social network graph of all terrorist groups? -Reduce uncertainty of decision making -Add methodological rigor to ARCYBER’s offensive/defensive operations - Neutralize terrorist threats on cyber ARCYBER - Brigadier General (Decision Maker)
- 113. Value Proposition Canvas - COCOM Products & Services Web/Desktop Application Target audience analysis, finding opportunities to optimize influence operations Customer Jobs -Loss of visibility on key communicators (when accounts shut down/opened) - No repository for high-level learnings Gains Pains Gain Creators Pain Relievers -Auto-populated list & real-time visualization of key accounts, dissemination networks -Allows more accurate targeting -Illustrates evolution of enemy info dissemination networks - Provides insight for mapping enemy “online terrain” Closure of visibility gap on key adversary accounts COCOM - Operator/Analyst (Actionable Insights)
- 114. Value Proposition Canvas - COCOM Products & Services Web/Desktop Application -Long term operational planning - Measuring effectiveness Customer Jobs - Unclear picture of how some adversary networks evolve, adapt over time Gains Pains Gain Creators Pain Relievers - Organizational chart of technology broken down by functional purpose - Visualization of key accounts, dissemination networks over time -Illustrates evolution of enemy info dissemination networks - Provides insight for measuring operations, mapping enemy “online terrain” Greater awareness of network evolution improves planning, indicators of operational effectiveness COCOM - Maj/LTC/Col. (O4-O6) (Operational Plans)
- 115. Value Proposition Canvas - COCOM Products & Services Web/Desktop Application - Judge potential effect and risk of operations - Approve/reject plans Customer Jobs - Less familiarity with internet as an operational environment - Needs high level summary of how adversaries use technology Gains Pains Gain Creators Pain Relievers - Organizational chart of technology broken down by functional purpose - Timeline-based viewer -Illustrates evolution of enemy info dissemination networks - Provides insight for mapping enemy “online terrain” Visualize high-level information about how regional adversaries use technology. COCOM - Brigadier General (O7) (Decision Maker)
- 116. APPENDIX APPENDIX
- 117. Hypotheses ❏ “C2”, “Recruiting”, “Propaganda” are sufficiently granular content categories. ❏ Advanced warning virality prediction is valuable for countering threats early. ❏ Content categorization is central to analyst workflow. ❏ Provide higher-level information around adversaries’ organizational structure Experiments ❏ Focused on getting sponsor to re-articulate original problem. ❏ Explicitly asked for thoughts on crowdsourcing mechanism. ❏ Interviewed end users about their tasks and current tools, as well as our own hypotheses Results ❏ Understanding terrorist groups is difficult as they are dynamic organizations, need some way of tracking how these organizations shift over time, and how they interact with each other ❏ Geospatial information is valuable, more so that social network analysis ❏ Government agencies are OK with crowdsourced tools for OSINT Actions Testing our two main MVPs: 1. Organizational chart of the various terrorist groups over time 2. Social network map of how various terrorist groups interact with each other Customer Discovery
- 118. ARCYBER Org Chart●Talk of CYBERCOM becoming its own COCOM ●Both CYBERCOM and NSA are technically led by ADM Rogers
- 119. Organizational chart of various terrorist groups over time and their social media platforms - Organizational structure of terrorist groups and affiliates - Sub-nodes display main channels for propaganda and platforms used - Time slider shows temporal change MVP ISIL Nusra Front C2 Propaganda Recruitment Google Maps Logistics Battle Whats App Public Private Twitter Telegram Twitter Time Slider APR 2015 MAY 2015
- 120. MVP Mockups
- 121. Disruptive Innovation Continuous Innovation Lean Means Getting Out of Your Office Horizon 2 Horizon 3 Speed & Urgency Lean Steve Blank • If you’re not talking to 100’s of customers, it’s not lean • If you’re not building iterative and incremental minimum viable products, it’s not lean
- 123. Help DOD/IC Tap University Talent Pool Create Tech National Service Help Gov’t Innovate at Speed Connect the DOD/IC Islands of Innovation
- 124. Thanks! Steve Blank: https://steveblank.com/about/ sblank@kandsranch.com Hacking for Defense Class: http://hacking4defense.stanford.edu Hacking for Defense Blogs: https://steveblank.com/category/hacking-for-defense/ Lean Open Source: https://steveblank.com/slides/ Lean in the HBR: https://hbr.org/2013/05/why-the-lean-start-up-changes-everything/ar/1
- 125. Innovation at 50x Steve Blank @sgblank www.steveblank.com 2/1/16