SlideShare une entreprise Scribd logo
1  sur  65
Télécharger pour lire hors ligne
SWIFT
       FOSS data triage platform




Crowdsourced Crisis Data   Media Analytics
       (Africa)             (Middle East)
May 6, 2009 Team Update


                  Where we are today?

Reeling from an awesome conference.
Enjoying the good company of InSTEDD.
International coworkers in person for the first time.
Swift is going live after months of collaborative design work.
Numerous deployments are scheduled in very near term.
Currently iterating “live” for votereport.in *gulp :)
Swift Update May 6
Swift Update May 6
Swift Update May 6
Huh?
the deployment problem
the deployment problem

     If you have to deploy, you
           have a problem
How might we imagine a better
    crisis reporting tool?

By designing a better crisis listening tool.
why
quot;At present, early warning units within the UN ... use manual
 labor to collect relevant information from online sources.
   Most units employ full-time staff for this, often
  meaning that 80% of an analyst’s time is actually used to
 collect pertinent articles and reports, leaving only 20% of
    the time for actual analysis, interpretation and policy
 recommendations. We can do better. Analysts ought
        to be spending 80% of their time analyzing.quot;
          Patrick Meier, “Crimson Hexagon: Early Warning 2.0?” February 17, 2009
    http://earlywarning.wordpress.com/2009/02/17/crimson-hexagon-early-warning-20
We need new listening tools.
              Ability to report eyewitness stories
              Ability to act on them



                                                         40000

                                                         30000

                                                     20000

                                                     10000

                                                     0

   1910     1940           1980            2010
Swift Update May 6
Swift Update May 6
Signal is Overwhelmed by Noise
How much do you need to know to act?
Well, how much data can you take?
Swift Update May 6
Swift Update May 6
What can we build when we focus
 on the user interface needs of citizen editors?

How does this changes our idea of an “analyst”?

   Can data entry actually make an impact?

    Could it actually be ... enjoyable to use?
What if we assume that citizen editors deserve the best?
What if we treat their data with the same respect?
STRUCTURED PUBLISHING
The revolution will not be transmitted via giant .xls files.
Swift Update May 6
Output: The Crisis API


The resulting crisis database is open, with distributed storage
         and standards-based portability, based on:

             ICAL / FOAF / Dublin Core / Etc.
                             via
               RSS / CSV / JSON / SQL / Etc.



           and all of it is handled by Freebase.com
Swift the method has evolved into a platform:


     1. data gathering engine (aggregator)
     2. data structuring tool (wiki)
     3. most importantly, an API for crisis data
Swift is an aggregator
        with entity extraction
By “roping together” relevant feeds, then parsing their
content, we can get a rich database of people, places
and organizations in real time.

We are working with
Freebase.com and Calais.

Swift is mostly a Rails app,
Twitter Vote Report.
Swift Update May 6
the sweeper
CITIZEN EDITORS




Good with Data? Now you can help.
Swift is designed for
Improving information findability in a crisis
Making it easier to find things that you didn't know you were looking for
Better understanding media from other parts of the world
Making urgent data more sharable (structured, published and accessible)
Making it more obvious what information is missing about an crisis
Promoting the work of eyewitnesses with prepared crisis editors
Expanding the grassroots reporting network
Preserving information across crises
Al Jazeera deployment
Indian Elections coverage
Meedan.net
swiftapp.org
Tracker




InSTEDD is good at listening.
Evolve
Vine
Real-timelines
Realtime Aggregators
Advanced analytics are mainstreaming
Our FOSS Toolkit is Growing Really Fast
But tending data is not new
Mapping is not new
From the Swift perspective, if you want to find
       the new opportunities for change ...


           Focus on speed and people.


We can work together faster than ever before.
Human-curated data can happen in realtime
with the help of entity recognition and reconciliation.

        Really, they just help us work faster.
Swift Update May 6
Recognition saves time
Reconciliation Saves Time
Swift Update May 6
Swift Update May 6
WHO ARE THE EDITORS?
Swift Update May 6
CITIZEN EDITORS

      Text
GEEK OUT A SEC
     Every incoming is parsed into an object with
                      attributes:
1.   URI.body (the text of the url)
2.   URI.rating (anyone can rate through a web UI)
3.   URI.submitters (anyone who linked to it)
4.   URI.history (every revision preserved)
5.   URI.tags (added by humans and machines)
Swift Update May 6
PFIF




people finder interface format
 used for 90,000 entries after Katrina
Grassroots reporting: Database-driven journalism and data
                        curation.
TWITCH GAMES
NEWS API, OH MY
Swift Update May 6
Weather Related Disasters
                                            Urban and Rural Fire
                                            Earthquake
                                            Economic Crisis
                                            Homelessness
                                            War
   Possibilities with speed:                Internal Displacement & Refugee
                                            Environmental Crisis
                                            Aviation Disasters
                                            Highway Accidents
What to do in a ___ based on my location?   Volcano
Report a ____ location                      Missing Person
Report a ___ accident.                      Structure Collapse
___ detection and reporting.                Railroad Disasters
                                            Health Disasters
“There was a ____. Are you ok?” alerts.
                                            Power Outage
Neighborhood-level ____ warnings.
                                            Explosions
                                            Floods
                                            AIDS
                                            Famine
                                            Landslide
                                            Stampede
                                            Foot and Mouth
                                            Avalanche
                                            Fatal Accident
                                            Tsunami Disaster
                                            Bird Flu
Swift’s realtime strengthens the Ushahidi
               alert cycle

 Subscribe to disaster keyword based
     alerts about an emergency
            eg: “san francisco, earthquake”
Possible with speed:
See Also

Crimson Hexagon state of the art NLP.

VRA's GeoMonitor, a natural language parser that reads the headlines of Reuters and AFP news
wires and codes to state quot;who did what, to who, where and when?quot; Rated quot;virtually identicalquot; to
human event summarizing.

JRC's European Media Monitor, which can parse thousands of different news sources but faces
limitations since analysts still need to read each article to understand the nature of the terrorist
event. http://emm.jrc.it/NewsBrief/clusteredition/en/latest.html

Tabari text parsing engine, quot;event data coder that has been used in at least five NSF-sponsored
projects and produced data used in a number of refereed articles in political science.quot;

FORECITE Forecasting of Crises and Instability Using Text-Based Events, developed by the US
Center for Army Analysis (CAA)

UN's HEWSweb A relatively new early warning system

Biowarn Textual analysis with infectious disease focus

Cewarn

FAST Comprehensive system by Swisspeace
A minimal implementation: an analogy

We’ll put a card table at the public library and
   work on reports with highlighters, put
everything into a card catalog and leave it in
                  the library.

Contenu connexe

Similaire à Swift Update May 6

Disaster Tech: What is working and what is coming
Disaster Tech: What is working and what is comingDisaster Tech: What is working and what is coming
Disaster Tech: What is working and what is comingguestf8e7a8
 
The Evolution of the Sahana System, Community and Standards @ Taiwan 2010
The Evolution of the Sahana System, Community and Standards @ Taiwan 2010The Evolution of the Sahana System, Community and Standards @ Taiwan 2010
The Evolution of the Sahana System, Community and Standards @ Taiwan 2010Chamindra de Silva
 
Social Media Analytics for Emergency Management - Telefonica Daedalus 2014
Social Media Analytics for Emergency Management -  Telefonica Daedalus 2014Social Media Analytics for Emergency Management -  Telefonica Daedalus 2014
Social Media Analytics for Emergency Management - Telefonica Daedalus 2014Sngular Meaning
 
Humanitarian Mapping - Interaction ICCC
Humanitarian Mapping - Interaction ICCCHumanitarian Mapping - Interaction ICCC
Humanitarian Mapping - Interaction ICCCAndrew Turner
 
Artificial Intelligence for Goods: Cases and Tools
Artificial Intelligence for Goods: Cases and ToolsArtificial Intelligence for Goods: Cases and Tools
Artificial Intelligence for Goods: Cases and ToolsOleksandr Krakovetskyi
 
Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...
Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...
Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...J T "Tom" Johnson
 
Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...
Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...
Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...Global Business Events
 
Humanitarian Mapping - InterAction ICCC
Humanitarian Mapping - InterAction ICCCHumanitarian Mapping - InterAction ICCC
Humanitarian Mapping - InterAction ICCCguestbfe342
 
Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...
Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...
Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...Global Risk Forum GRFDavos
 
Rfs & social media
Rfs & social mediaRfs & social media
Rfs & social mediaBGTT_SYD
 
Open Geoscience Knowledge Egu2014
Open Geoscience Knowledge Egu2014Open Geoscience Knowledge Egu2014
Open Geoscience Knowledge Egu2014NH Wikisaurus
 
Ushahidi esri juliana
Ushahidi esri julianaUshahidi esri juliana
Ushahidi esri julianaUshahidi
 
ETech2008 DisasterTech Robbins Maron 20080305a
ETech2008 DisasterTech Robbins Maron 20080305aETech2008 DisasterTech Robbins Maron 20080305a
ETech2008 DisasterTech Robbins Maron 20080305aJesse Robbins
 
IAEM11 2011 CrisisCommons VTC presentation
IAEM11 2011 CrisisCommons VTC presentationIAEM11 2011 CrisisCommons VTC presentation
IAEM11 2011 CrisisCommons VTC presentation@schuback
 
Real-Time Lightning Alerts from The Weather Network
Real-Time Lightning Alerts from The Weather NetworkReal-Time Lightning Alerts from The Weather Network
Real-Time Lightning Alerts from The Weather NetworkSafe Software
 
Leveraging technology in disaster management
Leveraging technology in disaster managementLeveraging technology in disaster management
Leveraging technology in disaster managementDhananjay Singh
 
Data Journalism Training @ Southern Metropolis Daily, Guangdong, China
Data Journalism Training @ Southern Metropolis Daily, Guangdong, ChinaData Journalism Training @ Southern Metropolis Daily, Guangdong, China
Data Journalism Training @ Southern Metropolis Daily, Guangdong, ChinaYolanda Ma Jinxin
 

Similaire à Swift Update May 6 (20)

Disaster Tech: What is working and what is coming
Disaster Tech: What is working and what is comingDisaster Tech: What is working and what is coming
Disaster Tech: What is working and what is coming
 
The Evolution of the Sahana System, Community and Standards @ Taiwan 2010
The Evolution of the Sahana System, Community and Standards @ Taiwan 2010The Evolution of the Sahana System, Community and Standards @ Taiwan 2010
The Evolution of the Sahana System, Community and Standards @ Taiwan 2010
 
InSTEDD
InSTEDDInSTEDD
InSTEDD
 
Social Media Analytics for Emergency Management - Telefonica Daedalus 2014
Social Media Analytics for Emergency Management -  Telefonica Daedalus 2014Social Media Analytics for Emergency Management -  Telefonica Daedalus 2014
Social Media Analytics for Emergency Management - Telefonica Daedalus 2014
 
Humanitarian Mapping - Interaction ICCC
Humanitarian Mapping - Interaction ICCCHumanitarian Mapping - Interaction ICCC
Humanitarian Mapping - Interaction ICCC
 
Artificial Intelligence for Goods: Cases and Tools
Artificial Intelligence for Goods: Cases and ToolsArtificial Intelligence for Goods: Cases and Tools
Artificial Intelligence for Goods: Cases and Tools
 
Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...
Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...
Analytic Journalism: Investing in an Intellectual Portfolio to Secure Journal...
 
Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...
Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...
Jason Witty, SVP & CISO at US Bank - Next eneration information security meet...
 
Humanitarian Mapping - InterAction ICCC
Humanitarian Mapping - InterAction ICCCHumanitarian Mapping - InterAction ICCC
Humanitarian Mapping - InterAction ICCC
 
Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...
Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...
Russ Johnson - Understanding Disasters: Geospatial Technologies in Risk Reduc...
 
Rfs & social media
Rfs & social mediaRfs & social media
Rfs & social media
 
Open Geoscience Knowledge Egu2014
Open Geoscience Knowledge Egu2014Open Geoscience Knowledge Egu2014
Open Geoscience Knowledge Egu2014
 
Ushahidi esri juliana
Ushahidi esri julianaUshahidi esri juliana
Ushahidi esri juliana
 
ETech2008 DisasterTech Robbins Maron 20080305a
ETech2008 DisasterTech Robbins Maron 20080305aETech2008 DisasterTech Robbins Maron 20080305a
ETech2008 DisasterTech Robbins Maron 20080305a
 
Disasters2.0
Disasters2.0Disasters2.0
Disasters2.0
 
IAEM11 2011 CrisisCommons VTC presentation
IAEM11 2011 CrisisCommons VTC presentationIAEM11 2011 CrisisCommons VTC presentation
IAEM11 2011 CrisisCommons VTC presentation
 
Real-Time Lightning Alerts from The Weather Network
Real-Time Lightning Alerts from The Weather NetworkReal-Time Lightning Alerts from The Weather Network
Real-Time Lightning Alerts from The Weather Network
 
Technologies for natural disaster
Technologies for natural disasterTechnologies for natural disaster
Technologies for natural disaster
 
Leveraging technology in disaster management
Leveraging technology in disaster managementLeveraging technology in disaster management
Leveraging technology in disaster management
 
Data Journalism Training @ Southern Metropolis Daily, Guangdong, China
Data Journalism Training @ Southern Metropolis Daily, Guangdong, ChinaData Journalism Training @ Southern Metropolis Daily, Guangdong, China
Data Journalism Training @ Southern Metropolis Daily, Guangdong, China
 

Swift Update May 6

  • 1. SWIFT FOSS data triage platform Crowdsourced Crisis Data Media Analytics (Africa) (Middle East)
  • 2. May 6, 2009 Team Update Where we are today? Reeling from an awesome conference. Enjoying the good company of InSTEDD. International coworkers in person for the first time. Swift is going live after months of collaborative design work. Numerous deployments are scheduled in very near term. Currently iterating “live” for votereport.in *gulp :)
  • 8. the deployment problem If you have to deploy, you have a problem
  • 9. How might we imagine a better crisis reporting tool? By designing a better crisis listening tool.
  • 10. why quot;At present, early warning units within the UN ... use manual labor to collect relevant information from online sources. Most units employ full-time staff for this, often meaning that 80% of an analyst’s time is actually used to collect pertinent articles and reports, leaving only 20% of the time for actual analysis, interpretation and policy recommendations. We can do better. Analysts ought to be spending 80% of their time analyzing.quot; Patrick Meier, “Crimson Hexagon: Early Warning 2.0?” February 17, 2009 http://earlywarning.wordpress.com/2009/02/17/crimson-hexagon-early-warning-20
  • 11. We need new listening tools. Ability to report eyewitness stories Ability to act on them 40000 30000 20000 10000 0 1910 1940 1980 2010
  • 15. How much do you need to know to act?
  • 16. Well, how much data can you take?
  • 19. What can we build when we focus on the user interface needs of citizen editors? How does this changes our idea of an “analyst”? Can data entry actually make an impact? Could it actually be ... enjoyable to use?
  • 20. What if we assume that citizen editors deserve the best?
  • 21. What if we treat their data with the same respect?
  • 22. STRUCTURED PUBLISHING The revolution will not be transmitted via giant .xls files.
  • 24. Output: The Crisis API The resulting crisis database is open, with distributed storage and standards-based portability, based on: ICAL / FOAF / Dublin Core / Etc. via RSS / CSV / JSON / SQL / Etc. and all of it is handled by Freebase.com
  • 25. Swift the method has evolved into a platform: 1. data gathering engine (aggregator) 2. data structuring tool (wiki) 3. most importantly, an API for crisis data
  • 26. Swift is an aggregator with entity extraction By “roping together” relevant feeds, then parsing their content, we can get a rich database of people, places and organizations in real time. We are working with Freebase.com and Calais. Swift is mostly a Rails app, Twitter Vote Report.
  • 29. CITIZEN EDITORS Good with Data? Now you can help.
  • 30. Swift is designed for Improving information findability in a crisis Making it easier to find things that you didn't know you were looking for Better understanding media from other parts of the world Making urgent data more sharable (structured, published and accessible) Making it more obvious what information is missing about an crisis Promoting the work of eyewitnesses with prepared crisis editors Expanding the grassroots reporting network Preserving information across crises
  • 35. Tracker InSTEDD is good at listening.
  • 37. Vine
  • 40. Advanced analytics are mainstreaming
  • 41. Our FOSS Toolkit is Growing Really Fast
  • 42. But tending data is not new
  • 44. From the Swift perspective, if you want to find the new opportunities for change ... Focus on speed and people. We can work together faster than ever before.
  • 45. Human-curated data can happen in realtime with the help of entity recognition and reconciliation. Really, they just help us work faster.
  • 51. WHO ARE THE EDITORS?
  • 54. GEEK OUT A SEC Every incoming is parsed into an object with attributes: 1. URI.body (the text of the url) 2. URI.rating (anyone can rate through a web UI) 3. URI.submitters (anyone who linked to it) 4. URI.history (every revision preserved) 5. URI.tags (added by humans and machines)
  • 56. PFIF people finder interface format used for 90,000 entries after Katrina
  • 57. Grassroots reporting: Database-driven journalism and data curation.
  • 61. Weather Related Disasters Urban and Rural Fire Earthquake Economic Crisis Homelessness War Possibilities with speed: Internal Displacement & Refugee Environmental Crisis Aviation Disasters Highway Accidents What to do in a ___ based on my location? Volcano Report a ____ location Missing Person Report a ___ accident. Structure Collapse ___ detection and reporting. Railroad Disasters Health Disasters “There was a ____. Are you ok?” alerts. Power Outage Neighborhood-level ____ warnings. Explosions Floods AIDS Famine Landslide Stampede Foot and Mouth Avalanche Fatal Accident Tsunami Disaster Bird Flu
  • 62. Swift’s realtime strengthens the Ushahidi alert cycle Subscribe to disaster keyword based alerts about an emergency eg: “san francisco, earthquake”
  • 64. See Also Crimson Hexagon state of the art NLP. VRA's GeoMonitor, a natural language parser that reads the headlines of Reuters and AFP news wires and codes to state quot;who did what, to who, where and when?quot; Rated quot;virtually identicalquot; to human event summarizing. JRC's European Media Monitor, which can parse thousands of different news sources but faces limitations since analysts still need to read each article to understand the nature of the terrorist event. http://emm.jrc.it/NewsBrief/clusteredition/en/latest.html Tabari text parsing engine, quot;event data coder that has been used in at least five NSF-sponsored projects and produced data used in a number of refereed articles in political science.quot; FORECITE Forecasting of Crises and Instability Using Text-Based Events, developed by the US Center for Army Analysis (CAA) UN's HEWSweb A relatively new early warning system Biowarn Textual analysis with infectious disease focus Cewarn FAST Comprehensive system by Swisspeace
  • 65. A minimal implementation: an analogy We’ll put a card table at the public library and work on reports with highlighters, put everything into a card catalog and leave it in the library.

Notes de l'éditeur