Crisis-affected populations are often able to maintain digital communications but in a sudden-onset crisis any aid organizations will have the least free resources to process such communications. Information that aid agencies can actually act on, ‘actionable’ information, will be sparse so there is great potential to (semi)automatically identify actionable communications. However, there are hurdles as the languages spoken will often be underresourced, have orthographic variation, and the precise definition of ‘actionable’ will be response-specific and evolving. We present a novel system that addresses this, drawing on 40,000 emergency text messages sent in Haiti following the January 12, 2010 earthquake, predominantly in Haitian Kreyol. We show that keyword/ngram-based models using streaming MaxEnt achieve up to F=0.21 accuracy. Further, we find current state-of-the-art subword models increase this substantially to F=0.33 accuracy, while modeling the spatial, temporal, topic and source contexts of the messages can increase this to a very accurate F=0.86 over direct text messages and F=0.90-0.97 over social media, making it a viable strategy for message prioritization.

1. Subword and spatiotemporal models
for identifying actionable information
in Haitian Kreyol
Robert Munro
Stanford University
CoNLL 2011
Munro, Robert. "Subword and spatiotemporal models for identifying actionable
information in Haitian Kreyol." Proceedings of the Fifteenth Conference on
Computational Natural Language Learning. Association for Computational Linguistics,
2011.
http://www.robertmunro.com/research/munro11kreyol.pdf

2. January 12, 2010

3. Messages start streaming in

4. Messages start streaming in

5. 80,000 messages

6. 80,000 messages

7. 80,000 messages

8. 80,000 messages

9. 80,000 messages

10. 80,000 messages

11. Crowdsourced (Mission 4636)

12. Feedback
 US Marines
◦ “Saving lives every day.”
 FEMA:
◦ “The most comprehensive and up-to-date
map available to the humanitarian
community.”
 World Food Program
◦ “We delivered food to an informal camp of
2500 people that you identified for us.”

13. Sudden-onset language
processing

14. Sudden-onset language
processing

15. Sudden-onset language
processing

16. Sudden-onset language
processing

17. Prioritization
 Only 2% of messages were
‘actionable’
◦ An identifiable location
◦ Medical, S+R, water, clustered food
requests, security, unaccompanied
children.
 How can we prioritize the
actionable items in the original
Haitian Kreyol?
 Can we leverage the models for

18. Evaluation data
 Mission 4636. 40,811 text-messages
sent to a free number, ‘4636’, in Haiti.
(predominantly in Haitian Kreyol, with
translations, UN-defined categories,
and geolocation)
 Radio Station. 7,528 text-messages
sent to a Haitian radio station.
 Twitter. 63,195 Haiti-related tweets.

19. Variation is the norm
mesi mèsi mèci meci merci
Kreyol French
Abbrv. Full Form Pattern Meaning
s’on se yon sVn is a
av`en av`eknou VvVn with us
relem rele mwen relem call me
wap ouap uVp you are
map mwen ap map I will be
zanmimzanmi mwen zanmim my friend
lavel lave li lavel to wash (it)

20. Evaluation data

21. Combining spatio-temporal
models

22. Streaming architecture
 Build from initial items
time
Model

23. Streaming architecture
 Predict (and evaluate) on incoming
items
◦ (penalty for training)
time
Model

24. Streaming architecture
 Repeat / retrain
time
Model

25. Streaming architecture
 Repeat / retrain
time
Model

26. Streaming architecture
 Repeat / retrain
time
Model

27. Streaming architecture
 Repeat / retrain
time
Model

28. Streaming architecture
 Repeat / retrain
time
Model

29. Features
 G : Words and ngrams
 W : Subword patterns
 P : Source of the message
 T : Time received
 C : Categories (c0,...,47)
 L : Location (longitude and latitude)
 L : Has-location (a location is written
in the message)

30. Features
 Subword models
◦ Full-forms and normalizations (Munro and
Manning 2010)
Abbrv. Full Form Pattern Meaning
s’on se yon sVn is a
av`en av`eknou VvVn with us
relem rele mwen relem call me
wap ouap uVp you are
map mwen ap map I will be
zanmimzanmi mwen zanmim my friend
lavel lave li lavel to wash (it)

31. Time / Space / source
 Timestamp (in place of discounting)
 Phone-number of sender
 Spatial tile-membership:

32. Additional streaming models
 Message contains an identifiable
location
time
Model

33. Additional streaming models
 Message contains an identifiable
location
 Prediction for 47 categories
time
Model
time
Model
time
Model
time
Model
…

34. Final streaming model
 Prediction for ‘is actionable’
time
Model

35. Final streaming model
 Prediction for ‘is actionable’
time
Model
time
Model
time
Model
time
Model
…
Combines features with predictions
from Category and Has-Location
models

36. Evaluation
 100 training epochs
◦ Calculated on predictions over epochs 2-
100
 Comparison of two-tier architecture
with Oracle ‘has-location’ and
‘Categories’
 Identification of actionable messages
in Radio Station and Twitter messages
 (full results in paper)

37. Feature-based improvements
Subwords and Source (G, T,W, P)
Temporal feature (G, T)
Words/Ngrams (G)
0.326
0.252
0.207

38. Combined models
All Features/Models
Spatial clusters(L)
Words/Ngrams (G)
0.855
0.756
0.207

39. Outperforming the oracle
Location (two-teir prediction)
Location (oracle)
Words/Ngrams (G)
0.310
0.274
0.207

40. Negative results from filtering
Oracle true-negfiltering
All Features/Models
Words/Ngrams (G)
0.428
0.855
0.207

41. Social media
Twitter
Radio station
All Features/Models
0.969
0.904
0.855

42. Conclusions (usability)
 Subword and spatio-temporal models
can give a 10-fold increase in
prioritization
 Adding multi-tiered streaming models
can give a 50-fold increase in
prioritization
 Cross-domain adaptation is possible
for need(le)-in-haystack information
extraction from social media

43. Questions?

44. Appendix: abstract
Crisis-affected populations are often able to maintain digital
communications but in a sudden-onset crisis any aid organizations
will have the least free resources to process such communications.
Information that aid agencies can actually act on, ‘actionable’
information, will be sparse so there is great potential to
(semi)automatically identify actionable communications. However,
there are hurdles as the languages spoken will often be
underresourced, have orthographic variation, and the precise
definition of ‘actionable’ will be response-specific and evolving.
We present a novel system that addresses this, drawing on 40,000
emergency text messages sent in Haiti following the January 12,
2010 earthquake, predominantly in Haitian Kreyol. We show that
keyword/ngram-based models using streaming MaxEnt achieve up to
F=0.21 accuracy. Further, we find current state-of-the-art subword
models increase this substantially to F=0.33 accuracy, while
modeling the spatial, temporal, topic and source contexts of the
messages can increase this to a very accurate F=0.86 over direct
text messages and F=0.90-0.97 over social media, making it a viable
strategy for message prioritization.