3. Motivation
People are sharing massive amounts of
information on the web
(Twitter, Flickr, Facebook, …)
How to do effective data consumption, not just
data creation
Geo-spatial situation awareness
Real time updates of the world state
From data to actionable knowledge
4. Concept
Understanding evolving world situations
by combining spatio-temporal-thematic
data coming from social media (e.g.
Twitter/Flickr).
„Iphone‟ social image for mainland USA. Jun 11, 2009
5. Social Pixels
Traditional Pixels
Photons aggregating at locations on CCD
Social Pixels
User interest aggregating at geo-locations
Create social Image, social Video…
Image/Media Processing operators Situation
Detection operators (e.g.
convolution, filtering, background subtraction)
6. Design principles
Humans as sensors
Social pixel approach
Visualization
Intuitive
query and mental model
Common spatio-temporal data representation
Data analysis using media processing
Combining media processing with declarative
query algebra
7. Overall Approach
1. Micro-event detection
2. Spatio-temporal aggregation using social pixel approach
3. Media processing engine
4. Query engine
8. Micro-event detection
Simple bag-of-words approach for detecting
what event is the user talking about.
e.g. „Sore throat‟, „Flu‟, „H1N1‟, …
Tweet: „caught sore-throat today…arrrgh !‟
Micro-event detected for user X.
Spatial
Temporal
Thematic
9. Spatio-temporal aggregation
using social pixels
Higher level abstractions have trade-offs with
lower level details
Percolate up what is necessary for the
application
Can be:
Count of tweets with the term
Average green channel value of images
Mean audio energy
Average monthly income, rainfall, population etc.
10. Data Model
Spatio-temporal element
stel = [s-t-coord, theme(s), value(s), pointer(s)]
E-mage
g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V =
N)
Temporal E-mage Set
TES= {(t1, g1), ..., (tn, gn)},
Temporal Pixel Set
TPS = {(t1, p1), ..., (tn, pn)},
12. 1. Selection Operation
Select part of E-mage based on predicate P
Input: Temporal E-mage Set TES =
{(t1, g1), …, (tn , gn)}
Output: Temporal E-mage Set TES‟
Spatial or Value predicate Pi on Emage
Pi(TES) =
{(t1, Pi(g1)), …, (tn, Pi(gn))}, where Pi(g) =
{(x, y) | y=g(x), if Pi(x,y) is true;
y=0, otherwise}
Boolean predicate Pt on time
Pt(TES) = {(t1‟ g1‟), …, (tm‟, gm‟)}, where P(ti‟)
is true, e.g. date = „2010-03-10‟
13. Selection Examples
Show last one week‟s E-mages of California
for topic „Obama‟
R=cal t <= 1wk theme= Obama(TES)
14. 2. Arithmetic Operation
Binary operations between two (or more) E-
mage Sets
(g1, g2) = g3(x, (v1(x), v2(x))), where
{+, -, *, /, max, min, convolution}, g1 and g2 are
the same size.
Example:
TES1=Temporal E-mage Set for „Unemployment
rate‟
TES2=Temporal E-mage Set for „normalized Gas
prices‟
TES3= (TES1, TES2)
15. 3. Aggregation Operation α
Aggregates multiple E-mages in TES based on
function .
(g1, g2) = g3(x, (v1(x), v2(x))), where
{+, *, mean, max, min}, g1 and g2 are the same
size.
Example:
Show the average emage of last one week‟s
emages from California for Obama.
α mean ( R=cal t <= 1wk theme= Obama(TES))
16. 4. Grouping Operation
Group stels in an E-mage g based on certain
function f
Input: Temporal E-mage Set TES = {(t1, g1), …, (tn
, gn)}
Output: Temporal E-mage Set TES‟
Function f essentially splits g, into multiple sub-e-
mages.
f(TES) = f((t1, g1)) … f((tn,gn)), where
f((ti, gi)) = {(ti , gi1‟), …, (ti , gik‟)}, and each gij‟ is a
sub-E-mage of g based on f
f {segmentation, clustering, blob-detection, etc.}
17. Grouping Example
Identify 3 clusters for each E-mage in the TES
set having last one week‟s E-mages of
California.
clustering, n=3( R=cal t <= 1wk(TES))
18. 5a. Characterization Op. (Spatial)
Represent each E-mage g based on a
characteristic C, and store result as a stel.
Input: Temporal E-mage Set TES =
{(t1, g1), …, (tn, gn)}
Output: Temporal Pixel Set TPS =
{(t1, p1), …, (tn, pn)}
C(TES) = {(t1, (g1)), …, (tn, (gn))}, where
(gi) is a pixel characterizing gi
C
{count, max, min, sum, average, coverage, epi
center, density, shape, growth_rate, periodicity
}
19. Characterization Examples
(Spatial)
Find the epicenter of each cluster E-mage in
the last one week‟s E-mages of USA from TES
epicenter ( clustering, n=3( R=USA t <= 1wk
theme=Obama(TES))
20. 5b. Characterization Op.
(Temporal)
Characterize a temporal pixel set, which is the
result of E-mage characterization
Input: Temporal Pixel Set TPS =
{(t1, p1), …, (tn, pn)}
Output: Temporal Pixel Set TPS‟
(TPS) = {(tk , ((t1, p1), …, (tk, pk))) | k
[2, n]}, where
{displacement, distance, velocity, speed, accel
eration, linear extrapolation, exponential
growth, exponential decay, etc.}
21. Temporal Characterization
Examples
Find the velocity of epicenter of each cluster E-
mage over the last one week‟s E-mages of
California from TES for theme Katrina
velocity ( epicenter ( clustering, n=3( R=Cal t <= 1wk theme =
Katrina (TES))))
22. 5. Pattern Matching
Pattern Matching (Spatial)
Compare the similarity between each E-mage
and a given pattern P
Input: Temporal E-mage Set TES =
{(t1, g1), …, (tn, gn)}, and pattern P
Output: Temporal Pixel Set TPS
P(TES) = {(t1, p1), …, (tn, pn)}, where each value
in pi represents the similarity between the E-mage
and the given pattern
Patterns (i.e. Kernels) can be loaded from a
library or be historical data samples.
23. Pattern Matching
Temporal Pattern matching:
Compare the similarity of the temporal value
changing with a given pattern, e.g.
„increasing‟, „decreasing‟, or „Enron‟s stock in
1999‟, …
Input: Temporal Pixel Set TPS =
{(t1, p1), …, (tn, pn)}, and a pattern P
Output: Temporal Pixel Set TPS‟
P(TPS) = {(tn , p)}, where v(x) in p is the
similarity value
24. Pattern Matching Examples
Compare the similarity between each E-mage
in the last one week‟s E-mages of California
from TES with radial decay
radial_decay( R=cal t <= 1wk theme = Obama (TES))
How close is the similarity above to pattern of
“Enron‟s stock price in 1999”?
Enron‟s stock( radial_decay( R=cal t <= 1wk(TES)))
25. Situation detection operators
S. No Operator Input Output
1 Selection Temporal Temporal
E-mage Set E-mage Set
2 Arithmetic & K*Temporal E-mage Temporal E-mage Set
Logical Set
3 Aggregation α Temporal E-mage set Temporal E-mage Set
4 Grouping Temporal E-mage Set Temporal E-mage Set
5 Characterization :
•Spatial •Temporal E-mage Set •Temporal Pixel Set
•Temporal •Temporal Pixel Set •Temporal Pixel Set
6 Pattern Matching
•Spatial •Temporal E-mage Set •Temporal Pixel Set
•Temporal •Temporal Pixel Set •Temporal Pixel Set
27. Implementation and results
Twitter feeds
Geo-coding user home location
Loops of location based queries for different
terms
Over 100 million tweets using „Spritzer‟ stream
(since Jun 2009), and the higher rate
„Gardenhose‟ stream since Nov, 2009.
Flickr feeds
API
Tags, RGB values from >800K images
33. iPhone theme AT&T
based e-mage, retail
Jun 2 to Jun 11 locations
. Convolution
Store
+ Aggregation
* catchment
area
Difference
Aggregate
interest Combination of operators
-
AT&T
total
catchmen
t area
<geoname>
Convolution
.
MAXIMA <name>College City</name>
Decision <lat>39.0057303</lat>
<lng>-122.0094129</lng>
Best Location is at <geonameId>5338600</geonameId>
*
<countryCode>US</countryCode>
Geocode [39, - <countryName>United
States</countryName>
122] , just north of <fcl>P</fcl>
Bay Area, CA <fcode>PPL</fcode>
<fclName>city, village,...</fclName>
<fcodeName>populated
place</fcodeName>
<population/>
Under-served <distance>1.0332</distance>
</geoname>
interest areas Store catchment
38. Year average Peak of green
At [35, -84], at the junction of Chattahoochee National Forest, Nantahala
National Forest, Cherokee National Forest and Great Smoky
Mountains National Park
39. Variations throughout the year
Total Energy
Jan Dec
Fall colors of New England
[R-G] channel data
0
Jan Dec
40. Conclusions
Combining spatio-temporal event data for
visualization, and analytics.
An e-mage representation of spatio-temporal
thematic data coming in real-time.
Defined operators for real-time situation
analysis
Applications in multiple domains
41. Challenges: Future work
Defining a (visual) query language using
operators
Scalability
Realtime data management for all possible topics
which user might be interested in
Automatic tweets from sensors
A reverse-911 like control/recommendation
mechanism
Creating an event web by connecting all event
related data