With the torrent of data available to us on the Internet, it's been increasingly difficult to separate the signal from the noise. We set out on a journey with a simple directive: Figure out a way to discover emerging technology trends. Through a series of experiments, trials, and pivots, we found our answer in the power of graph databases. We essentially built our "Emerging Tech Radar" on emerging technologies with graph databases being central to our discovery platform. Using a mix of NoSQL databases and open source libraries we built a scalable information digestion platform which touches upon multiple topics such as NLP, named entity extraction, data cleansing, cypher queries, multiple visualizations, and polymorphic persistence.

1. Discovering Emerging
Technology Through
Graph Analysis
GraphConnect | Chicago
June 2013

2. About Me
henry74@gmail.com || henry.hwangbo@us.pwc.com
@henry74
henry74
Founder / Director of PwC's Emerging Tech Lab

3. What is the Emerging Tech Lab?
We build stuff to help people get smart about applying technology to
solve problems
● Founded 3 years ago to identify and experiment with new
technologies relevant to but not widely adopted by the Enterprise
● Focuses on rapid prototyping & MVP build-outs for both
tactical internal projects and more creative, exploratory ideas
● Permanent core team, but operates a rotational program for
staff to provide them an opportunity for hands-on technical
experience, learning agile & lean principles, and exposure to a
startup-like environment

4. The Challenge

5. It usually starts with an idea…
“Build a platform to help discover emerging technologies.”

6. …followed by some pretty mock-ups…
…to raise expectations.

7. Envisioning success
● What are some emerging
technologies?
● How are they being used to solve
real problems?
● Who is talking about them?
● Who are the players?
● Are there related technologies?
● Get up to speed quickly
● Discover related topics
● Understand what is trending
● Find interesting applications
● See what's possible

8. What makes technology “emerging”?
● Cannot already be mainstream technology
● Needs to be more than a single event to be an emerging trend
● Must be growing in popularity, but not yet popular
● "Technology" could be a thing (e.g. nanotubes), but also an
aggregation or application of technologies (e.g. cloud
computing, quantified self)

9. The Journey

10. Initial design
Data Feeds
(RSS)
Pull &
Store Raw
Data
MongoDB
Analyze VisualizeSource
?
Postgres

11. Breaking ground
● Natural Language Processing
● Named Entity Recognition
● ???
● ???
● ???
● ???
● ???
Extract Text
Understand
Text
Discover
Insights

12. A bit more clarity
Data Feeds
(RSS)
Pull &
Store Raw
Data
MongoDB
Analyze VisualizeSource
?
3rd Party
APIs
Tag &
Update
Postgres

13. Digging a little deeper
● Natural Language Processing
● Named Entity Recognition
● Collocation?
● K-means clustering?
● Information Ontology?
● ???
● ???
Extract Text
Understand
Text
Discover
Insights

14. The Eureka moment...
…took a bit longer than it should have
Graphs are everywhere

15. Final design
Data Feeds
(RSS)
Pull &
Store Raw
Data
MongoDB
Analyze VisualizeSource
3rd Party
API
Tag &
Update
Neo4j Postgres

17. Lesson #1 - Graph data modeling is iterative
What should be a node, relationship, or a property? Depends on:
● What will you search on?
● How do you start your searches?
● How much data do you expect to have? What data?
Expect to change your graph based on:
● Experimentation
● Query syntax available to extract and aggregate graph data
● Query performance
TIP: Plan to reload your graph many times - save the raw data, start small,
use batch loading until you get it right
…but more flexible than traditional data modeling

18. Modeling the data
DO
C
P
P
C
K
K
C
T
C
DOC
P
P
C
K
K
O
T
Document are described by its
entities, concepts, and keywords
through relationships
This means:
● Document are related to other
documents through shared
entities, concepts, and keywords
● Concepts and entities are related
to each other through shared
documents
● Incoming relationships measures
# of referring documents
Simple, yet powerful
TAGGED_AS
RELATES_TO
REFERS_TO
CONTAINS
REFERS_TO

19. Lesson #2 - Connections are important
Highly connected data creates richer
graphs and increases potential for
discovering greater insights
BUT unnecessary connections can
create noise & extra work
Don't create artificial connections, but clean up data before importing when it
makes sense (e.g. networking, networks, network)
Prevent duplication which can impact your insights based on aggregation (e.g.
# of relationships) or certain patterns

20. Keeping it clean
Techniques Graph Benefits
Text extraction with
readability scoring
● Better named entity extraction
● Improve neighbor relevance
● Minimize invalid nodes & relationships
Similarity Hashing
● Improve validity of relationships
● Increase graph connectedness
Porter Stemming ● Improve graph connectedness

21. Lesson #3 - Understand Cypher
● Cypher experimentation opens up the possible
● SQL users will be at home - tabular results, similar
syntax
● Start without parameters, check with Neo4j shell,
move to parameterized queries for security &
performance (caching)
● Don't forget Lucene syntax
● Continues to evolve for the better - check new release
changes (http://docs.neo4j.org/refcard/1.9/)
● Let Cypher do the work

22. Useful Cypher Syntax
START with an index
MATCH defines your universe
WHERE filters it down
WITH combines multiple statements
HAS checks if a property exists
AS lets you name your return values
IN checks against an array
COLLECT aggregates into an array
ORDER just like SQL
LIMIT for performance

23. Prototype highlights
● 4 people & 4 months (first version)
● Data Stores - Neo4J, MongoDB, Redis, Postgres
● Visuals - D3.js, Vivagraph.js, Twitter Bootstrap
● Key Languages/Libraries - Ruby, Rails, Cypher,
Knockout.js, Amplify.js, HTML5, CSS3, jQuery,
Neography gem, Resque gem
● 3rd Party - Alchemy, OpenCalais, RSS feeds,
Wikipedia
● Concepts - natural language processing, named
entity extraction, text cleansing & de-duplication
(map/reduce), similarity hashing, large-scale
information retrieval
● 1M+ nodes, 3M+ relationships, 6M+ properties after
6 months

24. Emerging Tech Radar Demo

25. Tag Cloud / Search
DOC C
K
K
C
DOC
C
K
K
DOC
DOC
DOC
DOC
● Index keywords and search across keywords (tip: use Lucene syntax)
● Identify documents with strong relationships to keywords
● Locate concepts with strongest relationships to relevant documents
● Popularity based on number of incoming relationships

26. Emerging Index / Popularity / Doc List
DO
C
CDOC
(E)
OC
DOC
(NE)
DOC
(E)
DOC
(E)
DOC
(NE)
DOC
(E)
DOC
(NE)
DOC
(E)
Cloud computing (Concept) and Google (Org)
● Strong relationships with documents shared between concepts to filter
and rank relevant content
● Ratio and strength of relationships to quantify emerging index
● Popularity based on number of incoming relationships of each type of
document (emerging versus non-emerging)

27. Node Graph
DO
C
CK DOC OC
DOC
DOC
DOC
DOC DOC
DOC
● Existing relationships with documents shared between concepts to
filter relevant neighbors
● Strength of relationships based on # and weight for ranking relevance
(color)
C

28. The Takeaway

29. Final Thoughts
● Graphs makes it simple to generate complex insights - you don't
need to be a data scientist
● Graphs are a natural fit for anything connected...which is most
things (e.g. social media, internet of things, sensor data)
● Experimentation is the best way to learn the power of graphs
● Make graph databases a first class citizen in your technology
toolkit - many things can be solved better with a graph
The best way to discover emerging technologies is to try
them out

30. Thanks for Listening - Q & A
Special thanks to Max De Marzi for his neography gem (https://github.
com/maxdemarzi/neography) and ongoing advice, suggestions,
troubleshooting