Recommandé
Recommandé
Contenu connexe
En vedette
En vedette (20)
Similaire à The Five Graphs of Government: How Federal Agencies can Utilize Graph Technology
Similaire à The Five Graphs of Government: How Federal Agencies can Utilize Graph Technology (20)
Dernier
Dernier (20)
The Five Graphs of Government: How Federal Agencies can Utilize Graph Technology
- 1. WASHINGTON D.C. FEBRUARY 28, 2017 09:00-09:30 09:30-10:15 10:15-11:00 11:00-11:30 11:30-12:15 12:30-13:30 13:30-17:00 Breakfast and Registration How Government Agencies use Neo4j to Build the Next Generation of Applications and Services Intelligence Analysis with Neo4j Break Leveraging the Graph for Knowledge Architecture at NASA Lunch Training Session Agenda
- 3. Fred Kagan David Mesa Chief Knowledge Architect for NASA Kimberly Kagan Director Critical Threats Project American Enterprise Institute President Institute for the Study of War Today’s Guest Speakers
- 4. “Life can only be understood backwards; but it must be lived forwards.” -Søren Kierkegaard
- 6. Yellowstone National Park Ecosystem Known Influences Entered One-at-a-Time (Willow)-[:HABITAT_FOR]->(Lincoln’s Sparrow) (Aspen)-[:FOOD_FOR]->(Beaver) (Beaver Ponds)-[:HABITAT_FOR]->(Beaver) (Deer)-[:BROWSE_ON]->(Cottonwood) (Berry Shrubs)-[:FOOD_FOR]->(Bears) …
- 7. Yellowstone National Park Ecosystem Known Influences Revealed as a Graph
- 8. MATCH path = (:Animal {Entity:"Wolves"})-[*]->(:Landscape {Entity:"Rivers"}) WITH extract(node IN nodes(path) | node.Yellowstone) AS factor, rand() AS number RETURN factor AS How_Wolves_Affect_RiverStability ORDER BY number LIMIT 5 Yellowstone National Park Ecosystem Query for Trophic Cascades Conclusion:
- 9. 1. Where do graph databases fit into the overall data landscape? 2. What is a graph database & when is it useful? 3. Be inspired to find your next graph in government Takeaways from this Session:
- 10. Source: http://dataconomy.com/2014/06/understanding-big-data-ecosystem/ Big Data Landscape “Big Data Landscape 3.0”
- 11. Discrete Data Minimally connected data All You Really Need to Know (at least for today) Other NoSQL Relational DBMS Neo4j Graph DB Connected Data Focused on Data Relationships
- 12. e of Graphs has created some of the most successful companies in the wo
- 13. “Graph analysis is possibly the single most effective competitive differentiator for organizations pursuing data-driven operations and decisions after the design of data capture.” By the end of 2018, 70% of leading organizations will have one or more pilot or proof-of-concept efforts underway utilizing graph databases. Analyst Perspective “Forrester estimates that over 25% of enterprises will be using graph databases by 2017” IT Market Clock for Database Management Systems, 2014 https://www.gartner.com/doc/2852717/it-market-clock-database-management TechRadar™: Enterprise DBMS, Q1 2014 http://www.forrester.com/TechRadar+Enterprise+DBMS+Q1+2014/fulltext/-/E-RES106801 Making Big Data Normal with Graph Analysis for the Masses, 2015 http://www.gartner.com/document/3100219
- 15. Empowering Journalists To Make Sense of Data Taking mankind to MarsHelping Cure Cancer 2016: A Year in Graphs
- 16. SOFTWARE FINANCIAL SERVICES RETAIL MEDIA & OTHER SOCIAL NETWORKS TELECOM HEALTHCAR E 2016: A Year in Graphs
- 17. Real-Time Recommendations Dynamic Pricing Artificial Intelligence & IoT-applications Fraud Detection Network ManagementCustomer Engagement Supply Chain Efficiency Identity and Access Management 2016: A Year in Graphs
- 18. Graphs in Government 5.Planning 4.Science & Education 2.Resource Management 3.Oversight 1.Security 2016: A Year in Graphs
- 19. Some Perspective We are still here Journeying to here
- 21. A way of representing data DATA DATA
- 22. Relational Database Good for: • Well-understood data structures that don’t change too frequently A way of representing data • Known problems involving discrete parts of the data, or minimal connectivity DATA
- 23. Graph Database Relational Database A way of representing data Good for: • Dynamic systems: where the data topology is difficult to predict • Dynamic requirements: the evolve with the business • Problems where the relationships in data contribute meaning & value Good for: • Well-understood data structures that don’t change too frequently • Known problems involving discrete parts of the data, or minimal connectivity
- 24. 27 A unified view for ultimate agility • Easily understood • Easily evolved • Easy collaboration between business and IT #1 Benefit: Project Agility The Whiteboard Model Is the Physical Model
- 25. Connectedness and Size of Data Set ResponseTime Relational and Other NoSQL Databases 0 to 2 hops 0 to 3 degrees Thousands of connections 1000x Advantage Tens to hundreds of hops Thousands of degrees Billions of connections Neo4j “Minutes to milliseconds” #2 Benefit: “Minutes to Milliseconds” Real-Time Query Performance
- 26. “We found Neo4j to be literally thousands of times faster than our prior MySQL solution, with queries that require 10-100 times less code. Today, Neo4j provides eBay with functionality that was previously impossible.” - Volker Pacher, Senior Developer “Minutes to milliseconds” performance Queries up to 1000x faster than RDBMS or other NoSQL #3 Benefit: “Minutes to Milliseconds” Real-Time Query Performance
- 28. At Write Time: data is connected as it is stored At Read Time: Lightning-fast retrieval of data and relationships via pointer chasing Index free adjacency Magic Ingredient #1 of 3: Graph Optimized Memory & Storage
- 29. MATCH (:Person { name:“Dan”} ) -[:MARRIED_TO]-> (spouse) MARRIED_TO Dan Ann NODE RELATIONSHIP TYPE LABEL PROPERTY VARIABLE Magic Ingredient #2 of 3: A Productive and Powerful Graph Query Language
- 30. 3 3 Example HR Query in SQL The Same Query using Cypher MATCH (boss)-[:MANAGES*0..3]->(sub), (sub)-[:MANAGES*1..3]->(report) WHERE boss.name = “John Doe” RETURN sub.name AS Subordinate, count(report) AS Total Project Impact Less time writing queries • More time understanding the answers • Leaving time to ask the next question Less time debugging queries: • More time writing the next piece of code • Improved quality of overall code base Code that’s easier to read: • Faster ramp-up for new project members • Improved maintainability & troubleshooting Magic Ingredient #2 of 3: A Productive and Powerful Graph Query Language
- 31. Graph Transactions Over ACID Consistency Graph Transactions Over Non-ACID DBMSs 34 Maintains Integrity Over Time Becomes Corrupt Over Time Magic Ingredient #3 of 3: ACID Graph Writes
- 33. Graphs in Government 5.Planning 4.Science & Education 2.Resource Management 3.Oversight 1.Security
- 34. Graphs in Government 5.Planning 4.Science & Education 2.Resource Management 3.Oversight Law Enforcement 1.Security Cyber Security Intelligence
- 35. “Don’t consider traditional technology adequate to keep up with criminal trends” Market Guide for Online Fraud Detection, April 27, 2015
- 36. Law Enforcement Use Case: Information and Data Synchronization in Law Enforcement Law Enforcement Agencies use Neo4j to model the information into graphs to improve efficiency and make direct and implicit patterns readily apparent in real time. A suspect often appears in several different databases Financial recordsConvictions Adresses Vehicles Traffic cameras Arrests Police Reports Agency Records Public Records Traffic Records SUSPECT The Graphs In Government
- 37. The Graphs In Government 01 Bystander investigated due to deep connection found Use Case: Modeling Graphs in Investigations Neo4j is used by LE to track all parts of criminal investigations, including witnesses, suspects, forensic evidence, and locations. All related directly and indirectly. Law Enforcement
- 38. INVESTIGATE Revolving Debt Number of Accounts INVESTIGATE Normal behavior Fraud Detection With Discrete Analysis
- 39. Revolving Debt Number of Accounts Normal behavior Fraudulent pattern Fraud Detection With Connected Analysis
- 40. The Graphs In Government 01 ACCOUNT HOLDER 2 ACCOUNT HOLDER 1 ACCOUNT HOLDER 3 CREDIT CARD BANK ACCOUNT BANK ACCOUNT BANK ACCOUNT ADDRESS PHONE NUMBER PHONE NUMBER SSN 2 UNSECURED LOAN SSN 2 UNSECURED LOAN Law Enforcement Use Case: Modeling Fraud Rings as Graphs Organizing a fraud ring in the real world is relatively simple. A group of people share their personal information to create synthetic identities. For example with just 2 individuals sharing names and social security numbers can create 4 different identities. This can be discovered with connected analysis.
- 41. Cyber Security Attack Analysis Source: http://neo4j.com/graphgist/40caddf1d7537bce962e/ https://linkurio.us/graph-data-visualisation-cyber-security-threats-analysis/ UDP storm attack Domain Model Connected Domains
- 42. Graphs in Government 5.Planning 4.Science & Education Asset & Inventory Management Supply Chain Network & IT Operations 2.Resource Management 3.Oversight 1.Security
- 43. Network & IT Operations • Impact & Dependency Analysis • Root Cause Analysis • Network Design • Network Security Analysis • Network Asset Management (CMDB)
- 44. Supply Chain Example from Industry
- 46. Asset Graph: Financial Ownership • Impact & Dependency Analysis • Risk Assessment • Compliance Enforcement
- 48. The Graphs In Government 01 Withdraw Use Case: Combating Money Laundering With Graphs Neo4j is used to combat advanced money laundering schemes. Money laundering is all about how funds travel across a network of parties. Without graph analysis capabilities, some of these patterns can be impossible to detect. Washed in complex series of transfers Anti-Money Laundering Deposit
- 49. The Graphs In Government 01 The Cali Cartel Money Laundering Scheme Money Laundering
- 50. Source: http://neo4j.com/blog/analyzing-panama-papers-neo4j/ Case Study: “The Panama Papers” • The International Consortium of Investigative Journalists (ICIJ) exposed highly connected networks of offshore tax structures used by the world’s richest elites. • With 11,5 million documents, it’s the largest financial leak of all times. • The unfolded connections in “The Panama Papers” was a major news story 2016. The Graphs In Government 01 Money Laundering
- 52. Coming soon…
- 53. Patents, Papers, and Legislation Graph-Based Search 1) Patentula Search Demo: https://www.youtube.com/watch?v=GpHSO5j8nvQ 2) Visualizing and searching relationships between academic papers using Neo4j Graph database http://dspace.thapar.edu:8080/jspui/bitstream/10266/4014/1/801432008_Karan_cse_16-final-.pdf
- 54. Graphs in Government 5.Planning Lessons Learned Dependency Management Consolidation 4.Science & Education 2.Resource Management 3.Oversight 1.Security
- 57. Predicting WWI [Easley and Kleinberg]
- 59. THANK YOU!
- 60. WASHINGTON D.C. FEBRUARY 28, 2017 09:00-09:30 09:30-10:15 10:15-11:00 11:00-11:30 11:30-12:15 12:30-13:30 13:30-17:00 Breakfast and Registration How Government Agencies use Neo4j to Build the Next Generation of Applications and Services Intelligence Analysis with Neo4j Break Leveraging the Graph for Knowledge Architecture at NASA Lunch Training Session Agenda
Notes de l'éditeur
- If there’s one thing to remember for today, it’s this journey
- Frederick Kagan, Director, Critical Threats Project at American Enterprise Institute, and Kimberly Kagan, President at Institute for the Study of War
- I’d like to start the day with a koan… What Kierkegaard is talking about is Causality. Oftentimes we don’t understand what the effects will be because they’re so complex. --- Let me show you one of my favorite examples of complex causality in nature.
- Here is what one gets if one does precisely that. Finding the paper, reading it, and expressing the links as a graph, was given to one of our summer interns, who was able to build this graph in an afternoon. Source: http://gist.neo4j.org/?0ac320c799ce55089377
- Here is what one gets if one does precisely that. Finding the paper, reading it, and expressing the links as a graph, was given to one of our summer interns, who was able to build this graph in an afternoon. Source: http://gist.neo4j.org/?0ac320c799ce55089377
- The beauty of the graph however is in the questions it enables us to answer. For example, here we ask: what are all of the paths between “Wolves” and “Rivers”. It turns out there are quite a few, but that all four of the paths leading immediately into “Rivers” have the effect of promoting the rivers. We can easily conclude therefore that wolves can be expected to have an overall salutary effect upon the rivers… which was, after 15 years of experimental science, found to be true.
- And deriving value from data-relationships is exactly what some of the most successful companies in the world have done. Google created perhaps the most valuable advertising system of all time on top of their search-enginge, which is based on relationships between webpages. Linkedin created perhaps the most valuable HR-tool ever based on relationships amongst professional And this is also what pay-pal did, creating a peer-to-peer transaction service, based on relationships.
- How many of you feel you have a handle on the Big Data Landscape? So apparently according to this article in Dataconomy, it’s simply a matter of memorizing this.
- The “fruit salad” slide earlier outlined technologies that were mostly focused on dealing with data in discrete chunks. What’s interesting about the right side is that some of the largest & most successful tech companies in the last decade were possible thanks to their use of graphs. And deriving value from data-relationships is exactly what some of the most successful companies in the world did. Google created perhaps the most valuable advertising system of all time on top of their search-enginge, which is based on relationships between webpages. Linkedin created perhaps the most valuable HR-tool ever based on relationships amongst professional And this is also what pay-pal did, creating a peer-to-peer transaction service, based on relationships.
- Continuing on – We have receive very solid validation from these industry watchers that the market we are pursuing represents a huge opportunity and being anointed as the leader in this market that is likely to grow at this rate is very exciting.
- “Neo4j is the most popular graph database on the planet, so we have a privileged view on the massive adoption of graph databases. As an example, Neo4j is today used in verticals as diverse as <click> Software, <click>, Financial Services, <click> Retail, etc… by some of the biggest companies on the planet, across a wide range of use cases.”
- First, not everyone in the room would know what a graph is.
- What this means for your data structure
- Kick off with discussing major trends happening in enterprises.
- The query asks: “Find all direct reports and how many people they manage, up to three levels down”
- Keeping Your Graph Intact is Essential for Graph Operations This is great… Now let’s talk about reads. Some applications are ok with somewhat stale data. Some are not. Causal consistency gives you the choice. Titan example: Ghost Vertices: If a vertex gets deleted while it is concurrently being modified, the vertex might re-appear as a ghost. Stale Index entries: Index entries might point to nonexistent vertices in case of partial mutation persistence. Half-Edges: Only one direction of an edge gets persisted or deleted which might lead to the edge not being or incorrectly being retrieved. Uni-directed Ghost Edges: A uni-directed edge points to a deleted vertex.
- First, not everyone in the room would know what a graph is.
- Its obvious that traditional technologies which were aimed at individuals and their behavior are inadequate to detect and prevent sophisticated fraud rings. So why is that?
- So let’s take a look on how Data Synchronization in Law Enforcement could work modeled in a graph. For example: We have a suspect that might have prior convictions, arrests, and figures in police reports, and this could be stored in agency records.. A suspect might appear in many different databases. However these systems are not designed to relate to each other and here Neo4j and a graph database approach would be a very effective tool to augment existing systems. Having graph search capabilities across this data opens up for both targeted searches and advanced connected analysis.
- Neo4j is used by LE to track all parts of criminal investigations, including witnesses, suspects, forensic evidence, and locations. All of this is related directly and indirectly. Therefor connected analysis can give Law Enforcement agents an important insight of who and what to investigate… even implicit connections could unravel patterns that weren’t available before.
- [In this simple fraud detection approach to detect credit card fraud, it is relatively easy to spot outliers. But what if the fraudster commits fraud while still exhibiting normal behavior. Well - this is exactly how fraud rings operate]
- [A fraud ring rarely strays outside the normal behavior band. Instead they operate within normal limits and commit widespread fraud. This is very hard to detect by systems that are looking for outliers or activities outside the normal band.]
- Another important area for Law Enforcement is Fraud. Organizing a fraud ring in the real world is relatively simple. A group of people share their personal information to create synthetic identities. For example with just 2 individuals sharing names and social security numbers can create 4 different identities. This is something that can be discovered with connected analysis and the use of graphs.
- Today, agencies need to augment their discrete analysis capability with connected analysis. Whether you’re dealing with a fraud ring or stolen and synthetic identities, it’s extremely powerful to use a graph database Normally, your operational data is loaded into your fraud detection application which then conducts a wide range of discrete analysis to help your internal team to detect fraud. Neo4j helps extend this capability with connected analysis. You can load some of the same information inside Neo4j. Neo4j’s native graph model stores both the data and its relationships which can help your team detect known fraud patterns as well as discover new ones.
- Top Queries: 1. Trace dependencies up from servers all the way to applications and users 2. Trace dependencies across virtual and physical layers of infrastructure 3. Identify routes & alternate paths between various points in the network 4. Find the best, shortest, or least busy path, the best location in the network to introduce a new service
- Because money laundering is all about how funds travel across a network of parties. Without graph analysis capabilities, some of these patterns can be impossible to detect. What we see here a simple sketch of how this could be modelled. It’s deposits from different people and money that gets washed in a complex series of transfers.
- This is a real example of such complexities, done by the Department of Treasury’s Office of Foreign Assets Control… unfolding the Money Laundering Scheme of the Cali Cartel. And what we see here is a highly connected network of accounts, assets and people, which is perfect for graph analysis.
- One case study that we are very proud of is enabling The International Consortium of Investigative Journalists (ICIJ) to expose highly connected networks of offshore tax structures, through what they call the Panama Papers.
- Mars Curiosity Rover
- In the retail-example… this would probably look something like this. You will have systems in place to perform different functions, all of them probably crucial and doing the job btw, The problem is that these systems are designed to perform a very specific task. So in order to put the data from all these systems to good use effectively, in the recommendations-example we just showed you, in real-time you need to add the graph database-layer.
- Because building a system based on a foundation that doesn't handle connections naturally, is extremely difficult, and it would require so much time and money, that it is virtually impossible to justify.
- That WWI can be predicted without domain knowledge by iterating a graph and applying local structural constraints is nothing short of astonishing to me. Note how the network slides into a balanced labeling — and into World War I. You can pull in other dimensions easily here: geopolitics, weapons technology, genealogy (because the royals were not exactly blameless in this), Graphs rock. Sometimes humans not so much.
- Thank you for listening!