DataXDay - Exploring graphs: looking for communities & leaders
•
0 j'aime•192 vues
Ever been stuck in a data science use case where any approach seems too hard? Graph theory, describing a system just in terms of nodes and links, could be your answer! In the practical example we’ll show, we’ll try to find data science communities and their leaders in LinkedIn. Challenge accepted? Aurélia Nègre & Alberto Guggiola - Quantmetry https://dataxday.fr/
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à DataXDay - Exploring graphs: looking for communities & leaders
Similaire à DataXDay - Exploring graphs: looking for communities & leaders (20)
Plus de DataXDay Conference by Xebia
Plus de DataXDay Conference by Xebia (6)
Dernier
Dernier (20)
DataXDay - Exploring graphs: looking for communities & leaders
- 8. @DataXDay Aurélia Nègre Data Scientist anegre@quantmetry.com Alberto Guggiola Data Scientist aguggiola@quantmetry.com Graph Theory … looking for communities & finding the leaders… DataXDay 17th May 2017
- 9. @DataXDay@DataXDay Who are we? § 70 Consultants (Data Scientists, Architects, Engineers, Consultants & more …) § From proofs of concept to production § Fraud detection, predictive maintenance, customer insights … Aurélia Nègre & Alberto Guggiola © Quantmetry 2018 | Diffusion interdite sans accord
- 11. @DataXDay@DataXDay Some use cases of graph theory Spreading • Determine the speed of a spreading phenomenon • How to speed it up or to slow it down? Viral marketing, vaccination campaigns Dynamics & optimisation • Shortest path between two nodes? • Effects of modifying the structure? Transportation systems, social networks Domino effects • Resilience to random failures? • And to targeted attacks? Security systems, economics, infrastructures Structural importance • Which nodes are the most important or authoritatives? Who are the leaders? Google PageRank algorithm © Quantmetry 2018 | Diffusion interdite sans accord
- 13. @DataXDay@DataXDay Community detection: looking for a structure Community: Region having some degree of autonomy -> No unique formal definition! © Quantmetry 2018 | Diffusion interdite sans accord
- 14. @DataXDay@DataXDay Community detection: looking for a structure Community: Region having some degree of autonomy -> No unique formal definition! Which communities interact with each other? Which elements act as « bridges » between communities? © Quantmetry 2018 | Diffusion interdite sans accord
- 15. @DataXDay@DataXDay Cutting the bridges Gathering the most connected elements Two approaches for finding clusters Spectral clustering, Girvan Newman Fastgreedy, Louvain, Walktrap © Quantmetry 2018 | Diffusion interdite sans accord
- 16. @DataXDay@DataXDay Girvan Newman: a good algorithm on small graphs (<500 nodes), but a very high complexity Walktrap : much more efficient on large graphs Two examples Random walk on a network: path following randomly chosen edges on the graph Community « strength »: proportional to the time a random walker spends inside it Cut the bridges: iteratively remove links with highest betweenness Community are found when the graph becomes disconnected © Quantmetry 2018 | Diffusion interdite sans accord
- 17. @DataXDay@DataXDay ✅ Able to identify heterogenous communities ✅ Efficient on large graphs: complexity O(N logN) ✅ Available in most graph analytical libraries: ok as first try And the winner is... Louvain algorithm © Quantmetry 2018 | Diffusion interdite sans accord
- 18. @DataXDay@DataXDay ✅ Able to identify heterogenous communities ✅ Efficient on large graphs: complexity O(N logN) ✅ Available in most graph analytical libraries: ok as first try And the winner is... Louvain algorithm Modularity optimization Density of edges inside vs outside clusters 𝑄 = 1 2𝑚 & 𝐴() − 𝑘( 𝑘) 2𝑚 𝛿 () (𝑐(, 𝑐)) Local to global greedy From groups of nodes … … to groups of clusters © Quantmetry 2018 | Diffusion interdite sans accord
- 19. @DataXDay@DataXDay • I measure the capability to reconstruct real, known communities • Example of metrics: Normalized Mutual Information I observe the truth: the known communities Testing the algorithms and measuring the performances I create the truth: the Stochastic Block Model • I define the probability for each couple of nodes to be connected • In the simplest case: 𝑝() = ? 𝐴 𝑖𝑓 𝑖, 𝑗 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑒 𝑐𝑜𝑚𝑚𝑢𝑛𝑖𝑡𝑦 𝐵 < 𝐴 𝑖𝑓 𝑛𝑜𝑡 • More links inside communities as a consequence • Many observations can be generated to test algorithms © Quantmetry 2018 | Diffusion interdite sans accord
- 20. @DataXDay@DataXDay Look at modularity of best solution: if <0.3, not a real community structure Possible causes: • On generated data, intra and inter-community probability of links are too close • On real networks, the known communities do not influence the structure • The approximated solution is too far from the global optimum Possible follow-up: • NLP + graphs: groups of people discussing about a certain topic But sometimes, there is just no pattern to be discovered … © Quantmetry 2018 | Diffusion interdite sans accord
- 22. @DataXDay@DataXDay Which node is the most important? © Quantmetry 2018 | Diffusion interdite sans accord
- 23. @DataXDay@DataXDay Different ways of measuring nodes importance A global importance : the betweenness centrality A local importance : the degree Is the node « well connected »? Count its number of direct neighbours Is the node a « bridge »? Count number of shortest paths passing through it A well known, iterative metric : Google PageRank -> Is the node connected to many important nodes ? © Quantmetry 2018 | Diffusion interdite sans accord
- 25. @DataXDay@DataXDay Can provide information on profiles of nodes Combining centrality metrics & identifiying hierarchies © Quantmetry 2018 | Diffusion interdite sans accord
- 27. @DataXDay@DataXDay Several tools, depending on your objectives Non distributed analytical libraries Distributed analytical libraries Databases © Quantmetry 2018 | Diffusion interdite sans accord
- 29. @DataXDay@DataXDay Demo Time using LinkedIn data © Quantmetry 2018 | Diffusion interdite sans accord
- 30. @DataXDay@DataXDay • 3 blog articles (in french): – Introduction à une théorie aux applications multiformes (Alberto Guggiola) – Détection de communautés : théorie et retour d’expérience (Aurélia Nègre) – Comment identifier les rôles stratégiques des influenceurs d'un réseau ? (Ysé Wanono) • https://www.quantmetry.com/blog To go further... © Quantmetry 2018 | Diffusion interdite sans accord
- 31. The video of this presentation will be soon available at dataxday.fr Thanks to our sponsors Stay tuned by following @DataXDay