This document discusses big graph analytics engines and challenges in graph analysis. It describes Huawei's growth and collaboration efforts. Key challenges in graph analysis include very large graph sizes, irregular data access patterns, and real-time requirements. The document proposes a graph platform for smart big data that can manage graph topology and properties, provide basic graph algorithms and analytics, and support streaming graph updates and knowledge graph inference. Preliminary experiments show decent speedup for graph preprocessing and PageRank computation using the proposed graph data sharding and edge-set representation.
10. 10
Challenges
● Very large scale graphs for analysis
• 10B~1000B in terms of the number of vertices
• a few hundreds of properties, static and dynamic
• distributed communication introduces additional overhead
● Irregularity in graph data access
• Low data locality results in high disk/communication IO overhead
• Data access patterns are diverse among graph analysis algorithms
● Near real-time requirement
• Incorporate with incremental graph updates
• Approximate query & analysis should be considered
● Efficiency and productivity to balance
11. 11
Graph Platform for Smart Big Data
Infrastructure
Data
Management
Graph engines
Visualization
Analytics
Single Machine Cluster GPU Server Cloud
Structure
Management
Property
Management
Metadata
Management
Permission
Control
Basic Engine
Streaming Graph Graphical Model Hyper Graph
Bayes NetCommunity
Label propagationCentrality
Anomaly detection
Matching
Ego Feature
Max Flow
Dynamic Graph Vis Property Vis Large Graph Vis
Incremental Update
18. 18
Conclusion
● Many big data problems involve links among a lot of entities,
naturally represented as a graph
● Property graph is highly expressive
● Industry is looking for graph/graphical model engines for complex
network analysis, streaming graph, probabilistic graphical models,
and RDF graph computing
● Efficiency is the key in many industry graph analysis systems,
especially when the data volume is big
● Eventually, the graph engine should serve for AI Business systems