Minimizing cost in distributed multiquery processing applications
•Télécharger en tant que ODP, PDF•
0 j'aime•632 vues
A brief overview of several methods to optimize data movement in distributed multi-query processing applications.
![Outline ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (19)
Similaire à Minimizing cost in distributed multiquery processing applications
Similaire à Minimizing cost in distributed multiquery processing applications (20)
Plus de Luis Galárraga
Plus de Luis Galárraga (15)
Dernier
Dernier (20)
Minimizing cost in distributed multiquery processing applications
- 1. Minimizing Communication Cost in Distributed Multi-query Processing Jian Li, Amol Deshpande, Samir Khuller Department of Computer Science, University of Maryland Presented by: Luis Galárraga Saarland University July 7th, 2010
- 10. Tree topology
- 15. Distributed stream processing applications Common need: Minimize data movement!!
- 17. Justification
- 19. Tree topology
- 24. No restrictions on data size.
- 25. Query plans are part of the input.
- 26. Intermediate results sizes are known.
- 28. Topology, undirected weighted graph
- 29. Assignment of relations to nodes in the topology.
- 31. Each query comes with a plan in the form of a directed tree. Destination node Data sources involved Data size S i S j S i x S j w z(S i ) z(S j ) z(S i x S j )
- 32. More formally Given the topology graph G c and a set of trees representing the query plans, our goal is to find a data movement plan that minimizes the total communication cost incurred while executing the queries.
- 33. Problem formulation Topology G c Queries (10) S 1 S 2 S 1 x S 2 C (10) (7) S 4 S 1 x S 2 x S 4 (5) (100) (100) S 2 S 6 S 2 x S 6 D (10) (5) S 2 S 5 S 2 x S 5 B (10) (6) (8) B A C D E F S 2 S 1 S 3 S 4 S 6 S 5 (10) (10) (100) (8) (100)
- 42. For general topologies, aproximation algorithms are known.
- 48. Hyperedges can group any number of vertices.
- 50. Find a flow or mapping of maximum value:
- 51. Max-flow/Min-cut 3 / 3 2 / 3 2 / 2 3 / 3 0 / 2 1 / 4 2 / 2 3 / 3 Flow Capacity
- 53. The maximum value of an s-t flow is equal to the minimum capacity of an s-t cut.
- 54. Max-flow/Min-cut 3 / 3 2 / 3 3 / 3 0 / 2 1 / 4 2 / 2 3 / 3 2 / 2 Flow Capacity
- 58. w Max-flow/Min-cut in hypergraphs
- 60. We want to find a partition (S, T) on V such that:
- 71. Tree topology – Step 2 – Single query G c u G c v S 1 S 2 C S 4 (10) S 1 x S 2 (10) (7) S 1 x S 2 x S 4 (5) (100) H D B A C D E F S 2 S 1 S 3 S 4 S 6 S 5 (10) (10) (100) (8) (100) C C D C
- 75. Tree topology – Step 2 – Multiple queries S 1 S 1 x S 2 C (10) (7) S 4 S 1 x S 2 x S 4 (5) (100) (100) S 6 S 2 x S 6 D (5) S 2 S 5 S 2 x S 5 B (10) (6) (8)
- 79. Then add the edges according to this rule (for every edge in G c ):
- 88. A query-overlap graph H is a graph where the vertices correspond to the set of data items and each edge corresponds to a pair query.
- 92. Otherwise to Connected Facility Location Problem Isn't this problem NP-Hard? Yes, but good approximation algorithms do exist!!
- 94. Output: Bought edges (Steiner cost) Rented edges (Connection cost)
- 95. Connected Facility Location Problem Demands, D Facilities, F Rented edges
- 101. Tree topology
- 104. Multi-query approach compared with isolated optimization per query.
- 105. Normalized with the cost of the naïve approach.
- 106. Identical data size vs tri-modal distribution
- 108. Tree topology
- 109. Arbitrary topology – Pairs problem
- 112. Tree topology
- 118. Thank you!!
Notes de l'éditeur
- If more than one source resides in a node then we can just create a node per relation and link the nodes with weighted 0 edges. In the case of replication, it becomes part of the query plan optimization. In that case the tree query plan given to the algorithm should be the minimum weighted tree (of course only considering the weight edges in the topology). The shortest paths for every pair of nodes might be precomputed (we could use associativity for joins of 3 relations), then we only care about taking the groups of replicas such that their distance is the smallest, furthermore this is only done in the leaves of the query plan.
- Min-cut can be solved in polynomial time. Edmonds-Karp solves the problem in O(|V| * |E| ^2) Max-flow: Ford-Fulkerson O(|V| * f) Dinitz algorithm: O(|E| |V| ^ 2)