Task graphs or dependence graphs are used in runtime systems to schedule tasks for parallel execution. In problem domains such as dense linear algebra and signal process- ing, dependence graphs can be generated from a program by static analysis. However, in emerging problem domains such as graph analytics, the set of tasks and dependences between tasks in a program are complex functions of runtime values and cannot be determined statically. In this paper, we introduce a novel approach for exploiting parallelism in such programs. This approach is based on a data structure called the kinetic dependence graph (KDG), which consists of a dependence graph together with update rules that incrementally update the graph to reflect changes in the dependence structure whenever a task is completed. We have implemented a simple programming model that allows programmers to write these applications at a high level of abstraction, and a runtime within the Galois system that builds the KDG automatically and executes the program in parallel. On a suite of programs that are difficult to parallelize otherwise, we have obtained speedups of up to 33 on 40 cores, out-performing third-party implementations in many cases.

1. Kinetic Dependence Graphs
M. Amber Hassaan,
Donald Nguyen,
Keshav Pingali
The University of Texas at Austin

2. Dependence Graph Scheduling
2
Application Programmer
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]

3. Dependence Graph Scheduling
3
Application Programmer Compiler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]

4. Dependence Graph Scheduling
4
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]

5. Dependence Graph Scheduling
5
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]

6. Dependence Graph Scheduling
6
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library

7. Dependence Graph Scheduling
7
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model

8. Dependence Graph Scheduling
8
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

9. Dependence Graph Scheduling
9
σ0
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

10. Dependence Graph Scheduling
10
G0
σ0
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

11. Dependence Graph Scheduling
11
G0
σ0
w0
G1
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

12. Dependence Graph Scheduling
12
G0
σ0
w0
G1
w1 w2
G2
Execute in Parallel
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

13. Dependence Graph Scheduling
13
G0
σ0
w0
G1
w1 w2 wn-1
G2 Gn…
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

14. Dependence Graph Scheduling
14
G0
σ0
w0
G1
w1 w2 wn-1
G2 Gn…
w0 σ1
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

15. Dependence Graph Scheduling
15
G0
σ0
w0
G1
w1 w2 wn-1
G2 Gn…
w0 σ1
w1 w2 wn-1σ2 … σn
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

16. Dependence Graph Scheduling
16
G0
σ0
w0
G1
w1 w2 wn-1
G2 Gn…
w0 σ1
w1 w2 wn-1σ2 … σn
Scheduler
World
Program
World
Application Programmer Compiler RuntimeScheduler
i
j
1
1
2
3
2 3
A[N,N] = ...
for i in 1:N
for j in 1:N
A[i,j] =
A[i-1,j] + A[i,j-1]
Parallel Task Library
Parallel Programming Model
…

17. Limitations of Dependence Graphs
• Not always sufficient
– Discrete event simulation
– Billiard ball simulation
– Kruskal’s algorithm for MSTs
– Asynchronous Variational
Integrators
– …
• Ordered algorithms
– Have tasks with algorithm-specific
order in which tasks must appear
to execute
• Can use speculation
– But overheads can be high for
ordered algorithms
17

18. Discrete Event Simulation
18
2
A
C
B
3
8

19. Discrete Event Simulation
19
A
C
B
3
8

20. Discrete Event Simulation
20
4
A
C
B
3
8

21. Discrete Event Simulation
21
4
A
C
B
3
8
rw-set

22. Discrete Event Simulation
22
4
A
C
B
3
8

23. Discrete Event Simulation
23
4
A
C
B
3
8

24. Discrete Event Simulation
24
4
A
C
B
8
5

25. Discrete Event Simulation
25
A
C
D
B
3
2
8

26. Discrete Event Simulation
26
A
C
D
B
3
2
8

27. Discrete Event Simulation
27
A
C
D
B
3
2 32
8
8

28. Discrete Event Simulation
28
4
A
C
D
7
B
2 3
8
8

29. Discrete Event Simulation
29
4
A
C
D
7
B
2 3
4 7
round 0
round 18
8
8

30. Discrete Event Simulation
30
6A
C
D
B
2 3
4 7
round 0
round 18
8
8

31. Discrete Event Simulation
31
6A
C
D
B
2 3
4 7
6
round 0
round 1
round 2
8
8
8
8

32. Discrete Event Simulation
• Inadequacy of dependence graphs
– Tasks are created dynamically
– Not all sources are safe to execute
– Task executionmay require updating DAG
32
6A
C
D
B
2 3
4 7
6
round 0
round 1
round 2
8
8
8
8

33. • A dependence graph is:
– A DAG G for the program state σ
– An update rule U to produce the next G after
executing task w
• Remove source
33
σ0
w0 w1 w2 wn-1
w0 σ1
w1 w2 wn-1σ2 … σn
G0 G1 G2 Gn…
Execute sources
Dependence
Graph
Dependence Graph

34. Kinetic Dependence Graph
34
σ0
w0 w1 w2 wn-1
w0 σ1
w1 w2 wn-1σ2 … σn
G0 G1 G2 Gn…
Execute sources
• A kinetic dependence graph is:
– A DAG G for the program state σ
– A safe-source test P
– An update rule U to produce the next G after
executing task w
• Remove source, update other tasks’ dependencies, …
Dependence
Graph

35. Kinetic Dependence Graph
35
σ0
w0 σ1
w1 w2 wn-1σ2 … σn
• A kinetic dependence graph is:
– A DAG G for the program state σ
– A safe-source test P
– An update rule U to produce the next G after
executing task w
• Remove source, update other tasks’ dependencies, …
<σ0, G0> <σ1, G1> <σ2, G2> <σn, Gn>
w0 w1 w2 wn-1
…
Dependence
Graph
Kinetic

36. Kinetic Dependence Graph
36
σ0
w0 σ1
w1 w2 wn-1σ2 … σn
• A kinetic dependence graph is:
– A DAG G for the program state σ
– A safe-source test P
– An update rule U to produce the next G after
executing task w
• Remove source, update other tasks’ dependencies, …
<σ0, G0> <σ1, G1> <σ2, G2> <σn, Gn>
w0 w1 w2 wn-1
…
Dependence
Graph
Kinetic
Execute safe sources

37. Safe Sources
• Application-specific cone of influence
37
time
x
y

38. Safe Sources
• Application-specific cone of influence
38
time
x
y

39. Safe Sources
• Application-specific cone of influence
39
• In some algorithms, all sources are safe
– Stable source algorithm
time
x
y

40. General KDG Executor
40
a,1
c,3 d,4
b,2

41. General KDG Executor
41
1. Construct
initial DAG
a,1
c,3 d,4
b,2

42. General KDG Executor
42
1. Construct
initial DAG
a,1 b,2
c,3 d,4
σ0 ,< >
a,1
c,3 d,4
b,2

43. General KDG Executor
43
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
a,1
c,3 d,4
b,2

44. General KDG Executor
44
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
a,1
c,3 d,4
b,2

45. General KDG Executor
45
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
a,1
c,3 d,4
b,2

46. General KDG Executor
46
a,1
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
a,1
c,3 d,4
b,2

47. General KDG Executor
47
a,1
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
a,1
c,3 d,4
b,2

48. General KDG Executor
48
a,1
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
a,1
c,3 d,4
b,2

49. General KDG Executor
49
a,1
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
a,1
c,3 d,4
b,2

50. General KDG Executor
50
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
c,3 d,4
b,2

51. General KDG Executor
51
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
c,3 d,4
b,2

52. General KDG Executor
52
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
c,3 d,4
b,2

53. General KDG Executor
53
c,3 d,4
b,2
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
,< >
a,1
c,3 d,4
b,2

54. General KDG Executor
54
c,3 d,4
b,2
e,0
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
e,0
,< >
a,1
c,3 d,4
b,2

55. General KDG Executor
55
c,3 d,4
b,2
e,0
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
e,0
,< >
a,1
c,3 d,4
b,2

56. General KDG Executor
56
c,3 d,4
b,2
e,0
a,1
1. Construct
initial DAG
2. Apply safe
source test
3. Execute
4. Updaterw-sets
5. Add new tasks
a,1 b,2
c,3 d,4
σ0 ,< >
b,2
c,3 d,4
e,0
σ1 ,< >
a,1
a,1
c,3 d,4
b,2

57. KDG Specializations
• Each step needs a barrier
• Task’s rw-set potentially updated
several times prior to execution
57
Opportunities
KDG Runtime

58. KDG Specializations
• Each step needs a barrier
• Task’s rw-set potentially updated
several times prior to execution
58
Opportunities Program Properties
Application ProgrammerKDG Runtime

59. KDG Specializations
• Each step needs a barrier
• Task’s rw-set potentially updated
several times prior to execution
59
• Sources
– In general: safe source test could
inspect entire state
– Local safe source test
– Stable source (sources are safe)
• RW-sets
– In general: task could change
other tasks’ RW-sets
– Non-increasing RW-set
– Structure-based RW-set
• New tasks
– In general: task could create new
tasks at any priority
– Monotonic
– No new tasks
Opportunities Program Properties
Application ProgrammerKDG Runtime

60. KDG Specializations
• Each step needs a barrier
• Task’s rw-set potentially updated
several times prior to execution
60
• Sources
– In general: safe source test could
inspect entire state
– Local safe source test
– Stable source (sources are safe)
• RW-sets
– In general: task could change
other tasks’ RW-sets
– Non-increasing RW-set
– Structure-based RW-set
• New tasks
– In general: task could create new
tasks at any priority
– Monotonic
– No new tasks
Opportunities Program Properties
• Remove unnecessary barriers
given program properties
• Construct KDG for prefix of tasks
• Use different DAG representations
Specializations
Application ProgrammerKDG Runtime

61. KDG Specializations
• Each step needs a barrier
• Task’s rw-set potentially updated
several times prior to execution
61
• Sources
– In general: safe source test could
inspect entire state
– Local safe source test
– Stable source (sources are safe)
• RW-sets
– In general: task could change
other tasks’ RW-sets
– Non-increasing RW-set
– Structure-based RW-set
• New tasks
– In general: task could create new
tasks at any priority
– Monotonic
– No new tasks
Opportunities Program Properties
• Remove unnecessary barriers
given program properties
• Construct KDG for prefix of tasks
• Use different DAG representations
Specializations
Application ProgrammerKDG Runtime

62. KDGs in Action
• Application programmer
– Writes programs with ordered
loops
• Loop body must use provided
data structure library
• Loop properties provided via
annotations
• Loop body must read all
elements before modifying any
(cautious)
• KDG runtime (in Galois
system)
– Uses library API to track rw-
sets at runtime
– Selects executor based on
annotations
62
Graph g = ...
Set<Event> E = ...
@hasStructuredRWSets
@monotonic
foreach Event e in E orderedby e.t
process(e, g)
if *
E.push(newE)

63. Evaluation
• 7 applications
– From billiards simulation (hard-to-parallelize) to
tree traversal (well supported by prior work)
• 2 types of programs
– Other
• 3rd-party handwritten, application-specific, OpenMP
tasks, Cilk
– KDG-Auto
• Ordered loops + program property annotations
• 40-core, shared-memory machine
63

64. 1
6
11
16
21
26
31
Billiards DES MST AVI
SpeedupoverBestSequential
Other
KDG-Auto
64
n/a
< 1

65. 1
6
11
16
21
26
31
36
41
LU BFS Tree
SpeedupoverBestSequential
Other
KDG-Auto
65

66. 1
6
11
16
21
26
31
36
41
LU BFS Tree
SpeedupoverBestSequential
Other
KDG-Auto
66
OpenMP
tasks
Cilk

67. Summary
• Problem
– Dependence graph scheduling is insufficient for
many ordered programs
• Solution
– Develop general KDG executor
– Specialize executor according to small number of
programproperties
67
“It is a mistaketo try to look too far ahead. The chain of
destiny can only be grasped one link at a time.”
-Winston Churchill

68. Kinetic Dependence Graphs
M. Amber Hassaan,
Donald Nguyen,
Keshav Pingali
The University of Texas at Austin

69. 69