From High Dimensional Data to Big Data Presentation to MedicReS 5th World Congress on October 19,25,2015 in New York by Han Liu

1. From
High
Dimensional
Data
to
Big
Data
Han
Liu

2. Acknowledgement
2
Ethan Fang
Princeton University
Fang Han
JHU Biostatistics
Cun-hui Zhang
Rutgers Statistics

3. Big Data Movement
3

4. Big Data Movement
3

5. Big Data Movement
3

6. Big Data Movement
3

7. Big Data Movement
3

8. Big Data Movement
3

9. Outline
4
•
•
•

10. High Dimensional Multivariate Analysis
5

11. High Dimensional Multivariate Analysis
5
· · ·

12. High Dimensional Multivariate Analysis
5
· · ·

13. High Dimensional Multivariate Analysis
5
µ,
· · ·

14. High Dimensional Multivariate Analysis
5
µ,
· · ·

15. Gaussian Graphical Model
6
(µ, )
= ( , . . . , )

16. Gaussian Graphical Model
6
(µ, )
= ( , . . . , )
= ( , )

17. Gaussian Graphical Model
6
(µ, )
= ( , . . . , )
= ( , )

18. Gaussian Graphical Model
6
(µ, )
= ( , . . . , ) |
= ( , )

19. Gaussian Graphical Model
6
(µ, )
= ( , . . . , ) ( ) =|
= ( , )

20. Gaussian Graphical Model
6
(µ, )
= ( , . . . , ) ( ) =|
= ( , )
min log | | + , | |

21. Gaussian Graphical Model
6
(µ, )
= ( , . . . , ) ( ) =|
= ( , )
min log | | + , | |

22. Gaussian Graphical Model
6
(µ, )
= ( , . . . , ) ( ) =|
= ( , )
min log | | + , | |

23. Gaussian Graphical Model
6
(µ, )
= ( , . . . , ) ( ) =|
= ( , )
min log | | + , | |

24. Sparse Principal Component Analysis
7
= ( , . . . , )

25. Sparse Principal Component Analysis
7
= ( , . . . , )

26. Sparse Principal Component Analysis
7
= ( , . . . , )
max x = x x
(x) k

27. Sparse Principal Component Analysis
7
= ( , . . . , )
max x = x x
(x) k

28. High Dimensional Theory
8

29. High Dimensional Theory
8
= ( )

30. High Dimensional Theory
8
= ( )

31. High Dimensional Theory
8
= ( )
P ( ) = ( ) ( )

32. High Dimensional Theory
8
= ( )
P ( ) = ( ) ( )

33. Theoretical Foundations
9
R4
R3
R2
R5
R1 µ µ max = log
max = log
=
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>

34. Real Data are non-Gaussian
10
! "## $## (> K)
(> K)
> K

35. Real Data are non-Gaussian
10
! "## $## (> K)
(> K)
> K

36. Real Data are non-Gaussian
10
! "## $## (> K)
(> K)
> K

37. Transelliptical Distribution
11

38. Transelliptical Distribution
11
= ( , . . . , )
(µ, , { } = , )
, . . . ,
= ( ), . . . , ( )
| | / (y µ) (y µ)

39. Transelliptical Distribution
11
= ( , . . . , )
(µ, , { } = , )
, . . . ,
= ( ), . . . , ( )
| | / (y µ) (y µ)

40. Visualization
12

41. Special Cases
13
(µ, , { } = , )

42. Special Cases
13
(µ, , { } = , )
( ) = ,

43. Special Cases
13
(µ, , { } = , )
(µ, , )
( ) = ,

44. Special Cases
13
(µ, , { } = , )
(µ, , )
( ) = , ( ) =

45. Special Cases
13
(µ, , { } = , )
(µ, , ) (µ, , { } = )
( ) = , ( ) =

46. Special Cases
13
(µ, , { } = , )
(µ, , ) (µ, , { } = )
( ) = , ( ) =

47. 14
Identifiability Conditions
( ), . . . , ( ) (µ, , )

48. 14
Identifiability Conditions
( ), . . . , ( ) (µ, , )

49. 14
Identifiability Conditions
( ), . . . , ( ) (µ, , )
µ = ( ) = I

50. 14
Identifiability Conditions
( ), . . . , ( ) (µ, , )
µ = ( ) = I
E = E ( ) = µ Var( ) = Var[ ( )] =

51. 14
Identifiability Conditions
( ), . . . , ( ) (µ, , )
µ = ( ) = I
E = E ( ) = µ Var( ) = Var[ ( )] =

52. 15
Hierarchical Representation

53. 15
Hierarchical Representation
( , . . . , ) (µ, , { } = , )

54. 15
Hierarchical Representation
( , . . . , ) (µ, , )
= ( )
( , . . . , ) (µ, , { } = , )

55. 15
Hierarchical Representation
( , . . . , ) (µ, , )
= ( )
( , . . . , ) (µ, )
= µ + ( µ )
( , . . . , ) (µ, , { } = , )

56. 15
Hierarchical Representation
( , . . . , ) (µ, , )
= ( )
( , . . . , ) (µ, )
= µ + ( µ )
(·)
( , . . . , ) (µ, , { } = , )

57. 15
Hierarchical Representation
( , . . . , ) (µ, , )
= ( )
( , . . . , ) (µ, )
= µ + ( µ )
(·)
( , . . . , ) (µ, , { } = , )

58. 15
Hierarchical Representation
( , . . . , ) (µ, , )
= ( )
( , . . . , ) (µ, )
= µ + ( µ )
(·)
( , . . . , ) (µ, , { } = , )

59. 15
Hierarchical Representation
( , . . . , ) (µ, , )
= ( )
( , . . . , ) (µ, )
= µ + ( µ )
(·)
( , . . . , ) (µ, , { } = , )

60. Transelliptical Graphical Model
16
(µ, , { } = , )

61. Transelliptical Graphical Model
16
(µ, , { } = , )

62. Transelliptical Graphical Model
16
(µ, , { } = , )

63. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

64. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

65. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

66. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

67. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

68. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

69. Transelliptical Graphical Model
16
( ) =
(µ, , { } = , )

70. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )

71. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )

72. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )

73. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )

74. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )
µ, { } =

75. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )
µ, { } =
µ,

76. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )
µ, { } =
µ,

77. Semiparametric Inference
17
x , . . . , x (µ, , { } = , )
µ, { } =
µ,

78. Technical Requirements
18
R4
R3
R2
R5
R1 µ µ max = log
max = log
=
µ, µ,
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>

79. Estimating Mean
19

80. Estimating Mean
19
µ = · (x µ ) =

81. Estimating Mean
19
µ = · (x µ ) =

82. Estimating Mean
19
µ = · (x µ ) =

83. Estimating Mean
19
µ = · (x µ ) =
( ) = log + + /

84. Estimating Mean
19
µ = · (x µ ) =
( ) = log + + /
µ µ max = log
E <
R1

85. Estimating Covariance
20

86. Estimating Covariance
20
= · ·

87. Estimating Covariance
20
= · ·

88. Estimating Covariance
20
= · ·
= sin

89. Estimating Covariance
20
= ( )
<
(x x ) · (x x )
= · ·
= sin

90. Estimating Covariance
20
= ( )
<
(x x ) · (x x )
= · ·
E <
R2 max = log
= sin

91. 21
max
max = log

92. 21
max
max = log
= sin E

93. 21
max
max = log
= sin E

94. 21
max
max = log
= sin E

95. 21
max
max = log
= sin E

96. 21
max
max max E = log
max = log
= sin E

97. 21
max
max max E = log
max = log
= sin E

98. Transelliptical Theory
22

99. Transelliptical Theory
22
•
•
•

100. Transelliptical Theory
22
=
•
•
•

101. Transelliptical Theory
22
=
•
•
•
•
•
•

102. Transelliptical Theory
22
=
= s
•
•
•
•
•
•

103. Transelliptical Theory
22
=
= s
•
•
•
•
•
•

104. Proof of the Theory
23

105. Proof of the Theory
23
R4
R3
R2
R5
R1 µ µ max = log
max = log
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>
= log

106. Proof of the Theory
23
R4
R3
R2
R5
R1 µ µ max = log
max = log
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>
= log

107. Proof of the Theory
23
R4
R3
R2
R5
R1 µ µ max = log
max = log
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>
= log

108. Proof of the Theory
23
R4
R3
R2
R5
R1 µ µ max = log
max = log
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>
= log

109. Proof of the Theory
23
R4
R3
R2
R5
R1 µ µ max = log
max = log
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>
= log

110. Proof of the Theory
23
R4
R3
R2
R5
R1 µ µ max = log
max = log
sup
v
|v ( )v|
v
=
log
inf
v
|v v|
v
>
= log

111. Empirical Results
24

112. Empirical Results
24
( , ) = , =

113. Empirical Results
24
true graph transelliptical glasso
( , ) = , =

114. Empirical Results
24
true graph transelliptical glasso
FP
FN
( , ) = , =

115. Empirical Results
24
true graph transelliptical glasso
FP
FN
( , ) = , =

116. Efficiency Loss
25

117. Efficiency Loss
25
( , ) = , =

118. Efficiency Loss
25
ROC curve for graph recovery
1-FP
1-FN
( , ) = , =

119. Efficiency Loss
25
ROC curve for graph recovery
1-FP
1-FN
( , ) = , =

120. Efficiency Loss
25
ROC curve for graph recovery
1-FP
1-FN
( , ) = , =

121. Efficiency Loss
25
ROC curve for graph recovery
1-FP
1-FN
( , ) = , =

122. 26
Summary

123. 26
Summary
•
•
•

124. 26
Summary
•
•
•