The document discusses research on quantifying user satisfaction (QoE) in VoIP applications like Skype calls. It presents three key contributions: 1) Developing the first QoE measurement methodology based on analyzing large-scale Skype call data to correlate call duration with network quality factors like jitter and bit rate. 2) Proposing OneClick, a simple framework for crowdsourced QoE experiments based on user clicks to indicate dissatisfaction. 3) Introducing the first crowdsourcable QoE evaluation methodology to verify user judgments.

1. Network and Multimedia QoE Management Sheng-Wei (Kuan-Ta) Chen Institute of Information Science Academia Sinica

2. What is QoE? Quality of Experience = User Satisfaction in Using Computer/Communication Systems

3. What is QoE Management? Measurement Provisioning Measure user satisfaction Improve system design to provide more satisfactory user experience

4. Goal of QoE Management To Provide Satisfactory User Experience in Computer/Communication Systems

8. Our Research Focus Video Conferencing Online Entertainment VoIP

17. Quantifying User Satisfaction Collaborators: Chun-Ying Huang Polly Huang Chin-Laung Lei ( National Taiwan University) Sheng-Wei (Kuan-Ta) Chen Institute of Information Science, Academia Sinica Appeared on ACM SIGCOMM 2006

20. What path should Skype choose? Which path is “the best”? 3% 1% 2% loss rate 500 ms 30 Kbps 300 ms 20 Kbps 100 ms delay 10 Kbps avail bandwidth path Internet

22. A typical relationship between QoS and QoE QoS, e.g., network bandwidth QoE Hard to tell “very bad” from “extremely bad” Marginal benefit is small

30. Setting things up

34. Trace Summary 24 min 18 min 29 min Avg. Time 570 462 Total 369 209 Relayed 240 Hosts 253 Calls Direct Category Internet Direct sessions Relayed sessions

40. Effect of Jitter – Hypothesis Testing Null Hypothesis All the survival curves are equivalent Log-rank test: P < 1e-20 We have > 99.999% confidence claiming jitters are correlated with call duration The probability distribution of hanging up a call

41. Effect of Source Rate (the bandwidth Skype intended to use) Average session time (min)

46. The Logarithm Fits Better (Bit rate) After taking logarithm …

47. The Logarithm Fits Better (Jitter) After taking logarithm …

48. Final model & interpretation 0.18 0.09 0.13 std. err. 4.3e-02 0.36 RTT < 1e-20 1.55 log(jitter) < 1e-20 signif. -2.15 coef log(bit rate) variable A: bit rate = 20 Kbps B: bit rate = 15 Kbps, other factors same as A The hazard ratio between A and B can be computed by exp((log(15) – log(20)) × -2.15 ) ≈ 1.86  The probability B will hang up is 1.86 times the probability A will do so at any instant. Interpretation

49. Hang-up rate and USI Hang-up rate = User satisfaction index (USI) =

50. Actual and Predicted Time vs. USI Average session time (min)

51. The multi-path scenario is call hang-up rate a good indication of user satisfaction? BUT, 5.43 6.33 3.84 USI 3 Kbps 1 Kbps 2 Kbps jitter 500 ms 30 Kbps 300 ms 20 Kbps 100 ms RTT 10 Kbps avail bandwidth path Internet

53. User satisfaction: Validation Call duration ? intuition: call duration <-> satisfaction not confirmed yet

54. User satisfaction: One step further Speech interactivity Call duration ? now we’re going to check! intuition: interactive and tight speech activities in a cheerful conversation

61. Speech interactivity analysis Responsiveness: Avg. Response Delay: Avg. Burst Length: whether the other party responds how long before the other party responds how long does a speech burst last Responsiveness: Response delay: Burst length:

69. Case Study: ShenZhou Online

70. Traffic Trace Collection 4.7TB / 21.7TB / 26.5TB 325M / 336M / 661M 54,424 N2 4.7TB / 27.3TB / 32.0TB bytes (in/out/both) 342M / 353M / 695M packets (in/out/both) 57,945 conn. # N1 trace

71. Delay Jitter vs. Session Time (std. dev. of the round-trip times)

72. Hypothesis Testing -- Effect of Loss Rate Null Hypothesis: All the survival curves are equivalent Log-rank test: P < 1e-20 We have > 99.999% confidence claiming loss rates are correlated with game playing times high loss low loss med loss The CCDF of game session times

74. Final Model & Interpretation A: RTT = 200 ms B: RTT = 100 ms, other factors same as A Hazard ratio between A and B: exp((log(0.2) – log(0.1)) × 1.27 ) ≈ 2.4 A will more likely leave a game ( 2.4 times probability ) than B at any moment Interpretation < 1e-20 0.04 1.27 log(RTT) 0.01 0.01 0.03 Std. Err. 7e-13 0.09 log(sloss) < 1e-20 0.12 log(closs) < 1e-20 Signif. 0.68 Coef log(jitter) Variable

75. How good does the model fit?

76. Relative Influence of QoS Factors Server pakce loss = 15% Delay jitter = 45% Client packet loss = 20% Latency = 20%

79. App #2: Overlay Path Selection Internet 5.43 6.33 3.84 score 30 ms (A) 20 ms (G) 50 ms (P) jitter 1% (A) 200 ms (P) 1% (A) 150 ms (A) 5% (P) loss rate 100 ms (G) delay path

82. To be continued … Sheng-Wei (Kuan-Ta) Chen http://www.iis.sinica.edu.tw/~swc

83. OneClick -- A Framework for Measuring Network Quality of Experience Kuan-Ta Chen, Cheng-Chu Tu, Wei-Cheng Xiao Institute of Information Science, Academia Sinica Appeared on IEEE INFOCOM 2009

85. Relationship between QoS and QoE QoS, e.g., network bandwidth QoE Too bad to perceive Marginal benefit is small Comfort range

89. Our Ambition Identify a simple and yet efficient way to measure users’ satisfaction

95. Human as a QoE Rating System User Application QoS Application QoE Network Setting Click Events affect reflect vary this: observe this:

101. 1-2 Seconds Delay Response delay calibration needed!

103. Consistency of C(t+d) from Same User

104. Consistency of C(t+d) from Different Users Cross-user normalization needed!

105. Calibration and Normalization Added Response Delay Calibration Regression Modeling With Normalization User #1 User #2 OneClick Measurement

109. Qualitative Comparison [Validation] Network Loss Rate Bandwidth

111. Qualitative Comparison [Validation]

119. OneClick Online

121. To be continued … Kuan-Ta Chen http://www.iis.sinica.edu.tw/~swc

122. A Crowdsourceable QoE Evaluation Framework for Multimedia Content Kuan-Ta Chen Academia Sinica Chen-Chi Wu National Taiwan University Yu-Chun Chang National Taiwan University Chin-Laung Lei National Taiwan University Appeared on ACM Multimedia 2009

123. What is QoE? Quality of Experience = Users’ satisfaction about a service (e.g., multimedia content)

124. Quality of Experience Poor (underexposed) Good (exposure OK)

129. Contribution

131. Current Approach: MOS Rating Excellent? Good? Fair? Poor? Bad? Vote ?

132. Our Proposal: Paired Comparison Which one is better? Vote A B B

134. Choice Frequency Matrix 10 experiments, each containing C(4,2)=6 paired comparisons A B C D A B C D 0 4 2 1 6 0 3 0 8 7 0 1 9 10 9 0

136. Inferred QoE Scores 0 0.63 0.91 1

146. Concept Flow in Each Round

151. Inferred QoE Scores MP3 Compression Level VoIP Packet Loss Rate

153. Inferred QoE Scores Video Codec Concealment Scheme

156. http://mmnet.iis.sinica.edu.tw/link/qoe

158. Thank You! Kuan-Ta Chen Academia Sinica

160. Acknowledgements Chin-Laung Lei Chun-Ying Huang Polly Huang Yu-Chun Chang Te-Yuan Huang William Tu Hung-Hsuan Chen Chen-Chi Wu Wei-Cheng Xiao

161. Thank you! Sheng-Wei (Kuan-Ta) Chen http://www.iis.sinica.edu.tw/~swc Institute of Information Science, Academia Sinica