This document presents research on calibrating the DQX model for quality of experience (QoE) prediction in voice over IP (VoIP) calls. The researchers conducted experiments varying latency, packet loss, jitter, and bandwidth to collect user ratings. They calibrated the DQX model parameters to the data and compared it to other QoE models. The results showed that DQX is flexible but the influence factors require further tuning. Overall, DQX predicted mixed variable scenarios reasonably well compared to collected user ratings. Future work will explore additional variables, more test conditions and calibrating DQX for other services.

1. © 2015 UZH,
VoIP-based Calibration of the DQX Model
Christos Tsiaras, Manuel Rösch, Burkhard Stiller
Department of Informatics IFI, Communication Systems Group CSG,
University of Zürich UZH
[tsiaras,stiller]@ifi.uzh.ch manuel.roesch@uzh.ch
IFIP Networking 2015, Toulouse, France, May 20, 2015
QoE Models for VoIP
DQX and Goals
Experiments and Results
Conclusion

2. © 2015 UZH,
E-model (R)
 Ro
– Various noise sources
 Is
– Loud speech level
– Non-optimum Overall
Loudness Rating (OLR)
– Non-optimum Side Tone
Masking Rating (STMR)
 Id
– Delay
– Echo
 Ie
– Equipment impairment factor
 A
– Expectation
R = 0R − sI − dI − eI + A

3. © 2015 UZH,
IQX Hypothesis
IQX :QoE = α ×e−β×QoS
+γ
 1 degree of freedom
– β: curve gradient
 α and γ define the
min and max Mean
Opinion Score (MOS)
0-1 normalized value of a variable
MOS

4. © 2015 UZH,
DQX Model
 Increasing Variable (IV)
– The more you have the better it is
 Decreasing Variable (DV)
– The more you have the worst it is
 Mixed Variable
– Multiple variables affect QoE

5. © 2015 UZH,
DQX HOWTO
 Formalizing QoE in 6 steps
1. Identify variables that affect QoE
2. Characterize those variables
• Increasing variables (IV)
• Decreasing variables (DV)
1. Select the ideal/desired/expected/agreed value of a variable
2. Considering the service specifications select the best and
the worst value of each variable
3. Identify the effect of each variable’s variation
• Influence factors (m)
1. Identify the importance of each variable (wk)

6. © 2015 UZH,
DQX Model
ed (x) = 4e
−
x
x0





÷
m
ln4
3
+1QoE equation for DVs
ei (x) = 4(1−e
−
x
x0





÷
m
ln4
)+1QoE equation for IVs
E(X) =1+ 4
e i∨d( ) xk( ) −1
4







k=1
N
∏
wk
Generic QoE equation
Importance factor
Step 6
Influence factor
Step 5
Expected value
Step 3
Variables selection
Step 1
Variables characterization
Step 2
QoE QoE-related
variables values
Best and worst values
Step 4

7. © 2015 UZH,
DQX Model
Influence Factor m
Exponential functionLinear function Step function

8. © 2015 UZH,
Goals
 Define and calibrate the parameters of DQX in the
VoIP scenario
 Collect QoE-related feedback
 Develop a QoE measurement setup wrt
– Latency
– Packet loss
– Jitter
– Bandwidth
 Compare DQX with state of the art QoE models in
VoIP
– IQX Hypothesis
– E-model

9. © 2015 UZH,
Experiment Setup
Network
Emulation
• Jitter
• Latency
• Packet loss
• Bandwidth
Real-Time Communications (RTC)
Wide Area Network emulator (WANem)

10. © 2015 UZH,
Experimental Calls
 34 Subjects
 Places
– IFI UZH
– KS Willisau
 6 hours
– 541 data points
 45 different Scenarios
– 80% single variable
– 20% mixed variables

11. © 2015 UZH,
Evaluation
 Single variable scenarios
– Variables
• Latency
• Packet Loss
• Jitter
• Bandwidth
– m values
 Comparison
– DQX
– IQX
– E-Model
 Mixed variables scenario

12. © 2015 UZH,
min/max and Expected Variable Values x0
 Latency
– min value = 0 ms: no delay
– x0 = 150 ms: codec independent, ITU-T recommendation G.114 and G.1010
– max value = 1800 ms: satellite connection
 Jitter
– min value = 0 ms: no jitter
– x0 = 100 ms: no values for Opus in literature, Cisco recommendation
– max value = 1800 ms
 Packet Loss
– min value = 0%: no packet loss
– x0 = 5%: official Opus codec documentation
– max value = 50%
 Bandwidth
– min value = 0 kBit/s: no connectivity
– x0 = 64 kBit/s: default bandwidth for WebRTC according to its documentation
– max value = 140 kBit/s

13. © 2015 UZH,
Evaluation: Packet Loss

14. © 2015 UZH,
Evaluation: Latency

15. © 2015 UZH,
Evaluation: Jitter

16. © 2015 UZH,
Evaluation: Bandwidth
(m-:4.45, m+:0.47)

17. © 2015 UZH,
Influence Factor (m) Escalation
Variable’s Value

18. © 2015 UZH,
Influence Factor (m) Escalation - Bandwidth

19. © 2015 UZH,
Evaluation: Mixed Variables
 14 scenarios, unadjusted importance factor wk
 Mean Opinion Score (MOS) difference (Collected – DQX) : 0.53
 Standard Deviation: 0.68

20. © 2015 UZH,
Conclusion & Future Work
 Conclusion
– DQX is flexible
– Influence factor m is not constant
– Importance factors w and further calibration of the min, max, expected values
can improve the DQX results
– Critical thoughts
• Subjects: men between 20 and 25
• Headsets and duration of the test calls
• WebRTC, Browser Interoperability
 Future Work
– QoE measurement setup
• Other variables
• More tests
• Different services
– Videoconference
– Video streaming
– Further analysis of the m value and the formula for mixed variables

21. © 2015 UZH,
Thank you!
Q&A

22. © 2015 UZH,
# Steps from min to max Values

23. © 2015 UZH,
Collected MOS for Mixed Variables
Compared to the Calculated MOS

24. © 2015 UZH,
Used Software