Quality of Experience in emerging visual communications

1

Quality of Experience in
emerging visual communications
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch

Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne

Some old (still unanswered?) questions
What is the best way to apply PSNR to color images?
What is the best way to apply PSNR to video?
What are the most reliable and repeatable subjective evaluation
methodologies for image and video quality assessment?
How to measure quality (subjective evaluations or objective metrics)
of 3D image and video?
of UHD video?
of HDR image and video?
How to measure quality (subjective evaluations or objective metrics
of audiovisual content?
…


2

Qualinet in a nutshell

COST Action IC1003:
– European Network on Quality of Experience in
Multimedia Systems and Services

Period of activity:
– November 2011 to October 2014

33 countries (27+6) and 185 active researchers
More information:
– http://www.qualinet.eu


3

4

QUALINET in a nutshell
Certification
Products &
Services

Multimedia
Applications

QUALINET
International
Standards

ICT,
Psychology&
Neuroscience&
Humanities, …

Protocols &
Methodologies
& Metrics


Qualinet organization


5

Organisation of Qualinet


6

A fundamental and ancient concept

Aristotle classified every object of human
apprehension into 10 Categories
–
–
–
–
–
–
–
–
–
–

Substance
Quantity
Quality
Relation
Place
Time
Position
State
Action
Affection


7

User experience in multimedia


8

Qualinet white paper on Quality of Experience

White Paper produced by COST Action
IC1003 (Qualinet):
– Downloadable from http://www.qualinet.eu
– Latest version: V1.2, Novi Sad, March 2013

Several definitions of quality in multimedia
systems and services and other related
concepts
Qualinet databases
- 169 individual databases
- http://dbq-multimediatech.cz

9

Quality (of Experience) is like an elephant …

The blind men and the elephant: Poem by John Godfrey Saxe

10

User preference


11

Context


12

User centered evaluation


13

14

A simple model for QoE
User attributes
–
–
–
–

individual attributes – expectation, age, sex, personality, background…
sensorial attributes – including limitations and deficiencies
perceptual attributes
emotional attributes

System attributes

QoE

– technical attributes (as in QoS)

Contextual attributes
–
–
–
–

environmental attributes
device attributes
service attributes
content attributes

user


QoS
context

User and contextual attributes
Personas (user preference)
– Archetypical user representing the needs, behaviors

and goals of a particular group of users

Scenarios (context)
– Realistic usage environment


15

Quality of Experience in mobile multimedia
• Evaluation of quality of experience of video streaming in mobile
environment (living lab)
“Gesture and Touch Controlled Video Player Interface for Mobile
Devices” S. Buchinger, et al., in Proceeding of the ACM
Multimedia 2010 International Conference, (2010).


16

Emotional attributes
Study with 32 subjects
Valence-Arousal-Liking (VAL) emotional
modeling.
Elicitation using 40 music clips chosen to fill
the whole 2D VA space.
Subjective rating using SAM (Self
Assessment Manikin)


17

18

Classification in the VA space
Electroencephalography (EEG)
Physiological signals: blood flow, electrodermal activity
–EDA-, respiration,… (Physio.)
Multimedia content analysis (MCA)

Classification accuracy
Valence

Liking

EEG

0.56

0.58

0.5

Physio.

0.61

0.53

0.54

MCA

0.61

0.62

0.63

All


Arousal

0.65

0.62

0.63

19

Evolution of content
3D TV

B&W TV

Color TV

HD TV

UHD TV

?
HFR TV

HDR TV


UHDTV
PeopleOnStreet (3840x2160@30fps)

Traffic (3840x2048@30fps)

Sintel2 (3840x1744@24fps)

Sintel39* (3840x1744@24fps)


20

DS

Methodology
Double Stimulus Impairment Scale (DSIS) Variant II
Test
Video

Reference
Video

Test
Video

Age

21

1
Imperceptible

Reference
Video

Name

100
90

Perceptible
but not
annoying

80
70

Slightly
annoying

60

2s

5s

2s

5s

2s

5s

6s

40

(TOT= 34 s)

“Rate the level of annoyance of the visual defects that you see
in stimulus B, knowing that A is the reference video.”

Annoying

5s

30
20

Very
annoying

2s

50

10
0


PeopleOnStreet
PeopleOnStreet
100
90
80
70

MOS

60
50
40
30
20
10

AVC
HEVC

0
4

6

8

10

12
14
bitrate [Mbps]

16

18

20

22

At similar bit rates, HEVC outperforms AVC in 4 / 5 cases
Bit rate reduction
– BD-PSNR: 27.5%
– BD-MOS: 50.8%

22

23

Foveated video coding of UHDTV
Priority map

…

Localization result

Compression
(H.265/HEVC)

Blurred image


Gaussian pyramid of L levels

Example


24

Subjective evaluation experiment
7 test sequences
MJF content, Tears of Steel – 10 s
– Including multiple moving objects in scene
– UHD and HD resolution – separate sessions
–

Audio-visual source localization
–

Visual features: differential images
– Audio features: frame energy

H.265/HEVC coding
–

HM 12.1
– Different QP – 20, 30, 33

Subjective test
–

Perceived quality?
– Single stimulus - home like scenario
– Same distance for both resolutions


25

Results: Coding Efficiency & Subjective Quality
UHD C3

HD

C3

41% gain

9% gain

24% gain

19% gain

C6

C6

87% gain

9% gain

20% gain

5% gain


26

3D Quality ǂ Σ2D
Encode left/right images with
JPEG and different QPs (0-100)
Show images with decreasing
quality to the subjects
Determine limit of transparency
for left, right and stereo image
Compute PSNR of left and right
images and average for stereo
Find PSNR which corresponds
to the QP limit for each image
Average PSNRs for each image
across the individual subjects

27

3D Quality ǂ Σ2D


28

MVC assessment using PSNR as metric


29

MVC assessment by subjective evaluation


30

Left/right image [Campisi2007]

• Applies common 2D image quality metrics to left and right image
• Combines scores using average, main eye or visual acuity
approach

31

3D video content
 2 sets of spatio-temporal resolutions (8 different contents)
– Class A: 1920x1088p@25fps
– Class C: 1024x768p@30fps

4 target coding bit rates
22 different codecs + 2 anchors
YUV 4:2:0 uncompressed videos with 8 bits per sample


32

will stand on the screen for 1
seconds. The test subjects w
33
Evaluation methodology
quality votes will be expresse
Double Stimulus Impairment Scale (DSIS) evaluation

11-grade numerical categorical scale
Training

1
10
9
8
7
6
5
4
3
2
1
0

Test session: 24 test pairs + 3 dummy pairs + 1 ref vs. ref pair
Outlier detection


Exa

Results - Random stereo pair


34

Recent drivers behind HDR imaging

HDR sensors
– Backlit CMOS sensor
– Binary Pixel Imager


35

Recent drivers behind HDR imaging

HDR displays
– Modulated LED


36

Which tone-mapping?
Many subjective evaluations of tone-mapping to
find the best among those proposed in literature
– Not always consistent with each other

What happens if we perform subjective
evaluation of tone-mapping operators taking into
account explicitly the influence of content:
– Scenes with varying dynamic range shot at night

day, with dark and bright regions

And context:
– Environmental parameters (ambient illumination,

etc.)
– Devices (type of display, etc.)
– Content (type of content, etc.)


37

Subjective evaluation
Five state-of-the-art tone-mapping operators
– Drago
– Mantiuk
– Reinhard
– iCam
– Logarithmic

One controlled environment
– Eizo monitor in an ITU-R BT 500-11 compliant laboratory
– Passive subjects

Two uncontrolled environments
– iPad Tablet and Android mobile phone
– Active subjects


38

Different content


39

Evaluation protocol
Paired comparison between any two tonemapping operators applied to the same image
Scores: A>B, A=B, A<B
20 subjects (12 male, 8 female)
4 images with 10 paired comparisons for each
Training session to obtain more stable results
Reference versus reference
Randomization
Two short sessions to avoid visual fatigue and
loss of concentration (less than 15 min each)


40

Example comparison scores


41

Example subjective scores


42

Compression comparisons


43

44

HDR objective metrics
Visibility
VDP-2

SNR

38

1

36

0.9

0.8

34

0.7
32

VDP_pdet

SNR (dB)

0.6
30

28

BloomingGorse2
CanadianFalls
McKeesPub
MtRushmore2
WillyDesk

0.5

0.4
26
0.3
BloomingGorse2
CanadianFalls
McKeesPub
MtRushmore2
WillyDesk

24

22

20
0.5

1

1.5

2

2.5

3

3.5

4

4.5

0.2

0.1

5

Bpp


0
0.5

1

1.5

2

2.5

3
Bpp

3.5

4

4.5

5

Drunk scientist!


45

Measuring quality of experience through user sensing


46

User sensing through wearable devices


47

Thanks for your attention


48

Quality of Experience in emerging visual communications

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Viewers also liked

Viewers also liked (8)

Similar to Quality of Experience in emerging visual communications

Similar to Quality of Experience in emerging visual communications (20)

More from Touradj Ebrahimi

More from Touradj Ebrahimi (20)

Recently uploaded

Recently uploaded (20)

Quality of Experience in emerging visual communications

Editor's Notes