1. Quality of Experience:
What End-users Say About
Web Services?
Bipin
Upadhyaya

2. Introduction
• Web services provided by different vendors
have different qualities
• Quality of Service(QoS) refers to non-
functional requirements, such as performance
or availability requirements
• QoS deals with how a service behave at run-
time
2

3. Service Selection Approaches
QoS aware architecture filters the low quality web
services from a set of functionally equivalent services
3

4. Limitation of QoS
• Static Release
• Not frequently updated
• Environment & platform specific
• Runtime Monitoring
• Evaluation of QoS at client-side
• resource intensive, time consuming &
expensive
End users barely understand QoS attributes
4

5. User Experience
On average, users tell 15 people
about good experiences and 24
people about bad experiences
A word-of-mouth recommendation is the
primary factor behind
20% to 50%
of all purchasing decisions
Ref: http://www.mckinsey.com/insights/marketing_sales/a_new_way_to_measure_word-of-mouth_marketing

6. Reviews
• Web 2.0 has enabled people to
broadcast their experience to
the mass
• Online user reviews are
examples of such phenomenon
• Credible source of information
6

7. Heterogeneity in Reviews
7

8. Extract Relevant Information
from Reviews
• Extract service features such as
• Parts and Properties
• Opinions about the features are expressed as
• Adjectives and Adverbs
#Hotel
Pool
ElevatorFood
Breakfast
Staff
Location
Room
8

9. Overview of Our Approach
9
POS Tagged
Sentence
Extract
Sentence
& Tag POS
QoE Attributes
& Opinions
Extract
QoE &
Opinion
Aggregated
QoE
Attributes
Group
Similar QoE
Attributes

10. POS Tag for Review Sentences
Dropbox has great
synchronization and folder
sharing capability.
Dropbox/NNP has/VBZ
great/JJ synchronization/NN
&/CC folder/NN sharing/NN
capability/NN ./.
POS Tagger
10

11. Extract QoE Attributes and
Opinion
(great, synchronization)
(great, Folder sharing capability)
Dropbox/NNP has/VBZ great/JJ
synchronization/NN &/CC folder/NN
sharing/NN capability/NN ./.
Detect negation and reverse adjectives
Detect POS patterns
11

12. Polarity of Opinion
• SentiWordNet is used to assign a score of
opinion
• Each word in WordNet is assigned a score
• Allocates a sentiment score of positive,
negative or objective to words
12
(great, Synchronization)
(great, Folder sharing capability)
Great –> 0.75

13. Cluster QoE Attributes
Performance (6)
Download (3)
Install (2)Ease to Use(4)
Operation (4)
Sync (3)
Synchronization(4)
Back-up (5)
Store (3)
Fast (4)
Upload (3)
Download (3)
Install (2)Ease to Use (4)
Setup(2)
Sync (3)
Synchronization(4)
Back-up (5)
Store (3)
Performance (6)
Operation (4)
Fast (4)Upload (3)
Setup(2)
Clustering
13

14. Select Candidate Element
• Rank each word in a cluster based on
frequency and semantic similarity with other
words
• Candidate elements can be used by users to
specify preferences
Performance (6)
Operation (4)
Fast (4)
Upload (3)
QoE Attribute Rank
Performance 7.04
Operation 5.168
Upload 3.66
Fast 5.982
14

15. Research Questions
RQ1. Can our approach extract QoE from online
reviews?
RQ2. Does user sentiment about QoE attributes
changes over time?
RQ3. How do QoE attributes relate with QoS
attributes?
15

16. Case Study Setup
448 440 422 419
6980
5866
6611
4110
0
1000
2000
3000
4000
5000
6000
7000
8000
Trip Shopping Storage Mapping
Service
Sentences containing QoE
attributes and opinions
Sentences without QoE
attributes and opinions
• Collect reviews for a periods of 13 months
• Collect QoS attributes such as execution time
and availability for all services
16

17. RQ1. Can our approach extract
QoE from online reviews?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Travel Shopping Storage Mapping
Service
Recall
Precision
Extract QoE attributes with high precision
and recall
17
90%
72%

18. RQ2. Does user sentiment about
QoE attributes change over time?
18
User’s sentiment about QoE attributes
changes over time

19. RQ3. How do QoE attributes
relate with QoS attributes?
High correlation between QoS and QoE
Attributes indicates the possibility of using
QoE attributes for service selection.
19

20. 20