Part 2. Web sites quality evaluation projects with the 7 loci method

WEE-NET
Web Engineering Network
of Excellence
Summer School in La Plata
January 29 February 9 2007
Web Sites Quality Evaluation
Projects with the 7Loci method
Luisa Mich
etourism.economia.unitn.it

Schema
Goal of the talk: presents real web sites
quality evaluation projects
Yet another method! The 7 Loci meta-model
and method
WEE-NET
of Excellence
Summer School in La Plata, Argentina
Luisa Mich – The challenges of the NOW Economy
2
model A first project, to illustrate the method
Comparative evaluation projects
A multi-steps evaluation project

Evaluation of a ski consortium web
site
Goals of the owner:
to publicize the area
to promote the Consortium’s services
WEE-NET
of Excellence
3
to publicize the hotels belonging to the
Consortium
to sell the services offered by the
Consortium and by the hotels

WEIGHTS SCORES
IDENTITY
Does the site have strong Brand Identity or Charisma? 0.25 3
Does the site convey an adequate corporate image? 0.25 4
Does the site have good graphical and functional
0.25 4
characterization?
Is the site able to adapt to the user? 0.25 2
3.25
CONTENT
Does the information match the aims of the site? 0.25 4
Are the information and links useful? 0.25 4
Is the information accurate? 0.25 4
Are the sources of the information given? 0.25 3
3.75
SERVICES
Do the functions match the aims of the site owner? 0.25 3
Do the functions match the aims of the user? 0.25 2
Are the functions free of errors? 0.30 3
Are transactions secure and is privacy respected? 0.20 1
2.35
LOCATION
Is the site’s URL intuitive and easy to remember? 0.30 3
Does the site have good visibility, is it easy to find
0.30 3
WEE-NET
of Excellence
4
with a search engine?
Is it possible to contact the owner or the webmaster? 0.30 4
Is it possible to interact with other users of the site? 0.10 0
3.00
MANAGEMENT
Is the information up to date? 0.50 4
Do the dates of the last updating and creation of the
site appear?
0.20 0
Is the site well maintained? 0.25 3
Does the site use adequate technology? 0.25 3
3.50
USABILITY
Does using the site require standard HW and SW? 0.20 2
Are there devices to help the disabled? 0.20 2
Are page downloading times satisfactory? 0.20 2
Is it easy to navigate through the site? Is there a site
0.10 1
map or help on-line?
Is it possible to choose among languages? 0.20 3
Are the terms and symbols easy to understand? 0.10 3
2.20

WEE-NET
of Excellence
5
Home page - 1998

Evaluation:
August ‘98 - September ‘99
WEE-NET
of Excellence
6
Identity
4,00
2,00
0,00
Content
Services
Individuation
Usability
Management

A Study of Regional Tourist Boards
in the Alps
The project and its objective
The phases of the evaluation process
Main results
WEE-NET
of Excellence
7

The project and its objective
Study line of a triennial project of the
eTourism research group
Goals:
WEE-NET
of Excellence
8
To identify guidelines or
recommendations for the (re)design of a
RTB Web site
To define the characteristics that
determine the quality of the Web site of
a Public Tourist Board

The Alps
WEE-NET
of Excellence
9

Evaluation set-up
Scouting for web sites to evaluate
- to identify the official Alpine RTB sites to
examine
7 in Italy, 8 in Austria, 6 in Switzerland,
WEE-NET
of Excellence
10
3 in France, 1 in Slovenia (analysis of the
national site) and 1 in Germany (26 out
of 34)

Requirements analysis
To delineate the objectives of the RTB and
the needs and expectations of the tourists
Interviews (other lines of the research
group eTourism)
WEE-NET
of Excellence
11
Brainstorming sessions with domain and
evaluation experts
“Market analysis”, …

Evaluation Design
To define the evaluation model -
instantiate the 7Loci meta-model to assess
the quality of an RTB Web site
Brainstorming session with 3 senior and
WEE-NET
of Excellence
12
3 junior researchers - first version of
the model
Formulation of the elements of the
framework into Boolean questions -
reducing subjectivity

Evaluation Design, Cont'd
Revision of the table by two experts - to
reduce ambiguity in the questions; to
reorganize them; to determine which
elements should be considered for the level
of “excellence”
WEE-NET
of Excellence
13
Application of the table by three
researchers to a limited number of sites -
addition, withdrawal or reformulation of
some questions and validation

WEE-NET
of Excellence
14

To determine the survey modalities
Language to use for the assessment
- the official language
Order in which to evaluate the
WEE-NET
of Excellence
15
dimensions
- Identity to be evaluated after Content
and Services

To determine the survey modalities, Cont'd
Order in which to evaluate the sites
- parallel analysis of each dimension of
the sites coming from the same nation
WEE-NET
of Excellence
16
Tools to support evaluation (12% of the
questions)
- Xenu, Watson (freeware)

Realization of the evaluation
First step:
an initial visit to the site overall without
using the table
Second step:
WEE-NET
of Excellence
17
application of the RTB web site
evaluation model (ca. 2h 30’)

Main results
Problems seen with the majority of the RTB
web sites:
A lack of information or services which
were deemed important in the
WEE-NET
of Excellence
18
requirements identification
Insufficient application of some good
practices
Poor use of the potential offered by a
Web site

Lack of information or services
More than half of the sites give no
information
on the traditions and folklore of the
WEE-NET
of Excellence
19
region; e.g., Piemonte, RhoneAlpes
on the handicrafts and typical local
products; e.g., Veneto, Vorarlberg

Lack of information or services,
Cont'd
more than 50% of the sites give no
information public transport in the region;
e.g., Alto Adige-Suedtirol, Bayern
about 30% of the sites make no mention of
WEE-NET
of Excellence
20
the landscape, hiking trails in the
mountains or bike paths
about 20% fail to include information about
sporting activities different from skiing;
e.g., Piemonte, Veneto

Insufficient application of good
practices
Information updated at least for the season
when our evaluation took place (26%)
A high number of broken links: 8% of the
sites have a rate of broken links over 3%;
e.g. Graubuenden
WEE-NET
of Excellence
21
About 8% of sites require a plug-in to
download the home page; e.g. Alpi Liguria
85% of the sites take more than 15’’ to
download the home page (ex’d for a 28.8
Kbaud modem); e.g. Trentino, Tirol

Poor use of the potential offered by
a Web site
Only 15% of the sites offer the possibility to
print at least some of the contents; e.g.
Kaernten, Lombardia
In just over 40% of the sites it is possible
WEE-NET
of Excellence
22
to join a mailing list to receive updated
information or a newsletter via e-mail; e.g.
Bayern, Buergerland
Only a few sites allow the users to interact
with others; e.g. Bayern (forum)

Positive answers : a cross-national comparison
90,0%
80,0%
70,0%
60,0%
50,0%
40,0%
30,0%
20,0%
10,0%
WEE-NET
of Excellence
23
0,0%
Alto Adige
Friuli Venezia Giulia
Lombardia
Piemonte
Trentino
Valle d'Aosta
Veneto
Burgenland
Kärnten
Niederösterreich
Oberösterreich
Salzburger Land
Steiermark
Tirol
Voralberg
Berner Oberland
Canton de Vaud
Canton Ticino
Graubünden
Valais
Zentralschweiz
Provence Alpes-Côte d’Azur
Rhône-Alpes
Riviera Côte - d’Azur
Slovenjia
Bayern

Web site classification (PCA)
Average 1°CP Region
10,43 30,23 Canton Ticino
10,29 29,86 Niederösterreich T
10,29 29,58 Oberösterreich T
10,00 28,14 Tirol T
9,86 28,08 Steiermark T
9,86 27,80 Trentino
9,29 26,82 Bayern
9,57 26,69 Voralberg T
9,43 26,58 Burgenland T
WEE-NET
of Excellence
24
9,29 26,02 Kärnten
8,86 25,57 Graubünden
8,86 25,10 Provence Alpes-Côte d’Azur
8,71 24,60 Salzburger Land
8,57 24,60 Rhône-Alpes
8,86 24,59 Valle d'Aosta
8,71 24,15 Alto Adige
8,43 23,06 Valais
8,14 22,65 Riviera Côte - d’Azur
7,86 21,88 Friuli Venezia Giulia
7,43 21,32 Canton de Vaud
7,43 21,25 Slovenjia
7,57 21,12 Lombardia
7,14 20,06 Piemonte
6,43 17,18 Zentralschweiz
6,00 16,39 Veneto
5,57 14,05 Berner Oberland

“Coordination” Models
Integrated:
Kärnten, Niederösterreich, Salzburg, Steiermark,
Tirol (A); Ticino, Wallis, Régione du Leman,
Zentralschweiz, Graubünden (CH)
Partially integrated:
WEE-NET
of Excellence
25
Burgenland, Oberösterreich, Voralberg (A); Alto
Adige, Trentino, Valle d’Aosta (I); Provence Alpes-
Côte d’Azur, Rhône Alpes, Riviera Côte d’Azur (F);
Bayern (D)
Fragmented:
Berner Oberland (CH); Friuli V.G., Lombardia,
Piemonte, Veneto (I); Slovenia

Further analysis of the results
Comparison of the performances of the RTB
sites (Principal Component Analysis) to take
into account of the contribution of the
different dimensions of a web site and
further revise the evaluation model
WEE-NET
of Excellence
26
A deeper analysis to verify the existence of
different quality profiles (Correspondence
Analysis)

Other developments
Definition of the requirements for a high-quality
RTB web site (excellence profile)
laboratory test in a controlled
environment involving subjects that are
WEE-NET
of Excellence
27
representative of the various actors
Development of the tool to support the
evaluation of the quality of web sites: WET

Web Sites of hotels in the Alps
Results of a study on the quality of the Web
sites for hotels in the Italian Alpine region,
where most hotels are of small or medium
size
The goal was to determine whether and to
WEE-NET
of Excellence
28
what degree the hotels’ Web sites support
their activities of promotion and
commercialization by examining them
through the framework of destination
management

Context
The use of the Internet to organize
journeys and vacations has increased in
recent years
As regard the demands,
TIA found an increase in the percentage
WEE-NET
of Excellence
29
of Americans who book their
accommodations on line, increasing from
57% in 2002 to 71% in 2003
(Travel Industry Association of America, www.tia.org)
in Italy, online accommodations bookings
represented 15% of all guests in spring
2004 but 28.1% one year later
(ISNART, http://www.isnart.com)

Context, Cont'd
In terms of the offering, in recent years the
hotel sector has invested in the
development of Web sites. Nearly 100% of
hotels, especially if they belong to a chain,
either have their own site or are present on
WEE-NET
of Excellence
30
the site of an entity for tourist promotion or
of private tourism consortia

Scope of the research
As the aim was to evaluate Web sites of
hotels in the Italian Alps, the research
extended into the political regions of
Piemonte, Valle d’Aosta, Trentino Alto Adige
and Veneto (Province of Belluno)
WEE-NET
of Excellence
31
From the 3796 hotels in locations within
these regions, 2859 had an e-mail address.
This was the reference population from
which the sampling was extracted

Scope of the research, Cont'd
The sampling for a previous (2003-04)
study consisted of 325 hotels, 248 of which
responded to the e-mail questionnaire
Our research on the hotel Web sites
considered 204 of those 248 hotels,
WEE-NET
of Excellence
32
excluding 44 because they did not have
their own Web site

The evaluation schema, Cont'd
1° step: to identify the aims of the stakeholders –
hoteliers and tourists – that the Web site should meet
Hoteliers: in the Italian Alps where tourism is largely
non-mediated and where the offering is provided by
SMEs, the Web site of the single hotel can support:
promotional activities, e.g., by proposing special
WEE-NET
of Excellence
33
offers for specific groups;
e-commerce by facilitating booking and full or
partial payment on line;
from the perspective of the destination, the
creation of an image that is holistic and reflective of
the larger destination instead of just the single
business

When visiting a hotel site, tourists expect to find
a detailed description of the hotel: type, services,
special offers, pictures, news and events in the
area, cultural activities, sport and recreational
activities, contact information, etc.
It would be even more attractive for the tourist if
WEE-NET
of Excellence
34
the site allowed online booking and/or payment
Hotelier and tourist needs or requirements obtained
in this step were then translated by Web sites
analysts in a first set of questions. For example, to
investigate the information about the hotel, one of
the questions was: “Does the Web site contain
information on facilities available in the rooms?”

2° step: preliminary evaluation to reveal the
macroscopic characteristics and to then calibrate the
evaluation model
The preliminary inspective evaluation done by 2
experts in quality and in tourism showed that
there were notable differences among the hotel
WEE-NET
of Excellence
35
Web sites analyzed, that range from amateur
sites of just a few pages to professional sites; the
dimensions of the sites under examination proved
to be relatively contained, given that most of the
sites analyzed had fewer than 20 pages
These results were used to establish the number
(minimum and maximum) and the granularity of
the questions of the evaluation framework

3° step: choice of the 7Loci meta-model dimensions that
have to be investigated
As regard the hotel Web sites, we had to look for the
presence of those elements related to the expressed aims
of the hoteliers and tourists; they refer mainly to the
presence of information and services available at the hotel
WEE-NET
of Excellence
36
Web sites, that is to the dimensions Content and Services
Conceptual considerations, confirmed through the practical
use of the 7Loci meta-model, have also led to a focus on
these “semantic” dimensions: Content and Services issues
depend on the particular aspects that characterize a given
class of Web sites and allow to differentiate beyond quality
performances taken for granted by the user

The underlying assumption is that the aspects which we
are referring to as “syntactic”, which are general and
therefore are considered for all types of site and in all
domains, are a necessary condition but not sufficient in
themselves to achieve the aims of the site (e.g., the
correct functioning of the site - link, HTML, etc.; its
conformity with standards of language use or of
WEE-NET
of Excellence
37
accessibility as described in normatives such as the Web
Content Accessibility Guidelines of the W3C Consortium or
the U.S. Section 508 Standards (Section 508, n.d.), and
can generally be evaluated for the most part automatically
For a sake of completeness, a few questions were added
regarding the proper functioning of the links
(Maintenance) and about languages (Usability)

The evaluation framework was formulated
by means of an iterative process by a group
of experts in quality and in tourism
The first version of the framework was
tested in order to obtain questions that
WEE-NET
of Excellence
38
were unambiguous and dichotomous (high
objectivity in the answers - using the
framework to evaluate a given Web site,
different people will give the same answers
- but also a certain loss of information)

The evaluation model
WEE-NET
of Excellence
39

Realization of the evaluation
The inspectorial evaluation of the sites was
done in the spring of 2005
The twenty-three questions in the
evaluation framework were answered by a
quality and tourism expert visiting each of
WEE-NET
of Excellence
40
the 204 Web sites
We then calculated the final point totals,
giving one point for each positive: the
“issues” in the framework were all
considered equal in weight

Results
WEE-NET
of Excellence
41
none of the hotels reached the
maximum 23 points possible;
indeed more than half (66%)
have fewer than 13 points

Results, Cont'd
Trentino Aosta
11
Alto Adige
12,87
12,20
14
12
10
8
6
4
2
0
WEE-NET
of Excellence
Piemonte Bellunese
42
11,7 10,27
The average score for region

Risultati (cont.)
WEE-NET
of Excellence
43

WEE-NET
of Excellence
44

WEE-NET
of Excellence
45

Concluding remarks
There few sites that offer functions of e-commerce,
and the number of sites that fail to provide
indispensable information about the hotel and
territory cannot be ignored
The results of the evaluation project confirmed that
WEE-NET
of Excellence
46
for some aspects – e.g., the presence of versions of
the site in other languages – the hotels are able to
respond to new trends in the market, but are
nonetheless still bound by a management approach
that is highly individualistic and far from strategies
of destination management

Web site of the Alto Adige-
Suedtirol
www.suedtirol.info (2005-2006)
Multi-level evaluation:
Standard model to evaluate a set of Web
sites of the main competitors
WEE-NET
of Excellence
47
Detailed inspection to identify
weaknesses (ca.70 pag.)
Controlled experiment to check specific
functionalities (22 users in 4 market
segments)

Competitors’ Web sites
Dimensione Suedtirol Trentino Tirol Kaernten
Contenuto 0,78 0,81 0,81 0,63
Servizi 0,83 0,92 0,83 0,58
WEE-NET
of Excellence
48
Identità 0,80 1,00 1,00 0,87
Individuazione 0,42 0,75 0,92 0,83
Gestione 0,88 0,88 0,88 0,63
Usabilità 0,89 0,89 0,83 0,78

WEE-NET
of Excellence
49

Test navigazione utente n. 10
Giovane;Conosce Alto Adige
Tempi:
· questionario profilo utenti 6’
· svolgimento compiti 32’
· intervista finale 8
Test di navigazione
Compito Tempo Descrizione
00’ 06’’ Digitazione URL
00’ 13’’ Visualizzazione home page
WEE-NET
of Excellence
50
00’ 49’’ Clic sulla funzione “Cerca il tuo alloggio”
00’ 51’’ Viene effettuata una prima ricerca con i seguenti parametri
Località: Alta Badia
Arrivo: 14/4
Notti: 5
Persone: 2
Alloggio: hotel in ½ pensione
Visualizzazione dei risultati
Prenotazione camera
04’ 28’’ Viene effettuata una seconda ricerca con i seguenti parametri
Area: Dolomiti
Località: La Villa
Arrivo: 14/4
Notti: 5
Persone: 2
Alloggio: hotel in ½ pensione
Visualizzazione dei risultati

Conclusions
The method based on the 7Loci:
allowed to realize evaluation projects
characterized by a variety of aims, size,
number, type, goals, etc.
WEE-NET
of Excellence
51
provides a common conceptual
framework that supports the evaluator in
finding the right weight quality model

References
Mich L. et al (2003) Tourist Destinations and the
Quality of Web Sites: a Study of Regional Tourist
Boards in the Alps. In Proc. ICT in Tourism -
ENTER2003, Jan. 29–31, 2003, Helsinki
Mich L., Franch M., Web Sites of the hotels in the
WEE-NET
of Excellence
52
Alps: A goal-driven quality evaluation, Journal ITT,
2007

Part 2. Web sites quality evaluation projects with the 7 loci method

Recommandé

Recommandé

Contenu connexe

Similaire à Part 2. Web sites quality evaluation projects with the 7 loci method

Similaire à Part 2. Web sites quality evaluation projects with the 7 loci method (20)

Plus de Luisa Mich

Plus de Luisa Mich (20)

Dernier

Dernier (20)

Part 2. Web sites quality evaluation projects with the 7 loci method