More than Just Lines on a Map: Best Practices for U.S Bike Routes
Esomar 26 Questions - FAQ for Epinion Online Panel
1. WELCOME TO OUR WORLD EPINION
OF RESPONDENTS
EPINION ONLINE PANEL FAQ 2011
2. ESOMAR is the global organization for pro-
fessional market research agencies.
They have challenged the industry by for-
mulating 26 questions all research buyers
should ask when buying online surveys.
Here are our answers.
If you have further questions, don’t be
afraid to challenge us.
EPINION ONLINE PANEL FAQ 2
3. >3
What experience does your company sources which will provide us with the sort of
01 panelists that may at any given time be in
have with providing online samples for
market research? short supply in our pan we have created a
YEARS IN OPERATION global quarantine system that ensures that no
The Epinion sample only technology has been potential panelist wherever that person
in operation in Denmark for more than 3 years, (browser) is located around the world gets
and recently we passed our first milestone invited to our panel more than once pr. month.
which was 2 million completed interviews on As a result, for each of our country panels, we
>2.0
MILLION INTERVIEWS WAS
the system. We build our first online panel
back in 2001 and has been refining and devel-
oping our online methodologies and
measure and define quality in three dimen-
sions:
approaches ever since. Current size of active panelists
COMPLETED BY THIS SYSTEM
Current monthly inflow of new panelists
Current active recruitment channels
Please describe and explain the types (we aim at 75 or more open channels pr. coun-
02
of source(s) for the online sample that try)
you provide (are these databases,
actively managed panels, direct mar-
keting lists, web intercept sampling, What do you consider to be the primary
03
river sampling or other)? advantage of your sample over other
sample sources in the marketplace?
Our strategy is to resemble as closely as
possible the principle of equal sample- The competitive advantages of the Epinion
extraction probability known from traditional recruitment strategy in terms of delivering
off line research. Our recruitment ensures that superior quality sample originate from two
any online population over an extended period distinct traits of the business.
of time is equally exposed to invitations to join First, we keep very close attention to maintain-
the panel (in no cases, we allow people to opt ing a large number of different recruitment
in to the panel without being invited). We do channels in each country as this is the only
this by making sure that we have a large way to resemble the random probability sam-
number of open recruitment channels at all pling techniques that is used in good quality
times. Moreover, our system makes it possible offline interviewing. We use the term ‘resem-
for us to direct our recruitment efforts to bling’ off line sampling, as it is obviously not in
EPINION ONLINE PANEL FAQ 3
4. any real sense possible to create a known, If the sample source is a panel or data-
04
well defined, universe from which to draw base, is the panel or database used
respondents. However, representativeness is solely for market research? If not,
all about giving internet users an please explain?
(approximated) equal probability to be exposed
for an invitation to join the panel and later on The panel is only used for market research -
to answer surveys. The Epinion approach nothing else. The sample source is a panel
adheres strictly to this principle. consisting of panelists that have actively
Secondly, the unique Epinion sample builder opted in to participate in the Epinion panel.
makes it possible to stratify sample so
precisely that it is rarely necessary to post-
stratify data (weigh data) according to demo- How do you source groups that may be
05
graphics. This is so, because: hard-to-reach on the internet?
Regardless of what combination of Soci- We source these groups by contacting sites
odemographics that are used to define a that are visited primarily by hard to reach
sample (say, women 15-29 years of age), the target groups. For example, in many countries
system ensures that all the subsegments in young low-educated males are a hard to reach
the underlying sample-matrix (region, educa- target group. In these countries, we make a
tion, income etc.) are representatively distrib- special effort in reaching out to local football-
uted based on the latest statistics from the sites where we know that this group is over-
relevant National Bureau of Census. represented.
This is so because the system automati-
cally monitors whether the required number of What are people told when they are
06
interviews in each of the subsegments in the recruited?
samplematrix are reached. If not, more invites
are distributed to the specific subsegment in People are first asked a series of questions
question. This repetitive process ensures mainly related to background demographics
incredibly representative sample. and then they are faced with the following
In combination, these advantages make for question:
superior sample compared to traditional online Would you be interested in becoming a
sampling. member of Epinion's shared opinions panel?
EPINION ONLINE PANEL FAQ 4
5. 8.0
Yes, I would like to join Please describe the opt-in process
08
No, thank you.
The exact wording of the question is slightly In order to become a member of Epinion‘s
INTERVIEWS ON AVERAGE OF different for each person, since we have long internet panel you have to be invited by Epin-
A PANELIST LIFE TIME ago established that different types of argu- ion. We do not accept voluntarily signups at
ments appeal to different socio demographic our website or at other places. All respondents
groups. Young people should be recruited must complete a 4 step verification process to
using a different approach than old people. become members:
4
STEPS TO OPT-IN 07
If the sample comes from a panel,
When invited, first of all:
1. You have to accept our Terms & Condi-
what is your annual panel tions,
turnover/attrition/retention rate and 2. Then you have to accept to participate in
> Accept Terms and Conditions how is it calculated? our establishment survey,
3. You need to provide us with full demo-
> Accept Survey Participation We do not have an exact number for panel graphic background information in the estab-
retention. Actually, we don´t see panel reten- lishment survey,
> Provide demographic background tion as a goal in and of itself. We do not 4. Provide us with your e-mail address and
believe in the panel retention strategy of many then one more time, double confirm that you in
> Provide contact email address other online research companies. It may be deed wish to become a member of the panel.
good for profits and it is even used as a sign
of quality to build up a narrow pool of people Unless all of the above steps have been com-
making good money out of answering ques- pleted using Epinion recruitment system (no
tions. But data from professional panelists affiliate recruitment is allowed), we do not
does not mirror the average consumer. There- consider the applicant to be a member of
fore, we only use prize draws for small Epinion’s panel.
amounts of money as monetary incentives.
However, as we take care to give respondents
good feedback opportunities, appealing invites
and not too long surveys, we have many
panelists that complete multiple surveys. The
average lifetime number of completed surveys
pr. panelist is approximately 8 surveys.
EPINION ONLINE PANEL FAQ 5
6. Do you have a confirmation of identity difficult thing to handle when making cross-
09
procedure? Do you have procedures to country surveys). As a result, it is possible to
detect fraudulent respondents at the make exact comparisons across different
time of registration with the panel? If countries between for example: highly edu-
so, please describe? cated vs. low educated or between occupa-
tional categories.
Since we do not pay individual monetary In the early days of our panel-buildup, we
rewards to respondents, we can rely on e-mail profiled the panelists with lots of different
identification as our main identification proce- variables: Car ownership, newspaper-
dure. All invitations are distributed / linked to a subscription etc. However, it turned out that
unique e-mail address. Without access to this out of perhaps 150 variables we only made
e-mail address, you cannot participate. good use of a few. And often we found that
As part of the survey process we monitor the variables were obsolete when we actually
fraudulent respondents using a series of needed them. In the meantime, the panelist
methods described in greater detail in ques- have bought a new car, changed newspaper-
tion 24. Also, we select and invite panel mem- subscription etc. Therefore, we now take a
bers, they don’t select us. Respondents more dynamic approach to profiling meaning
cannot find ways to self recruit, thus making it that we in ad hoc surveys reach narrow target
more difficult for fraudulent respondents to groups by use of a sophisticated screening
create multiple accounts. system. When wording screeners we take
care to avoid self-selection bias. For instance
by using a wording like: What newspaper do
What profile data is kept on panel you read? (and then present a range of news-
10 papers where one of them is the newspaper in
members? For how many members is
this data collected and how often is question). Instead of asking: “Do you read
this data updated? newspaper A. Yes or No?”. Also, if clients
want to create a specific subpanel, we profile
The Respondents are profiled with socio- over a short period of time a large part of the
demographic criteria. The exact wording of the panel with the requested profile.
profile questions is developed carefully and is
in line with the acclaimed World Values Survey.
This makes sure that categories are compat-
ible across different countries (elsewise a very
EPINION ONLINE PANEL FAQ 6
7. 211,000 What is the size and/or the capacity of Census Bureau.
11
the panel, based on active panel mem- The system automatically monitors whether
bers on a given date? Can you provide the required number of interviews in each of
CURRENT ACTIVE PANEL an overview of active panelists by type the subsegments in the samplematrix are
MEMBERS of source? reached. If not, more invites are distributed to
At all times, one can see the current number the specific subsegment in question. This
of active panelists, the current inflow of new repetitive process ensures incredibly repre-
panelists and the number of active recruitment sentative sample.
Vietnam 41 channels in all of our country panels at
www.respomondo.com/panels.
Sweden 24 An active panelist is defined in a rather narrow Explain how people are invited to take
13
way meaning that is a panelist that have part in a survey. What does a typical
Finland 15 responded to a survey invitation within the invitation look like?
Norway last 6 invitations.
21
People are invited to surveys by way of email-
Ukraine 5 invitations. We work with acclaimed email
Please describe your sampling process newsletter-type companies and consistently
12
Denmark 105 including your exclusion procedures if work hard to maintain an industry-leading
applicable. Can samples be deployed mail-reputation which is crucially important for
MEMBERS OF ONLINE PANEL
as batches/replicates, by time zones, mail deliverability among leading email-
(IN THOUSAND)
geography, etc? If so, how is this con- providers (Hotmail, Yahoo etc.).
trolled? We operate with many different types of
invitations depending on the type of survey.
The sampling process is at the very heart of But under no circumstances can an invitation
what makes Epinion very special in terms of be sent out without active decline and panel-
online quality sampling. opt out links as well as the accept link itself.
Our clients or we can on behalf of the client
define a sample (say, women 15-29 years of
age) and then the system will ensure that all
the subsegments in the underlying sample-
matrix (region, education, income etc.) are
representatively distributed based on the
latest statistics from the relevant National
EPINION ONLINE PANEL FAQ 7
8. Please describe the nature of your How often are individual members
14 15
incentive system(s). How does this contacted for online surveys within a
vary by length of interview, respondent given time period? Do you keep data
characteristics, or other factors you on panelist participation history and
may consider? are limits placed on the frequency that
members are contacted and asked to
Over the cause of the years, we have done participate in a survey?
quite a bit of research on what incentives
work the best. As a result, we have structured Yes, we apply very stringent respondent
our incentive/motivational system around the quarantine practices. No respondent is invited
following: to more than one survey every 12 days. They
Monetary incentives: We almost always only only receive one reminder per survey invitation
use prize draws. We draw frequent winners We save full statistics on invitations sent,
but we award only small amounts of money at responses (accept, decline), mail-server statis-
a time. We´ve found this works better than tics, participation records on all respondents
infrequent, bigger, awards. Even more, our as long as they are members of the panel.
findings suggests that micropayments to all
participants do increase participation a little bit Is there a privacy policy in place? If so,
16
but first and foremost, it creates more flawed what does it state? Is the panel com-
answers, more fraud, and above all a pool of pliant with all regional, national and
professional panelists which in no meaningful local laws with respect to privacy, data
way can be said to be representative for the protection and children e.g. EU Safe
average consumer. Harbour, and COPPA in the US? What
Feedback/Motivational incentives. After each other research industry standards do
survey, each respondent are prompted to give you comply with e.g. ICC/ESOMAR
immediate feedback both by ranking the International Code on Market and
survey and by giving verbatim feedback. This Social Research, CASRO guidelines
provides the project manager with valuable etc.?
feedback, but even more importantly, it gives
respondents an opportunity to vent whatever We adhere to national and local laws, and
feelings they have about the survey. This adds beyond this, we follow ESOMAR´s International
to our general “know-and-appreciate-your- Code on Market and Social Research.
respondent” policy.
In some instances, we offer the respondent
access to selected results from the survey
and other specifically targeted incentives.
EPINION ONLINE PANEL FAQ 8
9. 30-45%
PARTICIPATE RATE
17
What data protection/security meas-
ures do you have in place?
19
Do you conduct online surveys with
children and young people? If so,
please describe the process for obtain-
Epinion is concerned with safeguarding the ing permission
information gathered through our Services. We
employ a variety of measures designed to In line with the ESOMAR code of conduct, we
protect your information from unauthorized do not commission surveys among children
access. We work only with wellknown and unless we have asked their parents for
1-25%
RANGE OF DROP-OUT RATE
professional database suppliers and only
authorized personnel are allowed to conduct
database operations.
permission.
Do you supplement your samples with
20
samples from other providers? How do
Do you apply a quality management you select these partners? Is it your
18
system? Please describe it policy to notify a client in advance
when using a third party provider? Do
As most of our survey processes is embedded you de-duplicate the sample when
10-15% in sophisticated IT-technology much of the
quality control is embodied in the
IT-development process itself. In this process,
using multiple sample providers?
Occasionally, we use some of our partners to
AVERAGE DROP-OUT RATE
a variety of techniques has been put to use in provide sample in countries where our own
manual as well as automated testing. panels are too small or over-extended. We only
In terms of work-procedures between different use Partners from a preferred supplier list.
departments and individuals and with respect That is, trusted partners with proven quality
to the content of deliveries designed and and service. As we do not split-sample
produced by our consultants or other staff, between our own panel and partner-panels, we
our processes and deliveries are internally do not need to deduplicate sample between
audited every 12 months. two different panels.
EPINION ONLINE PANEL FAQ 9
10. Do you have a policy regarding multi- gestions to improve the content of the survey
21
panel membership? What efforts do and we very often penalize surveys with high
you undertake to ensure that survey drop-out rates.
results are unbiased given that some It is our philosophy to maintain a healthy
individuals belong to multiple panels? relationship with the respondents to ensure
repeat business. In order to maintain that
Our panel management system automatically relationship, we often reject long surveys (or
controls that there are no duplicate emails. In insist that they broken down into several
addition, in each survey, by way of browser- independent surveys), and evaluate the
session, IP-address and cookie-tagging, we language used in the surveys using the
make sure that no respondent can answer any WYMAT-principle (Would Your Mother Approve
given survey (including tracking surveys) more This?).
than once.
Do you maintain individual level data
23
What are likely survey start rates, such as recent participation history,
22
drop-out and participation rates in date of entry, source, etc., on your
connection with a provided sample? panelists? Are you able to supply your
How are these computed? client with a per job analysis of such
individual level data?
Participation rate is usually between 30-45%.
This rate depends greatly on a series of We maintain a full record of all individual
parameters, like time of day the invitations are participants. Since we source only respond-
distributed, the day of the week, the target ents from our own in-house fully profiled
group selected, e-mail invitation text provided panel, we are able to retrieve a full log of all
etc. actions performed by our system in regards to
Drop-out rates are between 1-25%, with an a single respondent, as well as a full record of
average around 10-15%. The dropout rates all the respondent actions in relation to our
depend greatly on length of the survey, as survey invitations.
well as the content of the survey.
We continuously monitor drop-out rates,
accept-rates etc. on each survey. If perfor-
mance is below average, we introduce sug-
EPINION ONLINE PANEL FAQ 10
11. Do you use data quality analysis and rewarded as ‘payment’ for each completed
24
validation techniques to identify inat- survey.
tentive and fraudulent respondents? If We do not require that participants fit a spe-
yes, what techniques are used and at cific profile in order to receive the reward – in
what point in the process are they this case participate in the prize-draw. If you
applied? spend time answering our surveys, you partici-
pate in the prize-draws no matter if the partici-
We use a mix of different techniques to pant fits the target group or not.
identify fraudulent users. We combine ‘hard’ We have completely removed the incentive to
data from the participant’s computer/browser provide false answers in order to gain an
with behavioral data analysis in order to iden- advantage by earning money, points etc.
tify fraudulent respondents.
Hard data consist of: Session cookies,
browser cookies, IP-address validation, e-mail
validation (double opt-in required, no e-mail
equals no survey) + a wide range of different
browser plug-ins. We utilize a wide range of
information about the browser used as well as
the network connection used in order to verify
the participant. All these hard data are com-
bined with our ability to monitor deviations in
behavior when the participant completes the
surveys. We analyze features like: incoherent
response-patterns, average time spend devia-
tions, probability of completing open-text
answers etc. Together, all these information’s
create a credibility-index, which allows us to
monitor suspicious behavior closely.
But our most effective weapon against fraudu-
lent behavior is to remove the monetary incen-
tive to perform fraud. Epinion almost always
offers price-draws as a reward for participa-
tion. No cash rewards, rebates or points are
EPINION ONLINE PANEL FAQ 11
12. Do you measure respondent satisfaction? survey or questions that are difficult to understand) that requires immedi-
25
ate action is automatically forwarded to the project manager by e-mail.
Yes, after every survey is completed, all respondents are motivated to give The feedback collected from the respondents is then combined with hard
feedback. A typical feedback page looks like this (but will be modified to data about completion rates, average interview time etc. into an aggre-
suit different brands and languages): gated easy to read feedback report format.
We also use the invitation e-mails to motivate the respondents to provide
us with feedback, also those respondents who fail to complete the survey.
Epinion measures respondent satisfaction on five comparable parameters
as well as collecting open verbatim feedback from the participants of each
survey. This feedback goes directly to the client project manager. The
project manager can always access the respondent feedback report which
is updated in real time, as the respondent feedback comes in.
Critical feedback from the participants (like technical problems with the
EPINION ONLINE PANEL FAQ 12
13. What information do you provide to debrief your client after
26
the project has finished?
Along with the respondent feedback report shown above, the client can at
all times access the current completion report showing exactly the
progression of the survey in its lifecycle.
Dropout rates, participations rates, invitations rounds executed, success
rates etc. are all supplied in an easy to read interface.
Epinion also provides the project manager with a ‘General
Statistics‘-report benchmarking the survey performance of the project
manager with Epinion averages internationally.
The same report evaluates not only current performance, but also past
performance of the project manager’s surveys, and provides recommen-
dations on how to improve future survey projects.
See slide from benchmark report below
format.
EPINION ONLINE PANEL FAQ 13
