The document proposes a new model for advertising pre-testing based on prediction markets. It argues that current pre-testing methods using focus groups are unreliable and fail to capture diverse, independent opinions. The proposed model would use a prediction market approach to aggregate responses from a decentralized group. This would better align with the "wisdom of crowds" principle of tapping into collective intelligence. However, several challenges must be addressed such as how to provide real-world outcomes to judge predictions and how to motivate diverse strategies among participants in a single pre-testing session. The document also discusses moving from introspective individual responses to having participants predict how others would respond to ads.

1. Predictive Pre-testing:
A New Model for Ad Pre-Testing Based on Prediction Markets
Iqbal Mohammed
Twitter: @misentropy | Web: www.misentropy.com
misentropy@gmail.com
Abstract: In the advertising industry, there's
widespread agreement that ad pre-testing (or copy
testing) is a creativity killer. Current research
methodologies using focus groups to test innovative
ideas and products have poor predictive ability, often
throwing up lazy groupthink, post-rationalisations
and risk-averse generalisations. This paper argues for
a revamped model of ad pre-testing that harnesses
collective intelligence to improve forecasts of real
world response by a significant factor. Drawing
insights from a range of business implementations of
'wisdom of crowds', the paper presents prediction
markets as a tool to effectively aggregate the
responses of a research group that is characterised by
diversity of opinion, independent thinking and
decentralisation. The updated model takes care to
align the interests of research respondents with those
commissioning it by incorporating gamification
techniques to draw out deeply-held private
information about their cohorts and the world they
inhabit. The paper also suggests additional areas of
exploration and improvement while implementing
the concept into research practice.
Introduction
Few suspected culprits are as routinely prosecuted
and condemned in public as advertising pre-testing
is.
If the conversations in the plannersphere (the
collection of blogs by advertising planners) are
anything to go by, ad pre-testing has assumed
Rasputin-like proportions. A scoundrel that everyone
detests and one who routinely encounters every
known killing device – poison, bullets, lynching,
drowning, hanging, etc. - only to survive on.
Famous men have been called in to provide the
deathblow with their guillotine-edged words – Akio
Morita, Aldous Huxley, Henry Ford and David
Ogilvy. But to no avail.
Which is not to say, ad pre-testing doesn’t have its
defenders. Research agencies, who provide the
service, parade its benefits – while also pointing out
its limitations. The latter only serve to emphasize
how their own ‘much improved’ version of the ad pre-
testing model overcomes those limitations.
And then there are the clients. The reason why – it
seems – the discipline of advertising testing itself
exists in the first place. Despite the thundering words
of some of their ilk (“Nike never pre-tested any of its
campaigns, and we took the responsibility of what we
were creating rather than passing the buck” – Scott
Bedbury) – in popular perception, clients are only
too willing to forgo their own prerogative to decide
and pass it instead to a group of respondents with
nothing to do on a working day afternoon.
While the slugfest between its defenders and
opponents continues, the discipline itself - and the
widely used model of advertising pre-testing -
remains ignored. While research agencies routinely
add new trinkets to their services, the heart of the
contraption itself has undergone very little change.
In fact, in the words of the Advertising Research
Foundation – the industry body for market research
in the US – “For the most part, there's been no wide
scale significant innovation in copy testing and
tracking (except maybe data collection methods) in
50 years."
This paper argues that the time is ripe for the
thinking behind advertising pre-testing to be re-
visited. It presents an updated model incorporating
new paradigms of collaborative thinking and
decision-making that are currently gaining ground in
business.
But, first, here’s a round up of the common
charges against ad pre-testing – as it is practiced
now.
Criticisms of current ad pre-testing models
1. The correlation between the stated intent of
respondents and their actual behavior is very
low.
2. Pre-testing is a rational exercise that’s
fundamentally incapable of capturing emotional
responses to advertising.
3. Cognitive research shows that a significant part
of the human decision-making process takes
place in the subconscious. If that’s the case, a
conscious walkthrough of a consumer’s reactions
to advertising hardly has any bearing to their
ultimate actions.
4. Moderators often end up leading the respondents
to express certain points of view.
5. Respondents have a propensity to express snap
views and then post-rationalise.
6. Alpha respondents can often sway the opinions
of the rest of the group.
7. It’s human tendency to criticize and most of the
respondents are keen to do that – even if they
have nothing worthwhile to say.
8. The motivations of the respondents aren’t in tune
with those of the researchers – the respondents
are there for the money or for the free food or,
even worse, because they love the sound of their
own voice.

2. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 2
9. Respondents are often unable to visualize the
presented concepts (animatics, storyboards) in
finished form – a big concern when the ad being
tested is execution-based. In the words of Simon
Clift (global President of Marketing for Unilever -
Home & Personal Care), “To me the excessive
reliance on animatics is crazy – like choosing
your wife from a stick drawing."
10. Ads that are simple to understand and the ones
that stand out, tend do well.
11. By definition, new unfamiliar ideas are expected
to cause discomfort in an audience. So any new
thinking – which all advertising is expected to be
– won’t test too well. Thereby defeating the very
process of testing.
12. Ad pre-testing is reductive – trying to isolate
variables, whereas the effects of advertising itself
are emergent.
13. Pre-testing – instead of being used as a
diagnostic/optimization tool – often ends up
being used as a substitute for judgement.
14. The tendency among clients is to use pre-testing
as a tool to tackle the fear of making a mistake –
rather than to supplement the decision-making
process.
15. Some of the world’s most successful advertising
campaigns and products (Sony Walkman, Dyson
Vacuum, Aeron Chair, Seinfeld, computer
mouse) have failed research. (The research, in
these cases, was wisely ignored.)
16. Advertising pre-testing doesn’t match the
environment under which consumers watch
advertising – as ad breaks between blocks of TV
content. And in a synergistic rich real-life
environment incorporating press, outdoor, PR,
TV, radio, online, word of mouth, etc. Testing
that embeds ads in TV content costs a lot more.
17. The use of one-size-fits-all pre-testing models for
widely differing kinds of advertising.
18. Some studies claim that there’s a negative
correlation between ads that have pre-tested well
and those that have won IPA Effectiveness
Awards.
19. The Advertising Research Foundation chimes in
that “Because of emphasis on cognitive/rational
measures, current copy testing techniques cause
a regression to the mean, thereby reducing
advertising effectiveness.” In short, ARF
believes ad pre-testing reduces advertising
effectiveness.
20. Research companies feel the need to show value
to clients and thereby test the concepts to
destruction – only to create a report that’s many
times larger than the communication itself.
While the list itself seems endless, the above
criticisms about ad pre-testing fall under two broad
categories. The reliability (or lack of reliability) of
information gathering from respondents and the
subsequent use of research data itself – not as
something that informs the decision-making process
but as something that substitutes it.
Any attempt to create a better model of ad pre-
testing should address these two broad criticisms –
either tackle them effectively or define its usability
with greater clarity.
Wisdom of crowds?
Although none of its opponents or its defenders have
mentioned as much, but ad pre-testing is a
particularly bad implementation of a prediction
market using the ‘wisdom of crowds.’
Research agencies, in fact, have tended to de-
emphasize ad pre-testing’s predictive abilities and
instead focused on the diagnostic and optimization
possibilities. But the overwhelming belief that ad pre-
testing aims to provide predictive ability is difficult to
shake off. Even its nay-sayers seem to think that ad
pre-testing cannot predict market success – not that
ad pre-testing isn’t designed to predict market
success.
If indeed ad pre-testing is being viewed and used
as a predictive tool – then it’s particularly poorly
designed to either predict market success, or to tap
into the undeniable wisdom inherent in the crowd,
even in a group of focus group respondents.
James Surowiecki – who coined the phrase
‘Wisdom of Crowds’ and expanded on the concept in
a book of the same name – lists 3 important elements
that differentiate the wisdom from the folly of
crowds.
a. Diversity of opinion: If the group isn’t
representing a diverse set of opinions, chances
are that it’s going to have a one-sided take on
everything. In particular, every person should
bring his or her own unique ‘private’ or ‘tacit’
information to the group. This increases the
width and depth of information held by the
group – thereby maximizing its ability to
accurately predict outcomes.
b. Independence: The opinions of individuals
should be independent and not easily influenced
by those of others. In the absence of
independence, we get information cascades
which destroy the diversity of opinion within the
group.
c. Decentralization: A particular kind of
decentralization, – where self-motivated
individuals champion the information they
possess, rather than co-operate and compromise
explicitly – is also critical to the success of a
‘wisdom of crowds’ enterprise.
In addition to the above, a successful ‘wisdom of
crowds’ gathering should also incorporate some form
of mechanism (like a stock market, for eg.) that
aggregates the information within the group,
crunches it and provides predictive data that then
can be tested against a real outcome.
(Contrary to popular belief, this aggregating
mechanism need not always be overtly complicated
like a stock market. In promotion contests where
participants are asked to guess the number of balls in
a container – the average of all participants is almost
always better than the guess of the best performing
participant. In this case, the mechanism of averaging
serves to aggregate the collective knowledge of the

3. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 3
group and provides a close-to-accurate result –
provided the above 3 conditions are met.)
The current models of ad pre-testing flout all the
above 3 essential principles of the ‘wisdom of
crowds.’ Recruits are often defined with a few paltry
variables and end up resembling (and echoing) each
other more often than not. Pushy moderators and the
focus group itself (with the ubiquitous alpha
male/female) create powerful ‘information cascades’
that squeeze out all traces of independent thinking in
the group. And focus group dynamics doesn’t give
enough incentive for all the participants to champion
their own viewpoint passionately – resulting in
centralization of thought and opinion, rather than the
opposite.
Added to that, the aggregation mechanism used in
ad pre-testing is a poor collective tool. Instead of
dynamically mashing all the information inherent in
the group, it merely ends up representing passive
data.
Prediction markets
Decision markets, much like stock markets, are a
particularly elegant and well-designed means to
aggregate the wisdom of crowds. By providing the
means – diversity, independence and
decentralization - for the crowd to be smarter than
the individual, decision markets are finding
application in a wide variety of business situations.
Several companies have been putting decision
markets to good use. Yahoo has instituted in-house
decision markets to help it decide which technologies
hold promise for the future. Arcelor – the largest
steel producer in Europe and Latin America – uses
prediction markets to accurately project quarterly
variations in sales volume and prices of steel.
When HP ran prediction markets among its
employees to forecast computer workstation sales –
in 6 out of 8 cases they were more accurate than the
internal corporate forecasts.
Over the years, Hollywood Stock Exchange
(HSX.com) - a multiplayer online stock market game
trading in Hollywood movies and stars – has proved
uncannily accurate in the prediction of Oscar winners
and box office results. To date, there is no better
indication of a movie’s first week loot than the price
the movie is trading at on HSX. In fact, Hollywood
studios use the information from HSX to make
advertising and promotion decisions.
Challenges to setting up an ad pre-testing
decision market
But before we rush to design a prediction market
around ad pre-testing, there are many challenges to
be considered and overcome.
a. All decision markets rely on an infusion of
information from the real world – to enable it to
decide who the winners and losers are. For
example, when employees of Arcelor predict
variations in sales volume and price of steel, their
predictions are judged against the real market
figures when they come in. And the ones getting
closest to the target are deemed the winners.
But ad pre-testing has no provision for such
an infusion of information from the real world.
Since ad pre-testing is an exploration to seek
what might work well, the ads in question might
not even run – and most won’t run in the form
they are in. So it’s hard to attribute success or
failure to any prediction made during the course
of an ad pre-testing session. And even if such
information is forthcoming, it may take months
to arrive.
b. Most decision markets use a variation in wealth
in the market as a key motivating factor for
participants. Differing wealth (either in play
money or in real money) creates differing risk
profiles and thereby gives players a reason to dig
deep within themselves to seek out the all the
information they need to make the right decision.
Different levels of wealth also give rise to wide
range of strategies – rather than resorting to
simple ones.
Differences in wealth arise only when the
game has been iteratively played over a period of
time (most decision markets start with players
getting a fixed amount of money.) Since ad pre-
testing takes place over one session, it is difficult
to introduce the notion of wealth (either with real
money or play money) and an unequal
distribution of it. More so, to use it to enable
different investing strategies.
c. All the examples of decision markets mentioned
above are software-based and hosted online –
and run over prolonged periods of time. They
cost a fair bit in terms of investment, and by
creating a market for information – they require
that almost all information be shared with the
participants. Which is the reason why, most
corporate implementations of decision markets
are usually in-house and for employees only.
It seems, the only way to set up a massive
multiplayer decision market based on ad pre-
testing is to have companies (and often
competitors) co-operating across the board and
allowing their communication efforts to be
visible to everyone. Surely, not something that
will come to pass – now or ever.
Upstream or downstream?
Some of the augmentations to the ad pre-testing
model have attempted going further downstream –
ignoring what the individual is saying and listening
instead to the biological cues like MRI, EEG, etc. By
measuring a person’s biological responses rather
than his consciously aired views (or by corroborating
both), these models claim to capture reactions more
accurately.
These advances, of course, increase the cost
phenomenally – which is why they have few takers.
Plus, moving data collection further downstream
doesn’t really solve the problem of aggregating it into
a cohesive mashed-up whole.

4. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 4
One alternative to current thinking is to move the
respondent upstream - not to treat the respondent as
a representative of his own opinion but as a
barometer of other people’s opinions. By doing that,
you’re immediately removing the introspectiveness,
and associated issues, that tend to dog focus groups.
Instead you are focusing the respondents’ energies on
using all his knowledge about his peers and how they
would respond to the ad in question.
Carnie Mellon University professor Luis Von Ahn
has put a similar trick to use to solve the intractable
problem of how to label pictures on the net. Left to
themselves, people could additionally label a picture
of a rose as ‘pretty’, ‘red’, ‘flower’, ‘expensive’,
‘repulsive’ or anything else. Plus, they might make
typos, use a different language or even a slang term –
making the task of searching the picture back from
an archive extremely difficult. If you didn’t know the
mind of the labeler – then you would only be
guessing at a picture-label combination.
To counter this problem, Luis Von Ahn created a
game called ‘The ESP Game.’ Two randomly paired
online players are shown the same picture and asked
to generate labels for it – but with a twist. They are
both challenged to guess what labels the other
anonymous player is thinking of – and score a point
for every correct guess. So, personal preferences and
idiosyncrasies go out of the window, and both players
are now concentrating their energies on universally
accepted attributes of a rose, or whatever else they
are labeling. The game retains those labels that both
players successfully guessed and throws out all labels
that didn’t have a match – this also automatically
throws out typos and other mistakes, unless both the
players make the same typo, a miniscule probability.
It helped that it was an addictive game. Within 4
months of debuting the game, as many as 13,000
players had accurately produced 1.3 million labels for
some 300,000 images – all at no cost. Not only that,
because of the clever way in which the data was
collected, the labels tended to be subtle and nuanced
– capturing not just what’s in the picture but the
mood of it too. (A search for ‘funny’ returns pictures
of Ronald McDonald being hauled away by police and
Queen Elizabeth picking her nose.)
A few months ago, Luis von Ahn demo’d the game
to Google – who have now commercialized it as
Google Image Labeler. It is now being put to use to
make the company’s database of images better and
smarter.
The Upstream Advantage
As it turns out, taking a leaf out Luis von Ahn’s book
and turning the tables around solves many of the
crucial issues and challenges that our enterprise
encountered.
a. Asking each one of the focus group to guess
which particular ad the majority of the group will
end up liking provides an unexpected bonus –
the infusion of information from the real world
that was missing earlier. Their votes not only
provide their own individual guesses but when
aggregated provide the overall guess of the group
as a whole – which approximates for data from
the real world and serves to decide if the
individuals got it right or wrong.
b. Asking the respondents to guess what the others
in the group might end up liking (instead of what
they themselves like) also solves one of the
biggest criticisms of ad pre-testing. The
authenticity of the respondents’ responses. When
they are forthcoming on their own reactions
about the advertising, we have no way to
correlate that either their own subsequent
behavior or with their own genuine but unstated
reactions.
But when the question is posed as a game
(with an incentive thrown in), we have no reason
to doubt that they are putting their minds and
understanding of other consumers like them
(friends, colleagues, relatives) to good use.
c. One of the criticisms levelled against ad pre-
testing is the inability of respondents to imagine
what advertising in unfinished form - animatics,
ripomatics, storyboards – might end up looking
like. This, it’s rightly argued, results in skewed
testing.
One of the reasons why respondents are
unable to build on advertising in unfinished form
is because they don’t see themselves on the side
of the creators. They see it as outsiders – thereby
discounting all the potential and possibilities the
idea has.
But when involved in an ad pre-testing
exercise as a game to guess other people’s
reactions, most respondents are likely to imagine
each unfinished ad in the best light – to be able
to serve their own interests within the game. In
short, they are likely to assess each ad as an
insider – to see how other people might react to
it, and gain from that.
d. Another unexpected bonus of inverting the
paradigm is that it multiplies the number of
respondents immediately – though we still have
the same number of people in the room. Each
respondent is now bringing in information from
about a dozen of his peers and friends –
effectively widening the net for no increase in
cost.
e. Finally, the paradigm of a game where each one
is trying to guess what the overall group is
thinking also ensures that we don’t have
respondents who are just there for the pocket
money and free snacks. The snacks might still be
free – but as we’ll see, they will have to work hard
to get the money.
An outline of an ad pre-testing model based a
pari-mutuel prediction market
We begin by assembling a group of 10 to 12
individuals based on the demographic requirements.
But instead of merely ensuring the group conforms to
the demographic requirements – we ensure that
there’s also diversity (men, women, professions, ages,

5. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 5
hobbies, etc.) within the defined set. The more
diverse the group – the better the results will be.
Each individual in the group is handed a sum of
money, say $ 25 – real money that they need to
wager during the course of the session. What they
win, they take back with them. If they lose part or all
of it, they return empty-handed.
The session itself is broken into 5 rounds. Each
round consists of viewing a clutter tape of 5 to 6 ads
– after which the moderator asks all respondents to
vote for the ad that the majority in the group will like
the best. Along with the vote – written down on a
piece of paper and handed over to the moderator –
each respondent wagers $5 of his money on the
outcome in every round.
At the end of each round, the votes are tallied and
one of the ads shown in the round emerges as a
winner. All respondents who voted for the winner
will then share the total money wagered in that
round. For eg. if 6 people out of 10 voted for a
particular ad – then they get to share the spoils
(which in this round will be $ 50.) Each of the 6 will
get $8.33 back – while the remaining 4 will lose the $
5 they wagered.
The first 2 to 3 rounds ideally are dummy rounds
– to get the respondents warmed up and to ensure
they get the hang of things. By the end of round 3, the
respondents will have a fairly good idea of how the
system works – not to vote for the one they like, but
to vote for the one with the highest probability of the
group’s approval. In doing the latter, they are
considering a wider set of variables to judge the ads –
and not just one’s own biases and opinions.
The ad/ads that are being tested should ideally be
introduced round 4 onwards. They could be tested in
more than one round under differing criteria – which
of the ads will the group like the most, which
ad/product will the group most likely buy into or
which of these ads will the group consider as one that
stands out the most, etc. Or, alternatively they could
be tested in more than one round for the same
criteria, but surrounded by different sets of ads.
At the end of all the rounds, the winners take
home the cumulative money they have won in all the
rounds. It is possible that some of the respondents
will take home nothing, having lost each of the
rounds.
The results of the ad test rounds will indicate a
fairly accurate picture of the groups’ collective
reaction to the ads being tested. If all the hygiene
factors have been observed until then, the results will
indeed be predictive of how the ads will be received
by the market at large – because the group in
question is working together to arrive at that very
same prediction.
The results may often be unpalatable to the client
and agency in question. And in the currently
practiced form of ad pre-testing, that disappointment
is usually wished away under a maze of questions,
answers, interpretations – and doubts over
authenticity of the reactions and the methodology
involved.
Those routes are unavailable in the prediction
market model – the results are stark and clear more
often than not (except in the case of close voting).
And this is the biggest concern while implementing
prediction markets – the ability to handle the truth
that emerges. And because of clear and transparent
methodology, there’s very little one can do to pad the
truth.
Additional factors to consider while
implementing an ad pre-testing prediction
market
1. It’s worth emphasizing once again that the data
that turns up in a prediction market is very
unlike the reams of data that traditional pre-
testing throws up. It’s fairly terse and
transparent (in a collection of a certain set of ads,
the ad in question totaled so many votes) and
aggregated by the dynamics of the game rather
than by third party analysis.
Layers can be added to data results from a
prediction market by testing the same ad in more
than one round for different criteria – emotional
connect, ease of understanding, hip factor, or
buying response. The difference from traditional
pre-testing is that you ask the respondents which
of these ads will score high on emotional connect
(or whatever else) with the entire group – and
not just in their personal estimation.
2. While the model presented above mentions only
10-12 participants – there’s no upper limit to the
numbers one can recruit. An arbitrary limit,
however, will have to be imposed by other factors
– coordination and cost.
From the point of the prediction market, the
more the participants, the more accurately
predictive the results will be (as long as they
don’t compromise on the clause of diversity.)
3. Similarly, while the above model suggested 5
rounds of testing, one can theoretically add as
many rounds as one needs – thereby testing the
same ad on various parameters. The limiting
factor in this case will be cost.
Another factor to consider while using the
same ad in multiple rounds is information
leakage. If the results of the last round indicate
one winner, then participants might lazily pick
up the same one is subsequent rounds. One way
to stop that happening is to withhold the results
from the rounds with the same set of ads but
different criteria – until all the rounds are played
out.
4. It’s also a good idea to use the first 2 rounds to
test ads that are already in market and for which
market success numbers are already available.
This yields valuable data and enables calibration
of the group’s effectiveness and skew (if any),
particularly useful when judging the group’s
collective decisions on the ads being tested.
5. The first 2-3 rounds can also be used to create a
mood for the later rounds where ads will be
tested – either by featuring ads in the same

6. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 6
product category or ads featuring similar kinds
of execution. Animatics or other unfinished
forms of advertising can also be introduced in the
earlier rounds to enable the participants to warm
up to those techniques.
6. One drawback of putting all the participants in
one room is that it could lead to collusion and
game-fixing between the participants. Ideally, the
participants should be geographically dispersed
– preferably each in a different room.
Alternatively, 2 to 3 participants can be
present in each room and given the chance to
consult with each other before putting down their
vote. The discussion will make their decisions
more rounded, while the numbers will limit the
possibility of an information cascade within the
larger group.
7. An alternative way to set up the above prediction
market is to eliminate the people who don’t get
the right answers in each round. It then
mandates that you have a sufficiently large
number to begin with – to ensure there are more
than a handful left for the final rounds.
The advantage of eliminating people in each
round is that you’ll have the clued-in bunch left
behind – minus the nonclued-in ones to cloud
the voting. The drawback, on the other hand, is
that you run the risk of reducing the group to a
homogenous non-diverse one, when the final
rounds take place.
8. While the prediction market described above
creates variation in wealth as the rounds
progress (some participants are winning back
more than the $5 they invested in each round
and others are losing all of it) it doesn’t actively
allow the differing levels of wealth to be
leveraged during the rounds (the maximum and
minimum one can bet on each round is till $5) or
for it to affect tactics.
More complicated schemes can be introduced,
for eg. where each participant gets 3 votes per
round – and he can vote for multiple ads, while
also wagering different amounts of money on
each vote. Such schemes, however, might end up
confusing participants – and therefore making
the results seems awry.
9. Unlike traditional ad pre-testing that also serves
as a diagnostic and optimization tool, an ad pre-
testing prediction market only provides
predictive data about what the group thinks.
However, post the session, the participants –
especially the ones with the largest prize money,
indicating the best performers – can also be
quizzed in regular ad pre-testing format to
enunciate what they think about the ad/ads
being tested. After having spent a full session
considering the advertising in question from all
possible angles, they’ll be primed to provide
useful information on what they think works and
what doesn’t – and why.
10. In traditional pre-testing, the participants are led
to indicate additional insights into what part of
the advertising might not be working and why.
In an ad pre-testing prediction market, the
advertisers might have to do the hard work of
throwing up all alternatives – in the same ad or
as different concepts – right at the beginning.
Prediction markets don’t excel at throwing up
alternatives – but in choosing which one of the
alternatives works the best.
Some of the alternatives to consider – and test
– are different ways the same plot unfolds or
different call to action approaches, etc.
Conclusion
For a new ad pre-testing model to be considered as
an advance, we isolated two key improvement areas.
Enhance – by a huge quantum – the reliability of
information collected and, two, prevent the misuse of
results.
The ad pre-testing prediction market described
above scores very well on the first count. By inverting
the paradigm of introspection and by adding the
elements of a game, it provides the incentives
(financial and otherwise) for participants to seek and
arrive at the best predictions. Any skews the data
might throw up are more likely the result of the
group lacking diversity or numbers (or both) and are
unlikely to be the outcome of an individual’s
shortcomings.
By clearly focusing only on predictive data about
an ad’s likely market success – it also solves the
second requirement – the prevention of pre-testing’s
misuse. Clients and their agencies can choose to
accept or ignore the prediction markets results – but
with the process being completely transparent and
clear, they can hardly use it to suit a pre-conceived
agenda.
Advertisers seeking diagnostic and optimization
data for the ad campaign they are measuring have
two options. They can either run the additional
discussion sessions described in Point 9 of the
section above or choose to run a traditional ad pre-
testing exercise.
The availability of a prediction market as a pre-
testing option especially serves to clarify the objective
of a pre-testing exercise, right in the beginning. Are
we seeking predictive data to supersede our own
judgement or qualitative data that can inform our
decisions?
This clarity in objective then dictates which one of
the two methodologies we should pursue.
It’s a choice that wasn’t available. Until now.
ABOUT THE AUTHOR
IQBAL MOHAMMED is a brand and marketing
strategist whose area of specialization lies at the
intersection of advertising, information systems and
economics.
He is the winner of the WPP Atticus Award 2006
for best original published writing in the 'Branding
and Identity' category.
To subscribe to email updates of his latest papers,
visit www.misentropy.com/samizdat.html

7. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 7
REFERENCES
(A linked references list can be found online at
http://www.misentropy.com/references.html)
 James Surowiecki, 2004. The Wisdom of Crowds: Why
the many are smarter than the few. Abacus Publishers.
 Fredrik Sarnblad, 2006. Goodbye Innovation at
http://fredriksarnblad.wordpress.com/ 2006/05/09/
goodbye-innovation/
 ‘Workers, Place Your Bets’ Businessweek, August 3rd
2006 at
http://www.businessweek.com/technology/content/a
ug2006/tc20060803_012437.htm
 ‘Hollywood Games People Play’ Businessweek, August
7th 2006 at
http://www.businessweek.com/technology/content/a
ug2006/tc20060804_618481.htm
 Leland Maschmeyer, 2007. Predictive markets better
than ad testing? at
http://whistlethroughyourcomb.blogspot.com/2007_
03_01_archive.html
 Nigel Hollis, 2006. Is the Link pre-test the equivalent
of the Smith & Wesson Magnum 500? at
http://www.mb-blog.com/index.php/2006/07/26/is-
the-link-pre-test-the-equivalent-of-the-smith-wesson-
magnum-500/
 Jason Lonsdale, 2007. Great Quotes for Planners #22 -
Huxley on pre-testing at
http://memehuffer.typepad.com/meme_huffer/2007/
03/great_quote_for.html
 Jason Lonsdale, 2007. Great Quotes for Planners #41 -
Akio Morita on pre-testing at
http://memehuffer.typepad.com/meme_huffer/2007/
12/great-quotes-fo.html
 Ed Cotton, 2007. Arnold creatives show us just what
they think of focus groups at
http://www.influxinsights.com/blog/article/1625/arn
old-creatives-show-us-just-what-they-think-of-focus-
groups.html#comments
 Jason Oke, 2007. Pre-testing at
http://lbtoronto.typepad.com/lbto/2007/05/pretestin
g.html
 Nigel Hollis, 2007. In defense of pre-testing at
http://www.mb-blog.com/index.php/2007/05/18/in-
defense-of-pre-testing/
 Jason Oke, 2007. Pre-testing, part II at
http://lbtoronto.typepad.com/lbto/2007/05/pretestin
g_part.html
 Clive Thompson, 2007. ‘The Human Advantage’,
WIRED 15.07 at
http://www.wired.com/techbiz/it/magazine/15-
07/ff_humancomp
What do they know of cricket who only cricket know?
- CLR James, 'Beyond A Boundary'
ACKNOWLEDGEMENTS
Thanks to Leland Maschmeyer for seeding the thought of
advertising pre-testing using prediction markets via his
blog post ‘Predictive markets better than ad testing?’ dated
March 25, 2007 on his blog, Whistle Through Your Comb.
Though initially skeptical that predictive markets would
work in this context, I eventually changed my mind and
this paper is the outcome.
META
Suggested Citation: Mohammed, Iqbal, Predictive Pre-
Testing: An Outline for an Ad Pre-Testing Model Based on
Prediction Markets (September 8, 2008). Available at:
http://ssrn.com/abstract=1265089
This work is licensed under the Creative Commons
Attribution-NonCommercial-ShareAlike 2.5 License.
To view a copy of this license, visit
http://creativecommons.org/licenses/by-nc-sa/2.5/ or
send a letter to
Creative Commons, 543 Howard Street, 5th Floor, San
Francisco, California, 94105, USA.