The aim of this project was to develop and evaluate a prediction market tool that higher education institutions can use to rank outputs for potential REF submissions as part of their internal REF (Research Excellence Framework) planning. A prediction market is a bit like the stock market, except instead of investing in companies, participants invest in the outcomes of future events (in this case, ratings of research outputs). This presentation will give some background to the project and details of the prediction markets that have been tested with Units of Assessment at the University of Bristol. It will include a demo of the tool used and the lessons learned from the first round of markets.  

1. New methods of REF
assessment: Prediction
Markets
Jisc Research Analytics Webinar
1 July 2019
Jackie Thompson
University of Bristol

2. The Problem

3. Prediction markets: replication studies

4. The market interface
https://www.refpredictionmarkets.com/screencast.html

5. Prediction Market specs
• 21 participants – psychology department
• Postdocs and teaching/research academics
• Financial incentives
• 11 days
• 29 papers

6. Rationale
• Condenses crowd wisdom
• Dynamically weighted
• Incentives
• Fine-grained rankings
• Burden

7. Assessment methods
• Close reading 1 (Internal panel)
• Close reading 2 (external rater)
• Prediction market
• Machine learning

8. Results: Correlations
Spearman's rho
Internal CR External CR
Prediction
Market
Machine
Learning
Internal CR 1 .534** .732*** .655***
External CR 1 .601*** .739***
Prediction Market 1 .643***
Machine Learning 1

9. 5
7
9
11
13
0.4 0.6 0.8 1
5
7
9
11
13
0.4 0.6 0.8 1
5
7
9
11
13
5 7 9 11 13
External CR
Internal
CR
Prediction Market
External
CR
Prediction
Market
Machine Learning
0.4
0.6
0.8
1
1 2 3 4
5
7
9
11
13
1 2 3 4
5
7
9
11
13
1 2 3 4

10. Further plans
• User feedback mostly positive
• Rolling out beyond Bristol soon
(contact jackie.thompson@bristol.ac.uk)

11. Thank you!
Research Contributors:
Marcus Munafò (U. Bristol)
Ian Penton-Voak (U. Bristol)
Anna Dreber Almenberg
(Stockholm School of Economics)
Felix Holzmeister (U. Innsbruck)