Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
New methods of REF assessment: Prediction Markets Jisc Research Analytics Webinar 1 July 2019 Jackie Thompson University o...
The Problem
Prediction markets: replication studies
The market interface https://www.refpredictionmarkets.com/screencast.html
Prediction Market specs • 21 participants – psychology department • Postdocs and teaching/research academics • Financial i...
Rationale • Condenses crowd wisdom • Dynamically weighted • Incentives • Fine-grained rankings • Burden
Assessment methods • Close reading 1 (Internal panel) • Close reading 2 (external rater) • Prediction market • Machine lea...
Results: Correlations Spearman's rho Internal CR External CR Prediction Market Machine Learning Internal CR 1 .534** .732*...
5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 5 7 9 11 13 External CR Internal CR Prediction Market Exte...
Further plans • User feedback mostly positive • Rolling out beyond Bristol soon (contact jackie.thompson@bristol.ac.uk)
Thank you! Research Contributors: Marcus Munafò (U. Bristol) Ian Penton-Voak (U. Bristol) Anna Dreber Almenberg (Stockholm...
Prochain SlideShare
Chargement dans…5
×

Prediction markets

79 vues

Publié le

The aim of this project was to develop and evaluate a prediction market tool that higher education institutions can use to rank outputs for potential REF submissions as part of their internal REF (Research Excellence Framework) planning. A prediction market is a bit like the stock market, except instead of investing in companies, participants invest in the outcomes of future events (in this case, ratings of research outputs). This presentation will give some background to the project and details of the prediction markets that have been tested with Units of Assessment at the University of Bristol. It will include a demo of the tool used and the lessons learned from the first round of markets.  

Publié dans : Données & analyses
no profile picture user

  • Identifiez-vous pour voir les commentaires

  • Soyez le premier à aimer ceci

Prediction markets

  1. 1. New methods of REF assessment: Prediction Markets Jisc Research Analytics Webinar 1 July 2019 Jackie Thompson University of Bristol
  2. 2. The Problem
  3. 3. Prediction markets: replication studies
  4. 4. The market interface https://www.refpredictionmarkets.com/screencast.html
  5. 5. Prediction Market specs • 21 participants – psychology department • Postdocs and teaching/research academics • Financial incentives • 11 days • 29 papers
  6. 6. Rationale • Condenses crowd wisdom • Dynamically weighted • Incentives • Fine-grained rankings • Burden
  7. 7. Assessment methods • Close reading 1 (Internal panel) • Close reading 2 (external rater) • Prediction market • Machine learning
  8. 8. Results: Correlations Spearman's rho Internal CR External CR Prediction Market Machine Learning Internal CR 1 .534** .732*** .655*** External CR 1 .601*** .739*** Prediction Market 1 .643*** Machine Learning 1
  9. 9. 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 5 7 9 11 13 External CR Internal CR Prediction Market External CR Prediction Market Machine Learning 0.4 0.6 0.8 1 1 2 3 4 5 7 9 11 13 1 2 3 4 5 7 9 11 13 1 2 3 4
  10. 10. Further plans • User feedback mostly positive • Rolling out beyond Bristol soon (contact jackie.thompson@bristol.ac.uk)
  11. 11. Thank you! Research Contributors: Marcus Munafò (U. Bristol) Ian Penton-Voak (U. Bristol) Anna Dreber Almenberg (Stockholm School of Economics) Felix Holzmeister (U. Innsbruck)

×