Altmetrics journal club
•Télécharger en tant que PPT, PDF•
3 j'aime•895 vues
Journal Club analysis of Thelwall paper looking at possible association between article citation counts and altmetrics
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Altmetrics journal club
Similaire à Altmetrics journal club (20)
Plus de Anne Madden
Dernier
Dernier (20)
Altmetrics journal club
- 1. Thelwall, Mike, et al. "Do altmetrics work? Twitter and ten other social web services." PloS one 8.5 (2013): e64841. Anne Madden Journal Club 27th April 2015
- 2. • What are altmetrics? • How are they calculated? • Paper: Rationale and methodology • Results? • Discussion: do altmetrics “work”?
- 4. “No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped.” From the altmetrics manifesto. (Priem 2010)
- 6. Rationale for this study “To what extent do the altmetric indicators associate with citation counts?” Building some form of evidence base to underpin altmetrics?
- 8. Methodology • PubMed articles with a non-zero altmetric score • Sign test: “..each article is compared only against the two articles published immediately before and after it….” • Self-citations removed – why? • Altmetric scores gathered between July 2011 – 1st Jan 2013
- 9. Definition of success “..if the altmetric score is higher than the average altmetric score of the two adjacent articles and its citation score is higher than the average of the two adjacent articles OR the altmetric score is lower than the average altmetric score of the two adjacent articles and its citation score is lower than the average of the two adjacent articles.”
- 10. Results 1. Total no. of successes v no. of failures: overall success 2. No. of journals where success exceeds failure: success 3. Individual altmetric level: mixed results. Four altmetrics “significantly and positively correlate with citations..the correlation for Twitter is significant and negative”. • “..results provide strong evidence that six of the eleven altmetrics.. associate with citation counts. ..correlation effect size is unknown”
- 11. Experiment. Starting from non-zero citations: a very rough comparison • PubMed papers, with citations • Timeframe: mid to end 2013 • Search: “Influenza” [Title] • 20 papers selected at random • 14 had non-zero citations
- 14. Lowest altmetric score was for non-English paper (Chinese) Cut-off point where association between altmetric score and citation count lose significance.
- 15. Boon 2014, IATUL Proceedings: Altmetrics is an Indication of Quality Research or Just HOT Topics http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=2034&context=iatul
- 16. Discussion • Anonymity in most altmetric tools • Multiple accounts: the “Scholarly Selfie” • “Gaming” – could affect both altmetric and citation scores • Relative labour-intensity and engagement with paper
- 18. GAMING
- 20. Groups doing it for themselves “Early experience with JACR tweet chats demonstrates that organizing Twitter microblogging activities around topics of general interest to their target readership bears the potential for medical journals to increase their audiences and reach.” Hawkins, C. Matthew, et al. "The Impact of Social Media on Readership of a Peer-Reviewed Medical Journal." Journal of the American College of Radiology 11.11 (2014): 1038-1043.
- 21. Topical, timely, authoritative, clear message, relevance to a broad audience Appeals to a broader audience; the ready-made headline… 4 Wikipedia pages (aubergine bit)
- 22. Do altmetrics work? “Not everything that counts can be counted, and not everything that can be counted counts”
- 23. Neither altmetrics nor citation scores are evidence of quality.
- 25. “Although association preferences documented in our study theoretically could be a consequence of either mating or shoaling preferences in the different female groups investigated (should we cite the crappy Gabor paper here?), shoaling preferences are unlikely drivers of the documented patterns ……”
- 27. From Ben Goldacre on YouTube
- 28. Hopewell, Sally, et al. "Publication bias in clinical trials due to statistical significance or direction of trial results." The Cochrane Library (2009). “Trials with positive findings are more likely to be published and published quicker than trials with negative findings.” Failure to publish: “scientific misconduct” valid expectation on the part of trial participants that their participation will contribute to knowledge.
- 29. …which scholarly products are read, discussed, saved and recommended as well as cited. “..provide an insight into things we have not measured before….” Tattersall 2015 “…analyzing the impact of outputs in different formats.. as opposed to the analysis of only journal papers.” (Costas 2014) “…impact of scholarly material, with an emphasis on social media outlets…” (Priem 2014) …more timely data, showing evidence of impact in days instead of years ..the impact of web- native scholarly products …impacts on diverse audiences including scholars but also practitioners, clinicians, educators and the general public. (Piwowar 2013)
- 30. Scholarly journals or scholarly communication? “For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research “products” rather than “publications” in the biographical sketch section. This means that according to the NSF, a scientist’s worth is not dependent solely on publications.” Piwowar, Heather. "Altmetrics: Value all research products." Nature 493.7431 (2013): 159-159.
- 31. Djuricich, Alexander M., and Janine E. Zee-Cheng. "Live tweeting in medicine:'Tweeting the meeting'." International Review of Psychiatry 0 (2015): 1-7.
- 32. Cochran, Amalia, et al. "Use of Twitter to document the 2013 Academic Surgical Congress." Journal of Surgical Research 190.1 (2014): 36-40.
- 33. Aslam, Anoshé A., et al. "The Reliability of Tweets as a Supplementary Method of Seasonal Influenza Surveillance." Journal of medical Internet research 16.11 (2014). “….this study demonstrated increased accuracy in using Twitter as a supplementary surveillance tool for influenza as better filtering and classification methods yielded higher correlations for the 2013-2014 influenza season than those found for tweets in the previous influenza season, where emergency department ILI rates were better correlated to tweets than sentinel-provided ILI rates.”
- 34. • H-index outdated? Register for your ORCID Identifier (http://orcid.org). For more info see: http://libguides.ucd.ie/orcid • Create an author profile on Google Scholar Citations (http://scholar.google.com/citations) • Create an impactstory account for your biog. (http://impactstory.org/) From: Rathemacher, Andrée J. "Article-Level Metrics and Altmetrics: New Ways to Measure the Impact of Your Research." (2014). • When publishing ask for permission to store a copy in the Institutional Repository (http://www.lenus.ie)
- 35. “Do altmetrics work?” Has Thelwall proven his case? • Altmetrics were never intended to replace / anticipate / mimic citation counts. • Altmetrics have a far wider scope, with more focus on unpublished born-digital material and other audiences. Therefore, to prove they “work”, more multi-faceted research required.
Notes de l'éditeur
- Rough order of play.
- It’s very recent
- Use as a means of selecting from the masses of studies out there.
- Number of mentions Importance of sources Authoritativeness of authors Also: Degree of effort Engagement/understanding of paper
- Do altmetrics and citations measure the same phenomenon? Are altmetric scores an early indication of citation counts? Are altmetrics “valid proxies of either impact or utility”? Having mentioned that these are the questions that need answering, author then settled on establishing:…
- If you can prove that the red box relates to the green box, you can then make reasonable predictions with regard to later citation scores.
- only articles of approximately the same age, which are similarly exposed to the same citation delay and usage uptake biases, are compared to each other. Authors unlikely to have first heard of their own paper on social media – looking for causal relationship
- What would constitute “success”? With a sufficiently large cohort this could work.
- Google+ and Reddit were exceptions. Q&A, Pinners and LinkedIn not included as they didn’t have sufficiently high samples Again exclusions as a very wide range of journals was included, some with too low a sample to identify statistical differences Non-zero in at least one altmetric, not in all of them. Uptake of Twitter growing exponentially, tweeting right up to end Dec 2012.
- Took out a 30-day free trial with altmetric measuring tool which only went back 2 years.
- Most cited also had highest altmetric score. 13 out of 14 with non-zero citations also had non-zero altmetrics
- Possibly a cut-off point at which the both altmetric scores and citation counts lose significance. Or a change in the weighting is needed.
- Web of Science Predictive element
- It’s this easy to get an altmetric score. No need to even have read the paper.
- Kudos provides a platform for researchers to explain and share their work for wider audiences, and to measure the effect of these actions on a range of metrics including full text usage, altmetrics, and citations. basic service is free for researchers to use; publishers and institutions pay a fee for access to support tools, information on article and author performance within Kudoshttp://www.jisc-collections.ac.uk/UKSG/344/Kudos/?n=121b8a3f-e721-4bc0-8354-7ad4d47f99de
- The relevance of colours in altmetrics Other tips, mainly relating to Twitter, suggest best times to tweet and using humour or shock effects
- At the outset, we saw that “altmetrics as filter” was a key benefit.
- Am not using the example given in the paper. Unfortunately, popularity due to failure of peer-review, proof-reading and editing systems. Absolutely nothing to do with the intrinsic value of the paper.
- Neither are citations an indication of quality..
- Est.
- Publication bias Publishers chasing the papers that will boost JIFs, citation scores etc.
- “scientific misconduct” valid expectation on the part of trial participants that their participation will contribute to knowledge.
- “born digital” material
- An ideal tweet would include 3 features (Desai): Informative content Internal citations Positive sentiment score
- Make it easy and avoid being misquoted Include a 140 character or less slide in your presentation that sums up your key point.
- What else can it do? Nothing to do with papers on influenza being tweeted – just individuals talking about their own dose and that of their family, friends and neighbours.
- Some tools.