Motivating/rewarding reviewers

Among other things, researchers are expected to do research, publish the results of this research, and review the research of others. It is this reviewing part that I want to talk about today.

Reviewing is obviously one of the most important responsibilities of a researcher, one that can take a significant amount of time, but one that brings little reward, as it’s usually done for free. All a reviewer can show for it is a line on a CV saying “Reviewer for [put your favorite journal here]”. The purpose of this post is to propose a way to reward researchers who are good reviewers and spend a significant amount of time improving the work of others, often anonymously.

What if journals had awards for “Best reviewer of the year”? The laureate could then add this award on her CV, showing that she is doing a huge amount of service to her field. The award could be based on objective measures, such as the number of reviews returned in time, the number of reviews that concurred with the Editor-In-Chief’s decision, or could be more subjective, based on the Associate Editors and Editor-In-Chief assessments of the quality of the reviews they received. The award could be given with much ceremony at conference banquets, like awards for the best student paper, and perhaps with some money attached to it. Anonymity would not be broken, because all we would know about the laureate is that she reviewed N papers for journal X, not that she reviewed my paper submitted to journal X.

One could also think of a wall-of-fame type of thing, where reviewers would compete for the largest number of reviews returned in time, for instance. Or, to keep high levels of anonymity, give a way for a reviewer to know how her reviewing work compares to others: have I been reviewing more papers than 1%, 50%, 80% of the reviewers of this journal? If I see that I review less that my fellow researchers, perhaps I’ll be willing to accept the next invitation to review a paper. If I see that I review way more than my fellow researchers, perhaps I want to put that on my CV to show how altruistic I am.

Short of paying the reviewers for their reviews, which would perhaps be expensive for the smallest scientific societies, I think some type of reward/award system could be useful to appreciate the amount of time some researchers spend reviewing and improving the work of others. Given that systems for handling submissions and revisions such as “Manuscript Central” have all the stats available, that’s probably not very hard to do.

6 thoughts on “Motivating/rewarding reviewers

    1. Bastien Boussau Post author

      Thanks Bob! That’s extremely similar indeed. They post 300 names of reviewers, which seems like a lot to me though. I wonder how low the bar is to be among those reviewers, and would be curious to know the disparity among them.

      Reply
  1. Mark Holder

    Nice idea, Bastien. Reviewers certainly vary immensely in their thoroughness and timeliness. So it would really be nice to figure out how to reward those who do an exceptional job. I must admit that (as AE for Sys Bio) I do not always fill out the web form to comment on the quality of the reviews that I get. So if Sys Bio moved to this system, I’d have to start recording info about the peer review process more faithfully.
    I will say that I do pay attention to the # of reviews a person has done in the last year. MS Central maintains that info, and I try not to overburden those folks who are kind enough to provide reviews frequently (although Sys Bio has an “editorial board” for each AE, and those poor souls do get a lot of requests for reviews). Journals often choose associate editors based on people who are providing helpful reviews. I am sure that does not sound like much of an incentive (“review more to get a chance to have an official post of reviewing!”), but at least the editorships look more substantive on a CV.
    Ultimately, I hope that science moves toward open peer review (see https://peerj.com/about/policies-and-procedures/#open-peer-review for example). Something along those lines seems to be the best way for people to get credit for the work that they put in. There will still be cases in which reviewers will want/need to be anonymous. However, currently anonymity is the default even when there does not seem to be much need for it. Obviously anonymity makes it tricky to get credit for one’s work.

    Reply
    1. Bastien Boussau Post author

      Thanks Mark! If taking note of the quality of the reviews AEs get is too cumbersome, basic numerical metrics could help. All the AEs would need to do then is to ensure that the reviewer with the highest number of reviews is not also known for her poor reviews (which is unlikely, given that AEs keep using her!). Numerical metrics are obviously imperfect, but they would still measure a reviewer’s output.
      Regarding editorship, I agree that this looks nice on a CV. However, this is an honor that is usually obtained after a few years in the trade. For younger reviewers, e.g. postdocs, a way of showing their investment in the community on their CV, through a prize or just a publicly-available measure as on PeerJ, could be useful. And perhaps not too difficult to set up for a journal like Systematic Biology.
      Finally I agree that the PeerJ system looks very good!

      Reply
  2. John Hutchinson

    Nice post!
    Proceedings of the Royal Society B and other RS journals do the Reviewer of the Year, and like many other journals the editors (or in other journals, authors) can score reviewers based on (1) quality of review, and (2) how well within the deadline (~2 wks) the review is completed; those scores are used to choose the RoY (I’m an Assoc Editor). Scoring reviewers by whatever metric also rewards good reviews, as I tend to re-use reviewers who I know are good, so those reviewers have a stronger influence on shaping the quality of papers produced in their field.
    http://rspb.royalsocietypublishing.org/site/misc/Top_referees_2010.xhtml

    Also take note of The Peerage of Science, which is a new peer review model: a website where you can submit papers and have them review, then journals in Ecol/Evol Biol and other fields can “bid” for the papers w/reviews!
    http://www.peerageofscience.org/journals/

    Reply
    1. Bastien Boussau Post author

      Thanks John. I have been told that physics journals also have similar ways of rewarding good reviewers.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *