‘Friendly’ reviewers rate grant applications more highly

[Andy]

Peer reviewers are four times more likely to give a grant application an "excellent" or "outstanding" score rather than a "poor" or "good" one when they are chosen by the grant’s applicants, an analysis of Swiss funding applications has found.

The study, at the Swiss National Science Foundation (SNSF), was completed in 2016, and the SNSF acted quickly on its findings by banning grant applicants from being able to recommend referees.

The authors, who are affiliated with the SNSF, posted their results online at PeerJ Preprints1 on 19 March, and in their paper call on other funders to reconsider their funding processes.

“I think this practice should be abolished altogether,” says study co-author Anna Severin, a sociologist who studies peer review at the University of Bern. Other experts are also wary of the problems that author-picked peer reviewers might cause, but some question whether banning them altogether is the right step.

The study examined more than 38,000 reviews from nearly 13,000 SNSF grant applications by about 27,000 peer reviewers from all disciplines between 2006 and 2016. The authors found that reviewers nominated by applicants were more likely to give these applicants higher evaluation scores than referees chosen by the SNSF.

https://www.nature.com/articles/d41586-019-01198-3

 

[Wonko]

People are surprised by this why?

 

[Jonathan Edwards]

I doubt anybody is actually surprised. The question is what does it mean and what if anything should be done about it.

It cuts both ways. People with mediocre ideas will get their chums with the same mediocre ideas to support their papers. But if you have a genuinely original idea if you cannot choose at least some of the referees you are almost certain to get referees who either do not understand your work or as competitors are very keen to trash it.

The practice of allowing authors to choose some of the referees was based on an understanding of that second situation. For a sociologist writing a trivial study like this to say it should be abolished is just an indication that that sociologist probably has very little insight into the real situation.

Peer review is pretty much rubbish anyway so why bother arguing about this sort of thing. Why not have papers published by the academic institution where the work was done, with open post-publication review and discussion? You don't actually need referees.