Commission moots use of crowdsourced peer review for Horizon 2020 evaluations

The European Commission is seeking views via a public consultation on whether to adopt an open, internet-based system for peer review and says it will stage a trial if the idea gets enough support.

In the proposed pilot, research projects funded through the Horizon 2020 programme would be subject to general scrutiny, with people posting opinions on the progress of a given project and project participants, or indeed anyone, able to openly respond to comments.

The discussion would be tracked, considered by evaluators and fed into the review process.

“This collaborative and open approach should allow interested outsiders to contribute to the project with new input, and also allow problems to be identified,” the Commission says.

What the Commission is suggesting is a move to an ‘open access peer review model’. This is not to be confused with open peer review, where evaluators are publicly known and their reviews are entirely transparent.

The last say

“This would not be a fully open review as the expert reviewers would have the last say, but it would be a stepping stone towards it, to gauge the interest and identify potential benefits and problems,” the Commission says.

Today, Horizon 2020 reviews are carried out by Commission staff with the assistance, typically, of three experts per project.

EU research evaluation is single-blinded. This means that the reviewers know the identity of the applicants, but the applicants  do not know who the reviewers are. The reviewers remain anonymous, except to the Commission staff.

There are anecdotal accounts of scientists using open access peer review to evaluate their research. For instance a Harvard law professor Larry Lessig turned to the internet for help when updating his book, ‘Code and Other Laws of Cyberspace’.

The text of the first edition was posted to an online wiki, where users could add their own contributions. A year and a half later, after filtering all the comments, he produced a follow-up book, ‘Code Version 2.0’.

Cautious welcome

There was plenty of early reaction to the proposed pilot on the Commission’s website and Horizon 2020 LinkedIn account.

Muki Haklay, professor of geographical information science at University College London, likes the idea, as it would mean “expertise and comments are not confined only to the people in a consortium, or in the three reviewers who are selected to assess the project.”

However, Haklay foresees problems with the proposal. More consultation will make the evaluation process longer, may not result in constructive feedback, and might draw unfair comments, which he fears could sway a reviewer.  

Similarly, Machado Pinto Matilde, an associate professor of economics at the Carlos III University of Madrid said, “I understand the transparency upside but there are obvious disadvantages as well.”

Fabio Casati, professor of computer science at the University of Trento in Italy said, “I think project reviews as they are done today work reasonably well.” However, he supports experimentation, “So long as it does not become a marketing contest [for project managers].”

Several people noted that corporate R&D managers applying for funding from Horizon 2020 may not welcome the idea. A project can often contain information a company would rather keep out of the hands of competitors. For this reason, scientists sometimes ask reviewers to sign non-disclosure agreements.  

“Openness is not making [Horizon 2020] more attractive because it is against the interests of companies, [which] aim to make profit with the results,” commented Richard Smits, managing director at CSI chemical, a research consultancy based in Sofia, Bulgaria.

“I think the burden of having to prepare reports that can be made public will be bigger than the benefit of getting feedback for the project participants, so [companies] will not volunteer to open up the review process,” Smits said.