Removing numerical scoring when evaluating grant proposals found to have no effect on outcomes

Not using a numerical scoring system to review research grant proposals has little impact on the way that evaluators carry out their written assessment and the final outcome of the review, a new study has found.

The analysis compared proposal evaluations under the Innovative Training Networks (ITNs) funding scheme that was part of Horizon 2020’s Marie Skłodowska-Curie Actions (MSCA). This scheme is now known as Doctoral Networks under Horizon Europe.

In 2020, the numerical scoring element of the initial individual assessment phase was dropped, allowing a comparison with 2019 evaluations.

A team of researchers from the University of Split School of Medicine in Croatia looked at some 1,500 proposals in each of the years, and found that the change did not have a discernible impact on the evaluations.

“The removal of numerical scores from the individual phase does not have an effect on the final outcome, at least based on the characteristics we measured here,” said Ivan Buljan, the paper’s lead author.

He said that while more research is needed – the paper is also not yet peer-reviewed – it does show that evaluations do not necessarily need to include a score, which could remove some bias from the individual assessment phase.

Proposals also go through a group evaluation phase in which a consensus report is made, and this does still include a numerical score for the ITN grants. But with the individual assessors not having a fixed numerical score in mind going into that meeting, it could result in a fairer overall score.

Ana Marušić, co-author of the paper, said that usually when reviewers get together, there is a regression to the mean. “You can have people giving different marks but in the end when they come together, they will usually come to some kind of mean score rather than a median, which would be [better],” she said.

“Here we wanted to see whether it would change anything if the individual evaluators do not have a score in mind, but they just write what they think and then when they come to the consensus report they give a joint score.”

Both Buljan and Marušić are cautious in suggesting this study should pave the way for a wider abolition of numerical scoring across other calls, saying more studies must be done. But they say that it adds evidence to a growing movement in the research world for qualitative, rather than quantitative, processes.

The 2022 Agreement on Reforming Research Assessment that has been signed by over 600 organisations pushes for research assessment to be primarily based on qualitative evaluation, with quantitative measures used responsibly.

This is more aimed at trying to get away from the use of journal- and publication-based metrics when reviewing a researcher’s proposal, but the numerical scoring approach employed in proposal evaluations is in a similar area.

Marušić is in favour of this push for more qualitative assessment and is against the use of impact factors to assess individual research, to give out grants or to assess the prestige of a researcher. But she acknowledges the need to maintain objective quality in research assessment, which often comes in numerical form. Otherwise, there is the possibility of nepotism or corruption, with researchers publishing in local-language journals through their contacts and gaining the same prestige as a researcher published in international journals.

“I'm very much for qualitative assessment, but it has to be a complement to a basic threshold of objective criteria,” she said.

Lottery approach

One innovative approach to choosing which proposals get funding is the lottery method. The ITN call had a success rate of between 8 - 10%, making it highly competitive. Marušić said that when looking at proposals that hover around the cut-off line for funding, there was little to no difference in their quality. A solution could be to take those proposals around the cut-off mark and enter them into a lottery to see which ones receive funding.

“Around the cut-off line, it is a matter of chance whether you are over or under,” Marušić said. “So some argue a lottery would be fairer.”

The recent study into numerical scoring is part of a wider body of research that the team at the University of Split are doing to assess the impact of changes to research assessment. A previous study looked at the need to have consensus reports in-person, and found little difference between that and when the reports were put together through online calls.

The team works with David Pina from the European Research Executive Agency (REA), which manages research projects supported by Horizon Europe and previously Horizon 2020.

“We like to study such processes in order to provide evidence for future changes,” Marušić said.