European Research Council announces plan to update its evaluation system

In a landmark decision this week, the European Research Council (ERC) announced changes to its application forms and evaluation procedures that will be implemented starting with the 2024 calls for proposals.

ERC president Maria Leptin first announced the changes are coming at a meeting of the European Parliament’s research and industry committee last month. She emphasised they will not affect the core evaluation criteria of excellence, and that proposals will continue to be evaluated by panels of leading scientists and academics.

However, it was agreed by the ERC Scientific Council on Monday that the current CV and track record templates will be combined and simplified. Applicants will be able to add short narrative descriptions to describe their career paths, including unconventional career paths and outstanding contributions to the research community.

When the changes come into effect, principal investigators will no longer be required to submit personal profiles and ERC evaluators will give more weight to project proposals than past achievements of the applicant.

“In practical terms, we will be changing the format in which applicants submit the information about themselves,” Leptin told Science|Business.

The combined form will no longer ask applicants how much extramural funding they have attracted and how many PhDs and postdoc students they have supervised. “We've got rid of some of that stuff, which were metrics that don't provide the kind of context that is useful to our panels,” Leptin said.

All projects and applicants will be evaluated on scientific excellence only. The difference will be that researchers will be able to provide “more context to that excellence” which will be evaluated accordingly, said Leptin.

In addition to the usual criteria, such as research output and academic recognition, evaluators will also be looking at applicants’ broader impact in the scientific community, such as awards, prizes, lectures, contributions to open science, impact on improving teaching practices in their universities, engaging in peer review, and other relevant activities. But those criteria will apply in the later phases of the evaluation.

Leptin said that in the first phase of the ERC evaluation system, which is designed to select the most promising applications out of the thousands that are submitted every year, evaluators use a scoring system. As of 2024 this will give more weight to the project than to the CV of the researcher.

“After that step, [evaluators] don't really use scoring so much. They look at the project and then they look at the researcher to ask, does this person have the capacity to run the project,” Leptin said.

Then, in the interview phase, researchers will be probed on their ability to carry out their project, but publication metrics, such as impact factors, and the prestige of scientific journals listed in the CV will not be used “as proxies” for excellence. “This is made clear in the [evaluation] panel,” said Leptin.

What next?

The changes to the ERC evaluation system were announced at the same time that the ERC said that it would sign the agreement on reforming research assessment, which aims to get EU research organisations and funding agencies to upgrade their evaluation systems, to move away from metrics such as the number of papers a researcher has published, to look more at the quality of the overall body of work.

The European Commission has been pushing for a revamp to the way research is evaluated since last year, and the plans for reform are also included in the wider drive to reform and create a European Research Area.

Full details of the changes coming to the ERC evaluation process will be available in the upcoming work programme and guidance documents the funding agency is planning for 2024.

Leptin says the ERC is working on guidance documents for the evaluators, explaining why the agency will no longer be asking about the number of PhD students and postdocs an applicant has trained. The number of graduate students and postdocs doesn’t say whether applicants are good mentors. “We will have to work hard on getting that guidance right,” she said.

Leptin was not able to give a clear timeline for when the guidance documents will be published next year, but they should be ready before the 2024 calls are out. “We may get a first draft after Christmas,” she said. However, the funding agency will need additional time to gather feedback from its research assessment task force. “We will get lots of responses and we want to make sure we get everything right,” she said.