EU to go ahead with reform of its research assessment process in the face of scepticism

A number of national research organisations have been taken aback by the urgency of a request by the European Commission to appoint representatives to negotiate a new research assessment system in Europe.

The Commission says the current system of evaluating research is becoming outdated and reform is needed to make it more efficient. Instead of rewarding researchers for the number of papers published, citations and the prestige of the journals, a new system would be based on a “qualitative judgment with peer-review” that is supported by “a more responsible use of quantitative indicators,” the Commission told a meeting of member states earlier this month.

The reform of research assessment is included in the policy agenda of the revived European Research Area policy to create a single market for research. The Commission is now gathering support from a coalition of organisations to implement the assessment reform and to begin testing it out in the second quarter of 2022.

The Commission is anxious to deliver the ERA policy agenda as soon as possible and has teamed up with the current French presidency of the Council of the EU to speed things up.

The French presidency is preparing EU Council conclusions on open science, which could include an agreement on research assessment reform, and is organising a conference in Paris on the topic next Friday.

Diplomats familiar with the talks said the idea of Council conclusions on open science was not on the table in early December. The Commission published a “call for interest” in December, a week before Christmas, while the French government was making plans to include the file in its Council presidency agenda.

More than 100 organisations and institutes have indicated they are willing to participate in the coalition but some are not keen on rushing the process, and have called on the Commission to establish a subgroup in the ERA forum to debate the issue in detail.

German research institutes in particular are becoming concerned about the speed of this process, but they may take part in the negotiations, without joining the coalition, to avoid being left on the sidelines.

An informal alliance of big research organisations in Germany which are reluctant to join on such a short notice, was advised to get on the legislative train and try and steer it a little bit, rather than boycotting the process. Diplomats in Brussels say the reform can only be successful if it’s fully embraced by national research players.

The European University Association (EUA) and the League of European Research Universities (LERU) have published papers on the subject. The latest one by LERU was published today, highlighting that the assessment of researchers is “too strongly” focused on bibliometrics and past performance, despite efforts by some universities to broaden evaluation criteria to include qualitative metrics.

LERU acknowledges implementing the reform will not be an easy task. A more comprehensive assessment system could be more time consuming, both for applicants and evaluators. “This may lead to more pressure on researchers, who may experience the new assessment outlook as a way of increasing current performance standards,” the paper says.

Speaking at the launch event of the LERU paper, Kostas Glinos, head of the open science unit in the Commission’s directorate general for research said the Commission will hold off on taking any decisions on a new research assessment system until the coalition is organised.  “We do not want to definitively decide anything before the coalition is put together,” said Glinos, noting the coalition will be “self-governed” and participating organisations will be able to decide on the overall direction of the initiative.

The recognition that research assessment needs changing dates back to 2013, when an international coalition signed the DORA declaration on research assessment in San Francisco, hoping to reduce the weight of publication metrics in the evaluation of researchers.

More recently, the UK government has introduced a new “resume for research and innovation” as an alternative to traditional CVs which would allow researchers and innovators to highlight a broader set of contributions they have made to their field, going beyond journal metrics.

The European Research Council is also moving away from using the journal impact factor in their funding decisions. Research funding agencies in the open access advocacy group cOAlition S also committed to evaluate projects based on their “intrinsic merit” without taking into account the prestige of the publications the work has been published in.   

One size does not fit all

Research and technology organisations say a reform of research assessment should be planned more carefully and the Commission should clarify the difference between academic and research careers.

Muriel Attané, secretary general of EARTO said the reform proposal first started as a way to improve the assessment of academic careers, but has now been broadened to include the assessment of research performance.

But many researchers in Europe work outside academia and do not have to publish in prestigious science journals to advance in their careers.  Research and technology organisations beyond academia are being evaluated annually by regional and national funding bodies, with scientific publications being only one of many other criteria.

“We do not see where the added value is for our members to add an additional EU assessment to what we already believe is far more extensive and more impact-oriented than what is being done in [academia],” Attané told Science|Business.  “Some of our members report more than 80 to 100 [key performance indicators] per year already.”

Member state research and industry ministers already collect data on the impact of national and regional R&D investments. “We do not expect that those ministries will give the lead to the EU to have such discussion on [the] performance and targeted impact of [research and technology organisations] while those very much link with national priorities in terms of technology sovereignty and industrial policy,” said Attané.