Research organisations get closer to a consensus on assessment reform

After months of negotiations, research stakeholders in the EU are getting close to an agreement on a set of principles and commitments for reforming the way research outputs are evaluated.

The European Commission has written reform of research assessment into the policy agenda of the revived European Research Area plan to create a single market for research in the EU. After gathering a first round of input in 2021, the Commission wanted to give stakeholders and member states the opportunity to lead the consultation process and reach an agreement on new guidelines for evaluating researchers. 

A draft of the agreement dated 13 May and seen by Science|Business says signatories will pledge to implement four “core commitments” and to: recognise the diversity of research careers and contributions to science; base research assessment primarily on qualitative evaluation by peers; abandon the “inappropriate uses” of journal and publication metrics; and avoid the use of international rankings of research organisations in research assessment.

According to the draft agreement, by abandoning the inappropriate use of bibliometrics, research organisations will be able to “reduce the dominance of a narrow set of quantitative metrics”. In future publication metrics would complement qualitative evaluation criteria based on peer-review only in cases “where this is meaningful and relevant,” the draft agreement says.

The Commission and a large majority of the research community believe metrics such as the total number of publications and citations, as well as bibliometric measures such as the h-index, for evaluating the cumulative impact of an author's scholarly output and performance, are increasingly problematic. Researchers can exploit the system and advance their careers by chasing higher points in the publication game instead of focusing on the content of the research.

Paving the way to an agreement

The agreement is being drafted by a team of representatives from the European University Association (EUA) and Science Europe, together with Karen Stroobants, a former chemistry researcher who is now a policy adviser to various research institutions in Europe. The Commission took on the role of “facilitator”.

In addition to the drafting team, the Commission has helped set up a core group of 20 organisations at EU and national level to debate the details of the agreement. The Commission has also put together a list of more than 300 organisations which wish to be associated in the process.

Once the final details of the agreement are ironed out, research organisations will be invited to sign it. Then they will have the option to join a coalition of institutions that are willing to move forward with the reforms.

The Commission previously said that participating in the coalition will not be mandatory an organisations that do take part will be able to decide on the overall direction of the initiative. Speaking at an event of the League of European Research Universities (LERU) in January, Kostas Glinos, head of the open science unit in the Commission’s directorate general for research said the Commission does not want “to definitively decide anything before the coalition is put together”.

Some organisations, particularly in Germany, were wary of a potential top-down approach which would have meant the agreement on research assessment would be legally binding. Earlier this month, an alliance of Germany’s ten largest research organisations published a statement welcoming the talks on research assessment, but warned a reform needs to take into account the diversity of research systems and the autonomy of academic organisations, positioning themselves against an EU-wide system for evaluating research outputs.

The 13 May draft says the agreement is not legally binding but “represents a public commitment to contribute actively and constructively to reforming research assessment.” It also recognises that research organisations have the flexibility to choose how to implement the reforms.

According to members of the drafting team, recent progress in the talks has alleviated the concerns and the German statement and a subsequent position paper by the German Research Foundation (DFG) provided “constructive” feedback to the process. “We see no incompatibility between their report and the agreement we are drafting,” said Lidia Borrell-Damián, the secretary general of Science Europe.

Stephane Berghmans, director for R&I at the European University Association (EUA) said an online assembly meeting held on the 19th of May ended on a positive note. “We are getting the draft to almost a final version and we are starting to think about what the coalition will look like,” he said.

Once the agreement is finalised, signing research organisations will be able to join a coalition of those willing to put the reforms in practice. According to Berghmans, the coalition members will decide how to move forward and they will evaluate their own progress in moving to a fairer assessment system.

Another assembly meeting will be organised before the summer break, Borrell-Damián said. “I think it is very likely that the agreement will be reached before summer.”  Then stakeholders will “set up the basis for a light structure that will help taking the commitment forward,” she said.