White smoke: Stakeholders and Commission reach agreement on reforming research assessment

A consultation process between research stakeholders across Europe and the European Commission has come to a close today, after organisations involved published a list of “core commitments” towards reforming research assessment in Europe.

After months of negotiations, the European Commission and EU research organisations have agreed on a set of principles and commitments for reforming the way research outputs are evaluated.

A new way to evaluate research outputs would recognise the diversity of research careers and contributions to science; base research assessment primarily on qualitative evaluation by peers; abandon the “inappropriate uses” of journal and publication metrics; and avoid the use of international rankings of research organisations in research assessment.

The agreement has been drafted by a team of representatives from the European University Association (EUA) and Science Europe, together with Karen Stroobants, a former chemistry researcher who is now a policy adviser to various research institutions in Europe. The Commission took on the role of “facilitator”.

Instead of rewarding researchers for the number of papers published, citations and the prestige of the journals concerned, a new system should look at the quality of research, while reducing the reliance on journal metrics, the agreement says.

"‘Publish or perish’ and metrics have led us into a blind alley,” said Marc Schiltz, president of Science Europe. “Let’s start recognising the full breadth of value created by researchers."

The European Commission has written reform of research assessment into the policy agenda of the revived European Research Area (ERA) plan to create a single market for research in the EU. With this agreement in the bag, talks will continue in the ERA Forum with member states on how to implement the core commitments.

The EUA and Science Europe will come up with a system for collecting signatures that will be launched on 28 September at a conference organised by the European Commission, but research organisations can already indicate their intention to sign the agreement.

“It is now time to go beyond existing declarations and define clearly what we want for the future of research assessment,” said EUA president Michael Murphy. “Universities, researchers and all stakeholders need to choose how they want to be assessed and need to choose now.”

Uneven rollout

The rubber-stamping of the agreement by the EUA and Science Europe, along with the other stakeholders involved, is one step closer towards convincing research organisations and funders in EU member states to reform their assessment criteria.

Stephane Berghmans, director for research and innovation at the EUA, expects the rollout will be uneven across member states. Some organisations might want implement the agreement but they will not get to decide when they can do it, especially in countries with a heavily decentralised research and higher education systems.

EU science ministers signed off an agreement backing research assessment reform in Europe in June but, depending on national legal or administrative barriers, “some universities will not be able to [implement the agreement just yet],” Berghmans told Science|Business.

Research and university associations are already working on research assessment together with member states within the ERA Forum. For Berghmans, the meetings of the ERA Forum show that “there's a commitment from member states to help with this.”

In some European countries, such as the Netherlands and Norway, the reform process is already ongoing. In Luxembourg, the National Research Fund (FNR) has already implemented some of the reforms proposed in the agreement, by deemphasizing the importance of journal metrics and encouraging researchers to apply for grants with narrative CVs.

According to Schiltz, who in addition to his role at Science Europe is also CEO of FNR, the implementation of narrative CVs prompted positive reactions from both reviewers and applicants. The research assessment reform “is a game changer,” Schiltz said.

The agreement was put together after months of consultations among a coalition of a core group of 20 organisations and more than 350 organisations from across Europe that expressed their interest to become associated to the process. At a stakeholder assembly meeting on 8 July, more than 80% of participating organisations said they are very likely or likely to recommend signing the agreement. “That doesn't mean the signing will take place, but I think it's a very strong and good sign,” said Berghmans.

The EUA will continue to promote the reform to its members. “We plan to raise awareness among our membership and to establish a dialogue with them at national level to be able to address some of their concerns,” said Berghmans.

According to Lidia Borrell-Damián, secretary general of Science Europe, there are no hard deadlines by when signatories need to implement changes in their assessment systems, but some structure is needed so the proposed commitments are not left up in the air. Borrell-Damián said the coalition will try to coordinate and offer support in the rollout of the reforms “in full respect of the prerogatives and autonomy of every organisation.”

Governance

Organisations in the coalition will also debating a governance system for the research assessment reform but that is not ready yet. The governance was discussed at the stakeholder assembly earlier this month and organisations began drafting the structure of governing bodies.

A final version of the governance documents will be presented and discussed at the next stakeholder assembly which is planned for October 2022.