France helps Brussels move ahead with ‘disruptive’ plan for research assessment

France is now working with the European Commission and science and university associations to push for a new system of evaluating research in Europe, launching the Paris Call on research assessment.

The call, drafted by the French government’s Committee for Open Science, says evaluating researchers based solely on the number of highly cited papers they have published in journals with high impact factors underestimates the value of other contributions, lowers reproducibility of research, and discourages researchers from opening up their research and working with others.

The Paris Call fits with the Commission’s checklist of 20 priorities set out in the new policy agenda of the European Research Area, with science and research assessment first on the list.

The Commission is now gathering support to implement assessment reform and to begin testing it out in the second quarter of 2022. To date, more than 160 organisations have joined the initiative, but a list of the signatories published on 31 January shows the large German research organisations are still reluctant to get involved.

As the Commission notes, science increasingly relies on advanced technologies and the outputs reach beyond the scope of scientific papers. Researchers who produce datasets, software, algorithms and protocols need novel ways for their work to be recognised, EU research and innovation commissioner Mariya Gabriel told delegates at the open science conference organised by the French presidency of the Council of the EU last weekend.

The organisations that signed up to the coalition of those willing to get involved in the reform of research assessment are expected to agree new principles on which assessments are based and a timeline for implementing them.

Gabriel expects groups of universities to be the first to test the proposed reforms. “I expect that the European university alliances contribute and test the proposed reforms on the ground,” she said.

Jean-Eric Paquet, head of the Commission’s research directorate said he hopes the French conference will prompt more organisations to join the coalition, so it inclludes all 27 member states. At the moment only 22 are represented. “We are working hand in hand with the French presidency,” he said.

The Commission is working with Science Europe and the European University Association to gather feedback and write a first draft of the proposed reforms. Paquet said the work ahead will be very challenging because, “it is disruptive, it will change practices, habits and positions.”

Antoine Petit, director of CNRS, France’s national research agency, said assessment reform will also be debated by the G6 group of Europe’s largest research funders, which have put together a list of six priorities for mainstreaming open science and improving the evaluation of scientific excellence. “We push our points with decision makers and participants in research and innovation,” Paquet said.

New assessment for Horizon applications

The EU’s €95.5 billion Horizon Europe research programme will be changing the way proposals are assessed, with the Commission telling evaluators to focus less on journal metrics and more on the content of a scientist’s body of work.  

The European Research Council (ERC), which is funded through Horizon Europe banned the use of journal impact factors in evaluations after it signed the Declaration of Research Assessment (DORA) last year. DORA was established in 2013 in San Francisco and is the first international attempt to build consensus around a new research assessment system. “We're a signatory of DORA since last year, but impact factors have been discouraged for some time [at the ERC],” said ERC president Maria Leptin.

ERC evaluation panels are told not to look at other indicators, such as whether the applicant works in a high prestige institution, or whether he or she has a high citation index. Researchers can also submit so-called narrative resumes where they highlight their most relevant papers. “[Evaluators] are really encouraged to look at the essence of the work,” said Leptin.

The ERC Scientific Council is about to set up a task force on research assessment to improve the internal evaluation process and also to provide an exemplar for other funders. “We hope we will lead in implementing those reforms that come out of this consultation with the community,” Leptin said.