Research assessment: German funder raises concerns as EU reform talks come to a close

German research institutions are warning against a centralised reform of research assessment in Europe, as the European Commission and stakeholders get ready to sign a draft agreement on new guidelines for evaluating researchers.

In a position paper published on Wednesday, the German Research Foundation (DFG) says it is against a top down reform of the research assessment system in Europe, two weeks after an alliance of ten German research organisations, including the DFG, have warned they will not sign a draft agreement prepared jointly by EU member states, stakeholders and the Commission if the proposed reforms turn out to be binding.

“The DFG is committed to an open publishing system and a culture of assessment that is geared towards content,” said DFG president Katja Becker in a statement announcing the publication of the position paper.

The DFG says a change in the evaluation culture of research institutions is needed, but a reform cannot be successful if research managers do not fully trust the new system, which would discard publication metrics. “We want to promote confidence in this change so as to make it easier for academics to put quality first when it comes to the publication and assessment of scholarship,” said Becker.

The Commission and a number of research associations across Europe say the current system of evaluating research is becoming outdated and reform is needed to make it more efficient. Instead of rewarding researchers for the number of papers published, citations and the prestige of the journals concerned, a new system should look at the quality of research, while reducing the reliance on journal metrics.

These concerns are not new, nor limited to the EU. In 2013, an international coalition signed the DORA declaration on research assessment in San Francisco, hoping to reduce the weight of publication metrics in the evaluation of researchers.

The Commission has included a reform of research assessment in the policy agenda of the revived European Research Area plan for a single market for research in the EU. The Commission is now gathering support from a coalition of organisations to implement the assessment reform and to begin testing it out this year.

While it agrees that the careers of researchers have become “too systematically linked to the quantified assessment of very specific publication formats as accepted proof of academic achievement”, the DFG says it is up to research institutions to decide how to address that issue, arguing that reforms should stem directly from universities, research institutions, academies, and public research funders that fund research. “It is the task of these stakeholders to ensure responsible assessment of research and guarantee the appropriate development of the publication system in the future,” the position paper says.

The DFG acknowledges that assessing research by leaning heavily on publication metrics can have a negative impact on science and the humanities. Metrics such as the total number of publications and citations, as well as bibliometric measures such as the h-index, a metric for evaluating the cumulative impact of an author's scholarly output and performance, are increasingly problematic. Researchers can exploit the system and advance their careers by chasing to score higher points in the publication game instead of focusing on the content of the research

DFG recommends that research organisations and funding agencies come up with new ways to assess the quality of publications. According to its position paper, it is the responsibility of funding agencies and foundations to accept a broader range of publication. “The reputation of publication venues and bibliometric indicators, where they exist, should be removed from the canon of official assessment criteria and kept to a minimum in practical use.”

Tobias Grimm, head one of DFG’s life sciences divisions, said the reform of research assessment is an ongoing debate in Germany, but very few university networks signed the EU agreement. “The grassroot discussion is independent of this Commission initiative,” he said.

For Grimm, research organisations should rely less on publication metrics because “people try to manipulate [them].” He noted that DFG is about to change its reviewing system, it signed the DORA declaration and will soon release a new CV template and application guidelines for researchers that put more emphasis on the content of projects.

“At the same time, it’s quite important that the Commission leaves [the reform] in the ownership of science,” said Grimm. “We do not need an alignment for its own sake.”

The German research system has many very diverse players, from universities and funding agencies to networks of researchers which focus on specific areas of study, while the federal government has marginal responsibilities for education and science. The jurisdiction is split among 16 landers and a complex network of universities and research performing organisations, research funders and charities, with a strong commitment to institutional autonomy.

EU-led reform

The Commission began gathering opinions from stakeholders in early 2021 to move the reform forward, but German research institutes were not very keen to jump on the Brussels policy bandwagon and became concerned about a potential top-down binding agreement.

Last month, the Commission sent an email to a core group of research stakeholders who are participating in the negotiations, asking them to give input to a draft agreement by 26 April. The responses will inform talks to be held today at an online stakeholder assembly.

The agreement is being drafted by a team of representatives from the European University Association (EUA) and Science Europe, together with Karen Stroobants, a former chemistry researcher who is now a policy adviser to various research institutions in Europe. The Commission took on the role of “facilitator”.

In addition to the drafting team, the Commission has helped set up a core group of 20 organisations at EU and national level. The Commission has also put together a list of nearly 300 organisations who wished to sign the agreement on the reform.

The process is meant to help stakeholders and policymakers figure out what is the purpose of the reform, how it will be explained to researchers and what kind of indicators will complement journal metrics.

According to Jan Palmowski, the secretary general of the of Guild European Research-Intensive Universities, who is one of the 20 members in the core negotiating group, an agreement could be finalised very soon. “The plans are that this will be finished this side of summer so that it will be ready by early autumn,” Palmowski told Science|Business.

As the drafting process of the reform agreement comes to an end, Grimm says the DFG is has not signed the agreement because it wants to see the outcome of the negotiations before it gives a blank signature. “You had signatories to a document that was to be elaborated, you had a blank page to sign,” said Grimm, who is also a member of the core group. The alliance of ten German research organisations, including the DFG, has decided to hold off a decision to sign the agreement as yet.

Palmowski is confident more institutions will sign the agreement once it’s been completed, but then the Commission and stakeholders will have to decide how to monitor their progress in implementing the agreed reforms. Ensuring member states go along will be difficult, he says, because the EU has dozens of national and regional jurisdictions in higher education and research with different guidelines and laws, where institutions still rely heavily on quantitative indicators.

The main question now is how to draft the agreement in such a way that it is not restrictive and allows institutions to adapt, while still maintaining a sense of direction, said Palmowski.

As for the German question, Palmowski says the vast majority of faculties in member universities of the Guild are already working to ensure that metrics are not being used independently from qualitative assessments of researchers and their work. “If we say everything we did up until now is no good, German institutions will not come on board,” he said.