Viewpoint: novel ideas, bold bets

At the European Research Council (ERC) we know that ambitious, high-risk research pushes back the frontiers of knowledge. Supporting scientifically daring and novel projects means trusting researchers with bold ideas, those who challenge existing paradigms and push the boundaries of existing knowledge and methodologies. 

Evaluating such pioneering projects and selecting them for funding is a difficult task that all science funders face. We are pleased to see the debate on how to identify and support such work. A recent study covering the early years of the ERC (2007-13), reported in Science|Business, comes with relevant questions and input, even though its authors are cautious in their conclusions.   

The ERC welcomes academic analyses of its performance. As an organisation “run by scientists for scientists,” in the words of European Commission president Ursula von der Leyen, critical analysis is our daily business. One of the key tasks of our scientific council is to assess the processes and procedures we use to evaluate applications for grants, as well as the outputs and outcomes of the work we fund, and we have a task-force in place to help prepare an overhaul of some of these procedures. 

We also encourage academic researchers to look into our evaluation practices and results, and we facilitate independent research into our methods and outcomes. The study by our colleague and former scientific council member Reinhilde Veugelers, and her co-authors, on the ERC’s evaluation process from 2007 to 2013 is itself based on the data provided by the ERC.

The ERC, with an autonomous and independent scientific council, is widely recognised as a European success story. This is “shown by the EU’s position at the world scientific frontier,” as Reinhilde Veugelers stated in her recent policy memo to the new research commissioner. Many metrics and qualitative surveys indicate that ERC grantees perform well above average in making impactful discoveries. Their work is strongly overrepresented in the most highly cited publications, and among the successful projects funded by European Innovation Council Transition grants. It is also regularly recognised by international awards, some of them Nobel Prizes. 

The Quantum Internet Alliance is another telling example. This interactive group of researchers, start-ups and companies is building the world's first full-stack quantum internet prototype network, an endeavour initiated by ERC grantees in quantum communications and quantum computing. 

These examples show that, on the whole, the ERC peer review evaluation panels select promising projects and creative researchers for funding, and that these perform well compared to the overall research community.

Challenging questions

Veugelers and her co-authors took on important and challenging questions, which they analyse quantitatively and rigorously. Specifically, they ask whether, in 2007-13, ERC panels funded proposals by scientists with a track record of novel and pioneering findings at a higher or lower rate than researchers without such a record. They also asked whether the funded ERC applicants produced more such novel research than unfunded applicants in the post-application period.

The main challenge in this endeavour was to find an objective measure for what can be qualified as novel or pioneering. The measure used by Veugelers and her co-authors is a bibliometric proxy to gauge novelty, namely the applicant’s previous publications with “unusual journal combinations in the referenced papers.” The assumption is that this proxy is an indicator of the applicants’ general research approach and the research they propose for funding.

Both the scientific council of the ERC and our evaluation panels fully recognise that peer review, while the best available mechanism for evaluation, is not perfect. This is particularly true for assessing interdisciplinary and high-risk research. Through our regular assessment of the evaluation process, we have long been aware that proposals that straddle two or more research fields need to be judged by more than one expert evaluation panel. It is reasonable to assume that many of the applicants that fulfil the novelty definition used by Veugelers and her co-authors fall within this group. 

To the dismay of the panels and the scientific council, these projects have a slightly lower chance of being funded. This is an effect that nobody wants. We believe that this is one of the roots of the problem pointed out by the study and we are indeed seeking means to eliminate this tendency. At least part of this difficulty for interdisciplinary projects may be a technical problem that we seek to solve.

Veugelers also touched on the problem of assessing the “feasibility” of projects competing for funding, citing some panel members’ view that “this is such a large amount of money [that] we cannot afford that this would fail.” The scientific council has also recognised the feasibility criterion as being counter-productive when selecting high-risk projects with breakthrough potential and has hence removed it from the first step of the evaluation process.

So, has the ERC “failed to support applicants with a track record of novel research” and is therefore not “living up to its original aim of backing pioneering, high-risk, high-gain ideas,” as the Science|Business headline suggests? The authors of the study themselves are clear that this is not their conclusion. Their approach captures one specific aspect of research novelty, related to new combinations of knowledge areas, and defined by the types of referenced journals.

The conclusion is not that the ERC “does not fund” applicants with a history of novel research, but that the applicants with previous publications “displaying unusual combinations of referenced journals” have a slightly lower success rate in securing grants -- but many do secure grants! This difference is not seen among more senior applicants or those from top research environments. And as the authors remark, early-career ERC grantees produce on average more novel publications than they did before receiving the grant.

At the same time, the authors did not have access to the text of the research proposals nor to the evaluation panels’ comments, which, as they state, “leaves a large part of the probability of being selected unexplained” in this analysis. Since the study does not capture many other factors that might drive success rates, the authors are extremely cautious in their conclusions. 

This study is valuable nonetheless: it is important to have solid data on which we can base our decisions and actions. If we want to change an aspect of the world in which we live, we first need to understand the functioning of that world. Delivering this understanding in general is the essence and the value of fundamental research, and this also goes for this study, and for fine-tuning the processes at the ERC.  There is never room for complacency.

Maria Leptin is president of the European Research Council.