Commission lays out plans to manage low Horizon 2020 success rates

As anyone who has ever applied for research funding will know, getting the money is hard. But rarely has it been as true as in the case of the EU’s monstrously popular Horizon 2020, the programme which now leaves researchers with roughly a one in eight shot of winning a grant, according to new numbers released this week.

“It’s more popular than ever before. But with our success rates we’re heading to a situation where we have to be very careful not to scare away top researchers,” Robert-Jan Smits, the European Commission’s Director-General for Research and Innovation, told Science|Business.

In the UK, a researcher faces 30 per cent odds of success when applying to the major research councils. In the US, success rate stands between 22 and 24 per cent for the National Science Foundation and 18-21 per cent for the National Institutes of Health. Even in Australia, where public R&D funding has been cut, applicants to the National Health and Medical Research Council have a 21 per cent chance of success.

Under Horizon 2020, researchers’ odds for winning EU research money has tumbled to between 12 and 14 per cent, down from 19-21 per cent just two years ago.

For the top administrators in EU research, the demand is flattering but risks becoming a big political problem. Already, several critics of the Commission have been complaining that simply filling out the Horizon 2020 forms has become a major economic activity in its own right.

The Commission acknowledges the problem. “The concern I have is for top universities,” said Smits. “If they see success rates dropping below 10 per cent, they’ll stop applying, because they can get money anywhere.”

So, it’s a good time to step in and apply canny manoeuvres that could make a big difference in stemming the deluge, the Commission has decided. Smits and his colleagues have begun implementing a plan that doesn’t involve upending the whole system – but is intended to separate the losers from the winners faster. That may not change the overall odds of getting a grant; after all, only a drop in numbers of applicants or an increase in budget would do that. But the plan is intended to separate the losers and winners faster, so less time and money is invested in a losing application.

There are two main methods of “containment” – or rather, three if you count the demotivation some researchers might feel after reading about hugely competitive calls.

Two-stage proposals become the norm

Firstly, the two-stage evaluation procedure, used in several of Horizon 2020’s sub-programmes, will become a more widespread procedure for screening applications.  

As a rule of thumb, 80 per cent of proposals – those not considered strong enough to meet competition requirements – should be rejected in a short-form, stage one evaluation, Smits said. In stage two, where a longer application is required, at least 35 per cent of proposals should have a chance of success.

Smits acknowledges the process has not been perfect in the past. Evaluators have “been criticised for not giving researchers enough feedback after stage one.” There have also been complaints that success rates for the second stage of some competitions, notably health and the SME Instrument, have been very low, he added.

“So what we’re doing now is to clean that up,” said Smits. “I think first of all, we must give better feedback after stage one, so that those who go on to stage two have a better idea of how to improve their applications.” (This has to be balanced with the need to process evaluations fast, he added). 

A success rate of around 20 per cent for stage two “would not be fair”, said Smits, because “these are the people that are asked to make very detailed proposals.”

This threshold will be limber, with the Commission reserving some flexibility depending on the budget available, Smits said; but no proposal should go to the second stage without a one in three chance of being funded.

Requirements for stage one won’t change for researchers. There are no plans to ask for shorter proposal outlines, Smits confirmed. 

Evaluators told to get stricter on ‘impact’

The second main step planned by the Commission is to get stricter on rating the impact of a proposal for industrial technologies or a societal challenge.

Impact is one of the three criteria Horizon 2020 evaluators use to assess the quality of research proposals; the other two are excellence and the quality of implementation. In order to measure impact, several indicators are considered, such as, the capacity to innovate, the use of new knowledge, and contributions to the wider societal and economic impact. However, the key ones are those listed in the individual competitions themselves.

To be in with a shot of winning a grant, applicants will have to be sure that their expected impact is  “clearly defined” and rigorous. Brendan Hawdon, Head of Horizon 2020 Policy in Smit’s directorate-general, elaborated. “It’s all about the outcome,” he said. An applicant should say clearly: “Here’s what we want to come out of the project.” For an innovation project, for instance, increasing the world’s knowledge wouldn’t count as a concrete impact. By contrast, in a transport project, creating safe devices for a car, which would halve the number of lives lost on the road, might be a better example. Other impacts might be on technical standards, or the economy, he said.

Smits pointed to the way the Science Foundation Ireland handles impact in its grant applications. Among other things, he said, there is a two-stage evaluation process: First on the ‘excellence’ of the science proposed, and second – evaluated by a different set of experts – on the societal, economic or other impacts. He said that exact system wouldn’t work for the Commission, but the general focus on defining and assessing impact rigorously would. Of course, he added, “there is a risk that people will promise the moon. We have to be careful we aren’t taken for a ride” by over-optimistic applicants, he said. As a result, he said, in Horizon 2020 extra emphasis will be given to the evaluators carefully reviewing the impact claims.

This won’t be welcome by all researchers. Some have chafed at what they already see as too much emphasis on the impact criterion. Measuring the future value of an idea is hard, they complain. Under Horizon 2020, “impact has become very important but we need to make sure we are not becoming fundamentalists in assessing it,” said Sergio Bertolucci, the director for research and computing at CERN, at the Science|Business conference on Horizon 2020 in March.

The non-starter

Others in the research community have urged the Commission, if it really wants to improve success rates, to narrow the target: Define more precisely what kind of research or engineering they want to fund in each call, rather than wording the documents broadly.  Smits rejected that, as not leaving applicants enough freedom to decide for themselves what research topics are important. “What we are not going to do to deal with this demand is to make the calls more prescriptive. If you remember, one of the big criticisms of the previous research programme FP7 was that competitions were far too prescriptive and did not allow for enough creativity or innovative solutions to emerge,” Smits said.

Another option under debate: The European Research Council (ERC) has a rule that says if a proposal scores below a certain threshold, the organisations involved cannot apply again for two years. Smits said he hasn’t ruled that out for the rest of Horizon 2020. “We’re looking into it – we’re not there yet though,” said Smits.

Outside Europe

The other item high on Smits’ wish list is to boost participation from industrialised countries outside Europe – which on the face of it sounds counterintuitive, given how he’s trying to manage demand back home. But the big difference with so-called rich “third countries” is that they have to bring their own funding to the table. So, goes the Commission logic, it’s a win for both the EU and the foreign researchers if they apply jointly for funding.

Smits said there are negotiations underway with the Japanese and Korean governments, with a view to bumping up their participation rates in the programme. Japan has a rapidly ageing population in need of scientific and technical answers, and so have strong motivation to work with Europe. “The Koreans are masters of innovation but their big challenge is basic science. The reason they want to get involved is to generate new ideas,” he added.

The plan, Smits said, is for the Japanese and Korean governments to dedicate some funding that would automatically go to their researchers who are accepted in an EU Horizon 2020 project. Mexico already has a similar arrangement in place with the EU.

It’s South Korea’s latest courtship with EU research. In 2013, the country became the first non-European member of the Eurostars programme, which receives part of its funding from EU coffers, and its researchers became eligible to apply for ERC grants.

He said he would like to have greater cooperation with the US on research, but that country has so far thrown up a series of legal and diplomatic obstacles to its researchers who want to take part in the EU. And cooperation discussions had been underway with Russia, before the Ukrainian crisis. Moscow had been keen to set up a special fund for its researchers; but since EU economic sanctions began those discussions have dropped off to “a low level,” Smits said.