Do big public EU policy consultations work? For the most part

Two weeks ago, the European Commission concluded its biggest consultation on the past, present and future of EU research and innovation policy to date. During the three-month it was open feedback was gathered from 2,785 organisations and individuals on what works and doesn’t in the EU’s big research programmes. But will it feed into better policy?

To find out, Science|Business talked to a researcher who has spent years analysing EU’s big public consultations. Adriana Bunea is a professor in comparative politics at the University of Bergen and EU consultations are her bread and butter. In 2018, she was awarded a grant from the European Research Council (ERC), to dig deeper into these feedback exercises.

These big public consultations saw a big revamp in recent years. In 2016, the EU adopted the so-called Better Regulation agenda which dictates, among other things, how the Commission should consult stakeholders on the policies it proposes. This gave way to bigger and more elaborate consultations than ever and has allowed citizens to engage with EU policy throughout its entire cycle, starting from conception.

Bunea says the recent Horizon consultation was one of the most ambitious to date, combining policy evaluation and design. It asked all stakeholders to give their final assessment of the EU’s previous seven-year R&I framework, Horizon 2020, and their first impressions of the first half of the current €95.5 billion Horizon Europe.

Research organisations, universities and industry had a lot to say, and many pointed to a few key issues: an imbalance in funding for close-to-market versus fundamental research as well as unnecessary complexity plaguing the application process. The outcome, the Commission promises, will feed into the strategic plan for the second half of Horizon Europe and the next research framework, which starts in 2028.

Kamila Kozirog, policy analyst at the European University Association, sees the public consultation is a preamble to more targeted stakeholder consultations on the future research programmes, much like the ones the Commission held to shape up Horizon Europe.

“When Horizon Europe was designed, the Commission launched formal and informal consultations with stakeholders, and that was the platform for us to really contribute,” says Kozirog. “That’s where we can achieve the biggest impact and contribute to the design of the next framework programme.”

Information overload

The logic behind the exercise is simple: ask people to sit down and think about their past experiences with the programmes and look to the future of EU research and innovation policy. But with a myriad of complex policy questions, it’s not an exercise for the faint of heart, or the ill-informed. “The risk of information overload is real,” says Bunea.

At the same time, it is necessary to provide the scope for proper comment. Marta Agostinho, executive director of EU-LIFE alliance of research institutes in the life sciences, says many complex policy topics were addressed through multiple choice questions across the survey, making it difficult to contribute meaningful feedback.

Kozirog mirrors the concern. She notes the survey asked surprisingly little about Horizon Europe missions in particular, despite many stakeholders wanting to contribute feedback. “It didn’t really provide the opportunity to fully contribute on how to ameliorate them in the second half of Horizon Europe,” she says.

These issues are not exactly new for big scale consultations. Bunea’s team recently tried to analyse and aggregate the results of several such surveys, to try and find a data reduction technique and map the preferences of stakeholders across one or two dimensions, such as if they wanted more or less EU action, and what specific set of measures they preferred. But there was an issue: too many people struggle to answer all the questions and the results are riddled with huge numbers of “I don’t know” answers.

 “We discovered that there are so many ‘I don’t know’ answers that this became a dimension in itself,” says Bunea. “What this reveals is that open public consultations can be quite intensive on the side of the stakeholders.”

At the same time, the information gathered offers a wealth of information if it is properly digested. Yet, it’s often underused. The data is gathered, analysed across descriptive statistics, such as how many business groups say yes or no to one new rule or another, and then Commission officials write a consultation report, some of which ends up feeding into policy.

“A lot more can be done in terms of information coming from open consultations to make sense of what stakeholders really want,” says Bunea. “In some respects, it’s an underused instrument by policymakers.”

This time around, the Commission is getting help from an external consultant to analyse the feedback through descriptive statistics for yes and no questions, manual and automatic approaches for open-ended ones, and categorising submitted position papers in an issue matrix featuring the key issues.

Based on this, the Commission will produce three ‘synopsis reports’ together with Staff Working Documents on Horizon (to be delivered around the end of 2023), Horizon Europe (in 2024), and the analysis for the next strategic plan (in May 2023).

Better policy?

The big questions is whether  feedback is translated into better policy. But there is no easy answer to that because of the difficulty of tracing inputs received from stakeholders to legislative proposals.

Jan Palmowski, secretary general of the Guild of European Research-Intensive Universities, says the last time the Commission held this exercise, it felt short of the expectation of how much would be fed into the first Horizon Europe strategic plan. “One thing we would look forward to is a clearer sense of the connection between the strategic plan and university priorities,” says Palmowski.

At the same time, he acknowledges the last strategic plan was drawn up during a strategically complicated time, at the start of the COVID-19 pandemic, and is more positive about the next one. “We should give the colleagues in the Commission the benefit of a doubt and let’s see what they come up with,” said Palmowski.

Whichever way it goes, the public consultation is an important step in the assessment of Horizon Europe and the start of designing the next framework programme, notes Agostinho. But it should only be the beginning of the full consultation. The results should be coupled with more in depth dialogue in focus groups and meetings, as well as commitment to address the long-standing complaints routinely pointed at by stakeholders, such as the insufficient overall budget and lack of investment in fundamental research. “We are also concerned on how the analysis of the survey results will be used subsequently in the next steps of the co-creation process,” Agostinho says.

Some trends are clear. For one, Bunea says the consultations themselves give policy proposals legitimacy: if the Commission says an open consultation fed into a document or initiative, stakeholders tend to immediately be more supportive.

The bigger consultations have also helped increase citizen participation in the process and made the Commission a frontrunner in stakeholder engagement, ranking top of the charts in OECD assessments.

At the same time, opening up the policymaking process to audiences has exposed the Commission’s traditionally, in theory, apolitical policy development exercise to potential politicisation. But it is careful about letting politics in: a citizen’s comment is more likely to be taken into account if it is evidence rather than opinion based, according to Bunea’s research.

For now, it remains unclear how stakeholder inputs and general trends observed translate into policy solutions, options, proposals and initiatives. Bunea’s team is working on getting a clearer picture, but there are no clearcut answers as a lot depends on the policy area.