EU university metrics: growing in reputation abroad but divisive at home

The EU’s university ranking system U-multirank, is having success in selling its concept abroad, but there is still some doubt about the usefulness of the ranking system at home in Europe.

“We polled very mixed opinions among our members, [ranging] from positive to suspicious,” said Tia Loukkola, a director at the European University Association (EUA), an umbrella group for 850 higher education institutes across 47 countries.

The single biggest complaint from universities is the time-consuming effort required in pulling together data requested by U-multirank, which is set to release the second edition of its ranking by the end of March.

“All our universities have concerns regarding reliability and comparability,” Lukkola said. “The definitions for indicators used by the ranking don’t often match what universities use.”

As yet another alternative system for measuring the quality of universities, the U-multirank eschews traditional rankings, allowing students to select their own personal top list based on their study interests.

Students can use the U-multirank online tool to profile and compare universities, faculties and programmes out of 30 indicators in five areas: research, teaching and learning, international orientation, knowledge transfer, and regional engagement. Activities are assigned grades from A (very good) to E (weak).

Data wrinkles

“Some indicators just don’t work,” admitted Frank Ziegele, a researcher at Germany’s Centre for Higher Education, and one of the academics leading the €2 million project.

Statistical offices work with many different ways of collecting and presenting figures, so U-multirank’s data collection task was never going to be straight-forward.

When the project’s online tool launched last May, some of the data was very patchy. The employment rate of graduates from various bachelors and masters courses, which the ranking attempts to track, was a major data hole.

“Unemployment is a problem that’s remaining,” said Ziegele. “Next year we have to decide whether to leave it in or take it out.”

Another indicator with limited data is the number of graduates who have remained working in the region of their university. In all, two or three indicators might be taken out, Ziegele added.

Despite recounting universities’ sense of irritation with some of the data requirements, and a feeling that the benefit of the ranking system is not evident yet, Lukkola said that EUA members would not be discouraged from continuing to use it.

While common measurement methods between countries might be too much to hope for, there has been a surprising development in Spain. “There are discussions to harmonise data between the Spanish Education Ministry and U-multirank,” said Ziegele.

Who is participating?

The proof of the pudding for many was always the ranking’s success in attracting institutions from beyond Europe. “We’re exceeding plans for participation numbers,” Ziegele said.

For now, the project is not revealing the full list of institutions involved in the second edition of the U-multirank rankings, due. Nor is it saying a lot about the geographic spread.

But uptake is highest from India and Japan, said Ziegele. Luxembourg, the only European country that did not join in in the first round, will be represented by the University of Luxembourg, he added.

While Ziegele and his colleagues have spent the last nine months hot-footing it between conferences around the world, the project’s sought-after big fish, US and Chinese universities, have not yet been reeled in.

The 2015 edition will include rankings on 1,200 higher education institutions, with nearly 700 fully participating in U-multirank. For universities that do not participate, the project will compile publicly-available data drawn from bibliometric and patent databases.

As well as increasing participation rates, the ranking is introducing new subjects into its online tool. Physics, business, mechanical engineering, and electrical engineering were there to begin with, and medicine, psychology, and computer science will be added in the second edition.

“We want to keep adding more fields,” said Ziegele. In 2017, the ranking will include music, the subject with the highest mobility rate in the world.

Rankings, rankings everywhere

There is a multitude of ranking systems with over 150 country-based or specialist league tables. Like the EU, Spain and Bulgaria have tried publicly-backed ones, along with Pakistan, Thailand and Iran.

In the US, the Obama administration is preparing to roll out its own college metrics plan. As in Europe when U-multirank was unveiled, debate rages over which variables to include, the fear being whatever ones are selected will necessarily skew the ratings in that direction.

Under Obama’s plan, schools may be judged on graduation rates; the ability of their graduates to pay back their student loans; and the schools’ accessibility to low-income and first-generation students.