Excellence, Success and University League Tables

Excellence is surely a noble aim for universities - as for other institutions and activities.  However, since there are many excellences, and since those of universities will vary with the activities they undertake, it may be hard to say how good a university is.  Once we acknowledge the plurality of excellences that universities may seek, we can no longer imagine that they should aim simply to do better than other universities, although that can (but need not) be one of the results of striving.

Where standards are low, even the most successful may not be excellent; where they are high, many whose performance is excellent may not be successful. 

A good reason for taking the concept of excellence that comes in different forms seriously is that we are not then compelled to see the pursuit of excellence as a zero-sum game: we can imagine, indeed encourage, a world in which all universities do excellent teaching and research.   By contrast, we cannot even imagine a world in which all universities are equally successful in teaching or research, since success, unlike excellence, is a positional good.

The Measurement of Success

Many contemporary debates about universities focus not merely on success, but specifically on success as represented by scores on performance indicators, and by procedures used to amalgamate those scores into rankings.  Unsurprisingly, what starts out as an attempt to show us what universities are really like, and to provide objective evidence that could be useful in making difficult judgements that have to be made, can end up illustrating  Charles Goodhart’s well-known law: ‘Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.'

Goodhart articulated this law in a discussion of monetary policy, but it has since been very widely referred to in discussions of accountability, and especially of accountability in education.  A formulation that is particularly pertinent for judging university metrics runs ‘when a measure becomes a target it ceases to be an adequate measure’. These problems do not arise because of defects peculiar to one or another choice of performance indicators, or one or another choice of method for ranking or constructing a league table out of scores on those indicators, (although both sorts of choice create problems):  the problems are intrinsic to the approach, and unlikely to be dispelled by making different choices.

Any selection of performance indicators may be questionable because the required data are unavailable or patchy, or not genuinely comparable between institutions or between university systems.  They may be questionable, or even misleading, because they are only proxies for what we really want to know about: they are performance indicators, not genuine measures of performance. And any way of combining scores on these questionable indicators to create rankings and league tables involves many additional contestable assumptions.

None of this has deterred those who construct rankings. If we look back for a decade, we see the emergence and now widespread reference to the Shanghai Jiaotung University academic ranking of world universities and the Times Higher Education world university rankings.  As is well known, these did not rank many European Universities in the top 50.

And now the EU is considering a more differentiated ranking that will, it is intended, rank different aspects of universities separately, rather than providing a single composite ranking.  It is to be called U-Multirank.  Needless to say, this project too has met cogent criticism, in particular in a 2010 report of the League of European Research Universities - and also in a recent Select Committee Report of the House of Lords in the UK (in which I had no hand).

My own view is that it if U-Multirank is developed, it will still be open to anybody who finds it advantageous to devise a favourable way of aggregating the separate scores, just as the aggregated scores of current league tables are now commonly disaggregated by the public relations departments of universities in order to publicise the more favourable aspects of their scores.

Once comparative measures of university performance are compiled, it is easy to combine them in various ways to create rankings, and once that is done it is easy and tempting for institutional leaders and others to claim that selected representations of scores on these rankings should be seen as objective measures of institutional quality.

Scores on these rankings are often said to matter because they can inform - or at least   influence - prospective students and their even more prospective employers, or for wooing potentially mobile researchers.  They may be reflected - and may also be incorporated -  in reputational rankings.

University rankings can be used by governments and other funders or research collaborators, or businesses on research collaborations.  And they certainly lead some universities with lower rankings to seek ‘partnerships’ with universities whose profile will, they hope, raise their own: those near the top of the league tables are incessantly wooed by others with lesser ranking, in hope of landing advantageous collaborations.  So league tables are not a trivial matter.  However, they are neither a reliable nor a cost-free way of judging excellence.

Plurality and Diversity

Prima facie the case for measures and rankings may seem compelling.  Universities are now numerous and diverse; their spheres of operation are no longer provincial, but national and international; their aims stretch beyond tertiary education and research. And they are also supposed to kick-start technology transfer and innovation; to contribute to regional and national prosperity; to do research that has impact; to contribute to national influence by recruiting the ablest international students—and so on and on.

However, there is now little prospect of finding or rehabilitating any single view of what a university is or should be.  Essentialism is for the past, and we can at most expect to find family resemblances among the varied institutions that call themselves universities.

Universities to be sure, are among the institutions that provide post-secondary education or training, but so too do professional bodies, corporate training providers and government training organisations. We now operate with a highly elastic and non-prescriptive idea of a university.

Yet the real diversity among institutions, and their number, might be taken as showing that we need rankings more than we did in the more provincial past, so that we can tell what each university does and does well, what it does not do, or does not do well.  That is indeed what people often need to judge: but it is not obvious that rankings will help them to do so.

Those who devise rankings often suggest that they will be useful because students need information to decide where to study, and academics need information to decide where to apply for posts, seek collaborations, or contact colleagues.   However, in my experience it is vanishingly unlikely that students or academics will find the information that university rankings provide adequate, or even useful, for such purposes.

Prospective students need to know far more about specific courses at specific universities.  Researchers need to spend enormous energy trying to build departments and research teams, seeking out the right colleagues and collaborators, and attending the right conferences.  Anyone who defines their academic aim as merely as ‘joining a university that ranks highly in the league tables’ is aspiring to have a highly-rated brand, but avoiding the hard work of assessing the relevant evidence for excellence for the specific things that matter to them.

But if rankings are not generally useful to students and researchers, who and what are they for?  Cui bono?

Accountability and Rankings

It seems to me that rankings are mainly useful to those who seek to - perhaps need to - control universities.  That control is in the first place achieved by governments and other funders, and has a prospective and a retrospective aspect.  The prospective aspect is a matter of allocating funds for specified purposes and objectives.  The retrospective aspect is a matter of holding universities and other research institutions that are publicly funded to account for achieving the prescribed or agreed objectives.

Funders often use contested systems of funding for institutions and for research projects, and for this rankings may indeed be useful.  That, for example, has been the aim of the Research Assessment Exercise in the UK, in which ranking of research profiles for each department in each university for the immediate past period, provides one basis for some research funding allocations for the next cycle.

However it is important to note that research rankings rely on expert judgement to create the data that provide the basis for ranking.  The real work goes into peer review, and the compilation of rankings is often mainly an administrative task, used to differentiate the level of funding for institutions that score differentially.

This process consolidates the highly informative evidence provided by peer review into much less informative but comparative rankings.  Consequently the real use of rankings and league tables is driven by policy rather than by academic needs: rankings offer a more or less accepted justification for a differentiated allocation of funds, which might otherwise raise questions about fairness and favouritism.  This is not, of course, a negligible matter.

None of the above is meant as an argument against rigorous assessment of teaching or research performance.  Rigorous judgment matters.  Peer review matters. Integrity in marking and examining students matters.  But if we try to measure these complex processes – these forms of excellence— by scores on performance indicators, and to compile such scores into league tables, we are likely to obscure the evidence and may end up creating suspicion rather than trust.  If we persist in doing so, we simply show that we care more about relative success than about excellence.

Onora O’Neil is professor of philosophy at Cambridge University, a former principal of Newnham College, Cambridge, and founding President of the British Philosophical Association.

Thisis an edited version of a talk she gave at Aarhus University at the conference,‘Excell ence Revisited – The Value of Excellence’ one of the events of theDanish EU Presidency, 18 - 20 April 2012. The conference was a step in the development of the ‘Excellent Science’ objective of Horizon 2020.University