University rankings organisations need to do more

“The huge impact of rankings places considerable responsibility on their authors,” according to Jaroslaw Gowin, Poland’s minister of science and higher education, who was speaking at the opening of the IREG 2019 Conference in Bologna in early May.

He added that “the high quality of contributed data is an essential prerequisite for ensuring the high quality of any ranking”. Gowin challenged ranking organisations to find proper ways to measure the quality of teaching as well as collaboration between academia and industry.

The 2019 conference of the IREG Observatory on Academic Ranking and Excellence, “Rankings – A challenge to higher education?”, wasn’t a place for self-congratulation for ranking organisations.

Although all the speakers had no doubt that ranking was here to stay in the world of higher education, policy-makers like the Polish minister and also rectors, experts and rankers themselves, explored and dissected both the positive and negative sides of university rankings.

In her presentation, “In Pursuit of Prestige or Quality?”, Professor Ellen Hazelkorn highlighted the responsibilities of ranking organisations. She saw prestige as a driving and enduring force behind rankings.

However, she said one university’s prestige comes at the expense of others. Wealthy elite universities in global metropolises are able to maximise “absorptive capacity” in contrast to lower status institutions in sub-regional areas.

In pursuing international success for their universities, policy-makers can act to concentrate benefits and resources on certain institutions while priorities are shifted towards global research “excellence” rather than community-based strategies. In this game neither governments nor universities are “innocent victims”.

Rankings are not simply providing information but are playing on and amplifying structural inequalities within particular societies and globally. Hazelkorn pointed out that it is no longer good enough to talk about universities’ corporate social responsibility, and asked: isn’t it time we talked about the corporate social responsibility of the ranking organisations themselves?

Over-reliance on reputation

It is easy to say that a ranking is as good as the data it is built upon. Data was one of the main issues discussed in Bologna. At least three speakers criticised rankers for over-relying on reputation. Gowin, who has explored how reputation assessment is done and how researchers are selected, has come to the conclusion that reputational assessments are “not very transparent”.

Hazelkorn observed that prestige is long-lasting and that “reputational surveys are built on the principle that being known is key signifier of success”. Rankings expert Richard Holmes also voiced dissatisfaction with the fact that major global rankings such as Times Higher Education and QS put too much weight on the prestige factor.

Data lakes

Daniel Guhr from the Illuminate Consulting Group drew attention to another issue related to data that is increasing in prominence. The sheer volume of data elements involved total many billions. Data points and elements have become easily accessible, resulting in vast data lakes, he said. Who will navigate through these data lakes – rankers, data providers, consultants, institutional analysts, tech firms, others? Guhr asked.

He predicted that artificial intelligence-driven analytics – sitting on top of triple-digit billions of data elements – will start to upend the global university landscape in less than a decade from now.

Universities will have to rapidly move on from basic rankings analytics to a much more sophisticated approach – global performance metrics. Guhr warned: “Institutions which are moving to an evidence-based decision-making approach will outperform institutions that make decisions based on belief.”

Ways forward

The IREG Observatory on Academic Ranking and Excellence has started its own IREG Data Initiative aimed at making data, especially data provided by universities, more uniform for all rankings, thus making filing information easier for universities.

Luiz Claudio Costa, president of the IREG Observatory, highlighted three issues that are crucial for discussions involving rankings, universities, governments and society: 1. Data governance and data ownership; 2. Use of technology, especially artificial intelligence; and 3. Refining rankings indicators to encapsulate teaching quality and universities’ third mission, engaging with the needs of societies.

Rankings are no longer a mythical media phenomenon. Universities and policy-makers dissect and understand them well. While the number of international academic rankings has been growing, everybody knows rankings are drawing on the same sources of data, predominantly data of a bibliometric nature.

Universities and policy-makers are now asking more questions, and demanding that ranking organisations do more – especially when it comes to assessing the teaching and social responsibilities of higher education institutions.

Waldemar Siwinski is vice president of the IREG Observatory on Academic Ranking and Excellence.