Why SDG-focused Impact Rankings need to be contextualised

Times Higher Education (THE) delivered yet another bumper edition of its Impact Rankings on 21 April. There were exactly 1,239 institutions across 98 countries which submitted data, compared to 859 institutions in 2020 and 556 institutions in the inaugural edition of 2019.

The continued participation of universities in this ranking, together with an array of ongoing data sharing and benchmarking endeavours globally, are important developments given the importance of the United Nations Sustainable Development Agenda to our overall survival.

The THE Impact Rankings have become an important tool for universities to map their sustainable development journey. This is despite various methodological limitations and many things we all wish were not measured or done in the way that they are.

It is unfortunate that it is called ‘Impact Rankings’ as this obscures the fact that it is an annual assessment of universities’ performance against the United Nations’ Sustainable Development Goals (SDGs). Another way to see it is as a benchmarking exercise, as participating institutions are able to benchmark against others (regardless of their geography and standing) on any metric within the SDGs.

In total, there were 1,115 institutions which submitted data for four or more SDGs and therefore qualified to receive an overall score. Although there were 19 other institutions which submitted data for at least four SDGs, these institutions elected not to be publicly allocated an overall score.

Geographical distribution

To bring into context the geographical distribution of universities which participated in this ranking, let us compare the results against the 2021 THE World University Rankings (THE WUR).

First, there are significantly fewer universities from North America and Western Europe in the THE Impact Rankings (262 or 23.5%) compared to 622 or 40.7% of total ranked universities in the THE WUR.

Second, there are also fewer universities from East Asia and the Pacific region in the Impact Rankings (247 or 22.2%) compared to 382 or 25% of ranked universities in the THE WUR. Interestingly, there are only 15 universities from Africa ranked in the Impact Rankings compared to 21 in the THE WUR.

Across all other world regions, we see there are more universities ranked in the Impact Rankings than in the THE WUR.

Brendan O’Malley reported the strong representation of Russian universities – 75 received an overall score compared to 48 ranked in the THE WUR. Other countries which have significantly more universities ranked in THE Impact Rankings than THE WUR are Iraq (37 vs three), Pakistan (36 vs 17), Egypt (31 vs 21), Colombia (18 vs nine), Indonesia (18 vs nine) and Thailand (25 vs 17).

By contrast, countries which are under-represented in the Impact Rankings compared to the THE WUR include the United States – 45 universities received an overall rank compared to 181 ranked in the THE WUR, China (13 vs 91), the United Kingdom (50 vs 101), Japan (73 vs 116), Germany (six vs 48), Italy (six vs 48) and France (18 vs 41).

There are likely several reasons that would prompt some universities not to participate in the Impact Rankings. As this is a ranking not designed to measure an institution’s excellence nor its reputation, one reason may be that is not worth risking reputational damage in the event that research intensive or well-resourced universities do not perform as well as other institutions which rank lower in the major ranking schemas.

How many SDGs to submit for?

Universities can submit information for as many SDGs as they wish to. To receive an overall score and rank, institutions are required to submit data for a minimum of four SDGs, with one being SDG 17 (Partnership for the goals) which is compulsory for all.

Even though there are 262 universities that submitted data for all 17 SDGs compared to 164 in 2019, the figures represent the same proportion – 21% – of all participating institutions which did so. One possible explanation for not seeing a greater increase in the proportion of institutions which submitted for the 17 SDGs is the fact that participation in this ranking represents a significant investment of time, staff resources and coordination on the part of institutions.

Some 50.4% of institutions submitted data for four to eight SDGs, while 19.9% of institutions submitted information for between nine and 16 SDGs. The remaining 8.5% of participating institutions provided data for three or fewer SDGs.

What we observe is that there is no major difference in the number of SDGs institutions submitted data for across different economies. This means there is an overwhelming interest for universities across world regions to embark on a sustainable development journey.

Submitting data for fewer SDGs does not warrant higher standing. Knowing what SDGs are most suitable for and allied to a particular institution’s mission together with a long-term commitment to sustainable development are the key to success.

Preferred SDGs

The 17 SDGs are not all viewed equally across countries and world regions. Universities also seem to prefer some SDGs to others, in part a reflection of institutional mission, priority or hierarchical need, but also of the likelihood of ranking higher in the THE Impact Rankings.

For universities, SDG 4 (Quality education) is paramount and is the one most institutions submit for aside from SDG 17 (Partnership for the goals), which is the only mandatory SDG for participation in the THE Impact Rankings.

The vast majority or 78% of the 1,239 participating universities submitted for SDG 4, followed by SDG 3 (Good health and well-being) at 70%, then SDG 5 (Gender equality) at 63%. The least preferred SDGs were ‘Life on land’ (32%) and ‘Life below water’ (31%).

Interestingly, 53% of participating universities submitted for SDG 16 (Peace, justice and strong institutions). Universities play a pivotal role at the core of this SDG for improved governance, transparency and accountability at all levels of society.

Also, only 46% of participating universities submitted for SDG 13 (Climate action).

University leaders are urged to take a more proactive role in emphasising these SDGs in their overall mission and societal impact.

Expanding mandatory SDGs

The Impact Rankings is a ranking which is not necessarily going to be stable over time. We have seen movement at the top. For the first two years, the University of Auckland in New Zealand ranked first and it now ranks equal ninth. The University of Manchester in the United Kingdom (ranked eighth last year) now ranks first globally.

Another example is the University of Bologna in Italy which ranked ninth in the 2019 edition, sixth in 2020 and now 20th.

We also see that Queen’s University, Canada, unranked for the previous two years, ranks fifth overall. We are likely to see more newcomers (particularly among research-intensive and well-resourced institutions) enter this ranking in coming years that are likely to rank at the top. We have always argued for stability in rankings and year-on-year volatility needs to be mitigated.

One way to bring stability to this ranking is by increasing the number of mandatory SDGs from one to three or four and to base the overall score on them. One scenario is to make SDG 4 (Quality of education), SDG 16 (Peace, justice and strong institutions) and SDG 17 (Partnership for the goals) mandatory since these are core to universities’ institutional governance and partnerships.

Another way is to group the SDGs by their intended outcomes, whether these are social or individual well-being (SDG 1, SDG 2, SDG 3, SDG 5 and SDG 10); economic or supporting infrastructure (SDG6, SDG 7, SDG 8, SDG 9, SDG 11 and SDG 12) or preserving the environment (SDG 13, SDG 14 and SDG 15). The remaining three (SDG 4, SDG 16 and SDG 17) are focused on governance and partnership.

On this basis, we could create a composite index to derive an overall score, if deemed necessary (although this should not be mandatory).

As the ultimate goal of the 2030 Sustainable Development Agenda is not to leave anyone behind, a ranking of universities based on the SDGs is surely best served if it is contextualised according to a regional or national basis based on a like-to-like comparison.

The focus needs to be on world regions or groups of countries which have enough similarities to engender a useful comparison. This shift in emphasis could help to reduce regional asymmetries and contribute to an uplift in the overall quality of institutions, particularly in middle- and low-income economies.

Angel Calderon is principal adviser, planning and research, at RMIT University in Australia. He is a rankings expert and a Latin American specialist.