Teaching to the test

The OECD’s Assessment of Higher Education Learning Outcomes, or AHELO, project seeks to compare the quality of what students learn in different institutions and countries. AHELO is attempting to respond to a genuine problem in the measurement of higher education quality.

There are strong pressures for the measurement of how students benefit from higher education to ensure equitable higher education for all students regardless of which institution they study in.

The legitimacy of these demands needs to be recognised as students and societies invest considerable resources in higher education.

There are clear problems with existing national and international rankings of higher education, which tend to reinforce existing hierarchies based on institutional prestige rather than providing valid information about the quality of education that these institutions offer. AHELO will offer an alternative to rankings that will challenge rather than reinforce existing hierarchies, it has been claimed.

Skills assessment

In examining AHELO’s potential to provide a valid measure of students’ learning outcomes that could undermine institutional hierarchies, I will focus on its generic skills element because it is the one that is relevant to the vast majority of students.

The generic tests used by AHELO are adapted versions of the US Collegiate Learning Assessment, or CLA, in which students are asked to respond to case studies in order to assess skills such as ‘analytical reasoning and evaluation’, ‘problem solving’ and ‘writing effectiveness'. For example, in one task students are asked to provide responses to a mayor about a deformed catfish that has been found in a local lake.

There are a number of problems with the validity of this approach. First, despite the claims of its proponents, AHELO is not a direct measure of students’ learning. It is rather a measure of students’ ability to solve a particular set of problems unrelated to their degree programmes.

It is not even clear that AHELO tells us about students’ general ability to solve problems because it is based on the mistaken notion that students either have good or bad ‘reasoning’ or ‘problem-solving’ skills regardless of the particular task they are engaged in or the context in which they are undertaking this task.

In contrast, how students engage in a particular task is largely shaped by what is at stake for them in any particular performance and their interactions with other people and things within it.

This is because thinking always has an object and that object plays a crucial role in shaping thought processes, as does the cultural context in which thinking takes place.

Thus providing guidance to a mayor will be affected by the cultures in which students are operating and so is not a generic task at all: for example, preparing advice for the mayor of a large city in Russia is not the same task as preparing advice for the mayor of a small town in Colombia.

Second, there are different levels of familiarity with these kinds of generic tests across global higher education. One would expect students who have more experience of undertaking such tests and curricula that are more focused on developing generic skills, such as in the US, to perform better on such tests.

These two issues raise serious concerns about the potential of AHELO’s generic skills tests to provide valid comparisons of the quality of students learning outcomes' globally. This is because they are likely to tell us more about the cultural contexts in which students are operating and their level of experience of undertaking such generic skills tests than they are about what students have gained from studying in higher education.

Gaming the system

Third, even if AHELO succeeded it would not meet its aims. If AHELO were successful, then universities would clearly need to take their students’ performance in the generic skills tests very seriously. The predictable outcome is that they would invest time and resources in preparing their students to undertake these tests.

This would not tell us anything about the quality of students’ engagement with disciplinary and professional knowledge, but simply their ability to complete generic tests.

It would also not improve the quality of higher education, as it would simply make students better versed in completing generic comprehension exercises.

Finally, it would not reduce the impact of historical institutional hierarchies because wealthier institutions would have more resources to prepare their students for taking the assessments. Thus, if AHELO succeeded, the likelihood is that over time it would serve to reinforce existing hierarchies rather than challenge them.

All measures of the quality of students’ learning outcomes will simplify and give us a partial picture of what is going on. What is crucial is to be clear what elements are being used to create this picture and to question what they actually say about the experience of students.

In assessing the learning outcomes of higher education through an assessment of students’ generic skills, AHELO positions higher education as about the development of generic skills. This tells us nothing about how degree programmes give students access to particular bodies of knowledge that change their understanding of the world and themselves, which is the essence of higher education.

Unless we find ways of capturing the quality of higher education that capture this essence, then we risk undermining what makes higher education so attractive to students and societies.

Paul Ashwin is professor of higher education in the department of educational research at Lancaster University, UK. This is a shortened version of: Ashwin, P (2015) Missionary zeal: Some problems with the rhetoric, vision and approach of the AHELO project. European Journal of Higher Education, 5(4), 437-444.

COMMENT

Paul Ashwin’s article, “Teaching to the Test” (UWN 08 January 2016) conflates two critical issues: (a) The current concern about the ranking of global universities and (b) The competencies and skill sets of university graduates

Basically, those universities that have the fiscal, physical and academic resources clearly will rise, internationally, in any ranking. To raise an institution’s rank is an exceedingly costly endeavour. This is why, globally, many universities are forming consortia or are reaching articulations that allow research to be parsed/shared and to allow for student mobility to where specific needs can be best met. This can occur both in physical movements or, increasingly in virtual space.

As demands for post secondary education increases, globally, and knowledge gained is fungible and transferable across international boundaries, the idea of confining knowledge inside of a “ranked” university needs to be carefully evaluated, particularly for those countries who are resource constrained. The key is getting the best education for citizens and to be able to create the environmental conditions that keeps their own graduates and others gainfully employed and contributing to the quality of life of the country.

The idea of measuring the competencies of post secondary graduates by means that might separate the student’s capabilities separate from any ranking of institutions is specious. Ranked institutions have greater resources and thus provide opportunities in content aggregation as well as the soft skills and tacit knowledge not accessible to students at lower resourced institutions. This is a situation clearly understood by those who seek to employ graduates. In fact, decisions on where to recruit are often based on such evaluations of the institutions, their admissions policies, academic focus and history of students who have graduated.

The fact that there are cultural differences between countries, political entities and even the private sector, separate from levels of content capabilities has been problematic as is seen in all the international development efforts as well as with businesses regardless of whether or not the individual graduates from a ranked institution. The factors for individual success depend on many issues that can not be anticipated. The move, globally, to competency measures across all skills will help. But, again, students exposed to the riches set of resources whether in a single institution or the growing global collaborative experiences offers the best opportunity.

In a world where knowledge and experiences are fungible and transferable across geo/political boundaries, education is too important and expensive not to accelerate a global, collaborative approach.

Tom Abeles