Why ratings work (when they clearly don’t work)

Ratings and rankings are a key part of how schools use data. Indeed, compiling evaluative lists and ‘sorting’ students into hierarchies are some of the oldest forms of school data work. In this sense, we need to pay close attention to how digital technologies support the continuous production of ratings and rankings within our case study schools.

These forms of digital data will be varied. For instance, it is likely that we will find classroom teachers circulating regular end-of-course surveys on Google Forms and similar tools. These might well feature a succession of five-point Likert scale items asking students whether they found their classes to be ‘Engaging’, ’Very Engaging’ or perhaps ‘Not Very Engaging At All’.

At the same time, these students might be providing alternate ratings of the same classes on public websites such as ‘Rate My Teacher’. Elsewhere, the Australian government will be rating and ranking ‘whole school performance’ on its nationwide MySchool website, using annual data returns to present the school to parents and other ‘consumers’. In addition, combined ratings of all Australian schools will feature in international comparisons such as the OECD’s ‘PISA’ tables – leading to sweeping judgements such as the national education system suffering from a ‘tolerance for failure’.

Many educators and educationalists openly bemoan the prevalence of such measurements and metrics – not least because of the importance now attached to such data by policymakers and funders. Indeed, the process of rating and ranking is easy to critique on grounds of being “simplistic, obscurantist, inaccurate, and subjective” (Esposito and Stark 2019, p.8). Yet regardless of such pushback in education, such data practices and processes continue to grow in prominence. An important angle for our research to explore, therefore, is whythese seemingly problematic data processes and practices persist. What benefits do people get from these types of data? What might this tell us about the role that data now plays in contemporary education? What might this tell us about possible alternative ways of ‘doing data’ in schools?

In their recent TCS article, Esposito and Stark suggest that the everyday pervasiveness of ratings and rankings reflects the complex roles that data now plays in contemporary society. The crux of their argument is straight-forward – ratings and rankings can be easily ‘berated’ (especially by those being rated and ranked), yet they fulfil a much-needed role as orientation devices that can give us all a sense of what others are thinking, feeling and perceiving. In a world that is characterised by risk, uncertainly and contingency, it seems that ratings and rankings ‘work’ very well for us (i.e. as individuals needing to make sense of the world we live in). At the same time, we may also be well-aware that these ratings and rankings fail to ‘work’ satisfactorily in wider collective or societal terms, yet we continue to regularly refer to them nonetheless. Our relationship with data can be ambiguous, if not downright contradictory.

This argument raises a number of interesting ontological points with regards to data and schools. In particular, Esposito and Stark make the point that rating and ranking data is not meant to provide a mirror on reality, or be seen as an objective measure or ‘observation of an independent world’. Instead, these forms of data fulfil a collective function – working as a ‘social tool’ to orientate us toward what others are thinking. In this sense, any rating or ranking is a “second-order observation in which observations must take into account the observations of others. To this purpose they function well enough, not because they inform us about how things are but because they provide an orientation about what others observe” (p.3).

In this sense, conventional criticisms of rankings and ratings as ‘valid’ forms of data do not hold water. For example, as a rough-and-ready means of alerting us to the thoughts and opinions of others, “brutal simplification is necessary to bring information to a form in which it can offer guidance” (p.11). Similarly, if understood as second-order observations whose meanings are completely contingent (i.e. this is data that always has “the possibility to be otherwise”), then concerns over transparency or objectivity make little sense. Like the people who make them, any rating or ranking is an arbitrary and unreliable witness to what it purports to describe.

Thus, Esposito and Stark imply that any ‘problem’ we might have with ratings and rankings should lie in how these data are used, rather than with the nature of the data itself. To borrow one of the paper’s subheadings, it is worth shifting our mind-set (if possible) from the idea of ‘the ranked society’ to the more pluralistic notion of a ‘society of rankings’. In other words, we need to establish a completely different culture of how these ratings and rankings are understood and utilised.

For instance, imagine a school where ratings and rankings are commonly understood simply as a rough-and ready reference point and a quick way of gaining a general orientation on things – rather than being “conceived as an objective and indisputable disposition” (p.15). This would see schools treating data more in the spirit of a ‘straw poll’ rather than a cut-and-dried ‘performance rating’. Of course, the prevailing culture of most school systems will strongly favour the less nuanced use of any data that is generated. Schools are contexts where people are pushed to (mis)appropriate even the flimsiest proxies as hard-and-fast measurements. Nevertheless, we need to work hard to explore ways of challenging and changing these prevailing cultures of false objectivity.

This is a daunting challenge, yet not without precedent. Perhaps we might look toward other areas of schooling where more nuanced distinctions about the nature and purposes of different judgements are accepted (if not encouraged). For example, in terms of assessment of students’ work then schools have a clear and widely understood distinction between ‘formative’ and ‘summative’ assessments. Elsewhere, classrooms are still places where teachers rely on a ‘quick show of hands’ without taking the results any more seriously than they should. In a similar spirit, then, we need to explore ways to reframe rating and ranking data as tentative, indicative, subjective and generative.


[Notes on: Elena Esposito & David Stark (2019) What’s Observed in a Rating? Rankings as Orientation in the Face of Uncertainty.  Theory, Culture & Society forthcoming]