As objects that are intended to convey meaning, digital data can certainly be considered as a type of document. In this paper, Jones and McCoy set out a case for how documentation studies can provide a useful framework for analysing the datafication of students through educational data-systems and analytics.
The main pitch of the paper is considering how the documentation studies concepts of individuals being ‘made into’ data and how those data are ‘considered as’ individuals (Buckland 2013) might help frame vital questions concerning educational data practices. For example, they argues that the notion of individuals being ‘made into’ data raise a number of useful questions:
MATERIAL
- Are the data a document or exist as part of a corpus, and how do we know?
- What is the data source?
- In what form(s) do the data exist?
PROCESSES
- What visible and invisible documentation processes created the data?
- Were the documentation processes purposefully created or ad hoc, were they responsive or forward-thinking?
- How do uses of the data support other practices?
HUMAN AND NON-HUMAN ACTORS ISSUES
- Who directly and indirectly created the data?
- Who directly and indirectly uses the data?
- Who makes secondary uses of the data?
- In what ways do algorithms and artificial intelligence affect the data’s characteristics?
Similarly, the notion of how people are then ‘considered as’ these data can also be seen as raising a number of attendant questions. For example:
PURPOSES
- What evidence do these data provide, and can we verify their veracity?
- What end(s) do these data serve? .
- What role(s) do these data play in a sociotechnical system?
VALUES
- What social value do the data have? .
- What cultural value do the data have? .
- What political value do the data have?
ISSUES
- Is the data morally justifiable? .
- Can the data be ethically accessed and used? .
- How do the data create or perpetuate unjust systems? .
- How does the existence/use of the data create harms?
The paper then goes on to demonstrate the benefits of these approaches with regard to three critical concerns over the use of data in education. The first is the notion of data fallibility – in particular how the interests of institutional administrations shape the processes of making students into data – “effectively reducing them into fragmentary representations” (p.52). For Jones and McCoy, this raises questions over the specific sources of data being used, the choices of which data are to be included and not included in any models, in whose interests the analytics serve and what heuristics are used to justify the actions that the data analytics inform.
Secondly, the paper considers concerns over digital-data led ‘apophenia’ – i.e. the identification of patterns that do not actually exist. The critical concern here is the analysis of digital without the due involvement of individuals for who the phenomenon being represented is familiar. For example, a classroom teacher will be able to identify the contextual reasons and mitigating factors underpinning a data pattern that might be apparent in statistical terms but meaningless in social terms.
Here, then, Jones and McCoy illustrate how the documentation studies approach raises a number of questions regarding students being ‘made into’ data. For example, whether the analytics are being presented in tabular or visualisation form; whether the statistical processes were iteratively developed and vetted at each stage; what non-data science actors were involved in the construction of statistical models. Alternatively, in terms of how this data is then ‘considered as’, questions need to be asked of how intelligible the data outputs are for whom they are designed to be used; whether different actors share a common or diverse sense of how this data is ‘useful’.
Third, then, are critical concerns over the construction and use of ‘data doubles’. Questions raised here include how we know what data are actually included in the make-up of any profile or representation (i.e. its provenance), whether policies exist to specify what data should be included in student records, what can access these records and for what purposes.
In many ways, these are all questions that might reasonably already be included in any critical analysis of the datafication of schooling. Nevertheless, the documentation studies approach is neat way gathering together these concerns and focusing our attention on the essentially socially-constructed and institutionally-led nature of data ‘generation’ within a school. In particular, these questions push us toward a systematic consideration of the unintended consequences of digital data when applied in practice. In this sense, these are useful issues to be reminded of throughout our research.
Kyle Jones & Chase McCoy (2019) Reconsidering data in learning analytics: opportunities for critical research using a documentation studies framework, Learning, Media and Technology, 44(1):52-63