Should more public trust in data-driven systems be the goal?

There is growing awareness that experiences of data, automation and AI are shaped by structural inequalities. It has been shown that socially marginalised populations suffer the negative effects of data-related practices more than others. Well known examples include Virginia Eubanks’ book Automating Inequality: how high-tech tools profile, police and punish the poor,¹ in which she highlights the negative impacts of data-driven systems on people living in poverty, who often also belong to racialised communities. Research by Seeta Peña Gangadharan, in collaboration with Eubanks and others, on the Our Data Bodies project², also makes visible how data and automation are experienced as discriminatory by poor, minoritised communities. High profile reports published in ProPublica of the racism embedded in algorithmic criminal justice systems in the US provide further evidence that the effects of data-driven systems are not experienced equally by all.

A largely separate debate is underway about the importance of ensuring that there is public trust in data-driven and automated systems in order for them to be effective. Recent research relating to the COVID-19 pandemic is contributing to this debate. How the UK public gets information about COVID-19,³ by Professor Rasmus Kleis Neilson and Dr Richard Fletcher and Communicating uncertainly in data without undermining trust⁴  by Dr Sander van der Linden and Professor David Spiegelhalter, highlight the important role that trust plays in people’s engagements with COVID-19-related information and guidance. And findings from online deliberation exercises led by the Ada Lovelace Institute and others⁵ suggest that trust in COVID-19 technologies and the systems in they are embedded into is essential to their success.

But what is generally missing from these debates around trust is how structural inequalities shape the extent to which people trust and what people deem to be trustworthy. Both historically and more recently, it has been found that the wealthy and well-educated have higher levels of trust than more disadvantaged groups. For example, a review of research into public attitudes to health data sharing,⁶ published by Understanding Patient Data in 2018, found that ethnic minority groups are less likely than ethnic majority groups to trust that their health data will remain secure.

Although it is not widely acknowledged that distrust in data and data-driven systems is shaped by inequalities, we should not be surprised by this, because the worldviews of the (usually privileged) creators of such systems are embedded within them.

This happens in a range of ways. Some commentators note that proxies often contain assumptions about correlation that are based on biased reasoning. In machine learning, the selection of training data can unconsciously produce bias. A famous example is Amazon’s sexist automated recruitment tool, which learned to discriminate against female applicants because it ‘learned’ what a strong CV looks like from the CVs of existing technical staff, most of whom were male. Thus, words like ‘women’ (such as ‘captain of women’s football team’) appeared anomalous and pushed CVs containing them down the ranking. The tool was scrapped in 2018.

Data-driven systems are produced by humans and humans have values. These values get written into said systems, consciously or unconsciously. Values are sometimes biased or discriminatory, and that means they reproduce inequalities. It follows that systems that do not acknowledge inequalities are unlikely to be deemed trustworthy by people whose lives are marked by inequalities. Bronwyn Carlson, Indigenous Studies scholar, argues that data-driven systems need to earn the trust of socially disadvantaged populations.⁷ Consequently, distrusting data-driven systems is sometimes appropriate: as philosopher Annette Baier puts it: ‘trust-busting can be a morally proper goal’.⁸ In other words, distrusting is sometimes the right thing to do.

US researcher Ruha Benjamin uses the term ‘informed refusal’⁹ to make sense of the distrust she witnessed in her research on Black people’s engagements in health projects. Informed refusal is a counter to informed consent, she writes, which falsely assumes that ‘the transmission of information’ will result in ‘the granting of permission’. In the case of racialised communities, this assumption does not always hold up, and so we need to unpack ‘the racial logics of trust’, argues Benjamin. To do this, Benjamin draws on the argument that ‘the problem of distrusting citizens should be recast or reformulated as an issue of social justice’, put forward by Johnson and Melnikov¹⁰ in their reflections on Ukrainian society.

Taking these provocations seriously, our focus should not be on how to garner trust or counter what the Royal Statistical Society describes as the ‘data trust deficit’.¹¹ Rather, we should concern ourselves with why marginalised and minoritised communities should be expected to trust. This means focusing on the trustworthiness, or otherwise, of systems – biomedical in Benjamin’s case, data-driven or automated in ours.

The question of whether systems are deserving of trust has been the focus of literature on the public understanding of science,¹² where it is argued that efforts to increase public understanding as a way of minimising distrust (or that ‘the transmission of information’ will have the desired effects) are flawed. Once again, it is argued that we should not be focusing our attention on getting people to trust more, but rather on the systems themselves and whether they are trustworthy. Better still, we should seek to better understand the roots of distrust, through a lens that conceives of distrust as a matter of social justice, something that is shaped by structural inequalities and that is an appropriate response to systems developed for exclusive publics.

Trust is often assumed to be a positive emotion. Sociologist of trust Piotr Sztompka argues¹³ that trust is an orientation toward the future, which enables us to act. Writing about trust as a strategy for dealing with data anxieties, Sarah Pink and colleagues concur,¹⁴ arguing that trust in data is a feeling that enables people to move on and take action. Yet assuming that trust is positive and therefore desirable can delegitimise some groups’ ‘morally proper’ distrust and further entrench the inequalities that many of us would wish to challenge. Seeing trust as a privilege enjoyed by majority groups might help us to resist the temptation to believe that more trust should be our goal.

With thanks to colleagues Ros Williams and Hannah Ditchfield for inspirational conversations about the issues discussed here.

Further Reading:

To read more of the research on which this piece is based:

Hartman, T, Kennedy, H, Steedman, R, and Jones, R (2020) ‘Public perceptions of good data management: findings from a UK-based survey’, Big Data and Society. Available at: https://journals.sagepub.com/doi/full/10.1177/2053951720935616

Kennedy, H, Oman, S, Taylor, M, Bates, J and Steedman, R (2020) ‘Public understanding and perceptions of data practices: a review of existing research’. Available at: https://livingwithdata.org/current-research/publications/

Steedman, R, Kennedy, H and Jones, R (2020) ‘Complex ecologies of trust in data practices and data-driven systems’, Information, Communication and Society. Available at: https://www.tandfonline.com/doi/full/10.1080/1369118X.2020.1748090