Manipulation by algorithms. Exploring the triangle of unfair commercial practice, data protection, and privacy law

• Published date: 01 January 2023
• Author: Philipp Hacker
• Date: 01 January 2023
• DOI: http://doi.org/10.1111/eulj.12389

VARIETY: ORIGINAL ARTICLE

Manipulation by algorithms. Exploring the triangle

of unfair commercial practice, data protection, and

privacy law

Philipp Hacker*

Abstract

The optimisation of sales practices in consumer markets through machine learning not only harbours

the potential to better match consumer preferences with products, but also risks to facilitate the

exploitation of consumer weaknesses discovered via data analysis. More specifically, recent techno-

logical advances have brought us to the edge of mind-reading technologies, which automatically

analyse mental states and adapt offers accordingly, in potentially manipulative ways. This article

shows that, in market contexts, the challenges of manipulation by algorithm necessitate an inte-

grated understanding of unfair commercial practice, data protection, and privacy law. It maps the

interactions between these contiguous yet distinct fields of law, and draws on economics and com-

puter science to develop a novel framework to deal with algorithmic influence. Furthermore, it criti-

cally discusses the Commission proposals for the Digital Services Act and the Artificial Intelligence

Act, and suggests to complement them with more broadly applicable measures to mitigate algorith-

mic manipulation.

1|INTRODUCTION

For centuries, human beings have observed one another to evaluate how their counterparties think or feel, and to

strategically use that knowledge. Recent breakthroughs in machine learning, however, have taken that analytical

potential to a new level. At a fast pace, mind-reading technologies are being developed that automatically analyse

*Prof. Dr. Philipp Hacker, LL.M. (Yale), Chair for Law and Ethics of the Digital Society, European University Viadrina, European New School of Digital

Studies, Frankfurt (Oder). This paper benefitted from valuable comments by Prof. Hans-W. Micklitz, Prof. Sandra Wachterand audiences at Humboldt

University of Berlin, the University of Graz and the European New School of Digital Studies, as well as very helpful suggestions by three anonymous

reviewers and the Editor in Chief. Funding information: AXA Research Fund, Grant/Award Number: AXA Postdoctoral Scholarship. Funding information:

AXA Research Fund, Grant/Award Number AXA Postdoctoral Scholarship.

Received: 11 September 2020Accepted: 28 April 2021

DOI: 10.1111/eulj.12389

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and

reproduction in any medium, provided the original work is properly cited.

© 2021 The Author. European Law Journal published by John Wiley & Sons Ltd.

142 Eur Law J. 2023;29:142–175.

wileyonlinelibrary.com/journal/eulj

mental states, such as emotions, based on images, voice or video recordings.

1

What was once considered the exclu-

sive precinct of science fiction novels is now increasingly deployed in online transactions, brick-and-mortar stores,

and even in public spaces.

2

If advertising companies are betting on the potential of such technologies to match con-

sumers with products they are in the mood for,

3

privacy activists are sounding the alarm on what they consider as

intrusive surveillance of intimate body functions and mental states.

4

At the very least, mind-reading technologies

urge us to reconsider more broadly to what extent psychological traits and weaknesses of counterparties may be

revealed and acted upon in digital environments.

Generally speaking, such mental states can be exploited in two ways. First, general offers made indiscrimi-

nately to the public can be designed to disproportionately affect consumers which exhibit certain biases or

emotions. For example, contracts comprising teaser rates followed by steep price increases could be presented

to the general public, with only (or at least mostly) heavily myopic consumers accepting the deal.

5

In these

cases, vulnerable consumers self-select into the contract because certain psychological traits are present. Such

offers therefore constitute a psychologically informed screening device for a trader who is uninformed about

which consumers belong to what group.

6

Second, offers can be targeted, that is, made exclusively to specific

subgroups of consumers which the trader believes will be particularly receptive because they share certain char-

acteristics.

7

Hence, consumers do not self-select, but are pre-selected by the trader.

8

That strategy therefore

presupposes knowledge, by the trader, of subgroup membership. Such information is increasingly provided by

data collection and algorithmic modelling,

9

and it is therefore this type of targeting that the paper predomi-

nantly deals with.

10

While such technologies arguably raise important challenges in a number of legal fields,

11

this paper partic-

ularly considers applications in market contexts, and asks to what extent the specific targeting of cognitive

traits or emotional states may lead to a manipulation of consumer decision-making which runs afoul of EU

market regulation, including the proposed Digital Services Act (DSA)

12

and the proposed Artificial Intelligence

Act (AIA).

13

The application of machine learning to marketing and contracting contexts risks to exacerbate

exploitative practices as algorithmic models may—in ways intended or not intended by their users—capture

and optimise on expected consumer weaknesses in cognitive or emotional domains. However, as this paper

shows, current EU law struggles to address these issues within its existing framework. If anything, the case of

mind-reading technologies underscores the need for an integrated market order for the digital economy, in

1

For an overview, see C. Burr and N. Cristianini, ‘Can Machines Read Our Minds?’(2019) 29 Minds and Machines, 461.

2

See A. McStay, Emotional AI (Sage, 2018), ch. 8; and below, Section 2.1.

3

A. McStay, ‘Emotional AI, Soft Biometrics and the Surveillance of Emotional Life: An Unusual Consensus on Privacy’(2020) 7 Big Data & Society 6–7,

https://doi.org/10.1177/2053951720904386.

4

See, e.g. M. Whittaker et al., AI Now Report (2018), ainowinstitute.org/AI_Now_2018_Report.pdf,4,8.

5

H. Shui and L. Ausubel, ‘Time Inconsistency in the Credit Card Market’(2004) 14th Annual Utah Winter Finance Conference,https://eml.berkeley.edu/

webfac/dellavigna/e218_f05/ausubel.pdf,3–4,9 –10 (all websites last visited on 11 March 2021); see also below, n. 167.

6

On screening generally J. Stiglitz, ‘The theory of “screening”, education, and the distribution of income’(1975) 65 American Economic Review, 283.

7

This is often called behavioural targeting, see, e.g. F. Zuiderveen Borgesius, Improving Privacy Protection in the Area of Behavioural Targeting (Wolters

Kluwer, 2015), 15, 47.

8

See, e.g. R. Calo, ‘Digital Market Manipulation’(2014) 82 George Washington Law Review, 995, 1003 et seq., and references in n. 15.

9

See below, Section 2.2.

10

Unless otherwise noted, ‘targeting’refers to this practice. General offers, in turn, are discussed further in Sections 2.1.2 and 3.2.2.1; see also n. 167.

11

See e.g. with respect to face recognition technology, European Commission, ‘White Paper on AI’, COM(2020) 65 final, 21–22; Council of Europe,

Guidelines on Facial Recognition, T-PD(2020)03rev4, 2021; E. Kindt, ‘Having Yes, Using No? About the New Legal Regime for Biometric Data’(2017) 34.3

Computer Law & Security Review, 523; see also European Data Protection Supervisor (EDPS) ‘Opinion 3/2018 on online manipulation and personal data’

(2018); S. Stolton, ‘Commission under Pressure in EU Court over “Lie Detector Tech”’,EURACTIV (8 Feb. 2021), https://www.euractiv.com/section/digital/

news/aommission-under-pressure-over-lie-detector-tech-in-eu-courts/.

12

European Commission, Proposal for a Regulation of the EuropeanParliament and of the Council on a Single Market for Digital Services (Digital Services

Act), COM(2020) 825 final.

13

European Commission, Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence

(Artificial Intelligence Act), COM(2021) 206 final.

HACKER 143

which unfair commercial practice, data protection, and privacy law complement and mutually support one

another.

14

Indeed, manipulation by digital technologies is a growing concern among legal scholars.

15

More recently, the

proposals of the DSA and the AIA have also taken up worries about manipulation in digital spaces,

16

and may offer a

window of opportunity for policy proposals in this respect. This paper adds to this discourse by undertaking a sys-

tematic and in-depth assessment from the perspective of EU market law. A particular focus rests on the Unfair Com-

mercial Practices Directive (UCPD), the key EU law instrument for safeguarding informed and rational market

decisions. Even the DSA and the AIA, if eventually enacted, will not detract from the centrality of the UCPD for digi-

tal manipulation since, in this context, they primarily complement the UCPD with specific provisions concerning plat-

form transparency (DSA) or AI systems impacting consumers outside of commercial practices (AIA). However, as will

be discussed, the DSA and the AIA do fill important gaps, and merit close scrutiny concerning any future framework

for the mitigation of algorithmic manipulation.

In this endeavour, the article proceeds in four steps. First, it lays conceptual and interdisciplinary foundations for

the treatment of algorithmic manipulation by asking how manipulative practices differ from acceptable economic

behaviour (Section 2). In this, it draws on insights from behavioural economics and computer science to further refine

the concept of manipulation with respect to consumer weaknesses. Second, it explores the limits of harnessing

machine learning for emotional and cognitive targeting under the UCPD (Section 3). Third, it turns to data protection

law for an analysis of algorithmic manipulation under the General Data Protection Regulation (Section 4). Through-

out, the paper stresses the interactions and interdependencies between the different legal instruments considered.

Finally, it discusses the recent anti-manipulation initiatives in the DSA and the AIA and makes three concrete pro-

posals to update EU law in view of the challenges of mind-reading technologies, and algorithmic manipulation more

generally (Section 5).

2|INTERDISCIPLINARY AND CONCEPTUAL FOUNDATIONS

When regulating markets and technologies, and analysing concepts like manipulation, legal scholarship cannot isolate

itself from insights of neighbouring disciplines investigating these very terms and phenomena. Conceptually, the

delimitation of acceptable persuasion from inacceptable manipulation is hotly debated in moral philosophy and juris-

prudence.

17

In these debates, one may distinguish two dimensions of manipulation: manipulation as unawareness of

influence,

18

and manipulation as the impossibility of rational choice.

19

Normatively, both dimensions are relevant

14

For a similarly integrated view of EU market law, see also N. Helberger, F. Zuiderveen Borgesius and A. Reyna, ‘The Perfect Match? A Closer Look at the

Relationship between EU Consumer Law and Data Protection Law’(2017) 54 Common Market Law Review, 1427, 1439–1443; F. Costa-Cabral and

O. Lynskey, ‘Family Ties: the Intersection between Data Protection and Competition in EU Law’(2017) 54 Common Market Law Review, 11; P. Hacker,

‘Teaching Fairness to Artificial Intelligence: Existing and Novel Strategies against Algorithmic Discrimination under EU Law’(2018) 55 Common Market Law

Review, 1143; D. Clifford, I. Graef and P. Valcke, ‘Pre-formulated Declarations of Data Subject Consent—Citizen-Consumer Empowerment and the

Alignment of Data, Consumer and Competition Law Protections’(2019) 20 German Law Journal, 679.

15

See, e.g. Calo, n. 8; K. Yeung, ‘Hypernudge: Big Data as a Mode of Regulation by Design’, (2017) 20 Info. Comm. & Society, 118, 119; A. Nadler and

L. McGuigan, ‘An impulse to exploit: The behavioral turn in data driven marketing’(2018) 35 Critical Stud. Media Comm., 151, 161; T. Zarsky, ‘Privacy and

manipulation in the digital age’(2019) 20 Theoretical Inquiries in Law, 157, 158; D. Susser, B. Roessler and H. Nissenbaum, ‘Online manipulation: Hidden

influences in a digital world’(2019) 4 Georgetown Law Technology Review, 1; S. Wachter, ‘Affinity profiling and discrimination by association in online

behavioural advertising’(2020) 35 Berkeley Technology Law Journal (forthcoming); L. Willis, ‘Deception by design’(2020) 34 Harvard Journal of Law and

Technology, 115; N. Helberger et al., EU Consumer Protection 2.0, BEUC Report, 2021; J. Laux et al., ‘Neutralizing online behavioural advertising’(2021)

58 Common Market Law Review (forthcoming), https://ssrn.com/abstract=3822962; see also, more generally, Zuiderveen Borgesius, n. 7; F. Pasquale, The

Black Box Society (Harvard University Press, 2015).

16

See, e.g. Recital 57 DSA; Recitals 15, 70 AIA.

17

See, e.g. C. Coons and M. Weber, ‘Introduction’, in C. Coons and M. Weber (eds.), Manipulation: Theory and Practice (Oxford University Press, 2014), 1.

18

Susser, Roessler and Nissenbaum, n. 15, 17.

19

A. Wood, ‘Coercion, Manipulation, Exploitation’, in C. Coons and M. Weber (eds.), Manipulation: Theory and Practice (Oxford University Press, 2014),

17, 35; R. Noggle, ‘Manipulative actions: A conceptual and moral analysis’(1996) 33 Am. Phil. Q., 43, 44; see also Th. Hill, Autonomy and Self-Respect

(Cambridge University Press, 1991), 33.

144 HACKER