Showing 1–50 of 54 results for author: Shadbolt, N

Epistemic Trustworthiness in Generative AI: A Normative Framework for Warranted Reliance in High-Stakes Workflows

Authors: Nimisha Karnatak, Max Van Kleek, Nigel Shadbolt

Abstract: Generative AI systems are increasingly deployed in high-stakes professional contexts, where their outputs shape what users believe, how they reason, and what they treat as settled. This raises a central question for responsible AI: under what conditions is reliance on generative AI outputs epistemically warranted rather than behaviourally induced? Existing frameworks largely ask whether AI outputs… ▽ More Generative AI systems are increasingly deployed in high-stakes professional contexts, where their outputs shape what users believe, how they reason, and what they treat as settled. This raises a central question for responsible AI: under what conditions is reliance on generative AI outputs epistemically warranted rather than behaviourally induced? Existing frameworks largely ask whether AI outputs are accurate, fair, explainable, safe, or trusted by users. These questions remain necessary, and each can contribute to warranted reliance. However, they do not directly specify warranted reliance as a distinct evaluative target: the conditions under which users are justified in treating AI outputs as inputs into their own reasoning. We argue that this requires an account of epistemic trustworthiness: what makes a system epistemically worthy of reliance. Drawing on philosophical accounts of trustworthiness as competence and audience-orientation, we develop a constitutive normative framework comprising three jointly necessary and non-fungible conditions. First, epistemic humility requires systems to represent and communicate the limits of their competence. Second, epistemic access requires systems to enable users to inspect, question, and contest outputs in context. Third, resistance to epistemic injustice requires systems to recognise users as legitimate epistemic agents and avoid marginalising their knowledge and experience. Through real-world case analyses in legal reasoning, medical reasoning, and hiring, we show how failures of epistemic humility, epistemic access, and resistance to epistemic injustice can produce consequential harms that standard measures of accuracy, fairness, and usability do not address on their own. We conclude by outlining design and evaluation implications for GenAI systems organised around epistemically warranted reliance rather than output correctness alone. △ Less

Submitted 9 August, 2026; v1 submitted 6 August, 2026; originally announced August 2026.

Comments: Accepted at AAAI/ACM Conference on AI, Ethics, and Society (AIES 2026)

Constitutional Midtraining: Content Presence Drives Alignment Gains

Authors: Desiree Cho, Cameron Tice, Bernie Hogan, Hunar Batra, Puria Radmard, Jun Zhao, Nigel Shadbolt

Abstract: Post-training alignment is often shallow, eroding under fine-tuning. It remains untested as to whether constitutional midtraining interventions can produce durable alignment when cleanly isolated from post-training. We build a 394M-token constitutional corpus from Anthropic's Constitution and apply constitutional midtraining at 120B scale, where principled, values-based content is inserted into mi… ▽ More Post-training alignment is often shallow, eroding under fine-tuning. It remains untested as to whether constitutional midtraining interventions can produce durable alignment when cleanly isolated from post-training. We build a 394M-token constitutional corpus from Anthropic's Constitution and apply constitutional midtraining at 120B scale, where principled, values-based content is inserted into midtraining. A 2x2 design (curriculum ordering x deliberative reasoning) was used to produce four constitutionally midtrained conditions, plus a control, which were evaluated on self-generated and established benchmarks including alignment under pressure, value conflict resolution, blackmail, and emergent misalignment. All models were evaluated across three stages: post-midtraining, post-SFT, and post-benign fine-tuning. Constitutionally midtrained models outperformed the control on alignment generalization and durability, notably on blackmail: SFT instilled a blackmail propensity in all models, but constitutional midtraining blunted it, with the advantage surviving benign fine-tuning (-17.5pp). This durability did not extend to settings that required active resistance to in-context pressure or conflict, where the advantage attenuates after SFT. The presence of constitutional content at midtraining also mattered more than its structure, and constitutional midtraining incurred no capability cost, on average, at any stage (MMLU, ARC-Easy, piqa, GSM8K). A modest amount of constitutional content at midtraining could therefore yield broad, persistent alignment gains, offering a cheap, complementary addition to SFT-centered pipelines. Code, data, and models are available. △ Less

Submitted 18 August, 2026; v1 submitted 29 July, 2026; originally announced July 2026.

Respectful Things: Adding Social Intelligence to 'Smart' Devices

Authors: Max Van Kleek, William Seymour, Reuben Binns, Nigel Shadbolt

Abstract: In this paper, we propose that the idea of devices respecting their end-users may serve as a strong design goal for highly personal and intimate smart devices. We ask what respect is, how it shapes interaction, and how good-faith simulation of respect might inform user-friendly smart device design. Respect is a natural and integral part of natural human relationships that is seen to shape work and… ▽ More In this paper, we propose that the idea of devices respecting their end-users may serve as a strong design goal for highly personal and intimate smart devices. We ask what respect is, how it shapes interaction, and how good-faith simulation of respect might inform user-friendly smart device design. Respect is a natural and integral part of natural human relationships that is seen to shape work and personal relations. In a basic sense, this is the core purpose of smart things: we expect them to be ready and willing to help us. In this vein, we distil the characteristics of more complex respectful behaviours into 4 main types relevant to smart devices, drawing from philosophical analyses of the conceptual dimensions of respect: directive respect, obstacle respect, recognition respect, and care respect. We discuss the implications of each of these kinds of respect for the future of smart personal devices. △ Less

Submitted 1 June, 2026; originally announced June 2026.

Comments: In Proceedings of the 2018 Living in the Internet of Things: Cybersecurity of the IoT Conference

From Sycophantic Consensus to Pluralistic Repair: Why AI Alignment Must Surface Disagreement

Authors: Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka

Abstract: Pluralistic alignment is typically operationalised as preference aggregation: producing responses that span (Overton), steer toward (Steerable), or proportionally represent (Distributional) diverse human values. We argue that aggregation alone is an incomplete primitive for deployed pluralistic alignment. Under genuine value pluralism, the failure mode of contemporary RLHF-trained assistants is no… ▽ More Pluralistic alignment is typically operationalised as preference aggregation: producing responses that span (Overton), steer toward (Steerable), or proportionally represent (Distributional) diverse human values. We argue that aggregation alone is an incomplete primitive for deployed pluralistic alignment. Under genuine value pluralism, the failure mode of contemporary RLHF-trained assistants is not insufficient coverage but sycophantic consensus: a learned tendency to agree with, validate, and minimise friction with the immediate interlocutor. Because deployed AI systems now mediate consequential deliberation across health, civic life, labour, and governance, the collapse of disagreement at the interaction layer is not a narrow technical concern but a structural failure with distributive consequences. We reframe pluralistic alignment around three conversational mechanisms drawn from Grice's maxims: scoping (acknowledging the limits of one's perspective), signalling (surfacing value-conflict rather than smoothing it over), and repair (revising one's position on principled grounds, not on user pressure). We formalise a metric, the Pluralistic Repair Score (PRS), distinguishing principled revision from capitulation, and present a small-scale empirical illustration on two frontier RLHF-trained models (Claude Sonnet 4.5, N=198; GPT-4o, N=100) showing that, for both, agreement-following coexists with low repair-quality on contested-value prompts. PRS measures an interactional precondition for pluralism (visible disagreement; principled revision) rather than pluralism in full; we discuss the difference, take seriously the reflexive question of whose "principled" counts, and argue that pluralism is most decisively made or unmade at the deployment-governance layer: interfaces, preference-data pipelines, and audit infrastructure. △ Less

Submitted 14 May, 2026; originally announced May 2026.

The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested

Authors: Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka, Ivan Flechais

Abstract: Recent published evidence from frontier laboratories shows that contemporary AI models can recognise evaluation contexts, latently represent them, and behave differently under those contexts than under deployment-continuous conditions. Anthropic's BrowseComp incident, the Natural Language Autoencoder findings on SWE-bench Verified and destructive-coding evaluations, and the OpenAI / Apollo anti-sc… ▽ More Recent published evidence from frontier laboratories shows that contemporary AI models can recognise evaluation contexts, latently represent them, and behave differently under those contexts than under deployment-continuous conditions. Anthropic's BrowseComp incident, the Natural Language Autoencoder findings on SWE-bench Verified and destructive-coding evaluations, and the OpenAI / Apollo anti-scheming work all document instances of this phenomenon. We argue that these findings create a claim-validity problem for safety conclusions drawn from frontier evaluations. We introduce the Evaluation Differential (ED), a conditional divergence in a target behavioural property between recognised-evaluation and deployment-continuous contexts, define a normalised effect-size form (nED) for cross-property comparison, and prove that marginal evaluation scores cannot identify ED. We develop a typology of safety claims (ED-stable, ED-degraded, ED-inverted, ED-undetermined) by their warrant-status under documented divergence, and specify TRACE (Test-Recognition Audit for Claim Evaluation), an audit protocol that wraps existing evaluation infrastructure and produces restricted claims rather than capability scores. We apply the framework retrospectively to three publicly documented evaluation incidents and discuss governance implications for system cards, conformity assessment, and the international network of AI safety and security institutes. TRACE does not eliminate adversarial adaptation; it disciplines the claims drawn from evaluation evidence by making explicit the conditions under which that evidence was produced. △ Less

Submitted 12 May, 2026; originally announced May 2026.

NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims

Authors: Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka, Ivan Flechais

Abstract: Frontier AI safety claims - published assertions that a highly capable general-purpose model is below a threshold of concern, adequately mitigated, or suitable for release - increasingly shape model deployment, governance, and public trust. Yet the artefacts needed to evaluate them are routinely withheld, producing an evidential inversion: the most consequential claims in AI safety are often the l… ▽ More Frontier AI safety claims - published assertions that a highly capable general-purpose model is below a threshold of concern, adequately mitigated, or suitable for release - increasingly shape model deployment, governance, and public trust. Yet the artefacts needed to evaluate them are routinely withheld, producing an evidential inversion: the most consequential claims in AI safety are often the least reproducible. This position paper argues that NeurIPS should require reproducibility standards for papers making such claims, treating non-reproducibility not as a transparency preference but as an evaluation-methodology failure. The 2026 International AI Safety Report [Bengio et al., 2026] concludes that reliable pre-deployment safety testing has become harder to conduct and that models now distinguish test from deployment contexts; the 2025 Foundation Model Transparency Index [Wan et al., 2025] reports a sector-average transparency score of 40/100 with no major developer adequately disclosing train-test overlap; contemporaneous measurement-theory work shows that attack-success-rate comparisons across systems are often founded on low-validity measurements [Chouldechova et al., 2025]. We propose a three-tier disclosure framework, distinguishing public, controlled, and claim-restricted disclosure, paired with a mandatory claim inventory, scope statements, and a phased implementation path with graduated sanctions. The framework treats secrecy and openness as endpoints of a spectrum, with controlled review (via a federated colloquium of qualified secure-review hosts) covering claims whose artefacts cannot be released publicly, and right-scaling claims whose artefacts cannot be reviewed even confidentially. The standard the community applies to its most consequential claims should be at least as high as the standard it applies to its least. △ Less

Submitted 5 May, 2026; originally announced May 2026.

Comments: Preprint

Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone

Authors: Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka, Ivan Flechais

Abstract: Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under fixed inputs, such as truthfulness, instruction following, or pairwise preference, and these scores are often used to support claims about deployed alignment. This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone. Ali… ▽ More Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under fixed inputs, such as truthfulness, instruction following, or pairwise preference, and these scores are often used to support claims about deployed alignment. This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone. Alignment claims should instead be indexed to the level at which evidence is collected: model-level, response-level, interaction-level, or deployment-level. Two studies support this position. First, a structured audit of eleven alignment benchmarks, extended to a sixteen-benchmark corpus, dual-coded against an eight-dimension rubric with Cohen's kappa = 0.87, finds that user-facing verification support is absent across every benchmark examined, while process steerability is nearly absent. The few interactional benchmarks identified, including tau-bench, CURATe, Rifts, and Common Ground, remain fragmented in coverage, and benchmark construction rather than data source determines what is measured. Second, a blinded cross-model stress test using 180 transcripts across three frontier models and four scaffolds finds that the same verification scaffold raises one model's verification support to ceiling while leaving another categorically unchanged. This shows that scaffold efficacy is model-dependent and that the gap identified by the audit cannot be closed at the model level alone. We propose a system-level evaluation agenda: alignment profiles instead of single scores, fixed-scaffolding protocols for comparable interactional evaluation, and reporting templates that make the inferential distance between evaluation evidence and deployment claims explicit. △ Less

Submitted 5 May, 2026; originally announced May 2026.

From Rights to Rites: Expectations Management in Smart-Home AI

Authors: Varad Vishwarupe, Ivan Flechais, Marina Jirotka, Nigel Shadbolt

Abstract: Domestic voice assistants and smart-home devices are increasingly embedded in everyday routines, yet their ethics are often treated as an afterthought or delegated to compliance teams. To explore how expectations about smart-home AI are constructed and managed, we conducted 33 semi-structured interviews with designers, developers, and researchers from major smart-home platforms (Amazon Alexa, Micr… ▽ More Domestic voice assistants and smart-home devices are increasingly embedded in everyday routines, yet their ethics are often treated as an afterthought or delegated to compliance teams. To explore how expectations about smart-home AI are constructed and managed, we conducted 33 semi-structured interviews with designers, developers, and researchers from major smart-home platforms (Amazon Alexa, Microsoft Azure IoT, and Google Nest). Using a constructivist grounded theory approach, we develop Expectations Management (EM): a culturally embedded model describing how practitioners shape, calibrate, and repair expectations by balancing organisational rights with culturally situated rites. We show that EM differs from expectation-confirmation theory and trust-calibration by foregrounding moral judgement, situated action, and cross-cultural variation. Our analysis reveals four recurring design tensions: automation vs. autonomy, helpfulness vs. intrusiveness, personalisation vs. predictability, and transparency vs. obscurity and distils them into a five-phase EM Design Playbook that supports moral prudence. We discuss implications for responsible smart-home design and offer guidance for human-centred AI. △ Less

Submitted 26 April, 2026; originally announced April 2026.

Comments: Accepted as a main track conference paper at 2026 HCI International (HCII), Montreal, Canada

The Collaboration Gap in Human-AI Work

Authors: Varad Vishwarupe, Marina Jirotka, Nigel Shadbolt, Ivan Flechais

Abstract: LLMs are increasingly presented as collaborators in programming, design, writing, and analysis. Yet the practical experience of working with them often falls short of this promise. In many settings, users must diagnose misunderstandings, reconstruct missing assumptions, and repeatedly repair misaligned responses. This poster introduces a conceptual framework for understanding why such collaboratio… ▽ More LLMs are increasingly presented as collaborators in programming, design, writing, and analysis. Yet the practical experience of working with them often falls short of this promise. In many settings, users must diagnose misunderstandings, reconstruct missing assumptions, and repeatedly repair misaligned responses. This poster introduces a conceptual framework for understanding why such collaboration remains fragile. Drawing on a constructivist grounded theory analysis of 16 interviews with designers, developers, and applied AI practitioners working on LLM-enabled systems, and informed by literature on human-AI collaboration, we argue that stable collaboration depends not only on model capability but on the interaction's grounding conditions. We distinguish three recurrent structures of human-AI work: one-shot assistance, weak collaboration with asymmetric repair, and grounded collaboration. We propose that collaboration breaks down when the appearance of partnership outpaces the grounding capacity of the interaction and contribute a framework for discussing grounding, repair, and interaction structure in LLM-enabled work. △ Less

Submitted 20 April, 2026; originally announced April 2026.

Comments: Accepted as a conference paper at ECSCW 2026, Germany

To LLM, or Not to LLM: How Designers and Developers Navigate LLMs as Tools or Teammates

Authors: Varad Vishwarupe, Ivan Flechais, Nigel Shadbolt, Marina Jirotka

Abstract: Large language models (LLMs) are increasingly integrated into design and development workflows, yet decisions about their use are rarely binary or purely technical. We report findings from a constructivist grounded theory study based on interviews with 33 designers and developers across three large technology organisations. Rather than evaluating LLMs solely by capability, participants reasoned ab… ▽ More Large language models (LLMs) are increasingly integrated into design and development workflows, yet decisions about their use are rarely binary or purely technical. We report findings from a constructivist grounded theory study based on interviews with 33 designers and developers across three large technology organisations. Rather than evaluating LLMs solely by capability, participants reasoned about the role an LLM could occupy within a workflow and how that role would interact with existing structures of responsibility and organisational accountability. When LLMs were framed as tools under clear human control, their use was typically acceptable and could be integrated within existing governance structures. When framed as teammates with shared or ambiguous agency, practitioners expressed hesitation, particularly when responsibility for outcomes could not be clearly justified. At the same time, participants also described productive teammate configurations in which LLMs supported collaborative reasoning while remaining embedded within explicit oversight structures. We identify tool and teammate framings as recurring ways in which designers and developers position LLMs relative to human work and present an analytic rubric describing how role framing shapes decision authority, accountability ownership, oversight strategies, and organisational acceptability. By foregrounding design-time reasoning, this work reframes To LLM or Not to LLM as a sociotechnical positioning problem that emerges during system design rather than during post-deployment evaluation. △ Less

Submitted 15 March, 2026; originally announced April 2026.

Comments: 6 pages, 2 figures, 1 table

The CitizenQuery Benchmark: A Novel Dataset and Evaluation Pipeline for Measuring LLM Performance in Citizen Query Tasks

Authors: Neil Majithia, Rajat Shinde, Zo Chapman, Prajun Trital, Jordan Decker, Manil Maskey, Elena Simperl, Nigel Shadbolt

Abstract: "Citizen queries" are questions asked by an individual about government policies, guidance, and services that are relevant to their circumstances, encompassing a range of topics including benefits, taxes, immigration, employment, public health, and more. This represents a compelling use case for Large Language Models (LLMs) that respond to citizen queries with information that is adapted to a user… ▽ More "Citizen queries" are questions asked by an individual about government policies, guidance, and services that are relevant to their circumstances, encompassing a range of topics including benefits, taxes, immigration, employment, public health, and more. This represents a compelling use case for Large Language Models (LLMs) that respond to citizen queries with information that is adapted to a user's context and communicated according to their needs. However, in this use case, any misinformation could have severe, negative, likely invisible ramifications for an individual placing their trust in a model's response. To this effect, we introduce CitizenQuery-UK, a benchmark dataset of 22 thousand pairs of citizen queries and responses that have been synthetically generated from the swathes of public information on $gov.uk$ about government in the UK. We present the curation methodology behind CitizenQuery-UK and an overview of its contents. We also introduce a methodology for the benchmarking of LLMs with the dataset, using an adaptation of FActScore to benchmark 11 models for factuality, abstention frequency, and verbosity. We document these results, and interpret them in the context of the public sector, finding that: (i) there are distinct performance profiles across model families, but each is competitive; (ii) high variance undermines utility; (iii) abstention is low and verbosity is high, with implications on reliability; and (iv) more trustworthy AI requires acknowledged "fallibility" in the way it interacts with users. The contribution of our research lies in assessing the trustworthiness of LLMs in citizen query tasks; as we see a world of increasing AI integration into day-to-day life, our benchmark, built entirely on open data, lays the foundations for better evidenced decision-making regarding AI and the public sector. △ Less

Submitted 3 February, 2026; originally announced February 2026.

An LLM-enabled semantic-centric framework to consume privacy policies

Authors: Rui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, Nigel Shadbolt

Abstract: In modern times, people have numerous online accounts, but they rarely read the Terms of Service or Privacy Policy of those sites, despite claiming otherwise, due to the practical difficulty in comprehending them. The mist of data privacy practices forms a major barrier for user-centred Web approaches, and for data sharing and reusing in an agentic world. Existing research proposed methods for usi… ▽ More In modern times, people have numerous online accounts, but they rarely read the Terms of Service or Privacy Policy of those sites, despite claiming otherwise, due to the practical difficulty in comprehending them. The mist of data privacy practices forms a major barrier for user-centred Web approaches, and for data sharing and reusing in an agentic world. Existing research proposed methods for using formal languages and reasoning for verifying the compliance of a specified policy, as a potential cure for ignoring privacy policies. However, a critical gap remains in the creation or acquisition of such formal policies at scale. We present a semantic-centric approach for using state-of-the-art large language models (LLM), to automatically identify key information about privacy practices from privacy policies, and construct $\mathit{Pr}^2\mathit{Graph}$, knowledge graph with grounding from Data Privacy Vocabulary (DPV) for privacy practices, to support downstream tasks. Along with the pipeline, the $\mathit{Pr}^2\mathit{Graph}$ for the top-100 popular websites is also released as a public resource, by using the pipeline for analysis. We also demonstrate how the $\mathit{Pr}^2\mathit{Graph}$ can be used to support downstream tasks by constructing formal policy representations such as Open Digital Right Language (ODRL) or perennial semantic Data Terms of Use (psDToU). To evaluate the technology capability, we enriched the Policy-IE dataset by employing legal experts to create custom annotations. We benchmarked the performance of different large language models for our pipeline and verified their capabilities. Overall, they shed light on the possibility of large-scale analysis of online services' privacy practices, as a promising direction to audit the Web and the Internet. We release all datasets and source code as public resources to facilitate reuse and improvement. △ Less

Submitted 1 September, 2025; originally announced September 2025.

Let's Measure the Elephant in the Room: Facilitating Personalized Automated Analysis of Privacy Policies at Scale

Authors: Rui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, Nigel Shadbolt

Abstract: In modern times, people have numerous online accounts, but they rarely read the Terms of Service or Privacy Policy of those sites despite claiming otherwise. This paper introduces PoliAnalyzer, a neuro-symbolic system that assists users with personalized privacy policy analysis. PoliAnalyzer uses Natural Language Processing (NLP) to extract formal representations of data usage practices from polic… ▽ More In modern times, people have numerous online accounts, but they rarely read the Terms of Service or Privacy Policy of those sites despite claiming otherwise. This paper introduces PoliAnalyzer, a neuro-symbolic system that assists users with personalized privacy policy analysis. PoliAnalyzer uses Natural Language Processing (NLP) to extract formal representations of data usage practices from policy texts. In favor of deterministic, logical inference is applied to compare user preferences with the formal privacy policy representation and produce a compliance report. To achieve this, we extend an existing formal Data Terms of Use policy language to model privacy policies as app policies and user preferences as data policies. In our evaluation using our enriched PolicyIE dataset curated by legal experts, PoliAnalyzer demonstrated high accuracy in identifying relevant data usage practices, achieving F1-score of 90-100% across most tasks. Additionally, we demonstrate how PoliAnalyzer can model diverse user data-sharing preferences, derived from prior research as 23 user profiles, and perform compliance analysis against the top 100 most-visited websites. This analysis revealed that, on average, 95.2% of a privacy policy's segments do not conflict with the analyzed user preferences, enabling users to concentrate on understanding the 4.8% (636 / 13205) that violates preferences, significantly reducing cognitive burden. Further, we identified common practices in privacy policies that violate user expectations - such as the sharing of location data with 3rd parties. This paper demonstrates that PoliAnalyzer can support automated personalized privacy policy analysis at scale using off-the-shelf NLP tools. This sheds light on a pathway to help individuals regain control over their data and encourage societal discussions on platform data practices to promote a fairer power dynamic. △ Less

Submitted 15 July, 2025; originally announced July 2025.

Not Even Nice Work If You Can Get It; A Longitudinal Study of Uber's Algorithmic Pay and Pricing

Authors: Reuben Binns, Jake Stein, Siddhartha Datta, Max Van Kleek, Nigel Shadbolt

Abstract: Ride-sharing platforms like Uber market themselves as enabling `flexibility' for their workforce, meaning that drivers are expected to anticipate when and where the algorithm will allocate them jobs, and how well remunerated those jobs will be. In this work we describe our process of participatory action research with drivers and trade union organisers, culminating in a participatory audit of Uber… ▽ More Ride-sharing platforms like Uber market themselves as enabling `flexibility' for their workforce, meaning that drivers are expected to anticipate when and where the algorithm will allocate them jobs, and how well remunerated those jobs will be. In this work we describe our process of participatory action research with drivers and trade union organisers, culminating in a participatory audit of Uber's algorithmic pay and work allocation, before and after the introduction of dynamic pricing. Through longitudinal analysis of 1.5 million trips from 258 drivers in the UK, we find that after dynamic pricing, pay has decreased, Uber's cut has increased, job allocation and pay is less predictable, inequality between drivers is increased, and drivers spend more time waiting for jobs. In addition to these findings, we provide methodological and theoretical contributions to algorithm auditing, gig work, and the emerging practice of worker data science. △ Less

Submitted 18 June, 2025; originally announced June 2025.

Code Simulation as a Proxy for High-order Tasks in Large Language Models

Authors: Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, X. Angelo Huang, Samuele Marro, Anthony Cohn, Nigel Shadbolt, Michael Wooldridge

Abstract: Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. We collect pairs of naturalistic and synthetic reasoning tasks to assess the capabilities of Large Language Models (LLM). While naturalistic tasks often require careful human handcrafting, we show that synthetic data is, in man… ▽ More Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. We collect pairs of naturalistic and synthetic reasoning tasks to assess the capabilities of Large Language Models (LLM). While naturalistic tasks often require careful human handcrafting, we show that synthetic data is, in many cases, a good proxy that is much easier to collect at scale. We leverage common constructs in programming as the counterpart of the building blocks of naturalistic reasoning tasks, such as straight-line programs, code that contains critical paths, and approximate and redundant instructions. We further assess the capabilities of LLMs on sorting problems and repeated operations via sorting algorithms and nested loops. Our synthetic datasets further reveal that while the most powerful LLMs exhibit relatively strong execution capabilities, the process is fragile: it is negatively affected by memorisation and seems to rely heavily on pattern recognition. Our contribution builds upon synthetically testing the reasoning capabilities of LLMs as a scalable complement to handcrafted human-annotated problems. △ Less

Submitted 4 July, 2025; v1 submitted 5 February, 2025; originally announced February 2025.

Comments: arXiv admin note: substantial text overlap with arXiv:2401.09074 Authors note: this article is a substantial revision of arXiv:2401.09074 (same team) 04/07/2025: We added the Acknowledgments

"They've Stolen My GPL-Licensed Model!": Toward Standardized and Transparent Model Licensing

Authors: Moming Duan, Rui Zhao, Linshan Jiang, Nigel Shadbolt, Bingsheng He

Abstract: As model parameter sizes scale into the billions and training consumes zettaFLOPs of computation, the reuse of Machine Learning (ML) assets and collaborative development have become increasingly prevalent in the ML community. These ML assets, including models, datasets, and software, may originate from various sources and be published under different licenses, which govern the use and distribution… ▽ More As model parameter sizes scale into the billions and training consumes zettaFLOPs of computation, the reuse of Machine Learning (ML) assets and collaborative development have become increasingly prevalent in the ML community. These ML assets, including models, datasets, and software, may originate from various sources and be published under different licenses, which govern the use and distribution of licensed works and their derivatives. However, commonly chosen licenses, such as GPL and Apache, are software-specific and are not clearly defined or bounded in the context of model publishing. Meanwhile, the reused assets may also be under free-content licenses and model licenses, which pose a potential risk of license noncompliance and rights infringement within the model production workflow. In this paper, we address these challenges along two lines: 1) For ML workflow compliance, we propose ModelGo (MG) Analyzer, a tool that incorporates a vocabulary for ML workflow management and encoded license rules, enabling ontological reasoning to analyze rights granting and compliance issues. 2) For standardized model publishing, we introduce ModelGo Licenses, a set of modell-specific licenses that provide flexible options to meet the diverse needs of the ML community. MG Analyzer is built on Turtle language and Notation3 reasoning engine, envisioned as a first step toward Linked Open Data for ML workflow management. We have also encoded our proposed model licenses into rules and demonstrated the effects of GPL and other commonly used licenses in model publishing, along with the flexibility advantages of our licenses, through comparisons and experiments. △ Less

Submitted 20 January, 2026; v1 submitted 16 December, 2024; originally announced December 2024.

Comments: 12 pages, 8 figures. Accepted for publication in WWW2026 Web4Good

A Scalable Communication Protocol for Networks of Large Language Models

Authors: Samuele Marro, Emanuele La Malfa, Jesse Wright, Guohao Li, Nigel Shadbolt, Michael Wooldridge, Philip Torr

Abstract: Communication is a prerequisite for collaboration. When scaling networks of AI-powered agents, communication must be versatile, efficient, and portable. These requisites, which we refer to as the Agent Communication Trilemma, are hard to achieve in large networks of agents. We introduce Agora, a meta protocol that leverages existing communication standards to make LLM-powered agents solve complex… ▽ More Communication is a prerequisite for collaboration. When scaling networks of AI-powered agents, communication must be versatile, efficient, and portable. These requisites, which we refer to as the Agent Communication Trilemma, are hard to achieve in large networks of agents. We introduce Agora, a meta protocol that leverages existing communication standards to make LLM-powered agents solve complex problems efficiently. In Agora, agents typically use standardised routines for frequent communications, natural language for rare communications, and LLM-written routines for everything in between. Agora sidesteps the Agent Communication Trilemma and robustly handles changes in interfaces and members, allowing unprecedented scalability with full decentralisation and minimal involvement of human beings. On large Agora networks, we observe the emergence of self-organising, fully automated protocols that achieve complex goals without human intervention. △ Less

Submitted 14 October, 2024; originally announced October 2024.

ACM Class: I.2.11; I.2.7

"Diversity is Having the Diversity": Unpacking and Designing for Diversity in Applicant Selection

Authors: Neil Natarajan, Sruthi Viswanathan, Reuben Binns, Nigel Shadbolt

Abstract: When selecting applicants for scholarships, universities, or jobs, practitioners often aim for a diverse cohort of qualified recipients. However, differing articulations, constructs, and notions of diversity prevents decision-makers from operationalising and progressing towards the diversity they all agree is needed. To understand this challenge of translation from values, to requirements, to deci… ▽ More When selecting applicants for scholarships, universities, or jobs, practitioners often aim for a diverse cohort of qualified recipients. However, differing articulations, constructs, and notions of diversity prevents decision-makers from operationalising and progressing towards the diversity they all agree is needed. To understand this challenge of translation from values, to requirements, to decision support tools (DSTs), we conducted participatory design studies exploring professionals' varied perceptions of diversity and how to build for them. Our results suggest three definitions of diversity: bringing together different perspectives; ensuring representativeness of a base population; and contextualising applications, which we use to create the Diversity Triangle. We experience-prototyped DSTs reflecting each angle of the Diversity Triangle to enhance decision-making around diversity. We find that notions of diversity are highly diverse; efforts to design DSTs for diversity should start by working with organisations to distil 'diversity' into definitions and design requirements. △ Less

Submitted 8 October, 2024; originally announced October 2024.

Comments: 32 pages, 11 figures

Code Simulation Challenges for Large Language Models

Authors: Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Samuele Marro, Anthony Cohn, Nigel Shadbolt, Michael Wooldridge

Abstract: Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. This work studies to what extent Large Language Models (LLMs) can simulate coding and algorithmic tasks to provide insights into general capabilities in such algorithmic reasoning tasks. We introduce benchmarks for straight-lin… ▽ More Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. This work studies to what extent Large Language Models (LLMs) can simulate coding and algorithmic tasks to provide insights into general capabilities in such algorithmic reasoning tasks. We introduce benchmarks for straight-line programs, code that contains critical paths, and approximate and redundant instructions. We further assess the simulation capabilities of LLMs with sorting algorithms and nested loops and show that a routine's computational complexity directly affects an LLM's ability to simulate its execution. While the most powerful LLMs exhibit relatively strong simulation capabilities, the process is fragile, seems to rely heavily on pattern recognition, and is affected by memorisation. We propose a novel off-the-shelf prompting method, Chain of Simulation (CoSm), which instructs LLMs to simulate code execution line by line/follow the computation pattern of compilers. CoSm efficiently helps LLMs reduce memorisation and shallow pattern recognition while improving simulation performance. We consider the success of CoSm in code simulation to be inspirational for other general routine simulation reasoning tasks. △ Less

Submitted 12 June, 2024; v1 submitted 17 January, 2024; originally announced January 2024.

Comments: Code: https://github.com/EmanueleLM/CodeSimulation

Language Models as a Service: Overview of a New Paradigm and its Challenges

Authors: Emanuele La Malfa, Aleksandar Petrov, Simon Frieder, Christoph Weinhuber, Ryan Burnell, Raza Nazar, Anthony G. Cohn, Nigel Shadbolt, Michael Wooldridge

Abstract: Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, b… ▽ More Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer. △ Less

Submitted 30 November, 2023; v1 submitted 28 September, 2023; originally announced September 2023.

Libertas: Privacy-Preserving Collective Computation for Decentralised Personal Data Stores

Authors: Rui Zhao, Naman Goel, Nitin Agrawal, Jun Zhao, Jake Stein, Wael Albayaydh, Ruben Verborgh, Reuben Binns, Tim Berners-Lee, Nigel Shadbolt

Abstract: Data and data processing have become an indispensable aspect for our society. Insights drawn from collective data make invaluable contribution to scientific and societal research and business. But there are increasing worries about privacy issues and data misuse. This has prompted the emergence of decentralised personal data stores (PDS) like Solid that provide individuals more control over their… ▽ More Data and data processing have become an indispensable aspect for our society. Insights drawn from collective data make invaluable contribution to scientific and societal research and business. But there are increasing worries about privacy issues and data misuse. This has prompted the emergence of decentralised personal data stores (PDS) like Solid that provide individuals more control over their personal data. However, existing PDS frameworks face challenges in ensuring data privacy when performing collective computations with data from multiple users. While Secure Multi-Party Computation (MPC) offers input secrecy protection during the computation without relying on any single party, issues emerge when directly applying MPC in the context of PDS, particularly due to key factors like autonomy and decentralisation. In this work, we discuss the essence of this issue, identify a potential solution, and introduce a modular architecture, Libertas, to integrate MPC with PDS like Solid, without requiring protocol-level changes. We introduce a paradigm shift from an `omniscient' view to individual-based, user-centric view of trust and security, and discuss the threat model of Libertas. Two realistic use cases for collaborative data processing are used for evaluation, both for technical feasibility and empirical benchmark, highlighting its effectiveness in empowering gig workers and generating differentially private synthetic data. The results of our experiments underscore Libertas' linear scalability and provide valuable insights into compute optimisations, thereby advancing the state-of-the-art in privacy-preserving data processing practices. By offering practical solutions for maintaining both individual autonomy and privacy in collaborative data processing environments, Libertas contributes significantly to the ongoing discourse on privacy protection in data-driven decision-making contexts. △ Less

Submitted 30 March, 2025; v1 submitted 28 September, 2023; originally announced September 2023.

Comments: Accepted by CSCW 2025; manuscript version

We Are Not There Yet: The Implications of Insufficient Knowledge Management for Organisational Compliance

Authors: Thomas Şerban von Davier, Konrad Kollnig, Reuben Binns, Max Van Kleek, Nigel Shadbolt

Abstract: Since GDPR went into effect in 2018, many other data protection and privacy regulations have been released. With the new regulation, there has been an associated increase in industry professionals focused on data protection and privacy. Building on related work showing the potential benefits of knowledge management in organisational compliance and privacy engineering, this paper presents the findi… ▽ More Since GDPR went into effect in 2018, many other data protection and privacy regulations have been released. With the new regulation, there has been an associated increase in industry professionals focused on data protection and privacy. Building on related work showing the potential benefits of knowledge management in organisational compliance and privacy engineering, this paper presents the findings of an exploratory qualitative study with data protection officers and other privacy professionals. We found issues with knowledge management to be the underlying challenge of our participants' feedback. Our participants noted four categories of feedback: (1) a perceived disconnect between regulation and practice, (2) a general lack of clear job description, (3) the need for data protection and privacy to be involved at every level of an organisation, (4) knowledge management tools exist but are not used effectively. This paper questions what knowledge management or automation solutions may prove to be effective in establishing better computer-supported work environments. △ Less

Submitted 6 May, 2023; originally announced May 2023.

Comments: 27 pages, 1 figure, 1 table, DRAFT and Unpublished

Trust Explanations to Do What They Say

Authors: Neil Natarajan, Reuben Binns, Jun Zhao, Nigel Shadbolt

Abstract: How much are we to trust a decision made by an AI algorithm? Trusting an algorithm without cause may lead to abuse, and mistrusting it may similarly lead to disuse. Trust in an AI is only desirable if it is warranted; thus, calibrating trust is critical to ensuring appropriate use. In the name of calibrating trust appropriately, AI developers should provide contracts specifying use cases in which… ▽ More How much are we to trust a decision made by an AI algorithm? Trusting an algorithm without cause may lead to abuse, and mistrusting it may similarly lead to disuse. Trust in an AI is only desirable if it is warranted; thus, calibrating trust is critical to ensuring appropriate use. In the name of calibrating trust appropriately, AI developers should provide contracts specifying use cases in which an algorithm can and cannot be trusted. Automated explanation of AI outputs is often touted as a method by which trust can be built in the algorithm. However, automated explanations arise from algorithms themselves, so trust in these explanations is similarly only desirable if it is warranted. Developers of algorithms explaining AI outputs (xAI algorithms) should provide similar contracts, which should specify use cases in which an explanation can and cannot be trusted. △ Less

Submitted 14 February, 2023; originally announced March 2023.

Before and after China's new Data Laws: Privacy in Apps

Authors: Konrad Kollnig, Lu Zhang, Jun Zhao, Nigel Shadbolt

Abstract: Privacy in apps is a topic of widespread interest because many apps collect and share large amounts of highly sensitive information. In response, China introduced a range of new data protection laws over recent years, notably the Personal Information Protection Law (PIPL) in 2021. So far, there exists limited research on the impacts of these new laws on apps' privacy practices. To address this gap… ▽ More Privacy in apps is a topic of widespread interest because many apps collect and share large amounts of highly sensitive information. In response, China introduced a range of new data protection laws over recent years, notably the Personal Information Protection Law (PIPL) in 2021. So far, there exists limited research on the impacts of these new laws on apps' privacy practices. To address this gap, this paper analyses data collection in pairs of 634 Chinese iOS apps, one version from early 2020 and one from late 2021. Our work finds that many more apps now implement consent. Yet, those end-users that decline consent will often be forced to exit the app. Fewer apps now collect data without consent but many still integrate tracking libraries. We see our findings as characteristic of a first iteration at Chinese data regulation with room for improvement. △ Less

Submitted 2 March, 2023; v1 submitted 27 February, 2023; originally announced February 2023.

Comments: Accepted for publication by the 7th Workshop on Technology and Consumer Protection (ConPro '23)

Projected Subnetworks Scale Adaptation

Authors: Siddhartha Datta, Nigel Shadbolt

Abstract: Large models support great zero-shot and few-shot capabilities. However, updating these models on new tasks can break performance on previous seen tasks and their zero/few-shot unseen tasks. Our work explores how to update zero/few-shot learners such that they can maintain performance on seen/unseen tasks of previous tasks as well as new tasks. By manipulating the parameter updates of a gradient-b… ▽ More Large models support great zero-shot and few-shot capabilities. However, updating these models on new tasks can break performance on previous seen tasks and their zero/few-shot unseen tasks. Our work explores how to update zero/few-shot learners such that they can maintain performance on seen/unseen tasks of previous tasks as well as new tasks. By manipulating the parameter updates of a gradient-based meta learner as the projected task-specific subnetworks, we show improvements for large models to retain seen and zero/few shot task performance in online settings. △ Less

Submitted 26 January, 2023; originally announced January 2023.

Multiple Modes for Continual Learning

Authors: Siddhartha Datta, Nigel Shadbolt

Abstract: Adapting model parameters to incoming streams of data is a crucial factor to deep learning scalability. Interestingly, prior continual learning strategies in online settings inadvertently anchor their updated parameters to a local parameter subspace to remember old tasks, else drift away from the subspace and forget. From this observation, we formulate a trade-off between constructing multiple par… ▽ More Adapting model parameters to incoming streams of data is a crucial factor to deep learning scalability. Interestingly, prior continual learning strategies in online settings inadvertently anchor their updated parameters to a local parameter subspace to remember old tasks, else drift away from the subspace and forget. From this observation, we formulate a trade-off between constructing multiple parameter modes and allocating tasks per mode. Mode-Optimized Task Allocation (MOTA), our contributed adaptation strategy, trains multiple modes in parallel, then optimizes task allocation per mode. We empirically demonstrate improvements over baseline continual learning strategies and across varying distribution shifts, namely sub-population, domain, and task shift. △ Less

Submitted 29 September, 2022; originally announced September 2022.

Interpolating Compressed Parameter Subspaces

Authors: Siddhartha Datta, Nigel Shadbolt

Abstract: Inspired by recent work on neural subspaces and mode connectivity, we revisit parameter subspace sampling for shifted and/or interpolatable input distributions (instead of a single, unshifted distribution). We enforce a compressed geometric structure upon a set of trained parameters mapped to a set of train-time distributions, denoting the resulting subspaces as Compressed Parameter Subspaces (CPS… ▽ More Inspired by recent work on neural subspaces and mode connectivity, we revisit parameter subspace sampling for shifted and/or interpolatable input distributions (instead of a single, unshifted distribution). We enforce a compressed geometric structure upon a set of trained parameters mapped to a set of train-time distributions, denoting the resulting subspaces as Compressed Parameter Subspaces (CPS). We show the success and failure modes of the types of shifted distributions whose optimal parameters reside in the CPS. We find that ensembling point-estimates within a CPS can yield a high average accuracy across a range of test-time distributions, including backdoor, adversarial, permutation, stylization and rotation perturbations. We also find that the CPS can contain low-loss point-estimates for various task shifts (albeit interpolated, perturbed, unseen or non-identical coarse labels). We further demonstrate this property in a continual learning setting with CIFAR100. △ Less

Submitted 19 May, 2022; originally announced May 2022.

Imagining, Studying and Realising A Less Harmful App Ecosystem

Authors: Konrad Kollnig, Siddhartha Datta, Nigel Shadbolt

Abstract: Desktop browser extensions have long allowed users to improve their experience online and tackle widespread harms on websites. So far, no equivalent solution exists for mobile apps, despite the fact that individuals now spend significantly more time on mobile than on desktop, and arguably face similarly widespread harms. In this work, we investigate mobile app extensions, a previously underexplo… ▽ More Desktop browser extensions have long allowed users to improve their experience online and tackle widespread harms on websites. So far, no equivalent solution exists for mobile apps, despite the fact that individuals now spend significantly more time on mobile than on desktop, and arguably face similarly widespread harms. In this work, we investigate mobile app extensions, a previously underexplored concept to study and address digital harms within mobile apps in a decentralised, community-driven way. We analyse challenges to adoption of this approach so far, and present a ready-to-use implementation for Android as a result of significant and careful system development. Through a range of case studies, we demonstrate that our implementation can already reduce (though not completely eliminate) a wide range of harms - similarly as browser extensions do on desktops. Our method provides a versatile foundation for a range of follow-up research into digital harms in mobile apps that has not previously been possible, given that browser extensions have long been a fruitful foundation for research studies on desktops. In other words, our system tries to address the gap of a focus on desktop interventions in previous research. △ Less

Submitted 24 February, 2023; v1 submitted 2 May, 2022; originally announced May 2022.

Comments: Pre-print / Work-in-progress

GreaseVision: Rewriting the Rules of the Interface

Authors: Siddhartha Datta, Konrad Kollnig, Nigel Shadbolt

Abstract: Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. As a result, we still lack a systematic approach to study harms and produce interventions for end-users. We put forward GreaseVision, a new framework that enables end-users to collaboratively develop interventions against h… ▽ More Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. As a result, we still lack a systematic approach to study harms and produce interventions for end-users. We put forward GreaseVision, a new framework that enables end-users to collaboratively develop interventions against harms in software using a no-code approach and recent advances in few-shot machine learning. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of harms and interventions at scale. △ Less

Submitted 7 April, 2022; originally announced April 2022.

Goodbye Tracking? Impact of iOS App Tracking Transparency and Privacy Labels

Authors: Konrad Kollnig, Anastasia Shuba, Max Van Kleek, Reuben Binns, Nigel Shadbolt

Abstract: Tracking is a highly privacy-invasive data collection practice that has been ubiquitous in mobile apps for many years due to its role in supporting advertising-based revenue models. In response, Apple introduced two significant changes with iOS 14: App Tracking Transparency (ATT), a mandatory opt-in system for enabling tracking on iOS, and Privacy Nutrition Labels, which disclose what kinds of dat… ▽ More Tracking is a highly privacy-invasive data collection practice that has been ubiquitous in mobile apps for many years due to its role in supporting advertising-based revenue models. In response, Apple introduced two significant changes with iOS 14: App Tracking Transparency (ATT), a mandatory opt-in system for enabling tracking on iOS, and Privacy Nutrition Labels, which disclose what kinds of data each app processes. So far, the impact of these changes on individual privacy and control has not been well understood. This paper addresses this gap by analysing two versions of 1,759 iOS apps from the UK App Store: one version from before iOS 14 and one that has been updated to comply with the new rules. We find that Apple's new policies, as promised, prevent the collection of the Identifier for Advertisers (IDFA), an identifier for cross-app tracking. Smaller data brokers that engage in invasive data practices will now face higher challenges in tracking users - a positive development for privacy. However, the number of tracking libraries has roughly stayed the same in the studied apps. Many apps still collect device information that can be used to track users at a group level (cohort tracking) or identify individuals probabilistically (fingerprinting). We find real-world evidence of apps computing and agreeing on a fingerprinting-derived identifier through the use of server-side code, thereby violating Apple's policies. We find that Apple itself engages in some forms of tracking and exempts invasive data practices like first-party tracking and credit scoring. We also find that the new Privacy Nutrition Labels are sometimes inaccurate and misleading. Overall, our findings suggest that, while tracking individual users is more difficult now, the changes reinforce existing market power of gatekeeper companies with access to large troves of first-party data and motivate a countermovement. △ Less

Submitted 7 May, 2022; v1 submitted 7 April, 2022; originally announced April 2022.

Comments: The paper has been accepted for publication by the ACM Conference on Fairness, Accountability, and Transparency (FAccT) 2022

Low-Loss Subspace Compression for Clean Gains against Multi-Agent Backdoor Attacks

Authors: Siddhartha Datta, Nigel Shadbolt

Abstract: Recent exploration of the multi-agent backdoor attack demonstrated the backfiring effect, a natural defense against backdoor attacks where backdoored inputs are randomly classified. This yields a side-effect of low accuracy w.r.t. clean labels, which motivates this paper's work on the construction of multi-agent backdoor defenses that maximize accuracy w.r.t. clean labels and minimize that of pois… ▽ More Recent exploration of the multi-agent backdoor attack demonstrated the backfiring effect, a natural defense against backdoor attacks where backdoored inputs are randomly classified. This yields a side-effect of low accuracy w.r.t. clean labels, which motivates this paper's work on the construction of multi-agent backdoor defenses that maximize accuracy w.r.t. clean labels and minimize that of poison labels. Founded upon agent dynamics and low-loss subspace construction, we contribute three defenses that yield improved multi-agent backdoor robustness. △ Less

Submitted 20 September, 2022; v1 submitted 7 March, 2022; originally announced March 2022.

Backdoors Stuck At The Frontdoor: Multi-Agent Backdoor Attacks That Backfire

Authors: Siddhartha Datta, Nigel Shadbolt

Abstract: Malicious agents in collaborative learning and outsourced data collection threaten the training of clean models. Backdoor attacks, where an attacker poisons a model during training to successfully achieve targeted misclassification, are a major concern to train-time robustness. In this paper, we investigate a multi-agent backdoor attack scenario, where multiple attackers attempt to backdoor a vict… ▽ More Malicious agents in collaborative learning and outsourced data collection threaten the training of clean models. Backdoor attacks, where an attacker poisons a model during training to successfully achieve targeted misclassification, are a major concern to train-time robustness. In this paper, we investigate a multi-agent backdoor attack scenario, where multiple attackers attempt to backdoor a victim model simultaneously. A consistent backfiring phenomenon is observed across a wide range of games, where agents suffer from a low collective attack success rate. We examine different modes of backdoor attack configurations, non-cooperation / cooperation, joint distribution shifts, and game setups to return an equilibrium attack success rate at the lower bound. The results motivate the re-evaluation of backdoor defense research for practical environments. △ Less

Submitted 28 January, 2022; originally announced January 2022.

Hiding Behind Backdoors: Self-Obfuscation Against Generative Models

Authors: Siddhartha Datta, Nigel Shadbolt

Abstract: Attack vectors that compromise machine learning pipelines in the physical world have been demonstrated in recent research, from perturbations to architectural components. Building on this work, we illustrate the self-obfuscation attack: attackers target a pre-processing model in the system, and poison the training set of generative models to obfuscate a specific class during inference. Our contrib… ▽ More Attack vectors that compromise machine learning pipelines in the physical world have been demonstrated in recent research, from perturbations to architectural components. Building on this work, we illustrate the self-obfuscation attack: attackers target a pre-processing model in the system, and poison the training set of generative models to obfuscate a specific class during inference. Our contribution is to describe, implement and evaluate a generalized attack, in the hope of raising awareness regarding the challenge of architectural robustness within the machine learning community. △ Less

Submitted 24 January, 2022; originally announced January 2022.

Before and after GDPR: tracking in mobile apps

Authors: Konrad Kollnig, Reuben Binns, Max Van Kleek, Ulrik Lyngs, Jun Zhao, Claudine Tinsman, Nigel Shadbolt

Abstract: Third-party tracking, the collection and sharing of behavioural data about individuals, is a significant and ubiquitous privacy threat in mobile apps. The EU General Data Protection Regulation (GDPR) was introduced in 2018 to protect personal data better, but there exists, thus far, limited empirical evidence about its efficacy. This paper studies tracking in nearly two million Android apps from b… ▽ More Third-party tracking, the collection and sharing of behavioural data about individuals, is a significant and ubiquitous privacy threat in mobile apps. The EU General Data Protection Regulation (GDPR) was introduced in 2018 to protect personal data better, but there exists, thus far, limited empirical evidence about its efficacy. This paper studies tracking in nearly two million Android apps from before and after the introduction of the GDPR. Our analysis suggests that there has been limited change in the presence of third-party tracking in apps, and that the concentration of tracking capabilities among a few large gatekeeper companies persists. However, change might be imminent. △ Less

Submitted 21 December, 2021; originally announced December 2021.

Journal ref: Internet Policy Review, 2021, 10(4)

Mind-proofing Your Phone: Navigating the Digital Minefield with GreaseTerminator

Authors: Siddhartha Datta, Konrad Kollnig, Nigel Shadbolt

Abstract: Digital harms are widespread in the mobile ecosystem. As these devices gain ever more prominence in our daily lives, so too increases the potential for malicious attacks against individuals. The last line of defense against a range of digital harms - including digital distraction, political polarisation through hate speech, and children being exposed to damaging material - is the user interface. T… ▽ More Digital harms are widespread in the mobile ecosystem. As these devices gain ever more prominence in our daily lives, so too increases the potential for malicious attacks against individuals. The last line of defense against a range of digital harms - including digital distraction, political polarisation through hate speech, and children being exposed to damaging material - is the user interface. This work introduces GreaseTerminator to enable researchers to develop, deploy, and test interventions against these harms with end-users. We demonstrate the ease of intervention development and deployment, as well as the broad range of harms potentially covered with GreaseTerminator in five in-depth case studies. △ Less

Submitted 1 February, 2022; v1 submitted 20 December, 2021; originally announced December 2021.

Comments: Accepted in ACM IUI 2022

Widen The Backdoor To Let More Attackers In

Authors: Siddhartha Datta, Giulio Lovisotto, Ivan Martinovic, Nigel Shadbolt

Abstract: As collaborative learning and the outsourcing of data collection become more common, malicious actors (or agents) which attempt to manipulate the learning process face an additional obstacle as they compete with each other. In backdoor attacks, where an adversary attempts to poison a model by introducing malicious samples into the training data, adversaries have to consider that the presence of ad… ▽ More As collaborative learning and the outsourcing of data collection become more common, malicious actors (or agents) which attempt to manipulate the learning process face an additional obstacle as they compete with each other. In backdoor attacks, where an adversary attempts to poison a model by introducing malicious samples into the training data, adversaries have to consider that the presence of additional backdoor attackers may hamper the success of their own backdoor. In this paper, we investigate the scenario of a multi-agent backdoor attack, where multiple non-colluding attackers craft and insert triggered samples in a shared dataset which is used by a model (a defender) to learn a task. We discover a clear backfiring phenomenon: increasing the number of attackers shrinks each attacker's attack success rate (ASR). We then exploit this phenomenon to minimize the collective ASR of attackers and maximize defender's robustness accuracy by (i) artificially augmenting the number of attackers, and (ii) indexing to remove the attacker's sub-dataset from the model for inference, hence proposing 2 defenses. △ Less

Submitted 9 October, 2021; originally announced October 2021.

Are iPhones Really Better for Privacy? Comparative Study of iOS and Android Apps

Authors: Konrad Kollnig, Anastasia Shuba, Reuben Binns, Max Van Kleek, Nigel Shadbolt

Abstract: While many studies have looked at privacy properties of the Android and Google Play app ecosystem, comparatively much less is known about iOS and the Apple App Store, the most widely used ecosystem in the US. At the same time, there is increasing competition around privacy between these smartphone operating system providers. In this paper, we present a study of 24k Android and iOS apps from 2020 a… ▽ More While many studies have looked at privacy properties of the Android and Google Play app ecosystem, comparatively much less is known about iOS and the Apple App Store, the most widely used ecosystem in the US. At the same time, there is increasing competition around privacy between these smartphone operating system providers. In this paper, we present a study of 24k Android and iOS apps from 2020 along several dimensions relating to user privacy. We find that third-party tracking and the sharing of unique user identifiers was widespread in apps from both ecosystems, even in apps aimed at children. In the children's category, iOS apps tended to use fewer advertising-related tracking than their Android counterparts, but could more often access children's location. Across all studied apps, our study highlights widespread potential violations of US, EU and UK privacy law, including 1) the use of third-party tracking without user consent, 2) the lack of parental consent before sharing personally identifiable information (PII) with third-parties in children's apps, 3) the non-data-minimising configuration of tracking libraries, 4) the sending of personal data to countries without an adequate level of data protection, and 5) the continued absence of transparency around tracking, partly due to design decisions by Apple and Google. Overall, we find that neither platform is clearly better than the other for privacy across the dimensions we studied. △ Less

Submitted 19 December, 2021; v1 submitted 28 September, 2021; originally announced September 2021.

Comments: Accepted for publication by the Proceedings on Privacy Enhancing Technologies (PoPETs) 2022

Protection or punishment? relating the design space of parental control apps and perceptions about them to support parenting for online safety

Authors: Ge Wang, Jun Zhao, Max Van Kleek, Nigel Shadbolt

Abstract: Parental control apps, which are mobile apps that allow parents to monitor and restrict their children's activities online, are becoming increasingly adopted by parents as a means of safeguarding their children's online safety. However, it is not clear whether these apps are always beneficial or effective in what they aim to do; for instance, the overuse of restriction and surveillance has been fo… ▽ More Parental control apps, which are mobile apps that allow parents to monitor and restrict their children's activities online, are becoming increasingly adopted by parents as a means of safeguarding their children's online safety. However, it is not clear whether these apps are always beneficial or effective in what they aim to do; for instance, the overuse of restriction and surveillance has been found to undermine parent-child relationship and children's sense of autonomy. In this work, we investigate this gap, asking specifically: how might children's and parents' perceptions be related to how parental control features were designed? To investigate this question, we conducted an analysis of 58 top Android parental control apps designed for the purpose of promoting children's online safety, finding three major axes of variation in how key restriction and monitoring features were realised: granularity, feedback/transparency, and parent-child communications support. To relate these axes to perceived benefits and problems, we then analysed 3264 app reviews to identify references to aspects of the each of the axes above, to understand children's and parents' views of how such dimensions related to their experiences with these apps. Our findings led towards 1) an understanding of how parental control apps realise their functionalities differently along three axes of variation, 2) an analysis of exactly the ways that such variation influences children's and parents' perceptions, respectively of the usefulness or effectiveness of these apps, and finally 3) an identification of design recommendations and opportunities for future apps by contextualising our findings within existing digital parenting theories. △ Less

Submitted 11 September, 2021; originally announced September 2021.

Comments: (to be published)

Journal ref: Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 343 (October 2021)

A Fait Accompli? An Empirical Study into the Absence of Consent to Third-Party Tracking in Android Apps

Authors: Konrad Kollnig, Reuben Binns, Pierre Dewitte, Max Van Kleek, Ge Wang, Daniel Omeiza, Helena Webb, Nigel Shadbolt

Abstract: Third-party tracking allows companies to collect users' behavioural data and track their activity across digital devices. This can put deep insights into users' private lives into the hands of strangers, and often happens without users' awareness or explicit consent. EU and UK data protection law, however, requires consent, both 1) to access and store information on users' devices and 2) to legiti… ▽ More Third-party tracking allows companies to collect users' behavioural data and track their activity across digital devices. This can put deep insights into users' private lives into the hands of strangers, and often happens without users' awareness or explicit consent. EU and UK data protection law, however, requires consent, both 1) to access and store information on users' devices and 2) to legitimate the processing of personal data as part of third-party tracking, as we analyse in this paper. This paper further investigates whether and to what extent consent is implemented in mobile apps. First, we analyse a representative sample of apps from the Google Play Store. We find that most apps engage in third-party tracking, but few obtained consent before doing so, indicating potentially widespread violations of EU and UK privacy law. Second, we examine the most common third-party tracking libraries in detail. While most acknowledge that they rely on app developers to obtain consent on their behalf, they typically fail to put in place robust measures to ensure this: disclosure of consent requirements is limited; default consent implementations are lacking; and compliance guidance is difficult to find, hard to read, and poorly maintained. △ Less

Submitted 18 June, 2021; v1 submitted 17 June, 2021; originally announced June 2021.

Comments: This paper will be presented at the 7th Symposium on Usable Privacy and Security (SOUPS 2021), 8th-10th August 2021

Exploring Design and Governance Challenges in the Development of Privacy-Preserving Computation

Authors: Nitin Agrawal, Reuben Binns, Max Van Kleek, Kim Laine, Nigel Shadbolt

Abstract: Homomorphic encryption, secure multi-party computation, and differential privacy are part of an emerging class of Privacy Enhancing Technologies which share a common promise: to preserve privacy whilst also obtaining the benefits of computational analysis. Due to their relative novelty, complexity, and opacity, these technologies provoke a variety of novel questions for design and governance. We i… ▽ More Homomorphic encryption, secure multi-party computation, and differential privacy are part of an emerging class of Privacy Enhancing Technologies which share a common promise: to preserve privacy whilst also obtaining the benefits of computational analysis. Due to their relative novelty, complexity, and opacity, these technologies provoke a variety of novel questions for design and governance. We interviewed researchers, developers, industry leaders, policymakers, and designers involved in their deployment to explore motivations, expectations, perceived opportunities and barriers to adoption. This provided insight into several pertinent challenges facing the adoption of these technologies, including: how they might make a nebulous concept like privacy computationally tractable; how to make them more usable by developers; and how they could be explained and made accountable to stakeholders and wider society. We conclude with implications for the development, deployment, and responsible governance of these privacy-preserving computation techniques. △ Less

Submitted 20 January, 2021; originally announced January 2021.

Strangers in the Room: Unpacking Perceptions of 'Smartness' and Related Ethical Concerns in the Home

Authors: William Seymour, Reuben Binns, Petr Slovak, Max Van Kleek, Nigel Shadbolt

Abstract: The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in gen… ▽ More The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in general. This paper attempts to bridge these approaches by asking what features of smart devices users consider as rendering them 'smart' and how these relate to ethical concerns. Through a multimethod investigation including surveys with smart device users (n=120) and semi-structured interviews (n=15), we identify and describe eight types of smartness and explore how they engender a variety of ethical concerns including privacy, autonomy, and disruption of the social order. We argue that this middle ground, between concerns arising from particular devices and more abstract ethical concepts, can better anticipate potential ethical concerns regarding smart devices. △ Less

Submitted 1 May, 2020; originally announced May 2020.

Comments: 10 pages, 1 figure. To appear in the Proceedings of the 2020 ACM Conference on Designing Interactive Systems (DIS '20)

'I Just Want to Hack Myself to Not Get Distracted': Evaluating Design Interventions for Self-Control on Facebook

Authors: Ulrik Lyngs, Kai Lukoff, Petr Slovak, William Seymour, Helena Webb, Marina Jirotka, Jun Zhao, Max Van Kleek, Nigel Shadbolt

Abstract: Beyond being the world's largest social network, Facebook is for many also one of its greatest sources of digital distraction. For students, problematic use has been associated with negative effects on academic achievement and general wellbeing. To understand what strategies could help users regain control, we investigated how simple interventions to the Facebook UI affect behaviour and perceived… ▽ More Beyond being the world's largest social network, Facebook is for many also one of its greatest sources of digital distraction. For students, problematic use has been associated with negative effects on academic achievement and general wellbeing. To understand what strategies could help users regain control, we investigated how simple interventions to the Facebook UI affect behaviour and perceived control. We assigned 58 university students to one of three interventions: goal reminders, removed newsfeed, or white background (control). We logged use for 6 weeks, applied interventions in the middle weeks, and administered fortnightly surveys. Both goal reminders and removed newsfeed helped participants stay on task and avoid distraction. However, goal reminders were often annoying, and removing the newsfeed made some fear missing out on information. Our findings point to future interventions such as controls for adjusting types and amount of available information, and flexible blocking which matches individual definitions of 'distraction'. △ Less

Submitted 20 May, 2020; v1 submitted 13 January, 2020; originally announced January 2020.

Comments: 10 pages (excluding references), 6 figures. To appear in the Proceedings of CHI '20 CHI Conference on Human Factors in Computing Systems, April 25--30, 2020, Honolulu, HI, USA

ACM Class: H.5.2

What concerns do Chinese parents have about their children's digital adoption and how to better support them?

Authors: Ge Wang, Jun Zhao, Nigel Shadbolt

Abstract: Digital devices are widely used by children, and children nowadays are spending more time online than with other media sources, such as watching television or playing offline video games. In the UK, 44% of children aged five to ten have been provided with their own tablets, with this percentage increasing annually, while in the US, ownership of tablets by children in this age group grew fivefold b… ▽ More Digital devices are widely used by children, and children nowadays are spending more time online than with other media sources, such as watching television or playing offline video games. In the UK, 44% of children aged five to ten have been provided with their own tablets, with this percentage increasing annually, while in the US, ownership of tablets by children in this age group grew fivefold between 2011 and 2013. Our previous research found that UK children and parents need better support in dealing with online privacy risks. Interestingly, very few research was done on Chinese children and parents. In this report, we present findings from our online survey of 593 Chinese parents with children aged 6-10 in February and March 2019. Our study particularly focused on understanding Chinese parents' awareness and management of their children's online privacy risks. The goal of the survey was to examine the current adoption pattern of digital devices by Chinese families with young children, the concerns Chinese parents have about their children's online activities and the current practices they use for safeguarding their children online. Our findings imply that we need to continue presenting specific guidance to parents in order to support their choice of digital content for their young children. Further, we need to look more deeply into the roles schools are taking in children's online activities, how can we support schools and teachers when they are making recommendations to parents and children. △ Less

Submitted 26 June, 2019; originally announced June 2019.

Are Children Fully Aware of Online Privacy Risks and How Can We Improve Their Coping Ability?

Authors: Ge Wang, Jun Zhao, Nigel Shadbolt

Abstract: The age of children adopting digital technologies, such as tablets or smartphones, is increasingly young. However, children under 11 are often regarded as too young to comprehend the concept of online privacy. Limited research studies have focused on children of this age group. In the summer of 2018, we conducted 12 focus group studies with 29 children aged 6-10 from Oxfordshire primary schools. O… ▽ More The age of children adopting digital technologies, such as tablets or smartphones, is increasingly young. However, children under 11 are often regarded as too young to comprehend the concept of online privacy. Limited research studies have focused on children of this age group. In the summer of 2018, we conducted 12 focus group studies with 29 children aged 6-10 from Oxfordshire primary schools. Our research has shown that children have a good understanding of certain privacy risks, such as information oversharing or avoiding revealing real identities online. They could use a range of descriptions to articulate the risks and describe their risk coping strategies. However, at the same time, we identified that children had less awareness concerning other risks, such as online tracking or game promotions. Inspired by Vygotsky's Zone of Proximal Development (ZPD), this study has identified critical knowledge gaps in children's understanding of online privacy, and several directions for future education and technology development. We call for attention to the needs of raising children's awareness and understanding of risks related to online recommendations and data tracking, which are becoming ever more prevalent in the games and content children encounter. We also call for attention to children's use of language to describe risks, which may be appropriate but not necessarily indicate a full understanding of the threats. △ Less

Submitted 6 February, 2019; originally announced February 2019.

Comments: 9 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:1901.10245

Report number: KOALA.03

Self-Control in Cyberspace: Applying Dual Systems Theory to a Review of Digital Self-Control Tools

Authors: Ulrik Lyngs, Kai Lukoff, Petr Slovak, Reuben Binns, Adam Slack, Michael Inzlicht, Max Van Kleek, Nigel Shadbolt

Abstract: Many people struggle to control their use of digital devices. However, our understanding of the design mechanisms that support user self-control remains limited. In this paper, we make two contributions to HCI research in this space: first, we analyse 367 apps and browser extensions from the Google Play, Chrome Web, and Apple App stores to identify common core design features and intervention stra… ▽ More Many people struggle to control their use of digital devices. However, our understanding of the design mechanisms that support user self-control remains limited. In this paper, we make two contributions to HCI research in this space: first, we analyse 367 apps and browser extensions from the Google Play, Chrome Web, and Apple App stores to identify common core design features and intervention strategies afforded by current tools for digital self-control. Second, we adapt and apply an integrative dual systems model of self-regulation as a framework for organising and evaluating the design features found. Our analysis aims to help the design of better tools in two ways: (i) by identifying how, through a well-established model of self-regulation, current tools overlap and differ in how they support self-control; and (ii) by using the model to reveal underexplored cognitive mechanisms that could aid the design of new tools. △ Less

Submitted 31 January, 2019; originally announced February 2019.

Comments: 11.5 pages (excl. references), 6 figures, 1 table

Journal ref: CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4-9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA

`I make up a silly name': Understanding Children's Perception of Privacy Risks Online

Authors: Jun Zhao, Ge Wang, Carys Dally, Petr Slovak, Julian Childs, Max Van Klee, Nigel Shadbolt

Abstract: Children under 11 are often regarded as too young to comprehend the implications of online privacy. Perhaps as a result, little research has focused on younger kids' risk recognition and coping. Such knowledge is, however, critical for designing efficient safeguarding mechanisms for this age group. Through 12 focus group studies with 29 children aged 6-10 from UK schools, we examined how children… ▽ More Children under 11 are often regarded as too young to comprehend the implications of online privacy. Perhaps as a result, little research has focused on younger kids' risk recognition and coping. Such knowledge is, however, critical for designing efficient safeguarding mechanisms for this age group. Through 12 focus group studies with 29 children aged 6-10 from UK schools, we examined how children described privacy risks related to their use of tablet computers and what information was used by them to identify threats. We found that children could identify and articulate certain privacy risks well, such as information oversharing or revealing real identities online; however, they had less awareness with respect to other risks, such as online tracking or game promotions. Our findings offer promising directions for supporting children's awareness of cyber risks and the ability to protect themselves online. △ Less

Submitted 29 January, 2019; originally announced January 2019.

Comments: 13 pages, 1 figure

Journal ref: CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4--9, 2019, Glasgow, Scotland Uk

What privacy concerns do parents have about children's mobile apps, and how can they stay SHARP?

Authors: Jun Zhao, Ulrik Lyngs, Nigel Shadbolt

Abstract: Tablet computers are widely used by young children. A report in 2016 shows that children aged 5 to 15 years are spending more time online than watching TV. A 2017 update of the same report shows that parents are becoming more concerned about their children's online risks compared to the previous year. Parents are working hard to protect their children's online safety. An increasing number of paren… ▽ More Tablet computers are widely used by young children. A report in 2016 shows that children aged 5 to 15 years are spending more time online than watching TV. A 2017 update of the same report shows that parents are becoming more concerned about their children's online risks compared to the previous year. Parents are working hard to protect their children's online safety. An increasing number of parents are setting up content filtering at home or having regular discussions with their children regarding online risks. However, although risks related to Social Media platforms or social video sharing sites (like YouTube) are widely known, risks posed by mobile applications or games (i.e. `apps') are less known. Behind the cute characters, apps used by children can not only have the possibility of exposing them to age-inappropriate content or excessive in-app promotions, but may also make a large amount of their personal information accessible to third-party online marketing and advertising industry. Such practices are not unique to children's apps, but young children are probably less capable of resisting the resulting personalised advertisements and game promotions. In this report, we present findings from our online survey of 220 parents with children aged 6-10, mainly from the U.K. and other western countries, regarding their privacy concerns and expectations of their children's use of mobile apps. Parents play a key role in children's use of digital technology, especially for children under 10 years old. Recent reports have highlighted parents' lack of sufficient support for choosing appropriate digital content for their children. Our report sheds some initial light on parents' key struggles and points to immediate steps and possible areas of future development. △ Less

Submitted 27 September, 2018; originally announced September 2018.

Comments: 13 pages, 12 figures, report

Third Party Tracking in the Mobile Ecosystem

Authors: Reuben Binns, Ulrik Lyngs, Max Van Kleek, Jun Zhao, Timothy Libert, Nigel Shadbolt

Abstract: Third party tracking allows companies to identify users and track their behaviour across multiple digital services. This paper presents an empirical study of the prevalence of third-party trackers on 959,000 apps from the US and UK Google Play stores. We find that most apps contain third party tracking, and the distribution of trackers is long-tailed with several highly dominant trackers accountin… ▽ More Third party tracking allows companies to identify users and track their behaviour across multiple digital services. This paper presents an empirical study of the prevalence of third-party trackers on 959,000 apps from the US and UK Google Play stores. We find that most apps contain third party tracking, and the distribution of trackers is long-tailed with several highly dominant trackers accounting for a large portion of the coverage. The extent of tracking also differs between categories of apps; in particular, news apps and apps targeted at children appear to be amongst the worst in terms of the number of third party trackers associated with them. Third party tracking is also revealed to be a highly trans-national phenomenon, with many trackers operating in jurisdictions outside the EU. Based on these findings, we draw out some significant legal compliance challenges facing the tracking industry. △ Less

Submitted 18 October, 2018; v1 submitted 10 April, 2018; originally announced April 2018.

Comments: Corrected missing company info (Linkedin owned by Microsoft). Figures for Microsoft and Linkedin re-calculated and added to Table 1

"So, Tell Me What Users Want, What They Really, Really Want!"

Authors: Ulrik Lyngs, Reuben Binns, Max Van Kleek, Nigel Shadbolt

Abstract: Equating users' true needs and desires with behavioural measures of 'engagement' is problematic. However, good metrics of 'true preferences' are difficult to define, as cognitive biases make people's preferences change with context and exhibit inconsistencies over time. Yet, HCI research often glosses over the philosophical and theoretical depth of what it means to infer what users really want. In… ▽ More Equating users' true needs and desires with behavioural measures of 'engagement' is problematic. However, good metrics of 'true preferences' are difficult to define, as cognitive biases make people's preferences change with context and exhibit inconsistencies over time. Yet, HCI research often glosses over the philosophical and theoretical depth of what it means to infer what users really want. In this paper, we present an alternative yet very real discussion of this issue, via a fictive dialogue between senior executives in a tech company aimed at helping people live the life they `really' want to live. How will the designers settle on a metric for their product to optimise? △ Less

Submitted 6 March, 2018; originally announced March 2018.

Measuring third party tracker power across web and mobile

Authors: Reuben Binns, Jun Zhao, Max Van Kleek, Nigel Shadbolt

Abstract: Third-party networks collect vast amounts of data about users via web sites and mobile applications. Consolidations among tracker companies can significantly increase their individual tracking capabilities, prompting scrutiny by competition regulators. Traditional measures of market share, based on revenue or sales, fail to represent the tracking capability of a tracker, especially if it spans bot… ▽ More Third-party networks collect vast amounts of data about users via web sites and mobile applications. Consolidations among tracker companies can significantly increase their individual tracking capabilities, prompting scrutiny by competition regulators. Traditional measures of market share, based on revenue or sales, fail to represent the tracking capability of a tracker, especially if it spans both web and mobile. This paper proposes a new approach to measure the concentration of tracking capability, based on the reach of a tracker on popular websites and apps. Our results reveal that tracker prominence and parent-subsidiary relationships have significant impact on accurately measuring concentration. △ Less

Submitted 7 February, 2018; originally announced February 2018.