-
Security and Privacy in Agentic AI: Grand Challenges and Future Directions
Authors:
Adam Jenkins,
Agnieszka Kitkowska,
Caterina Maidhof,
Diego Paracuellos,
Francesco Sovrano,
Gonzalo Gabriel Mendez,
Guillermo Suarez-Tangil,
Hana Kopecka,
Isabel Wagner,
Isabel Barbera,
Javier Carnerero-Cano,
Jide Edu,
Jose Luis Martin-Navarro,
Jose Such,
Josep Domingo-Ferrer,
Juan Carlos Carrillo,
Kopo Marvin Ramokapane,
Mark Cote,
Pablo Vellosillo,
Ramon Ruiz-Dolz,
Rongjun Ma,
Ruba Abu-Salma,
Sameer Patil,
William Seymour,
Xiao Zhan
Abstract:
We present key challenges and future research directions in the security and privacy of agentic AI, based on a horizon-scanning exercise that brought together thirty leading international experts from academia, industry, and government to engage in focused discussions and collaborative exercises on the emerging risks associated with the growing agency of AI.
We present key challenges and future research directions in the security and privacy of agentic AI, based on a horizon-scanning exercise that brought together thirty leading international experts from academia, industry, and government to engage in focused discussions and collaborative exercises on the emerging risks associated with the growing agency of AI.
△ Less
Submitted 27 July, 2026; v1 submitted 7 July, 2026;
originally announced July 2026.
-
Respectful Things: Adding Social Intelligence to 'Smart' Devices
Authors:
Max Van Kleek,
William Seymour,
Reuben Binns,
Nigel Shadbolt
Abstract:
In this paper, we propose that the idea of devices respecting their end-users may serve as a strong design goal for highly personal and intimate smart devices. We ask what respect is, how it shapes interaction, and how good-faith simulation of respect might inform user-friendly smart device design. Respect is a natural and integral part of natural human relationships that is seen to shape work and…
▽ More
In this paper, we propose that the idea of devices respecting their end-users may serve as a strong design goal for highly personal and intimate smart devices. We ask what respect is, how it shapes interaction, and how good-faith simulation of respect might inform user-friendly smart device design. Respect is a natural and integral part of natural human relationships that is seen to shape work and personal relations. In a basic sense, this is the core purpose of smart things: we expect them to be ready and willing to help us. In this vein, we distil the characteristics of more complex respectful behaviours into 4 main types relevant to smart devices, drawing from philosophical analyses of the conceptual dimensions of respect: directive respect, obstacle respect, recognition respect, and care respect. We discuss the implications of each of these kinds of respect for the future of smart personal devices.
△ Less
Submitted 1 June, 2026;
originally announced June 2026.
-
Beliefs and Misconceptions around Integrated Conversational AI
Authors:
William Seymour,
Adam Jenkins,
Mark Cote,
Jose Such
Abstract:
LLM-driven conversational AI is beginning to disappear into the background, shifting from something used directly towards something increasingly integrated into existing workflows. In the process, markers of origin and training are smoothed away as LLMs become commodified in the eyes of users. We explore how people approach using a web browser with conversational AI built in, focusing on how they…
▽ More
LLM-driven conversational AI is beginning to disappear into the background, shifting from something used directly towards something increasingly integrated into existing workflows. In the process, markers of origin and training are smoothed away as LLMs become commodified in the eyes of users. We explore how people approach using a web browser with conversational AI built in, focusing on how they develop their understanding and determine whether to trust its outputs. We conducted a study where 20 participants used the Copilot AI features in Microsoft Edge to conduct information retrieval and planning tasks. Participants relied on a combination of existing perceptions of LLMs and internet search, tracing the effect of beliefs about how Copilot generated answers on prompting strategies. The inclusion of citations increased the trustworthiness of answers without participants feeling the need to be check them, with participants often reaching for the same information sources as the CAI when fact-checking.
△ Less
Submitted 14 May, 2026;
originally announced May 2026.
-
Shifting Engagement With Cybersecurity: How People Discover and Share Cybersecurity Content at Work and at Home
Authors:
William Seymour,
Martin J. Kraemer
Abstract:
Cybersecurity awareness is shaped by a wide range of professional and personal experiences, including information and training at work and the sharing of news and other content at home. In order to explore how people discover cybersecurity content and the effect that participation in workplace training may have on this we present an online study of 1200 participants from the UK, US, France, and Ge…
▽ More
Cybersecurity awareness is shaped by a wide range of professional and personal experiences, including information and training at work and the sharing of news and other content at home. In order to explore how people discover cybersecurity content and the effect that participation in workplace training may have on this we present an online study of 1200 participants from the UK, US, France, and Germany. Those undertaking cybersecurity training at work showed reduced intention to share information at home, shifting the focus towards the workplace. They were also more likely to recall cybersecurity information shared by their employer than from any other source, which in turn correlated with content type and distribution channel. We critically reflect on this shift, highlighting opportunities to improve cybersecurity information sharing at work and at home.
△ Less
Submitted 23 February, 2026;
originally announced February 2026.
-
Realistic CDSS Drug Dosing with End-to-end Recurrent Q-learning for Dual Vasopressor Control
Authors:
Will Y. Zou,
Jean Feng,
Alexandre Kalimouttou,
Jennifer Yuntong Zhang,
Christopher W. Seymour,
Romain Pirracchio
Abstract:
Reinforcement learning (RL) applications in Clinical Decision Support Systems (CDSS) frequently encounter skepticism because models may recommend inoperable dosing decisions. We propose an end-to-end offline RL framework for dual vasopressor administration in Intensive Care Units (ICUs) that directly addresses this challenge through principled action space design. Our method integrates discrete, c…
▽ More
Reinforcement learning (RL) applications in Clinical Decision Support Systems (CDSS) frequently encounter skepticism because models may recommend inoperable dosing decisions. We propose an end-to-end offline RL framework for dual vasopressor administration in Intensive Care Units (ICUs) that directly addresses this challenge through principled action space design. Our method integrates discrete, continuous, and directional dosing strategies with conservative Q-learning and incorporates a novel recurrent modeling using a replay buffer to capture temporal dependencies in ICU time-series data. Our comparative analysis of norepinephrine dosing strategies across different action space formulations reveals that the designed action spaces improve interpretability and facilitate clinical adoption while preserving efficacy. Empirical results on eICU and MIMIC demonstrate that action space design profoundly influences learned behavioral policies. Compared with baselines, the proposed methods achieve more than 3x expected reward improvements, while aligning with established clinical protocols.
△ Less
Submitted 24 November, 2025; v1 submitted 1 October, 2025;
originally announced October 2025.
-
Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information
Authors:
Xiao Zhan,
Juan Carlos Carrillo,
William Seymour,
Jose Such
Abstract:
LLM-based Conversational AIs (CAIs), also known as GenAI chatbots, like ChatGPT, are increasingly used across various domains, but they pose privacy risks, as users may disclose personal information during their conversations with CAIs. Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes. However, a novel and particularly concerning type of malicious LLM appli…
▽ More
LLM-based Conversational AIs (CAIs), also known as GenAI chatbots, like ChatGPT, are increasingly used across various domains, but they pose privacy risks, as users may disclose personal information during their conversations with CAIs. Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes. However, a novel and particularly concerning type of malicious LLM application remains unexplored: an LLM-based CAI that is deliberately designed to extract personal information from users.
In this paper, we report on the malicious LLM-based CAIs that we created based on system prompts that used different strategies to encourage disclosures of personal information from users. We systematically investigate CAIs' ability to extract personal information from users during conversations by conducting a randomized-controlled trial with 502 participants. We assess the effectiveness of different malicious and benign CAIs to extract personal information from participants, and we analyze participants' perceptions after their interactions with the CAIs. Our findings reveal that malicious CAIs extract significantly more personal information than benign CAIs, with strategies based on the social nature of privacy being the most effective while minimizing perceived risks. This study underscores the privacy threats posed by this novel type of malicious LLM-based CAIs and provides actionable recommendations to guide future research and practice.
△ Less
Submitted 13 June, 2025;
originally announced June 2025.
-
Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
Authors:
David Rodriguez,
William Seymour,
Jose M. Del Alamo,
Jose Such
Abstract:
User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remai…
▽ More
User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI's usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, while customization tends to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation.
△ Less
Submitted 19 December, 2025; v1 submitted 3 February, 2025;
originally announced February 2025.
-
Speculating About Multi-user Conversational Interfaces and LLMs: What If Chatting Wasn't So Lonely?
Authors:
William Seymour,
Emilee Rader
Abstract:
The advent of LLMs means that CUIs are cool again, but what isn't so cool is that we're doomed to use them alone. The one user, one account, one device paradigm has dominated the design of CUIs and is not going away as new conversational technologies emerge. In this provocation we explore some of the technical, legal, and design difficulties that seem to make multi-user CUIs so difficult to implem…
▽ More
The advent of LLMs means that CUIs are cool again, but what isn't so cool is that we're doomed to use them alone. The one user, one account, one device paradigm has dominated the design of CUIs and is not going away as new conversational technologies emerge. In this provocation we explore some of the technical, legal, and design difficulties that seem to make multi-user CUIs so difficult to implement. Drawing inspiration from the ways that people manage messy group discussions, such as parliamentary and consensus-based paradigms, we show how LLM-based CUIs might be well suited to bridging the gap. With any luck, this might even result in everyone having to sit through fewer poorly run meetings and agonising group discussions - truly a laudable goal!
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Healthcare Voice AI Assistants: Factors Influencing Trust and Intention to Use
Authors:
Xiao Zhan,
Noura Abdi,
William Seymour,
Jose Such
Abstract:
AI assistants such as Alexa, Google Assistant, and Siri, are making their way into the healthcare sector, offering a convenient way for users to access different healthcare services. Trust is a vital factor in the uptake of healthcare services, but the factors affecting trust in voice assistants used for healthcare are under-explored and this specialist domain introduces additional requirements. T…
▽ More
AI assistants such as Alexa, Google Assistant, and Siri, are making their way into the healthcare sector, offering a convenient way for users to access different healthcare services. Trust is a vital factor in the uptake of healthcare services, but the factors affecting trust in voice assistants used for healthcare are under-explored and this specialist domain introduces additional requirements. This study explores the effects of different functional, personal, and risk factors on trust in and adoption of healthcare voice AI assistants (HVAs), generating a partial least squares structural model from a survey of 300 voice assistant users. Our results indicate that trust in HVAs can be significantly explained by functional factors (usefulness, content credibility, quality of service relative to a healthcare professional), together with security, and privacy risks and personal stance in technology. We also discuss differences in terms of trust between HVAs and general-purpose voice assistants as well as implications that are unique to HVAs.
△ Less
Submitted 11 January, 2024; v1 submitted 9 January, 2024;
originally announced January 2024.
-
Voice App Developer Experiences with Alexa and Google Assistant: Juggling Risks, Liability, and Security
Authors:
William Seymour,
Noura Abdi,
Kopo M. Ramokapane,
Jide Edu,
Guillermo Suarez-Tangil,
Jose Such
Abstract:
Voice applications (voice apps) are a key element in Voice Assistant ecosystems such as Amazon Alexa and Google Assistant, as they provide assistants with a wide range of capabilities that users can invoke with a voice command. Most voice apps, however, are developed by third parties - i.e., not by Amazon/Google - and they are included in the ecosystem through marketplaces akin to smartphone app s…
▽ More
Voice applications (voice apps) are a key element in Voice Assistant ecosystems such as Amazon Alexa and Google Assistant, as they provide assistants with a wide range of capabilities that users can invoke with a voice command. Most voice apps, however, are developed by third parties - i.e., not by Amazon/Google - and they are included in the ecosystem through marketplaces akin to smartphone app stores but with crucial differences, e.g., the voice app code is not hosted by the marketplace and is not run on the local device. Previous research has studied the security and privacy issues of voice apps in the wild, finding evidence of bad practices by voice app developers. However, developers' perspectives are yet to be explored.
In this paper, we report a qualitative study of the experiences of voice app developers and the challenges they face. Our findings suggest that: 1) developers face several risks due to liability pushed on to them by the more powerful voice assistant platforms, which are linked to negative privacy and security outcomes on voice assistant platforms; and 2) there are key issues around monetization, privacy, design, and testing rooted in problems with the voice app certification process. We discuss the implications of our results for voice app developers, platforms, regulators, and research on voice app development and certification.
△ Less
Submitted 15 November, 2023;
originally announced November 2023.
-
Who are CUIs Really For? Representation and Accessibility in the Conversational User Interface Literature
Authors:
William Seymour,
Xiao Zhan,
Mark Cote,
Jose Such
Abstract:
The theme for CUI 2023 is 'designing for inclusive conversation', but who are CUIs really designed for? The field has its roots in computer science, which has a long acknowledged diversity problem. Inspired by studies mapping out the diversity of the CHI and voice assistant literature, we set out to investigate how these issues have (or have not) shaped the CUI literature. To do this we reviewed t…
▽ More
The theme for CUI 2023 is 'designing for inclusive conversation', but who are CUIs really designed for? The field has its roots in computer science, which has a long acknowledged diversity problem. Inspired by studies mapping out the diversity of the CHI and voice assistant literature, we set out to investigate how these issues have (or have not) shaped the CUI literature. To do this we reviewed the 46 full-length research papers that have been published at CUI since its inception in 2019. After detailing the eight papers that engage with accessibility, social interaction, and performance of gender, we show that 90% of papers published at CUI with user studies recruit participants from Europe and North America (or do not specify). To complement existing work in the community towards diversity we discuss the factors that have contributed to the current status quo, and offer some initial suggestions as to how we as a CUI community can continue to improve. We hope that this will form the beginning of a wider discussion at the conference.
△ Less
Submitted 8 June, 2023;
originally announced June 2023.
-
Why Are Conversational Assistants Still Black Boxes? The Case For Transparency
Authors:
Trung Dong Huynh,
William Seymour,
Luc Moreau,
Jose Such
Abstract:
Much has been written about privacy in the context of conversational and voice assistants. Yet, there have been remarkably few developments in terms of the actual privacy offered by these devices. But how much of this is due to the technical and design limitations of speech as an interaction modality? In this paper, we set out to reframe the discussion on why commercial conversational assistants d…
▽ More
Much has been written about privacy in the context of conversational and voice assistants. Yet, there have been remarkably few developments in terms of the actual privacy offered by these devices. But how much of this is due to the technical and design limitations of speech as an interaction modality? In this paper, we set out to reframe the discussion on why commercial conversational assistants do not offer meaningful privacy and transparency by demonstrating how they \emph{could}. By instrumenting the open-source voice assistant Mycroft to capture audit trails for data access, we demonstrate how such functionality could be integrated into big players in the sector like Alexa and Google Assistant. We show that this problem can be solved with existing technology and open standards and is thus fundamentally a business decision rather than a technical limitation.
△ Less
Submitted 8 June, 2023;
originally announced June 2023.
-
Predicting Privacy Preferences for Smart Devices as Norms
Authors:
Marc Serramia,
William Seymour,
Natalia Criado,
Michael Luck
Abstract:
Smart devices, such as smart speakers, are becoming ubiquitous, and users expect these devices to act in accordance with their preferences. In particular, since these devices gather and manage personal data, users expect them to adhere to their privacy preferences. However, the current approach of gathering these preferences consists in asking the users directly, which usually triggers automatic r…
▽ More
Smart devices, such as smart speakers, are becoming ubiquitous, and users expect these devices to act in accordance with their preferences. In particular, since these devices gather and manage personal data, users expect them to adhere to their privacy preferences. However, the current approach of gathering these preferences consists in asking the users directly, which usually triggers automatic responses failing to capture their true preferences. In response, in this paper we present a collaborative filtering approach to predict user preferences as norms. These preference predictions can be readily adopted or can serve to assist users in determining their own preferences. Using a dataset of privacy preferences of smart assistant users, we test the accuracy of our predictions.
△ Less
Submitted 21 February, 2023;
originally announced February 2023.
-
Legal Obligation and Ethical Best Practice: Towards Meaningful Verbal Consent for Voice Assistants
Authors:
William Seymour,
Mark Cote,
Jose Such
Abstract:
To improve user experience, Alexa now allows users to consent to data sharing via voice rather than directing them to the companion smartphone app. While verbal consent mechanisms for voice assistants (VAs) can increase usability, they can also undermine principles core to informed consent. We conducted a Delphi study with experts from academia, industry, and the public sector on requirements for…
▽ More
To improve user experience, Alexa now allows users to consent to data sharing via voice rather than directing them to the companion smartphone app. While verbal consent mechanisms for voice assistants (VAs) can increase usability, they can also undermine principles core to informed consent. We conducted a Delphi study with experts from academia, industry, and the public sector on requirements for verbal consent in VAs. Candidate requirements were drawn from the literature, regulations, and research ethics guidelines that participants rated based on their relevance to the consent process, actionability by platforms, and usability by end-users, discussing their reasoning as the study progressed. We highlight key areas of (dis)agreement between experts, deriving recommendations for regulators, skill developers, and VA platforms towards crafting meaningful verbal consent mechanisms. Key themes include approaching permissions according to the user's ability to opt-out, minimising consent decisions, and ensuring platforms follow established consent principles.
△ Less
Submitted 19 January, 2023;
originally announced January 2023.
-
A Design Philosophy for Agents in the Smart Home
Authors:
William Seymour
Abstract:
The home is often the most private space in people's lives, and not one in which they expect to be surveilled. However, today's market for smart home devices has quickly evolved to include products that monitor, automate, and present themselves as human. After documenting some of the more unusual emergent problems with contemporary devices, this body of work seeks to develop a design philosophy fo…
▽ More
The home is often the most private space in people's lives, and not one in which they expect to be surveilled. However, today's market for smart home devices has quickly evolved to include products that monitor, automate, and present themselves as human. After documenting some of the more unusual emergent problems with contemporary devices, this body of work seeks to develop a design philosophy for intelligent agents in the smart home that can act as an alternative to the ways that these devices are currently built. This is then applied to the design of privacy empowering technologies, representing the first steps from the devices of the present towards a more respectful future.
△ Less
Submitted 30 November, 2022;
originally announced November 2022.
-
Ignorance is Bliss? The Effect of Explanations on Perceptions of Voice Assistants
Authors:
William Seymour,
Jose Such
Abstract:
Voice assistants offer a convenient and hands-free way of accessing computing in the home, but a key problem with speech as an interaction modality is how to scaffold accurate mental models of voice assistants, a task complicated by privacy and security concerns. We present the results of a survey of voice assistant users (n=1314) measuring trust, security, and privacy perceptions of voice assista…
▽ More
Voice assistants offer a convenient and hands-free way of accessing computing in the home, but a key problem with speech as an interaction modality is how to scaffold accurate mental models of voice assistants, a task complicated by privacy and security concerns. We present the results of a survey of voice assistant users (n=1314) measuring trust, security, and privacy perceptions of voice assistants with varying levels of online functionality explained in different ways. We then asked participants to re-explain how these voice assistants worked, showing that while privacy explanations relieved privacy concerns, trust concerns were exacerbated by trust explanations. Participants' trust, privacy, and security perceptions also distinguished between first party online functionality from the voice assistant vendor and third party online functionality from other developers, and trust in vendors appeared to operate independently from device explanations. Our findings point to the use of analogies to guide users, targeting trust and privacy concerns, key improvements required from manufacturers, and implications for competition in the sector.
△ Less
Submitted 23 November, 2022;
originally announced November 2022.
-
RISE: Robust Individualized Decision Learning with Sensitive Variables
Authors:
Xiaoqing Tan,
Zhengling Qi,
Christopher W. Seymour,
Lu Tang
Abstract:
This paper introduces RISE, a robust individualized decision learning framework with sensitive variables, where sensitive variables are collectible data and important to the intervention decision, but their inclusion in decision making is prohibited due to reasons such as delayed availability or fairness concerns. A naive baseline is to ignore these sensitive variables in learning decision rules,…
▽ More
This paper introduces RISE, a robust individualized decision learning framework with sensitive variables, where sensitive variables are collectible data and important to the intervention decision, but their inclusion in decision making is prohibited due to reasons such as delayed availability or fairness concerns. A naive baseline is to ignore these sensitive variables in learning decision rules, leading to significant uncertainty and bias. To address this, we propose a decision learning framework to incorporate sensitive variables during offline training but not include them in the input of the learned decision rule during model deployment. Specifically, from a causal perspective, the proposed framework intends to improve the worst-case outcomes of individuals caused by sensitive variables that are unavailable at the time of decision. Unlike most existing literature that uses mean-optimal objectives, we propose a robust learning framework by finding a newly defined quantile- or infimum-optimal decision rule. The reliable performance of the proposed method is demonstrated through synthetic experiments and three real-world applications.
△ Less
Submitted 11 November, 2022;
originally announced November 2022.
-
A Systematic Review of Ethical Concerns with Voice Assistants
Authors:
William Seymour,
Xiao Zhan,
Mark Cote,
Jose Such
Abstract:
Since Siri's release in 2011 there have been a growing number of AI-driven domestic voice assistants that are increasingly being integrated into devices such as smartphones and TVs. But as their presence has expanded, a range of ethical concerns has been identified around the use of voice assistants, such as the privacy implications of having devices that are always listening and the ways that the…
▽ More
Since Siri's release in 2011 there have been a growing number of AI-driven domestic voice assistants that are increasingly being integrated into devices such as smartphones and TVs. But as their presence has expanded, a range of ethical concerns has been identified around the use of voice assistants, such as the privacy implications of having devices that are always listening and the ways that these devices are integrated into the existing social order of the home. This has created a burgeoning area of research across a range of fields including computer science, social science, and psychology. This paper takes stock of the foundations and frontiers of this work through a systematic literature review of 117 papers on ethical concerns with voice assistants. In addition to analysis of nine specific areas of concern, the review measures the distribution of methods and participant demographics across the literature. We show how some concerns, such as privacy, are operationalized to a much greater extent than others like accessibility, and how study participants are overwhelmingly drawn from a small handful of Western nations. In so doing we hope to provide an outline of the rich tapestry of work around these concerns and highlight areas where current research efforts are lacking.
△ Less
Submitted 23 June, 2023; v1 submitted 8 November, 2022;
originally announced November 2022.
-
When It's Not Worth the Paper It's Written On: A Provocation on the Certification of Skills in the Alexa and Google Assistant Ecosystems
Authors:
William Seymour,
Mark Cote,
Jose Such
Abstract:
The increasing reach and functionality of voice assistants has allowed them to become a general-purpose platform for tasks like playing music, accessing information, and controlling smart home devices. In order to maintain the quality of third-party skills and to protect children and other members of the public from inappropriate or malicious skills, platform providers have developed content polic…
▽ More
The increasing reach and functionality of voice assistants has allowed them to become a general-purpose platform for tasks like playing music, accessing information, and controlling smart home devices. In order to maintain the quality of third-party skills and to protect children and other members of the public from inappropriate or malicious skills, platform providers have developed content policies and certification procedures that skills must undergo prior to public release. Unfortunately, research suggests that these measures have been ineffective at curating voice assistant platforms, with documented instances of skills with significant security and privacy problems. This provocation paper outlines how the underlying architectures of these platforms had turned skill certification into a seemingly intractable problem, as well as how current certification methods fall short of their full potential. We present a roadmap for improving the state of skill certification on contemporary voice assistant platforms, including research directions and actions that need to be taken by platform vendors. Promoting this change in domestic voice assistants is especially important, as developers of commercial and industrial assistants or other similar contexts increasingly look to these devices for norms and conventions.
△ Less
Submitted 22 June, 2022;
originally announced June 2022.
-
Can you meaningfully consent in eight seconds? Identifying Ethical Issues with Verbal Consent for Voice Assistants
Authors:
William Seymour,
Mark Cote,
Jose Such
Abstract:
Determining how voice assistants should broker consent to share data with third party software has proven to be a complex problem. Devices often require users to switch to companion smartphone apps in order to navigate permissions menus for their otherwise hands-free voice assistant. More in line with smartphone app stores, Alexa now offers "voice-forward consent", allowing users to grant skills a…
▽ More
Determining how voice assistants should broker consent to share data with third party software has proven to be a complex problem. Devices often require users to switch to companion smartphone apps in order to navigate permissions menus for their otherwise hands-free voice assistant. More in line with smartphone app stores, Alexa now offers "voice-forward consent", allowing users to grant skills access to personal data mid-conversation using speech. While more usable and convenient than opening a companion app, asking for consent 'on the fly' can undermine several concepts core to the informed consent process. The intangible nature of voice interfaces further blurs the boundary between parts of an interaction controlled by third-party developers from the underlying platforms. This provocation paper highlights key issues with current verbal consent implementations, outlines directions for potential solutions, and presents five open questions to the research community. In so doing, we hope to help shape the development of usable and effective verbal consent for voice assistants and similar conversational user interfaces.
△ Less
Submitted 22 June, 2022;
originally announced June 2022.
-
Respect as a Lens for the Design of AI Systems
Authors:
William Seymour,
Max Van Kleek,
Reuben Binns,
Dave Murray-Rust
Abstract:
Critical examinations of AI systems often apply principles such as fairness, justice, accountability, and safety, which is reflected in AI regulations such as the EU AI Act. Are such principles sufficient to promote the design of systems that support human flourishing? Even if a system is in some sense fair, just, or 'safe', it can nonetheless be exploitative, coercive, inconvenient, or otherwise…
▽ More
Critical examinations of AI systems often apply principles such as fairness, justice, accountability, and safety, which is reflected in AI regulations such as the EU AI Act. Are such principles sufficient to promote the design of systems that support human flourishing? Even if a system is in some sense fair, just, or 'safe', it can nonetheless be exploitative, coercive, inconvenient, or otherwise conflict with cultural, individual, or social values. This paper proposes a dimension of interactional ethics thus far overlooked: the ways AI systems should treat human beings. For this purpose, we explore the philosophical concept of respect: if respect is something everyone needs and deserves, shouldn't technology aim to be respectful? Despite its intuitive simplicity, respect in philosophy is a complex concept with many disparate senses. Like fairness or justice, respect can characterise how people deserve to be treated; but rather than relating primarily to the distribution of benefits or punishments, respect relates to how people regard one another, and how this translates to perception, treatment, and behaviour. We explore respect broadly across several literatures, synthesising perspectives on respect from Kantian, post-Kantian, dramaturgical, and agential realist design perspectives with a goal of drawing together a view of what respect could mean for AI. In so doing, we identify ways that respect may guide us towards more sociable artefacts that ethically and inclusively honour and recognise humans using the rich social language that we have evolved to interact with one another every day.
△ Less
Submitted 15 June, 2022;
originally announced June 2022.
-
Consent on the Fly: Developing Ethical Verbal Consent for Voice Assistants
Authors:
William Seymour,
Mark Cote,
Jose Such
Abstract:
Determining how voice assistants should broker consent to share data with third party software has proven to be a complex problem. Devices often require users to switch to companion smartphone apps in order to navigate permissions menus for their otherwise hands-free voice assistant. More in line with smartphone app stores, Alexa now offers "voice-forward consent", allowing users to grant skills a…
▽ More
Determining how voice assistants should broker consent to share data with third party software has proven to be a complex problem. Devices often require users to switch to companion smartphone apps in order to navigate permissions menus for their otherwise hands-free voice assistant. More in line with smartphone app stores, Alexa now offers "voice-forward consent", allowing users to grant skills access to personal data mid-conversation using speech.
While more usable and convenient than opening a companion app, asking for consent 'on the fly' can undermine several concepts core to the informed consent process. The intangible nature of voice interfaces further blurs the boundary between parts of an interaction controlled by third-party developers from the underlying platforms. We outline a research agenda towards usable and effective voice-based consent to address the problems with brokering consent verbally, including our own work drawing on the GDPR and work on consent in Ubicomp.
△ Less
Submitted 21 April, 2022;
originally announced April 2022.
-
Exploring Interactions Between Trust, Anthropomorphism, and Relationship Development in Voice Assistants
Authors:
William Seymour,
Max Van Kleek
Abstract:
Modern conversational agents such as Alexa and Google Assistant represent significant progress in speech recognition, natural language processing, and speech synthesis. But as these agents have grown more realistic, concerns have been raised over how their social nature might unconsciously shape our interactions with them. Through a survey of 500 voice assistant users, we explore whether users' re…
▽ More
Modern conversational agents such as Alexa and Google Assistant represent significant progress in speech recognition, natural language processing, and speech synthesis. But as these agents have grown more realistic, concerns have been raised over how their social nature might unconsciously shape our interactions with them. Through a survey of 500 voice assistant users, we explore whether users' relationships with their voice assistants can be quantified using the same metrics as social, interpersonal relationships; as well as if this correlates with how much they trust their devices and the extent to which they anthropomorphise them. Using Knapp's staircase model of human relationships, we find that not only can human-device interactions be modelled in this way, but also that relationship development with voice assistants correlates with increased trust and anthropomorphism.
△ Less
Submitted 4 August, 2021;
originally announced August 2021.
-
Strangers in the Room: Unpacking Perceptions of 'Smartness' and Related Ethical Concerns in the Home
Authors:
William Seymour,
Reuben Binns,
Petr Slovak,
Max Van Kleek,
Nigel Shadbolt
Abstract:
The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in gen…
▽ More
The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in general. This paper attempts to bridge these approaches by asking what features of smart devices users consider as rendering them 'smart' and how these relate to ethical concerns. Through a multimethod investigation including surveys with smart device users (n=120) and semi-structured interviews (n=15), we identify and describe eight types of smartness and explore how they engender a variety of ethical concerns including privacy, autonomy, and disruption of the social order. We argue that this middle ground, between concerns arising from particular devices and more abstract ethical concepts, can better anticipate potential ethical concerns regarding smart devices.
△ Less
Submitted 1 May, 2020;
originally announced May 2020.
-
Does Siri Have a Soul? Exploring Voice Assistants Through Shinto Design Fictions
Authors:
William Seymour,
Max Van Kleek
Abstract:
It can be difficult to critically reflect on technology that has become part of everyday rituals and routines. To combat this, speculative and fictional approaches have previously been used by HCI to decontextualise the familiar and imagine alternatives. In this work we turn to Japanese Shinto narratives as a way to defamiliarise voice assistants, inspired by the similarities between how assistant…
▽ More
It can be difficult to critically reflect on technology that has become part of everyday rituals and routines. To combat this, speculative and fictional approaches have previously been used by HCI to decontextualise the familiar and imagine alternatives. In this work we turn to Japanese Shinto narratives as a way to defamiliarise voice assistants, inspired by the similarities between how assistants appear to 'inhabit' objects similarly to kami. Describing an alternate future where assistant presences live inside objects, this approach foregrounds some of the phenomenological quirks that can otherwise easily become lost. Divorced from the reality of daily life, this approach allows us to reevaluate some of the common interactions and design patterns that are common in the virtual assistants of the present.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Informing the Design of Privacy-Empowering Tools for the Connected Home
Authors:
William Seymour,
Martin J. Kraemer,
Reuben Binns,
Max Van Kleek
Abstract:
Connected devices in the home represent a potentially grave new privacy threat due to their unfettered access to the most personal spaces in people's lives. Prior work has shown that despite concerns about such devices, people often lack sufficient awareness, understanding, or means of taking effective action. To explore the potential for new tools that support such needs directly we developed Are…
▽ More
Connected devices in the home represent a potentially grave new privacy threat due to their unfettered access to the most personal spaces in people's lives. Prior work has shown that despite concerns about such devices, people often lack sufficient awareness, understanding, or means of taking effective action. To explore the potential for new tools that support such needs directly we developed Aretha, a privacy assistant technology probe that combines a network disaggregator, personal tutor, and firewall, to empower end-users with both the knowledge and mechanisms to control disclosures from their homes. We deployed Aretha in three households over six weeks, with the aim of understanding how this combination of capabilities might enable users to gain awareness of data disclosures by their devices, form educated privacy preferences, and to block unwanted data flows. The probe, with its novel affordances-and its limitations-prompted users to co-adapt, finding new control mechanisms and suggesting new approaches to address the challenge of regaining privacy in the connected home.
△ Less
Submitted 24 January, 2020;
originally announced January 2020.
-
'I Just Want to Hack Myself to Not Get Distracted': Evaluating Design Interventions for Self-Control on Facebook
Authors:
Ulrik Lyngs,
Kai Lukoff,
Petr Slovak,
William Seymour,
Helena Webb,
Marina Jirotka,
Jun Zhao,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Beyond being the world's largest social network, Facebook is for many also one of its greatest sources of digital distraction. For students, problematic use has been associated with negative effects on academic achievement and general wellbeing. To understand what strategies could help users regain control, we investigated how simple interventions to the Facebook UI affect behaviour and perceived…
▽ More
Beyond being the world's largest social network, Facebook is for many also one of its greatest sources of digital distraction. For students, problematic use has been associated with negative effects on academic achievement and general wellbeing. To understand what strategies could help users regain control, we investigated how simple interventions to the Facebook UI affect behaviour and perceived control. We assigned 58 university students to one of three interventions: goal reminders, removed newsfeed, or white background (control). We logged use for 6 weeks, applied interventions in the middle weeks, and administered fortnightly surveys. Both goal reminders and removed newsfeed helped participants stay on task and avoid distraction. However, goal reminders were often annoying, and removing the newsfeed made some fear missing out on information. Our findings point to future interventions such as controls for adjusting types and amount of available information, and flexible blocking which matches individual definitions of 'distraction'.
△ Less
Submitted 20 May, 2020; v1 submitted 13 January, 2020;
originally announced January 2020.
-
Informing The Future of Data Protection in Smart Homes
Authors:
Martin J Kraemer,
William Seymour,
Reuben Binns,
Max Van Kleek,
Ivan Flechais
Abstract:
Recent changes to data protection regulation, particularly in Europe, are changing the design landscape for smart devices, requiring new design techniques to ensure that devices are able to adequately protect users' data. A particularly interesting space in which to explore and address these challenges is the smart home, which presents a multitude of difficult social and technical problems in an i…
▽ More
Recent changes to data protection regulation, particularly in Europe, are changing the design landscape for smart devices, requiring new design techniques to ensure that devices are able to adequately protect users' data. A particularly interesting space in which to explore and address these challenges is the smart home, which presents a multitude of difficult social and technical problems in an intimate and highly private context. This position paper outlines the motivation and research approach of a new project aiming to inform the future of data protection by design and by default in smart homes through a combination of ethnography and speculative design.
△ Less
Submitted 17 June, 2019;
originally announced October 2019.