-
Calming Robot Pitches? Exploring the Influence of Robot Voice Pitch on Children's Stress Levels
Authors:
Nina G. M. van Roij,
Emilia I. Barakova,
Briana Isaila,
Aoju Chen
Abstract:
This study examined whether variations in robot speech pitch influence children's stress levels during a robot-guided game. Although lower-pitched voices have been shown to facilitate stress regulation in human communication, it remains unclear whether this effect generalizes to synthetic voices in child-robot interactions. Twenty-seven Dutch children aged 8-12 years were randomly assigned to inte…
▽ More
This study examined whether variations in robot speech pitch influence children's stress levels during a robot-guided game. Although lower-pitched voices have been shown to facilitate stress regulation in human communication, it remains unclear whether this effect generalizes to synthetic voices in child-robot interactions. Twenty-seven Dutch children aged 8-12 years were randomly assigned to interact with a Zenbo Junior II robot using either a lower-pitched or a higher-pitched voice. The interaction consisted of an introduction followed by a timed LEGO-building game. Stress levels, measured with an adapted version of CAM-S, increased during the game, confirming the stress-inducing nature of the task. No differences emerged between pitch conditions. These findings suggest that the benefits of lower pitch in reducing stress may not directly translate to child-robot interactions. Possible explanations include children's developing sensitivity to emotional tone, mismatches between the robot's voice and appearance, or the use of fixed pitch changes that sound unnatural, since real speech varies dynamically across multiple dimensions. Future research examining combinations of prosodic cues (beyond pitch alone) could provide further insights and help inform robot voice design for effective stress regulation support for children.
△ Less
Submitted 20 August, 2026;
originally announced August 2026.
-
Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems
Authors:
Sheng Li,
Jing Li,
Felix Schijve,
Jun Hu,
Emilia Barakova
Abstract:
Automatic speech recognition (ASR) has become a critical component of modern robotic systems because it is one of the most natural and intuitive ways for humans to interact with robots. A commonly used method is to directly use API services online. But is that all we can do? This article provides an overview of how ASR technologies are integrated into various intelligent robots and machines. We di…
▽ More
Automatic speech recognition (ASR) has become a critical component of modern robotic systems because it is one of the most natural and intuitive ways for humans to interact with robots. A commonly used method is to directly use API services online. But is that all we can do? This article provides an overview of how ASR technologies are integrated into various intelligent robots and machines. We discuss the evolution of speech recognition from established approaches to state-of-the-art deep learning models, such as OpenAI's Whisper. We also list large-scale datasets and open source toolkits that have been widely used in both industry and academia. We structure the survey around ASR model families, deployment strategies in robotics (especially ROS-based, cloud-based, and hybrid solutions), and several real-world robotic platforms. Finally, we outline the challenges of deploying robust speech recognition in robots and discuss future directions, including multimodal interaction in diverse and dynamic environments. This paper can help social robotics researchers better navigate the emerging domain of language-based natural human-robot interaction.
△ Less
Submitted 13 July, 2026;
originally announced July 2026.
-
Beyond Explainable AI (XAI): An Overdue Paradigm Shift and Post-XAI Research Directions
Authors:
Saleh Afroogh,
Syed Ishtiaque Ahmed,
Petra Ahrweiler,
David Alvarez-Melis,
Mansur Maturidi Arief,
Emilia Barakova,
Falco J. Bargagli-Stoffi,
Erdem Biyik,
Hanjie Chen,
Xiang 'Anthony' Chen,
Robert Alan Clements,
Keeley Crockett,
Amit Dhurandhar,
Fethiye Irmak Dogan,
Mollie Dollinger,
Motahhare Eslami,
Aldo A Faisal,
Arya Farahi,
Melanie F. Pradier,
Saadia Gabriel,
Diego Garcia-Olano,
Marzyeh Ghassemi,
Shaona Ghosh,
Hatice Gunes,
Ehsan Hajiramezanali
, et al. (24 additional authors not shown)
Abstract:
This study provides a cross-disciplinary examination of Explainable Artificial Intelligence (XAI) approaches-focusing on deep neural networks (DNNs) and large language models (LLMs)-and identifies empirical and conceptual limitations in current XAI. We discuss critical symptoms that stem from deeper root causes (i.e., two paradoxes, two conceptual confusions, and five false assumptions). These fun…
▽ More
This study provides a cross-disciplinary examination of Explainable Artificial Intelligence (XAI) approaches-focusing on deep neural networks (DNNs) and large language models (LLMs)-and identifies empirical and conceptual limitations in current XAI. We discuss critical symptoms that stem from deeper root causes (i.e., two paradoxes, two conceptual confusions, and five false assumptions). These fundamental problems within the current XAI research field reveal three insights: experimentally, XAI exhibits significant flaws; conceptually, it is paradoxical; and pragmatically, further attempts to reform the paradoxical XAI might exacerbate its confusion-demanding fundamental shifts and new research directions. To move beyond XAI's limitations, we propose a four-pronged synthesized paradigm shift toward reliable and certified AI development. These four components include: verification-focused Interactive AI (IAI) to establish scientific community protocols for certifying AI system performance rather than attempting post-hoc explanations, AI Epistemology for rigorous scientific foundations, User-Sensible AI to create context-aware systems tailored to specific user communities, and Model-Centered Interpretability for faithful technical analysis-together offering comprehensive post-XAI research directions.
△ Less
Submitted 25 May, 2026; v1 submitted 27 February, 2026;
originally announced February 2026.
-
Towards Emotion Co-regulation with LLM-powered Socially Assistive Robots: Integrating LLM Prompts and Robotic Behaviors to Support Parent-Neurodivergent Child Dyads
Authors:
Jing Li,
Felix Schijve,
Sheng Li,
Yuye Yang,
Jun Hu,
Emilia Barakova
Abstract:
Socially Assistive Robotics (SAR) has shown promise in supporting emotion regulation for neurodivergent children. Recently, there has been increasing interest in leveraging advanced technologies to assist parents in co-regulating emotions with their children. However, limited research has explored the integration of large language models (LLMs) with SAR to facilitate emotion co-regulation between…
▽ More
Socially Assistive Robotics (SAR) has shown promise in supporting emotion regulation for neurodivergent children. Recently, there has been increasing interest in leveraging advanced technologies to assist parents in co-regulating emotions with their children. However, limited research has explored the integration of large language models (LLMs) with SAR to facilitate emotion co-regulation between parents and children with neurodevelopmental disorders. To address this gap, we developed an LLM-powered social robot by deploying a speech communication module on the MiRo-E robotic platform. This supervised autonomous system integrates LLM prompts and robotic behaviors to deliver tailored interventions for both parents and neurodivergent children. Pilot tests were conducted with two parent-child dyads, followed by a qualitative analysis. The findings reveal MiRo-E's positive impacts on interaction dynamics and its potential to facilitate emotion regulation, along with identified design and technical challenges. Based on these insights, we provide design implications to advance the future development of LLM-powered SAR for mental health applications.
△ Less
Submitted 14 July, 2025;
originally announced July 2025.
-
Petting Pen for Stress Awareness and Management in Children
Authors:
Jing Li,
Pinhao Wang,
Emilia Barakova,
Jun Hu
Abstract:
We found that children in elementary school often experience stress during task performance. Limited coping skills and lack of stress awareness restrict children's ability to manage their stress. Many designs and studies have proposed different stress detection and intervention solutions. Still, they often overlook the potential of enhancing everyday objects and actively sensing stress-related beh…
▽ More
We found that children in elementary school often experience stress during task performance. Limited coping skills and lack of stress awareness restrict children's ability to manage their stress. Many designs and studies have proposed different stress detection and intervention solutions. Still, they often overlook the potential of enhancing everyday objects and actively sensing stress-related behavioral data during human-product interaction. Therefore, we propose Petting pen as an interactive robotic object for children to manage their stress during task performance. It detects and validates stress and further intervenes in stress during a process of natural writing and relaxation interactions. The design is an iteration based on our previous research results of a stress-aware pen, enhanced with tactile needs, robotic interaction, and integration of behavioral and bio-sensing capabilities. Petting pen is supposed to bridge the gap between robots and everyday objects in mental health applications for children.
△ Less
Submitted 28 March, 2025; v1 submitted 18 March, 2025;
originally announced March 2025.
-
The Chef's Hat Simulation Environment for Reinforcement-Learning-Based Agents
Authors:
Pablo Barros,
Anne C. Bloem,
Inge M. Hootsmans,
Lena M. Opheij,
Romain H. A. Toebosch,
Emilia Barakova,
Alessandra Sciutti
Abstract:
To achieve social interactions within Human-Robot Interaction (HRI) environments is a very challenging task. Most of the current research focuses on Wizard-of-Oz approaches, which neglect the recent development of intelligent robots. On the other hand, real-world scenarios usually do not provide the necessary control and reproducibility which are needed for learning algorithms. In this paper, we p…
▽ More
To achieve social interactions within Human-Robot Interaction (HRI) environments is a very challenging task. Most of the current research focuses on Wizard-of-Oz approaches, which neglect the recent development of intelligent robots. On the other hand, real-world scenarios usually do not provide the necessary control and reproducibility which are needed for learning algorithms. In this paper, we propose a virtual simulation environment that implements the Chef's Hat card game, designed to be used in HRI scenarios, to provide a controllable and reproducible scenario for reinforcement-learning algorithms.
△ Less
Submitted 12 March, 2020;
originally announced March 2020.
-
It's Food Fight! Introducing the Chef's Hat Card Game for Affective-Aware HRI
Authors:
Pablo Barros,
Alessandra Sciutti,
Anne C. Bloem,
Inge M. Hootsmans,
Lena M. Opheij,
Romain H. A. Toebosch,
Emilia Barakova
Abstract:
Emotional expressions and their changes during an interaction affect heavily how we perceive and behave towards other persons. To design an HRI scenario that makes possible to observe, understand, and model affective interactions and generate the appropriate responses or initiations of a robot is a very challenging task. In this paper, we report our efforts in designing such a scenario, and to pro…
▽ More
Emotional expressions and their changes during an interaction affect heavily how we perceive and behave towards other persons. To design an HRI scenario that makes possible to observe, understand, and model affective interactions and generate the appropriate responses or initiations of a robot is a very challenging task. In this paper, we report our efforts in designing such a scenario, and to propose a modeling strategy of affective interaction by artificial intelligence deployed in autonomous robots. Overall, we present a novel HRI game scenario that was designed to comply with the specific requirements that will allow us to develop the next wave of affective-aware social robots that provide adequate emotional responses.
△ Less
Submitted 9 March, 2020; v1 submitted 25 February, 2020;
originally announced February 2020.
-
A Deep Neural Model Of Emotion Appraisal
Authors:
Pablo Barros,
Emilia Barakova,
Stefan Wermter
Abstract:
Emotional concepts play a huge role in our daily life since they take part into many cognitive processes: from the perception of the environment around us to different learning processes and natural communication. Social robots need to communicate with humans, which increased also the popularity of affective embodied models that adopt different emotional concepts in many everyday tasks. However, t…
▽ More
Emotional concepts play a huge role in our daily life since they take part into many cognitive processes: from the perception of the environment around us to different learning processes and natural communication. Social robots need to communicate with humans, which increased also the popularity of affective embodied models that adopt different emotional concepts in many everyday tasks. However, there is still a gap between the development of these solutions and the integration and development of a complex emotion appraisal system, which is much necessary for true social robots. In this paper, we propose a deep neural model which is designed in the light of different aspects of developmental learning of emotional concepts to provide an integrated solution for internal and external emotion appraisal. We evaluate the performance of the proposed model with different challenging corpora and compare it with state-of-the-art models for external emotion appraisal. To extend the evaluation of the proposed model, we designed and collected a novel dataset based on a Human-Robot Interaction (HRI) scenario. We deployed the model in an iCub robot and evaluated the capability of the robot to learn and describe the affective behavior of different persons based on observation. The performed experiments demonstrate that the proposed model is competitive with the state of the art in describing emotion behavior in general. In addition, it is able to generate internal emotional concepts that evolve through time: it continuously forms and updates the formed emotional concepts, which is a step towards creating an emotional appraisal model grounded in the robot experiences.
△ Less
Submitted 1 August, 2018;
originally announced August 2018.
-
Left/Right Hand Segmentation in Egocentric Videos
Authors:
Alejandro Betancourt,
Pietro Morerio,
Emilia Barakova,
Lucio Marcenaro,
Matthias Rauterberg,
Carlo Regazzoni
Abstract:
Wearable cameras allow people to record their daily activities from a user-centered (First Person Vision) perspective. Due to their favorable location, wearable cameras frequently capture the hands of the user, and may thus represent a promising user-machine interaction tool for different applications. Existent First Person Vision methods handle hand segmentation as a background-foreground problem…
▽ More
Wearable cameras allow people to record their daily activities from a user-centered (First Person Vision) perspective. Due to their favorable location, wearable cameras frequently capture the hands of the user, and may thus represent a promising user-machine interaction tool for different applications. Existent First Person Vision methods handle hand segmentation as a background-foreground problem, ignoring two important facts: i) hands are not a single "skin-like" moving element, but a pair of interacting cooperative entities, ii) close hand interactions may lead to hand-to-hand occlusions and, as a consequence, create a single hand-like segment. These facts complicate a proper understanding of hand movements and interactions. Our approach extends traditional background-foreground strategies, by including a hand-identification step (left-right) based on a Maxwell distribution of angle and position. Hand-to-hand occlusions are addressed by exploiting temporal superpixels. The experimental results show that, in addition to a reliable left/right hand-segmentation, our approach considerably improves the traditional background-foreground hand-segmentation.
△ Less
Submitted 21 July, 2016;
originally announced July 2016.
-
Unsupervised Understanding of Location and Illumination Changes in Egocentric Videos
Authors:
Alejandro Betancourt,
Natalia Díaz-Rodríguez,
Emilia Barakova,
Lucio Marcenaro,
Matthias Rauterberg,
Carlo Regazzoni
Abstract:
Wearable cameras stand out as one of the most promising devices for the upcoming years, and as a consequence, the demand of computer algorithms to automatically understand the videos recorded with them is increasing quickly. An automatic understanding of these videos is not an easy task, and its mobile nature implies important challenges to be faced, such as the changing light conditions and the u…
▽ More
Wearable cameras stand out as one of the most promising devices for the upcoming years, and as a consequence, the demand of computer algorithms to automatically understand the videos recorded with them is increasing quickly. An automatic understanding of these videos is not an easy task, and its mobile nature implies important challenges to be faced, such as the changing light conditions and the unrestricted locations recorded. This paper proposes an unsupervised strategy based on global features and manifold learning to endow wearable cameras with contextual information regarding the light conditions and the location captured. Results show that non-linear manifold methods can capture contextual patterns from global features without compromising large computational resources. The proposed strategy is used, as an application case, as a switching mechanism to improve the hand-detection problem in egocentric videos.
△ Less
Submitted 27 March, 2017; v1 submitted 30 March, 2016;
originally announced March 2016.