Showing 1–50 of 57 results for author: Tsukada, M

Toward Blockage-Resilient 6G-V2X Connectivity: Semi-Distributed Bandit with Dynamic Arm Set for mmWave HetNets

Authors: Weiqi Chi, Bo Qian, Hanlin Wu, Donghui Li, Haibo Zhou, Manabu Tsukada

Abstract: The vision for 6G vehicle-to-everything (V2X) communications demands reliable, adaptive connectivity for fully autonomous driving across complex dynamic environments. Millimeter-wave (mmWave) user association (UA) in heterogeneous vehicular networks presents a particularly demanding instance of this problem, where dynamic blockages and rapid channel variations continuously undermine the stationary… ▽ More The vision for 6G vehicle-to-everything (V2X) communications demands reliable, adaptive connectivity for fully autonomous driving across complex dynamic environments. Millimeter-wave (mmWave) user association (UA) in heterogeneous vehicular networks presents a particularly demanding instance of this problem, where dynamic blockages and rapid channel variations continuously undermine the stationary reward assumptions of traditional multi-armed bandit (MAB) frameworks. This paper proposes a fully distributed blockage-aware non-stationary dynamic bandit algorithm (BAND) and its semi-distributed extension S-BAND for cooperative learning across vehicles. Blockage prediction is incorporated into the change-detection (CD) mechanism to suppress false alarms, while a dynamic base station (BS) set management scheme balances exploration and exploitation across large-scale BS deployments without requiring centralized channel state information (CSI) acquisition or offline training. In S-BAND, vehicles accumulate BS reward estimates as local knowledge and periodically upload them to the macro base station (MBS), which aggregates them into cluster-based central knowledge. A trajectory-aligned knowledge (TAK) region is proposed to capture the spatial correlation of mmWave channel characteristics. A knowledge inheritance fidelity (KIF) metric is introduced to quantify knowledge transfer quality. Simulation results on a realistic urban topology show that BAND and S-BAND achieve 34.9% and 59.4% regret reduction relative to a centralized MAB baseline, with performance gains sustained across blockage rates ranging from 10% to 50%. The proposed TAK region consistently outperforms the traditional K-means clustering scheme under both fidelity criteria. △ Less

Submitted 5 August, 2026; originally announced August 2026.

QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment

Authors: Arian Kheirandish, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: Video instance segmentation (VIS) requires models to detect, segment, and track object identities across frames, and most methods enforce temporal consistency through video-level supervision. Image-only training approaches, with MinVIS as one prominent example, have challenged this assumption, reaching competitive VIS without video training by treating frames as independent images and associating… ▽ More Video instance segmentation (VIS) requires models to detect, segment, and track object identities across frames, and most methods enforce temporal consistency through video-level supervision. Image-only training approaches, with MinVIS as one prominent example, have challenged this assumption, reaching competitive VIS without video training by treating frames as independent images and associating instances only at inference. The field has nonetheless moved toward ever more elaborate video-trained trackers, which depend on costly identity-consistent annotations, leaving the image-only direction under-explored. A diagnostic analysis identifies object query quality as the bottleneck: queries trained only to localize objects within a frame drift apart across frames and destabilize association. QueenVIS introduces a query-centric framework for strengthening image-trained VIS. During single-frame training, we enrich Mask2Former queries with two auxiliary heads: a feature-prediction loss that aligns each query with the pooled backbone descriptor of its instance, and a center-prediction loss that injects spatial structure. Both heads are discarded at inference, adding zero parameters, and temporal identity is maintained by a training-free query-propagation and memory-bank scheme. On YouTube-VIS and OVIS with a ResNet-50 backbone, QueenVIS improves over MinVIS, up to +6.7 AP on YouTube-VIS, +4.8 AP on OVIS, and +10.3 AP on the long-sequence YouTube-VIS split. QueenVIS achieves 50.9 AP on YouTube-VIS and remains competitive with recent video-supervised state-of-the-art, without processing a single video clip during training. Our findings suggest that strengthening the discriminative power and temporal stability of object queries is an important, underexplored axis for VIS. Code and models: https://github.com/ArianKheir/QueenVIS △ Less

Submitted 27 July, 2026; originally announced July 2026.

Comments: Project website: https://aut-aisl.github.io/QueenVIS-Project-Page/

DispatchRAG: Grounding Emergency Dispatch Decisions in Real-World Protocols from Traffic Accident Video

Authors: Muhammad Sulthan Adhipradhana, Ehsan Javanmardi, Naren Bao, Manabu Tsukada

Abstract: Assessing the severity of a traffic accident scenario is important to decide which emergency service to dispatch. Missing an ambulance dispatch on a pedestrian accident is a fatal issue that can lead to death. Recently, Vision-Language Models (VLMs) have been a promising tool for accident reasoning, yet many VLMs are not grounded in real-life accident response protocols, making them not usable in… ▽ More Assessing the severity of a traffic accident scenario is important to decide which emergency service to dispatch. Missing an ambulance dispatch on a pedestrian accident is a fatal issue that can lead to death. Recently, Vision-Language Models (VLMs) have been a promising tool for accident reasoning, yet many VLMs are not grounded in real-life accident response protocols, making them not usable in accident severity assessment off-the-shelf. We introduced DispatchRAG, an accident assessor and dispatcher framework grounded in real-life Japanese traffic-accident response protocols, designed to enhance VLMs to generate an appropriate emergency response during an emergency scenario. Utilizing a RAG-based retrieval mechanism to retrieve the most relevant accident protocol and an LLM-powered reasoner to suggest the most proper response. To support evaluation, we introduce Accident Dispatch Dataset, a comprehensive dataset of accident assessment and emergency response according to Japanese accident response protocols adapted from the MM-AU dataset. We validate our framework on the Accident Dispatch Dataset, showing strong performance across various accident scenarios compared to the baseline VLM, pointing toward integration in autonomous vehicles that can automatically report both their own and nearby accidents. △ Less

Submitted 25 July, 2026; originally announced July 2026.

Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving

Authors: Yun Li, Jiachen Gong, Simon Thompson, Ehsan Javanmardi, Qunli Zhang, Zifan Zeng, Shiming Liu, Peng Wang, Zixuan Guo, Manabu Tsukada

Abstract: Large language models bring instruction following and scene reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop agents hide this gap by invoking the model on alternate simulation ticks and replaying the previous command in between, so half of all control outputs ignore the newest observations. We present a fast-slow a… ▽ More Large language models bring instruction following and scene reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop agents hide this gap by invoking the model on alternate simulation ticks and replaying the previous command in between, so half of all control outputs ignore the newest observations. We present a fast-slow architecture that removes this compromise. A frozen 7B vision-language backbone acts as the slow system, digesting navigation instructions and visual history at low frequency while exposing its per-layer key-value cache as a standing representation of the scene. A lightweight action expert acts as the fast system, attending to this cache and to the current camera frame at every simulation tick to regress waypoints in a single forward pass. Since the cache lags behind the world at deployment, we train the expert under randomized staleness, aligning training with asynchronous execution. On LangAuto-Short routes in CARLA, our system produces fresh control at every 50 ms simulation tick and lifts route completion from 37.0 to 94.0 over the frame-skipping baseline. A frame-skip ablation with the same expert separates the two factors at work: the expert raises the driving score on its own, while per-tick freshness raises completion from 82.1 to 94.0 and cuts red-light violations by a third. Trained on a single town, the expert transfers zero-shot to two unseen towns, holding 84-94% route completion where the baseline reaches 31-41%. It reduces open-loop waypoint error by nearly a factor of four compared to the backbone's own action head, at a per-tick model cost of 32 ms that is independent of history length on a single consumer GPU. △ Less

Submitted 17 July, 2026; originally announced July 2026.

Comments: 13 pages, 5 figures, 4 tables

ACM Class: I.2.9; I.2.10

How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation

Authors: Saba Fathi, Fardin Ayar, Maryam Abdolali, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: Diffusion models have recently been repurposed for zero-shot classification, giving rise to diffusion classifiers that identify the best-matching text prompt by minimizing the noise-prediction error. Despite their growing adoption, how these models make classification decisions remains poorly understood. We introduce ASOB-Bench, a bias evaluation for diffusion classifiers along three dimensions: A… ▽ More Diffusion models have recently been repurposed for zero-shot classification, giving rise to diffusion classifiers that identify the best-matching text prompt by minimizing the noise-prediction error. Despite their growing adoption, how these models make classification decisions remains poorly understood. We introduce ASOB-Bench, a bias evaluation for diffusion classifiers along three dimensions: Attribute binding, Size-Order bias, and Background dependency. These dimensions serve not as an exhaustive taxonomy but as targeted probes of how the text-conditioned reconstruction-error score reaches a decision. Such a perspective is well studied for discriminative vision-language models, yet remains overlooked for diffusion classifiers. Extending an existing framework with five new attribute categories on newly constructed datasets, we find diffusion classifiers are less prone to attribute misbinding than an OpenCLIP baseline; on the established ComCo benchmark they are substantially more susceptible to size-order shortcuts; and on ImageNet-B they suffer far larger accuracy drops, revealing heavy reliance on background over foreground cues. Reconstruction-error heatmaps and U-Net cross-attention visualizations expose the mechanism behind each bias. Because diffusion classifiers share the same denoiser as text-to-image models, these single-pass diagnostics also point toward analogous failure modes in generation. Overall, diffusion classifiers exhibit a distinct bias profile from vision-language models, offering guidance for building more robust diffusion-based models. △ Less

Submitted 4 July, 2026; originally announced July 2026.

Blockage-Aware Non-stationary Dynamic Bandit for User Association in mmWave V2X Networks

Authors: Weiqi Chi, Manabu Tsukada

Abstract: In millimeter-wave (mmWave) vehicular networks, dense base station (BS) deployments expand the user association (UA) decision space while dynamic blockages cause link quality fluctuations, posing critical challenges for effective mobility management. Traditional Multi-Armed Bandit (MAB) frameworks assume stationary reward distributions and fail to handle the rapid context-reward mapping shifts cau… ▽ More In millimeter-wave (mmWave) vehicular networks, dense base station (BS) deployments expand the user association (UA) decision space while dynamic blockages cause link quality fluctuations, posing critical challenges for effective mobility management. Traditional Multi-Armed Bandit (MAB) frameworks assume stationary reward distributions and fail to handle the rapid context-reward mapping shifts caused by vehicle mobility and transient blockages. To address this, we propose Blockage-Aware Non-stationary Dynamic Bandit (BAND), a fully distributed, channel state information (CSI)-free mobility management framework for mmWave vehicular networks, formulating UA as a non-stationary contextual bandit problem, enabling online adaptive optimization without requiring central coordination or offline training. BAND employs a cumulative sum-based change detection (CUSUM-CD) to dynamically narrow the active BS set, reducing exploration overhead while tracking reward distribution shifts. Proactive blockage detection suppresses transient signal degradation in the reward estimation process. Simulations demonstrate over 40% regret reduction and up to 33.1% network communication rate improvement compared with hypercube-based contextual bandit baselines, with robustness validated across varying blockage rates and network configurations. △ Less

Submitted 6 June, 2026; originally announced June 2026.

LoViF 2026 The First Challenge on Holistic Quality Assessment for 4D World Model (PhyScore)

Authors: Wei Luo, Yiting Lu, Xin Li, Haoran Li, Fengbin Guan, Chen Gao, Xin Jin, Yong Li, Zhibo Chen, Sijing Wu, Kang Fu, Yunhao Li, Ziang Xiao, Huiyu Duan, Jing Liu, Qiang Hu, Xiongkuo Min, Guangtao Zhai, Manxi Sun, Zixuan Guo, Yun Li, Ziyang Chen, Manabu Tsukada, Zhengyang Li, Zhenglin Du , et al. (10 additional authors not shown)

Abstract: This paper reports on the LoViF 2026 PhyScore challenge, a competition on holistic quality assessment of world-model-generated videos across both 2D and 4D generation settings. The challenge is motivated by a central gap in current evaluation practice: perceptual quality alone is insufficient to judge whether generated dynamics are physically plausible, temporally coherent, and consistent with inp… ▽ More This paper reports on the LoViF 2026 PhyScore challenge, a competition on holistic quality assessment of world-model-generated videos across both 2D and 4D generation settings. The challenge is motivated by a central gap in current evaluation practice: perceptual quality alone is insufficient to judge whether generated dynamics are physically plausible, temporally coherent, and consistent with input conditions. Participants are required to build a metric that jointly predicts four dimensions, i.e., Video Quality, Physical Realism, Condition-Video Alignment, and Temporal Consistency. Depart from that, participants also need to localize physical anomaly timestamps for fine-grained diagnosis. The benchmark dataset contains 1,554 videos generated by seven representative world generative models, organized into three tracks (text-2D, image-to-4D, and video-to-4D) and spanning 26 categories. These categories explicitly cover physics-relevant scenarios, including dynamics, optics, and thermodynamics, together with diverse real-world and creative content. To ensure label reliability, scores and anomaly timestamps are produced through trained human annotation with an additional automated quality-control pass. Evaluation is based on both score prediction and anomaly localization, with a composite protocol that combines TimeStamp_IOU and SRCC/PLCC. This report summarizes the challenge design and provides method-level insights from submitted solutions. △ Less

Submitted 6 May, 2026; originally announced May 2026.

NTIRE 2026 3D Restoration and Reconstruction in Real-world Adverse Conditions: RealX3D Challenge Results

Authors: Shuhong Liu, Chenyu Bao, Ziteng Cui, Xuangeng Chu, Bin Ren, Lin Gu, Xiang Chen, Mingrui Li, Long Ma, Marcos V. Conde, Radu Timofte, Yun Liu, Ryo Umagami, Tomohiro Hashimoto, Zijian Hu, Yuan Gan, Tianhan Xu, Yusuke Kurose, Tatsuya Harada, Junwei Yuan, Gengjia Chang, Xining Ge, Mache You, Qida Cao, Zeliang Li , et al. (81 additional authors not shown)

Abstract: This paper presents a comprehensive review of the NTIRE 2026 3D Restoration and Reconstruction (3DRR) Challenge, detailing the proposed methods and results. The challenge seeks to identify robust reconstruction pipelines that are robust under real-world adverse conditions, specifically extreme low-light and smoke-degraded environments, as captured by our RealX3D benchmark. A total of 279 participa… ▽ More This paper presents a comprehensive review of the NTIRE 2026 3D Restoration and Reconstruction (3DRR) Challenge, detailing the proposed methods and results. The challenge seeks to identify robust reconstruction pipelines that are robust under real-world adverse conditions, specifically extreme low-light and smoke-degraded environments, as captured by our RealX3D benchmark. A total of 279 participants registered for the competition, of whom 33 teams submitted valid results. We thoroughly evaluate the submitted approaches against state-of-the-art baselines, revealing significant progress in 3D reconstruction under adverse conditions. Our analysis highlights shared design principles among top-performing methods and provides insights into effective strategies for handling 3D scene degradation. △ Less

Submitted 29 April, 2026; v1 submitted 5 April, 2026; originally announced April 2026.

Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Authors: Yun Li, Yidu Zhang, Simon Thompson, Ehsan Javanmardi, Manabu Tsukada

Abstract: Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), w… ▽ More Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems. △ Less

Submitted 2 April, 2026; originally announced April 2026.

Comments: 18 pages, 6 figures, 4 tables

Beam-aware Kernelized Contextual Bandits for User Association and Beamforming in mmWave Vehicular Networks

Authors: Xiaoyang He, Manabu Tsukada

Abstract: Timely channel information is necessary for vehicles to determine both the serving base station (BS) and the beamforming vector, but frequent estimation of fast-fading mmWave channels incurs significant overhead. To address this challenge, we propose a Beam-aware Kernelized Contextual Upper Confidence Bound (BKC-UCB) algorithm that estimates instantaneous transmission rates without additional chan… ▽ More Timely channel information is necessary for vehicles to determine both the serving base station (BS) and the beamforming vector, but frequent estimation of fast-fading mmWave channels incurs significant overhead. To address this challenge, we propose a Beam-aware Kernelized Contextual Upper Confidence Bound (BKC-UCB) algorithm that estimates instantaneous transmission rates without additional channel measurements by exploiting historical contexts such as vehicle location and velocity, together with past observed transmission rates. Specifically, BKC-UCB leverages kernel methods to capture the nonlinear relationship between context and transmission rate by mapping contexts into a reproducing kernel Hilbert space (RKHS), where linear learning becomes feasible. Rather than treating each beam as an independent arm, the beam index is embedded into the context, enabling BKC-UCB to exploit correlations among beams to accelerate convergence. Furthermore, an event-triggered information sharing mechanism is incorporated into BKC-UCB, enabling information exchange only when significant explorations are conducted to improve learning efficiency with limited communication overhead. △ Less

Submitted 8 March, 2026; originally announced March 2026.

An Open-Source Modular Benchmark for Diffusion-Based Motion Planning in Closed-Loop Autonomous Driving

Authors: Yun Li, Simon Thompson, Yidu Zhang, Ehsan Javanmardi, Manabu Tsukada

Abstract: Diffusion-based motion planners have achieved state-of-the-art results on benchmarks such as nuPlan, yet their evaluation within closed-loop production autonomous driving stacks remains largely unexplored. Existing evaluations abstract away ROS 2 communication latency and real-time scheduling constraints, while monolithic ONNX deployment freezes all solver parameters at export time. We present an… ▽ More Diffusion-based motion planners have achieved state-of-the-art results on benchmarks such as nuPlan, yet their evaluation within closed-loop production autonomous driving stacks remains largely unexplored. Existing evaluations abstract away ROS 2 communication latency and real-time scheduling constraints, while monolithic ONNX deployment freezes all solver parameters at export time. We present an open-source modular benchmark that addresses both gaps: using ONNX GraphSurgeon, we decompose a monolithic 18,398 node diffusion planner into three independently executable modules and reimplement the DPM-Solver++ denoising loop in native C++. Integrated as a ROS 2 node within Autoware, the open-source AD stack deployed on real vehicles worldwide, the system enables runtime-configurable solver parameters without model recompilation and per-step observability of the denoising process, breaking the black box of monolithic deployment. Unlike evaluations in standalone simulators such as CARLA, our benchmark operates within a production-grade stack and is validated through AWSIM closed-loop simulation. Through systematic comparison of DPM-Solver++ (first- and second-order) and DDIM across six step-count configurations (N in {3, 5, 7, 10, 15, 20}), we show that encoder caching yields a 3.2x latency reduction, and that second-order solving reduces FDE by 41% at N=3 compared to first-order. The complete codebase will be released as open-source, providing a direct path from simulation benchmarks to real-vehicle deployment. △ Less

Submitted 1 March, 2026; originally announced March 2026.

Comments: 8 pages, 5 figures

Peeking Ahead of the Field Study: Exploring VLM Personas as Support Tools for Embodied Studies in HCI

Authors: Xinyue Gui, Ding Xia, Mark Colley, Yuan Li, Vishal Chauhan, Anubhav Anubhav, Zhongyi Zhou, Ehsan Javanmardi, Stela Hanbyeol Seo, Chia-Ming Chang, Manabu Tsukada, Takeo Igarashi

Abstract: Field studies are irreplaceable but costly, time-consuming, and error-prone, which need careful preparation. Inspired by rapid-prototyping in manufacturing, we propose a fast, low-cost evaluation method using Vision-Language Model (VLM) personas to simulate outcomes comparable to field results. While LLMs show human-like reasoning and language capabilities, autonomous vehicle (AV)-pedestrian inter… ▽ More Field studies are irreplaceable but costly, time-consuming, and error-prone, which need careful preparation. Inspired by rapid-prototyping in manufacturing, we propose a fast, low-cost evaluation method using Vision-Language Model (VLM) personas to simulate outcomes comparable to field results. While LLMs show human-like reasoning and language capabilities, autonomous vehicle (AV)-pedestrian interaction requires spatial awareness, emotional empathy, and behavioral generation. This raises our research question: To what extent can VLM personas mimic human responses in field studies? We conducted parallel studies: 1) one real-world study with 20 participants, and 2) one video-study using 20 VLM personas, both on a street-crossing task. We compared their responses and interviewed five HCI researchers on potential applications. Results show that VLM personas mimic human response patterns (e.g., average crossing times of 5.25 s vs. 5.07 s) lack the behavioral variability and depth. They show promise for formative studies, field study preparation, and human data augmentation. △ Less

Submitted 17 February, 2026; originally announced February 2026.

Comments: Accepted to CHI 2026

Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback

Authors: Seyed Amir Hosseini, Maryam Abdolali, Amirhosein Tavakkoli, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: Preference-based reinforcement learning (PBRL) offers a promising alternative to explicit reward engineering by learning from pairwise trajectory comparisons. However, real-world preference data often comes from heterogeneous annotators with varying reliability; some accurate, some noisy, and some systematically adversarial. Existing PBRL methods either treat all feedback equally or attempt to fil… ▽ More Preference-based reinforcement learning (PBRL) offers a promising alternative to explicit reward engineering by learning from pairwise trajectory comparisons. However, real-world preference data often comes from heterogeneous annotators with varying reliability; some accurate, some noisy, and some systematically adversarial. Existing PBRL methods either treat all feedback equally or attempt to filter out unreliable sources, but both approaches fail when faced with adversarial annotators who systematically provide incorrect preferences. We introduce TriTrust-PBRL (TTP), a unified framework that jointly learns a shared reward model and expert-specific trust parameters from multi-expert preference feedback. The key insight is that trust parameters naturally evolve during gradient-based optimization to be positive (trust), near zero (ignore), or negative (flip), enabling the model to automatically invert adversarial preferences and recover useful signal rather than merely discarding corrupted feedback. We provide theoretical analysis establishing identifiability guarantees and detailed gradient analysis that explains how expert separation emerges naturally during training without explicit supervision. Empirically, we evaluate TTP on four diverse domains spanning manipulation tasks (MetaWorld) and locomotion (DM Control) under various corruption scenarios. TTP achieves state-of-the-art robustness, maintaining near-oracle performance under adversarial corruption while standard PBRL methods fail catastrophically. Notably, TTP outperforms existing baselines by successfully learning from mixed expert pools containing both reliable and adversarial annotators, all while requiring no expert features beyond identification indices and integrating seamlessly with existing PBRL pipelines. △ Less

Submitted 26 January, 2026; originally announced January 2026.

Comments: Equal contribution: Seyed Amir Hosseini and Maryam Abdolali. Corresponding author: Maryam Abdolali (maryam.abdolali@kntu.ac.ir)

SUG-Occ: Explicit Semantics and Uncertainty Guided Sparse Learning for Efficient 3D Occupancy Prediction

Authors: Hanlin Wu, Pengfei Lin, Ehsan Javanmardi, Naren Bao, Bo Qian, Hao Si, Manabu Tsukada

Abstract: 3D semantic occupancy prediction has emerged as a critical perception task for autonomous driving due to its ability to offer voxel-level semantic and geometric understanding of the environment. However, such a refined representation for large-scale scenes incurs prohibitive computation, posing a significant challenge to practical real-time deployment. To address this, we propose SUGOcc, an explic… ▽ More 3D semantic occupancy prediction has emerged as a critical perception task for autonomous driving due to its ability to offer voxel-level semantic and geometric understanding of the environment. However, such a refined representation for large-scale scenes incurs prohibitive computation, posing a significant challenge to practical real-time deployment. To address this, we propose SUGOcc, an explicit semantics and uncertainty guided sparse learning framework for efficient occupancy prediction, which exploits the inherent sparsity of 3D scenes to reduce redundant computation while maintaining geometric and semantic integrity. Specifically, we first utilize semantic and uncertainty priors to suppress image projections from free space while employing explicit unsigned distance encoding to enhance geometric consistency, thereby producing a structurally sparse representation. Secondly, we introduce a cascade sparse completion module to enable efficient coarse-to-fine reasoning over the sparse representation via hyper cross sparse convolution, generative upsampling and adaptive pruning. Finally, we propose an object contextual representation (OCR) based mask decoder that refines the voxel-wise predictions through lightweight query-context interactions, thereby avoiding expensive attention operations over volumetric features. Extensive experiments on SemanticKITTI and Occ3D-Nuscenes benchmark demonstrate that the proposed approach outperforms the baselines, achieving notable improvements in both accuracy and efficiency across datasets. △ Less

Submitted 28 March, 2026; v1 submitted 16 January, 2026; originally announced January 2026.

Trajectory Planning for UAV-Based Smart Farming Using Imitation-Based Triple Deep Q-Learning

Authors: Wencan Mao, Quanxi Zhou, Tomas Couso Coddou, Manabu Tsukada, Yunling Liu, Yusheng Ji

Abstract: Unmanned aerial vehicles (UAVs) have emerged as a promising auxiliary platform for smart agriculture, capable of simultaneously performing weed detection, recognition, and data collection from wireless sensors. However, trajectory planning for UAV-based smart agriculture is challenging due to the high uncertainty of the environment, partial observations, and limited battery capacity of UAVs. To ad… ▽ More Unmanned aerial vehicles (UAVs) have emerged as a promising auxiliary platform for smart agriculture, capable of simultaneously performing weed detection, recognition, and data collection from wireless sensors. However, trajectory planning for UAV-based smart agriculture is challenging due to the high uncertainty of the environment, partial observations, and limited battery capacity of UAVs. To address these issues, we formulate the trajectory planning problem as a Markov decision process (MDP) and leverage multi-agent reinforcement learning (MARL) to solve it. Furthermore, we propose a novel imitation-based triple deep Q-network (ITDQN) algorithm, which employs an elite imitation mechanism to reduce exploration costs and utilizes a mediator Q-network over a double deep Q-network (DDQN) to accelerate and stabilize training and improve performance. Experimental results in both simulated and real-world environments demonstrate the effectiveness of our solution. Moreover, our proposed ITDQN outperforms DDQN by 4.43\% in weed recognition rate and 6.94\% in data collection rate. △ Less

Submitted 21 December, 2025; originally announced December 2025.

EIA-SEC: Improved Actor-Critic Framework for Multi-UAV Collaborative Control in Smart Agriculture

Authors: Quanxi Zhou, Wencan Mao, Yilei Liang, Manabu Tsukada, Yunling Liu, Jon Crowcroft

Abstract: The widespread application of wireless communication technology has promoted the development of smart agriculture, where unmanned aerial vehicles (UAVs) play a multifunctional role. We target a multi-UAV smart agriculture system where UAVs cooperatively perform data collection, image acquisition, and communication tasks. In this context, we model a Markov decision process to solve the multi-UAV tr… ▽ More The widespread application of wireless communication technology has promoted the development of smart agriculture, where unmanned aerial vehicles (UAVs) play a multifunctional role. We target a multi-UAV smart agriculture system where UAVs cooperatively perform data collection, image acquisition, and communication tasks. In this context, we model a Markov decision process to solve the multi-UAV trajectory planning problem. Moreover, we propose a novel Elite Imitation Actor-Shared Ensemble Critic (EIA-SEC) framework, where agents adaptively learn from the elite agent to reduce trial-and-error costs, and a shared ensemble critic collaborates with each agent's local critic to ensure unbiased objective value estimates and prevent overestimation. Experimental results demonstrate that EIA-SEC outperforms state-of-the-art baselines in terms of reward performance, training stability, and convergence speed. △ Less

Submitted 21 December, 2025; originally announced December 2025.

FM-EAC: Feature Model-based Enhanced Actor-Critic for Multi-Task Control in Dynamic Environments

Authors: Quanxi Zhou, Wencan Mao, Manabu Tsukada, John C. S. Lui, Yusheng Ji

Abstract: Model-based reinforcement learning (MBRL) and model-free reinforcement learning (MFRL) evolve along distinct paths but converge in the design of Dyna-Q [1]. However, modern RL methods still struggle with effective transferability across tasks and scenarios. Motivated by this limitation, we propose a generalized algorithm, Feature Model-Based Enhanced Actor-Critic (FM-EAC), that integrates planning… ▽ More Model-based reinforcement learning (MBRL) and model-free reinforcement learning (MFRL) evolve along distinct paths but converge in the design of Dyna-Q [1]. However, modern RL methods still struggle with effective transferability across tasks and scenarios. Motivated by this limitation, we propose a generalized algorithm, Feature Model-Based Enhanced Actor-Critic (FM-EAC), that integrates planning, acting, and learning for multi-task control in dynamic environments. FM-EAC combines the strengths of MBRL and MFRL and improves generalizability through the use of novel feature-based models and an enhanced actor-critic framework. Simulations in both urban and agricultural applications demonstrate that FM-EAC consistently outperforms many state-of-the-art MBRL and MFRL methods. More importantly, different sub-networks can be customized within FM-EAC according to user-specific requirements. △ Less

Submitted 17 December, 2025; originally announced December 2025.

UTrice: Unifying Primitives in Differentiable Ray Tracing and Rasterization via Triangles for Particle-Based 3D Scenes

Authors: Changhe Liu, Ehsan Javanmardi, Naren Bao, Alex Orsholits, Manabu Tsukada

Abstract: Ray tracing 3D Gaussian particles enables realistic effects such as depth of field, refractions, and flexible camera modeling for novel-view synthesis. However, existing methods trace Gaussians through proxy geometry, which requires constructing complex intermediate meshes and performing costly intersection tests. This limitation arises because Gaussian-based particles are not well suited as unifi… ▽ More Ray tracing 3D Gaussian particles enables realistic effects such as depth of field, refractions, and flexible camera modeling for novel-view synthesis. However, existing methods trace Gaussians through proxy geometry, which requires constructing complex intermediate meshes and performing costly intersection tests. This limitation arises because Gaussian-based particles are not well suited as unified primitives for both ray tracing and rasterization. In this work, we propose a differentiable triangle-based ray tracing pipeline that directly treats triangles as rendering primitives without relying on any proxy geometry. Our results show that the proposed method achieves significantly higher rendering quality than existing ray tracing approaches while maintaining real-time rendering performance. Moreover, our pipeline can directly render triangles optimized by the rasterization-based method Triangle Splatting, thus unifying the primitives used in novel-view synthesis. △ Less

Submitted 3 December, 2025; originally announced December 2025.

Comments: 13 pages, 10 figures, submitted to CVPR2026

Towards Robust LiDAR Localization: Deep Learning-based Uncertainty Estimation

Authors: Minoo Dolatabadi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: LiDAR-based localization and SLAM often rely on iterative matching algorithms, particularly the Iterative Closest Point (ICP) algorithm, to align sensor data with pre-existing maps or previous scans. However, ICP is prone to errors in featureless environments and dynamic scenes, leading to inaccurate pose estimation. Accurately predicting the uncertainty associated with ICP is crucial for robust s… ▽ More LiDAR-based localization and SLAM often rely on iterative matching algorithms, particularly the Iterative Closest Point (ICP) algorithm, to align sensor data with pre-existing maps or previous scans. However, ICP is prone to errors in featureless environments and dynamic scenes, leading to inaccurate pose estimation. Accurately predicting the uncertainty associated with ICP is crucial for robust state estimation but remains challenging, as existing approaches often rely on handcrafted models or simplified assumptions. Moreover, a few deep learning-based methods for localizability estimation either depend on a pre-built map, which may not always be available, or provide a binary classification of localizable versus non-localizable, which fails to properly model uncertainty. In this work, we propose a data-driven framework that leverages deep learning to estimate the registration error covariance of ICP before matching, even in the absence of a reference map. By associating each LiDAR scan with a reliable 6-DoF error covariance estimate, our method enables seamless integration of ICP within Kalman filtering, enhancing localization accuracy and robustness. Extensive experiments on the KITTI dataset demonstrate the effectiveness of our approach, showing that it accurately predicts covariance and, when applied to localization using a pre-built map or SLAM, reduces localization errors and improves robustness. △ Less

Submitted 23 September, 2025; originally announced September 2025.

Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers

Authors: Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: Streaming visual transformers like StreamVGGT achieve strong 3D perception but suffer from unbounded growth of key value (KV) memory, which limits scalability. We propose a training-free, inference-time token eviction policy that bounds memory by discarding redundant tokens while keeping the most informative ones. Our method uses significantly less memory with little to no drop in accuracy: on 7-S… ▽ More Streaming visual transformers like StreamVGGT achieve strong 3D perception but suffer from unbounded growth of key value (KV) memory, which limits scalability. We propose a training-free, inference-time token eviction policy that bounds memory by discarding redundant tokens while keeping the most informative ones. Our method uses significantly less memory with little to no drop in accuracy: on 7-Scenes with long sequences it reduces peak memory from 18.63 GB to 9.39 GB while accuracy and completeness drop by only 0.003. Under strict memory budgets, eviction enables denser frame sampling, which improves reconstruction accuracy compared to the baseline. Experiments across video depth estimation (Sintel, KITTI), 3D reconstruction (7-Scenes, NRGBD), and camera pose estimation (Sintel, TUM-dynamics) show that our approach closely matches StreamVGGT at a fraction of the memory and makes long-horizon streaming inference more practical. △ Less

Submitted 2 October, 2025; v1 submitted 22 September, 2025; originally announced September 2025.

Comments: project page: https://soroush-mim.github.io/projects/evict3r/

You Share Beliefs, I Adapt: Progressive Heterogeneous Collaborative Perception

Authors: Hao Si, Ehsan Javanmardi, Manabu Tsukada

Abstract: Collaborative perception enables vehicles to overcome individual perception limitations by sharing information, allowing them to see further and through occlusions. In real-world scenarios, models on different vehicles are often heterogeneous due to manufacturer variations. Existing methods for heterogeneous collaborative perception address this challenge by fine-tuning adapters or the entire netw… ▽ More Collaborative perception enables vehicles to overcome individual perception limitations by sharing information, allowing them to see further and through occlusions. In real-world scenarios, models on different vehicles are often heterogeneous due to manufacturer variations. Existing methods for heterogeneous collaborative perception address this challenge by fine-tuning adapters or the entire network to bridge the domain gap. However, these methods are impractical in real-world applications, as each new collaborator must undergo joint training with the ego vehicle on a dataset before inference, or the ego vehicle stores models for all potential collaborators in advance. Therefore, we pose a new question: Can we tackle this challenge directly during inference, eliminating the need for joint training? To answer this, we introduce Progressive Heterogeneous Collaborative Perception (PHCP), a novel framework that formulates the problem as few-shot unsupervised domain adaptation. Unlike previous work, PHCP dynamically aligns features by self-training an adapter during inference, eliminating the need for labeled data and joint training. Extensive experiments on the OPV2V dataset demonstrate that PHCP achieves strong performance across diverse heterogeneous scenarios. Notably, PHCP achieves performance comparable to SOTA methods trained on the entire dataset while using only a small amount of unlabeled data. △ Less

Submitted 11 September, 2025; originally announced September 2025.

A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving

Authors: Hanlin Wu, Pengfei Lin, Ehsan Javanmardi, Naren Bao, Bo Qian, Hao Si, Manabu Tsukada

Abstract: 3D semantic occupancy prediction is an emerging perception paradigm in autonomous driving, providing a voxel-level representation of both geometric details and semantic categories. However, its effectiveness is inherently constrained in single-vehicle setups by occlusions, restricted sensor range, and narrow viewpoints. To address these limitations, collaborative perception enables the exchange of… ▽ More 3D semantic occupancy prediction is an emerging perception paradigm in autonomous driving, providing a voxel-level representation of both geometric details and semantic categories. However, its effectiveness is inherently constrained in single-vehicle setups by occlusions, restricted sensor range, and narrow viewpoints. To address these limitations, collaborative perception enables the exchange of complementary information, thereby enhancing the completeness and accuracy of predictions. Despite its potential, research on collaborative 3D semantic occupancy prediction is hindered by the lack of dedicated datasets. To bridge this gap, we design a high-resolution semantic voxel sensor in CARLA to produce dense and comprehensive annotations. We further develop a baseline model that performs inter-agent feature fusion via spatial alignment and attention aggregation. In addition, we establish benchmarks with varying prediction ranges designed to systematically assess the impact of spatial extent on collaborative prediction. Experimental results demonstrate the superior performance of our baseline, with increasing gains observed as range expands. Our code is available at https://github.com/tlab-wide/Co3SOP}{https://github.com/tlab-wide/Co3SOP. △ Less

Submitted 16 January, 2026; v1 submitted 20 June, 2025; originally announced June 2025.

Towards Efficient Roadside LiDAR Deployment: A Fast Surrogate Metric Based on Entropy-Guided Visibility

Authors: Yuze Jiang, Ehsan Javanmardi, Manabu Tsukada, Hiroshi Esaki

Abstract: The deployment of roadside LiDAR sensors plays a crucial role in the development of Cooperative Intelligent Transport Systems (C-ITS). However, the high cost of LiDAR sensors necessitates efficient placement strategies to maximize detection performance. Traditional roadside LiDAR deployment methods rely on expert insight, making them time-consuming. Automating this process, however, demands extens… ▽ More The deployment of roadside LiDAR sensors plays a crucial role in the development of Cooperative Intelligent Transport Systems (C-ITS). However, the high cost of LiDAR sensors necessitates efficient placement strategies to maximize detection performance. Traditional roadside LiDAR deployment methods rely on expert insight, making them time-consuming. Automating this process, however, demands extensive computation, as it requires not only visibility evaluation but also assessing detection performance across different LiDAR placements. To address this challenge, we propose a fast surrogate metric, the Entropy-Guided Visibility Score (EGVS), based on information gain to evaluate object detection performance in roadside LiDAR configurations. EGVS leverages Traffic Probabilistic Occupancy Grids (TPOG) to prioritize critical areas and employs entropy-based calculations to quantify the information captured by LiDAR beams. This eliminates the need for direct detection performance evaluation, which typically requires extensive labeling and computational resources. By integrating EGVS into the optimization process, we significantly accelerate the search for optimal LiDAR configurations. Experimental results using the AWSIM simulator demonstrate that EGVS strongly correlates with Average Precision (AP) scores and effectively predicts object detection performance. This approach offers a computationally efficient solution for roadside LiDAR deployment, facilitating scalable smart infrastructure development. △ Less

Submitted 9 April, 2025; originally announced April 2025.

Comments: Accepted by IEEE Intelligent Vehicles Symposium (IV 2025)

Where Do You Go? Pedestrian Trajectory Prediction using Scene Features

Authors: Mohammad Ali Rezaei, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: Accurate prediction of pedestrian trajectories is crucial for enhancing the safety of autonomous vehicles and reducing traffic fatalities involving pedestrians. While numerous studies have focused on modeling interactions among pedestrians to forecast their movements, the influence of environmental factors and scene-object placements has been comparatively underexplored. In this paper, we present… ▽ More Accurate prediction of pedestrian trajectories is crucial for enhancing the safety of autonomous vehicles and reducing traffic fatalities involving pedestrians. While numerous studies have focused on modeling interactions among pedestrians to forecast their movements, the influence of environmental factors and scene-object placements has been comparatively underexplored. In this paper, we present a novel trajectory prediction model that integrates both pedestrian interactions and environmental context to improve prediction accuracy. Our approach captures spatial and temporal interactions among pedestrians within a sparse graph framework. To account for pedestrian-scene interactions, we employ advanced image enhancement and semantic segmentation techniques to extract detailed scene features. These scene and interaction features are then fused through a cross-attention mechanism, enabling the model to prioritize relevant environmental factors that influence pedestrian movements. Finally, a temporal convolutional network processes the fused features to predict future pedestrian trajectories. Experimental results demonstrate that our method significantly outperforms existing state-of-the-art approaches, achieving ADE and FDE values of 0.252 and 0.372 meters, respectively, underscoring the importance of incorporating both social interactions and environmental context in pedestrian trajectory prediction. △ Less

Submitted 23 January, 2025; originally announced January 2025.

Comments: Accepted by 2024 International Conference on Intelligent Computing and its Emerging Applications

Recognition-Oriented Low-Light Image Enhancement based on Global and Pixelwise Optimization

Authors: Seitaro Ono, Yuka Ogino, Takahiro Toizumi, Atsushi Ito, Masato Tsukada

Abstract: In this paper, we propose a novel low-light image enhancement method aimed at improving the performance of recognition models. Despite recent advances in deep learning, the recognition of images under low-light conditions remains a challenge. Although existing low-light image enhancement methods have been developed to improve image visibility for human vision, they do not specifically focus on enh… ▽ More In this paper, we propose a novel low-light image enhancement method aimed at improving the performance of recognition models. Despite recent advances in deep learning, the recognition of images under low-light conditions remains a challenge. Although existing low-light image enhancement methods have been developed to improve image visibility for human vision, they do not specifically focus on enhancing recognition model performance. Our proposed low-light image enhancement method consists of two key modules: the Global Enhance Module, which adjusts the overall brightness and color balance of the input image, and the Pixelwise Adjustment Module, which refines image features at the pixel level. These modules are trained to enhance input images to improve downstream recognition model performance effectively. Notably, the proposed method can be applied as a frontend filter to improve low-light recognition performance without requiring retraining of downstream recognition models. Experimental results demonstrate that our method improves the performance of pretrained recognition models under low-light conditions and its effectiveness. △ Less

Submitted 7 January, 2025; originally announced January 2025.

Comments: accepted to VISAPP2025

Neural Error Covariance Estimation for Precise LiDAR Localization

Authors: Minoo Dolatabadi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

Abstract: Autonomous vehicles have gained significant attention due to technological advancements and their potential to transform transportation. A critical challenge in this domain is precise localization, particularly in LiDAR-based map matching, which is prone to errors due to degeneracy in the data. Most sensor fusion techniques, such as the Kalman filter, rely on accurate error covariance estimates fo… ▽ More Autonomous vehicles have gained significant attention due to technological advancements and their potential to transform transportation. A critical challenge in this domain is precise localization, particularly in LiDAR-based map matching, which is prone to errors due to degeneracy in the data. Most sensor fusion techniques, such as the Kalman filter, rely on accurate error covariance estimates for each sensor to improve localization accuracy. However, obtaining reliable covariance values for map matching remains a complex task. To address this challenge, we propose a neural network-based framework for predicting localization error covariance in LiDAR map matching. To achieve this, we introduce a novel dataset generation method specifically designed for error covariance estimation. In our evaluation using a Kalman filter, we achieved a 2 cm improvement in localization accuracy, a significant enhancement in this domain. △ Less

Submitted 5 January, 2025; originally announced January 2025.

Comments: Accepted by 2024 International Conference on Intelligent Computing and its Emerging Applications

LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training

Authors: Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi, Mohammad Rahmati

Abstract: Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied for cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit… ▽ More Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied for cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS).This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points. △ Less

Submitted 30 December, 2024; originally announced December 2024.

Comments: Accepted by 2024 International Conference on Intelligent Computing and its Emerging Applications

eRSS-RAMP: A Rule-Adherence Motion Planner Based on Extended Responsibility-Sensitive Safety for Autonomous Driving

Authors: Pengfei Lin, Ehsan Javanmardi, Yuze Jiang, Dou Hu, Shangkai Zhang, Manabu Tsukada

Abstract: Driving safety and responsibility determination are indispensable pieces of the puzzle for autonomous driving. They are also deeply related to the allocation of right-of-way and the determination of accident liability. Therefore, Intel/Mobileye designed the responsibility-sensitive safety (RSS) framework to further enhance the safety regulation of autonomous driving, which mathematically defines r… ▽ More Driving safety and responsibility determination are indispensable pieces of the puzzle for autonomous driving. They are also deeply related to the allocation of right-of-way and the determination of accident liability. Therefore, Intel/Mobileye designed the responsibility-sensitive safety (RSS) framework to further enhance the safety regulation of autonomous driving, which mathematically defines rules for autonomous vehicles (AVs) behaviors in various traffic scenarios. However, the RSS framework's rules are relatively rudimentary in certain scenarios characterized by interaction uncertainty, especially those requiring collaborative driving during emergency collision avoidance. Besides, the integration of the RSS framework with motion planning is rarely discussed in current studies. Therefore, we proposed a rule-adherence motion planner (RAMP) based on the extended RSS (eRSS) regulation for non-connected and connected AVs in merging and emergency-avoiding scenarios. The simulation results indicate that the proposed method can achieve faster and safer lane merging performance (53.0% shorter merging length and a 73.5% decrease in merging time), and allows for more stable steering maneuvers in emergency collision avoidance, resulting in smoother paths for ego vehicle and surrounding vehicles. △ Less

Submitted 4 September, 2024; originally announced September 2024.

Comments: 12 pages, 19 figures, submitted to an IEEE journal

Large Language Models for Human-like Autonomous Driving: A Survey

Authors: Yun Li, Kai Katsumata, Ehsan Javanmardi, Manabu Tsukada

Abstract: Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of Autonomous Driving (AD). As AD systems evolve from rule-based and optimization-based methods to learning-based techniques like deep reinforcement learning, they are now poised to embrace a third and more advanced category: knowled… ▽ More Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of Autonomous Driving (AD). As AD systems evolve from rule-based and optimization-based methods to learning-based techniques like deep reinforcement learning, they are now poised to embrace a third and more advanced category: knowledge-based AD empowered by LLMs. This shift promises to bring AD closer to human-like AD. However, integrating LLMs into AD systems poses challenges in real-time inference, safety assurance, and deployment costs. This survey provides a comprehensive and critical review of recent progress in leveraging LLMs for AD, focusing on their applications in modular AD pipelines and end-to-end AD systems. We highlight key advancements, identify pressing challenges, and propose promising research directions to bridge the gap between LLMs and AD, thereby facilitating the development of more human-like AD systems. The survey first introduces LLMs' key features and common training schemes, then delves into their applications in modular AD pipelines and end-to-end AD, respectively, followed by discussions on open challenges and future directions. Through this in-depth analysis, we aim to provide insights and inspiration for researchers and practitioners working at the intersection of AI and autonomous vehicles, ultimately contributing to safer, smarter, and more human-centric AD technologies. △ Less

Submitted 27 July, 2024; originally announced July 2024.

Comments: 8 pages, 2 figures, accepted at IEEE Intelligent Transportation Systems Conference (ITSC) 2024

Accurate Cooperative Localization Utilizing LiDAR-equipped Roadside Infrastructure for Autonomous Driving

Authors: Yuze Jiang, Ehsan Javanmardi, Manabu Tsukada, Hiroshi Esaki

Abstract: Recent advancements in LiDAR technology have significantly lowered costs and improved both its precision and resolution, thereby solidifying its role as a critical component in autonomous vehicle localization. Using sophisticated 3D registration algorithms, LiDAR now facilitates vehicle localization with centimeter-level accuracy. However, these high-precision techniques often face reliability cha… ▽ More Recent advancements in LiDAR technology have significantly lowered costs and improved both its precision and resolution, thereby solidifying its role as a critical component in autonomous vehicle localization. Using sophisticated 3D registration algorithms, LiDAR now facilitates vehicle localization with centimeter-level accuracy. However, these high-precision techniques often face reliability challenges in environments devoid of identifiable map features. To address this limitation, we propose a novel approach that utilizes road side units (RSU) with vehicle-to-infrastructure (V2I) communications to assist vehicle self-localization. By using RSUs as stationary reference points and processing real-time LiDAR data, our method enhances localization accuracy through a cooperative localization framework. By placing RSUs in critical areas, our proposed method can improve the reliability and precision of vehicle localization when the traditional vehicle self-localization technique falls short. Evaluation results in an end-to-end autonomous driving simulator AWSIM show that the proposed method can improve localization accuracy by up to 80% under vulnerable environments compared to traditional localization methods. Additionally, our method also demonstrates robust resistance to network delays and packet loss in heterogeneous network environments. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: Accepted by IEEE Intelligent Transportation Systems Conference (ITSC) 2024

Zero-Knowledge Proof of Distinct Identity: a Standard-compatible Sybil-resistant Pseudonym Extension for C-ITS

Authors: Ye Tao, Hongyi Wu, Ehsan Javanmardi, Manabu Tsukada, Hiroshi Esaki

Abstract: Pseudonyms are widely used in Cooperative Intelligent Transport Systems (C-ITS) to protect the location privacy of vehicles. However, the unlinkability nature of pseudonyms also enables Sybil attacks, where a malicious vehicle can pretend to be multiple vehicles at the same time. In this paper, we propose a novel protocol called zero-knowledge Proof of Distinct Identity (zk-PoDI,) which allows a v… ▽ More Pseudonyms are widely used in Cooperative Intelligent Transport Systems (C-ITS) to protect the location privacy of vehicles. However, the unlinkability nature of pseudonyms also enables Sybil attacks, where a malicious vehicle can pretend to be multiple vehicles at the same time. In this paper, we propose a novel protocol called zero-knowledge Proof of Distinct Identity (zk-PoDI,) which allows a vehicle to prove that it is not the owner of another pseudonym in the local area, without revealing its actual identity. Zk-PoDI is based on the Diophantine equation and zk-SNARK, and does not rely on any specific pseudonym design or infrastructure assistance. We show that zk-PoDI satisfies all the requirements for a practical Sybil-resistance pseudonym system, and it has low latency, adjustable difficulty, moderate computation overhead, and negligible communication cost. We also discuss the future work of implementing and evaluating zk-PoDI in a realistic city-scale simulation environment. △ Less

Submitted 3 May, 2024; v1 submitted 20 March, 2024; originally announced March 2024.

Comments: Accepted for publication at IEEE IV 2024

A Rule-Compliance Path Planner for Lane-Merge Scenarios Based on Responsibility-Sensitive Safety

Authors: Pengfei Lin, Ehsan Javanmardi, Yuze Jiang, Manabu Tsukada

Abstract: Lane merging is one of the critical tasks for self-driving cars, and how to perform lane-merge maneuvers effectively and safely has become one of the important standards in measuring the capability of autonomous driving systems. However, due to the ambiguity in driving intentions and right-of-way issues, the lane merging process in autonomous driving remains deficient in terms of maintaining or ce… ▽ More Lane merging is one of the critical tasks for self-driving cars, and how to perform lane-merge maneuvers effectively and safely has become one of the important standards in measuring the capability of autonomous driving systems. However, due to the ambiguity in driving intentions and right-of-way issues, the lane merging process in autonomous driving remains deficient in terms of maintaining or ceding the right-of-way and attributing liability, which could result in protracted durations for merging and problems such as trajectory oscillation. Hence, we present a rule-compliance path planner (RCPP) for lane-merge scenarios, which initially employs the extended responsibility-sensitive safety (RSS) to elucidate the right-of-way, followed by the potential field-based sigmoid planner for path generation. In the simulation, we have validated the efficacy of the proposed algorithm. The algorithm demonstrated superior performance over previous approaches in aspects such as merging time (Saved 72.3%), path length (reduced 53.4%), and eliminating the trajectory oscillation. △ Less

Submitted 19 March, 2024; originally announced March 2024.

Comments: Submitted to IEEE IROS 2024

RaceMOP: Mapless Online Path Planning for Multi-Agent Autonomous Racing using Residual Policy Learning

Authors: Raphael Trumpp, Ehsan Javanmardi, Jin Nakazato, Manabu Tsukada, Marco Caccamo

Abstract: The interactive decision-making in multi-agent autonomous racing offers insights valuable beyond the domain of self-driving cars. Mapless online path planning is particularly of practical appeal but poses a challenge for safely overtaking opponents due to the limited planning horizon. To address this, we introduce RaceMOP, a novel method for mapless online path planning designed for multi-agent ra… ▽ More The interactive decision-making in multi-agent autonomous racing offers insights valuable beyond the domain of self-driving cars. Mapless online path planning is particularly of practical appeal but poses a challenge for safely overtaking opponents due to the limited planning horizon. To address this, we introduce RaceMOP, a novel method for mapless online path planning designed for multi-agent racing of F1TENTH cars. Unlike classical planners that rely on predefined racing lines, RaceMOP operates without a map, utilizing only local observations to execute high-speed overtaking maneuvers. Our approach combines an artificial potential field method as a base policy with residual policy learning to enable long-horizon planning. We advance the field by introducing a novel approach for policy fusion with the residual policy directly in probability space. Extensive experiments on twelve simulated racetracks validate that RaceMOP is capable of long-horizon decision-making with robust collision avoidance during overtaking maneuvers. RaceMOP demonstrates superior handling over existing mapless planners and generalizes to unknown racetracks, affirming its potential for broader applications in robotics. Our code is available at http://github.com/raphajaner/racemop. △ Less

Submitted 16 September, 2024; v1 submitted 11 March, 2024; originally announced March 2024.

Comments: IEEE/RSJ International Conference on Intelligent Robots and Systems 2024

Improving Low-Light Image Recognition Performance Based on Image-adaptive Learnable Module

Authors: Seitaro Ono, Yuka Ogino, Takahiro Toizumi, Atsushi Ito, Masato Tsukada

Abstract: In recent years, significant progress has been made in image recognition technology based on deep neural networks. However, improving recognition performance under low-light conditions remains a significant challenge. This study addresses the enhancement of recognition model performance in low-light conditions. We propose an image-adaptive learnable module which apply appropriate image processing… ▽ More In recent years, significant progress has been made in image recognition technology based on deep neural networks. However, improving recognition performance under low-light conditions remains a significant challenge. This study addresses the enhancement of recognition model performance in low-light conditions. We propose an image-adaptive learnable module which apply appropriate image processing on input images and a hyperparameter predictor to forecast optimal parameters used in the module. Our proposed approach allows for the enhancement of recognition performance under low-light conditions by easily integrating as a front-end filter without the need to retrain existing recognition models designed for low-light conditions. Through experiments, our proposed method demonstrates its contribution to enhancing image recognition performance under low-light conditions. △ Less

Submitted 7 January, 2025; v1 submitted 12 January, 2024; originally announced January 2024.

Comments: accepted to VISAPP2024

Zero-Knowledge Proof of Traffic: A Deterministic and Privacy-Preserving Cross Verification Mechanism for Cooperative Perception Data

Authors: Ye Tao, Ehsan Javanmardi, Pengfei Lin, Jin Nakazato, Yuze Jiang, Manabu Tsukada, Hiroshi Esaki

Abstract: Cooperative perception is crucial for connected automated vehicles in intelligent transportation systems (ITSs); however, ensuring the authenticity of perception data remains a challenge as the vehicles cannot verify events that they do not witness independently. Various studies have been conducted on establishing the authenticity of data, such as trust-based statistical methods and plausibility-b… ▽ More Cooperative perception is crucial for connected automated vehicles in intelligent transportation systems (ITSs); however, ensuring the authenticity of perception data remains a challenge as the vehicles cannot verify events that they do not witness independently. Various studies have been conducted on establishing the authenticity of data, such as trust-based statistical methods and plausibility-based methods. However, these methods are limited as they require prior knowledge such as previous sender behaviors or predefined rules to evaluate the authenticity. To overcome this limitation, this study proposes a novel approach called zero-knowledge Proof of Traffic (zk-PoT), which involves generating cryptographic proofs to the traffic observations. Multiple independent proofs regarding the same vehicle can be deterministically cross-verified by any receivers without relying on ground truth, probabilistic, or plausibility evaluations. Additionally, no private information is compromised during the entire procedure. A full on-board unit software stack that reflects the behavior of zk-PoT is implemented within a specifically designed simulator called Flowsim. A comprehensive experimental analysis is then conducted using synthesized city-scale simulations, which demonstrates that zk-PoT's cross-verification ratio ranges between 80 % to 96 %, and 80 % of the verification is achieved in 2 s, with a protocol overhead of approximately 25 %. Furthermore, the analyses of various attacks indicate that most of the attacks could be prevented, and some, such as collusion attacks, can be mitigated. The proposed approach can be incorporated into existing works, including the European Telecommunications Standards Institute (ETSI) and the International Organization for Standardization (ISO) ITS standards, without disrupting the backward compatibility. △ Less

Submitted 13 December, 2023; originally announced December 2023.

Roadside LiDAR Assisted Cooperative Localization for Connected Autonomous Vehicles

Authors: Yuze Jiang, Ehsan Javanmardi, Jin Nakazato, Manabu Tsukada, Hiroshi Esaki

Abstract: Advancements in LiDAR technology have led to more cost-effective production while simultaneously improving precision and resolution. As a result, LiDAR has become integral to vehicle localization, achieving centimeter-level accuracy through techniques like Normal Distributions Transform (NDT) and other advanced 3D registration algorithms. Nonetheless, these approaches are reliant on high-definitio… ▽ More Advancements in LiDAR technology have led to more cost-effective production while simultaneously improving precision and resolution. As a result, LiDAR has become integral to vehicle localization, achieving centimeter-level accuracy through techniques like Normal Distributions Transform (NDT) and other advanced 3D registration algorithms. Nonetheless, these approaches are reliant on high-definition 3D point cloud maps, the creation of which involves significant expenditure. When such maps are unavailable or lack sufficient features for 3D registration algorithms, localization accuracy diminishes, posing a risk to road safety. To address this, we proposed to use LiDAR-equipped roadside unit and Vehicle-to-Infrastructure (V2I) communication to accurately estimate the connected autonomous vehicle's position and help the vehicle when its self-localization is not accurate enough. Our simulation results indicate that this method outperforms traditional NDT scan matching-based approaches in terms of localization accuracy. △ Less

Submitted 11 July, 2024; v1 submitted 14 November, 2023; originally announced November 2023.

Comments: Accepted by 2023 International Conference on Intelligent Computing and its Emerging Applications

Clothoid Curve-based Emergency-Stopping Path Planning with Adaptive Potential Field for Autonomous Vehicles

Authors: Pengfei Lin, Ehsan Javanmardi, Manabu Tsukada

Abstract: The Potential Field (PF)-based path planning method is widely adopted for autonomous vehicles (AVs) due to its real-time efficiency and simplicity. PF often creates a rigid road boundary, and while this ensures that the ego vehicle consistently operates within the confines of the road, it also brings a lurking peril in emergency scenarios. If nearby vehicles suddenly switch lanes, the AV has to ve… ▽ More The Potential Field (PF)-based path planning method is widely adopted for autonomous vehicles (AVs) due to its real-time efficiency and simplicity. PF often creates a rigid road boundary, and while this ensures that the ego vehicle consistently operates within the confines of the road, it also brings a lurking peril in emergency scenarios. If nearby vehicles suddenly switch lanes, the AV has to veer off and brake to evade a collision, leading to the "blind alley" effect. In such a situation, the vehicle can become trapped or confused by the conflicting forces from the obstacle vehicle PF and road boundary PF, often resulting in indecision or erratic behavior, even crashes. To address the above-mentioned challenges, this research introduces an Emergency-Stopping Path Planning (ESPP) that incorporates an adaptive PF (APF) and a clothoid curve for urgent evasion. First, we design an emergency triggering estimation to detect the "blind alley" problem by analyzing the PF distribution. Second, we regionalize the driving scene to search the optimal breach point on the road PF and the final stopping point for the vehicle by considering the possible motion range of the obstacle. Finally, we use the optimized clothoid curve to fit these calculated points under vehicle dynamics constraints to generate a smooth emergency avoidance path. The proposed ESPP-based APF method was evaluated by conducting the co-simulation between MATLAB/Simulink and CarSim Simulator in a freeway scene. The simulation results reveal that the proposed method shows increased performance in emergency collision avoidance and renders the vehicle safer, in which the duration of wheel slip is 61.9% shorter, and the maximum steering angle amplitude is 76.9% lower than other potential field-based methods. △ Less

Submitted 19 August, 2023; originally announced August 2023.

Comments: 14 pages, 20 figures, journal paper in submission

Occlusion-Aware Path Planning for Collision Avoidance: Leveraging Potential Field Method with Responsibility-Sensitive Safety

Authors: Pengfei Lin, Ehsan Javanmardi, Jin Nakazato, Manabu Tsukada

Abstract: Collision avoidance (CA) has always been the foremost task for autonomous vehicles (AVs) under safety criteria. And path planning is directly responsible for generating a safe path to accomplish CA while satisfying other commands. Due to the real-time computation and simple structure, the potential field (PF) has emerged as one of the mainstream path-planning algorithms. However, the current PF is… ▽ More Collision avoidance (CA) has always been the foremost task for autonomous vehicles (AVs) under safety criteria. And path planning is directly responsible for generating a safe path to accomplish CA while satisfying other commands. Due to the real-time computation and simple structure, the potential field (PF) has emerged as one of the mainstream path-planning algorithms. However, the current PF is primarily simulated in ideal CA scenarios, assuming complete obstacle information while disregarding occlusion issues where obstacles can be partially or entirely hidden from the AV's sensors. During the occlusion period, the occluded obstacles do not possess a PF. Once the occlusion is over, these obstacles can generate an instantaneous virtual force that impacts the ego vehicle. Therefore, we propose an occlusion-aware path planning (OAPP) with the responsibility-sensitive safety (RSS)-based PF to tackle the occlusion problem for non-connected AVs. We first categorize the detected and occluded obstacles, and then we proceed to the RSS violation check. Finally, we can generate different virtual forces from the PF for occluded and non-occluded obstacles. We compare the proposed OAPP method with other PF-based path planning methods via MATLAB/Simulink. The simulation results indicate that the proposed method can eliminate instantaneous lateral oscillation or sway and produce a smoother path than conventional PF methods. △ Less

Submitted 12 June, 2023; originally announced June 2023.

Comments: Submitted to IEEE ITSC 2023

Potential Field-based Path Planning with Interactive Speed Optimization for Autonomous Vehicles

Authors: Pengfei Lin, Ehsan Javanmardi, Jin Nakazato, Manabu Tsukada

Abstract: Path planning is critical for autonomous vehicles (AVs) to determine the optimal route while considering constraints and objectives. The potential field (PF) approach has become prevalent in path planning due to its simple structure and computational efficiency. However, current PF methods used in AVs focus solely on the path generation of the ego vehicle while assuming that the surrounding obstac… ▽ More Path planning is critical for autonomous vehicles (AVs) to determine the optimal route while considering constraints and objectives. The potential field (PF) approach has become prevalent in path planning due to its simple structure and computational efficiency. However, current PF methods used in AVs focus solely on the path generation of the ego vehicle while assuming that the surrounding obstacle vehicles drive at a preset behavior without the PF-based path planner, which ignores the fact that the ego vehicle's PF could also impact the path generation of the obstacle vehicles. To tackle this problem, we propose a PF-based path planning approach where local paths are shared among ego and obstacle vehicles via vehicle-to-vehicle (V2V) communication. Then by integrating this shared local path into an objective function, a new optimization function called interactive speed optimization (ISO) is designed to allow driving safety and comfort for both ego and obstacle vehicles. The proposed method is evaluated using MATLAB/Simulink in the urgent merging scenarios by comparing it with conventional methods. The simulation results indicate that the proposed method can mitigate the impact of other AVs' PFs by slowing down in advance, effectively reducing the oscillations for both ego and obstacle AVs. △ Less

Submitted 12 June, 2023; originally announced June 2023.

Comments: Submitted to IEEE IECON 2023

Time-to-Collision-Aware Lane-Change Strategy Based on Potential Field and Cubic Polynomial for Autonomous Vehicles

Authors: Pengfei Lin, Ehsan Javanmardi, Ye Tao, Vishal Chauhan, Jin Nakazato, Manabu Tsukada

Abstract: Making safe and successful lane changes (LCs) is one of the many vitally important functions of autonomous vehicles (AVs) that are needed to ensure safe driving on expressways. Recently, the simplicity and real-time performance of the potential field (PF) method have been leveraged to design decision and planning modules for AVs. However, the LC trajectory planned by the PF method is usually lengt… ▽ More Making safe and successful lane changes (LCs) is one of the many vitally important functions of autonomous vehicles (AVs) that are needed to ensure safe driving on expressways. Recently, the simplicity and real-time performance of the potential field (PF) method have been leveraged to design decision and planning modules for AVs. However, the LC trajectory planned by the PF method is usually lengthy and takes the ego vehicle laterally parallel and close to the obstacle vehicle, which creates a dangerous situation if the obstacle vehicle suddenly steers. To mitigate this risk, we propose a time-to-collision-aware LC (TTCA-LC) strategy based on the PF and cubic polynomial in which the TTC constraint is imposed in the optimized curve fitting. The proposed approach is evaluated using MATLAB/Simulink under high-speed conditions in a comparative driving scenario. The simulation results indicate that the TTCA-LC method performs better than the conventional PF-based LC (CPF-LC) method in generating shorter, safer, and smoother trajectories. The length of the LC trajectory is shortened by over 27.1\%, and the curvature is reduced by approximately 56.1\% compared with the CPF-LC method. △ Less

Submitted 12 June, 2023; originally announced June 2023.

Comments: Accepted in IEEE Intelligent Vehicles Symposium (IV) 2023

Flowsim: A Modular Simulation Platform for Microscopic Behavior Analysis of City-Scale Connected Autonomous Vehicles

Authors: Ye Tao, Ehsan Javanmardi, Jin Nakazato, Manabu Tsukada, Hiroshi Esaki

Abstract: As connected autonomous vehicles (CAVs) become increasingly prevalent, there is a growing need for simulation platforms that can accurately evaluate CAV behavior in large-scale environments. In this paper, we propose Flowsim, a novel simulator specifically designed to meet these requirements. Flowsim offers a modular and extensible architecture that enables the analysis of CAV behaviors in large-s… ▽ More As connected autonomous vehicles (CAVs) become increasingly prevalent, there is a growing need for simulation platforms that can accurately evaluate CAV behavior in large-scale environments. In this paper, we propose Flowsim, a novel simulator specifically designed to meet these requirements. Flowsim offers a modular and extensible architecture that enables the analysis of CAV behaviors in large-scale scenarios. It provides researchers with a customizable platform for studying CAV interactions, evaluating communication and networking protocols, assessing cybersecurity vulnerabilities, optimizing traffic management strategies, and developing and evaluating policies for CAV deployment. Flowsim is implemented in pure Python in approximately 1,500 lines of code, making it highly readable, understandable, and easily modifiable. We verified the functionality and performance of Flowsim via a series of experiments based on realistic traffic scenarios. The results show the effectiveness of Flowsim in providing a flexible and powerful simulation environment for evaluating CAV behavior and data flow. Flowsim is a valuable tool for researchers, policymakers, and industry professionals who are involved in the development, evaluation, and deployment of CAVs. The code of Flowsim is publicly available on GitHub under the MIT license. △ Less

Submitted 9 June, 2023; originally announced June 2023.

zk-PoT: Zero-Knowledge Proof of Traffic for Privacy Enabled Cooperative Perception

Authors: Ye Tao, Yuze Jiang, Pengfei Lin, Manabu Tsukada, Hiroshi Esaki

Abstract: Cooperative perception is an essential and widely discussed application of connected automated vehicles. However, the authenticity of perception data is not ensured, because the vehicles cannot independently verify the event they did not see. Many methods, including trust-based (i.e., statistical) approaches and plausibility-based methods, have been proposed to determine data authenticity. However… ▽ More Cooperative perception is an essential and widely discussed application of connected automated vehicles. However, the authenticity of perception data is not ensured, because the vehicles cannot independently verify the event they did not see. Many methods, including trust-based (i.e., statistical) approaches and plausibility-based methods, have been proposed to determine data authenticity. However, these methods cannot verify data without a priori knowledge. In this study, a novel approach of constructing a self-proving data from the number plate of target vehicles was proposed. By regarding the pseudonym and number plate as a shared secret and letting multiple vehicles prove they know it independently, the data authenticity problem can be transformed to a cryptography problem that can be solved without trust or plausibility evaluations. Our work can be adapted to the existing works including ETSI/ISO ITS standards while maintaining backward compatibility. Analyses of common attacks and attacks specific to the proposed method reveal that most attacks can be prevented, whereas preventing some other attacks, such as collusion attacks, can be mitigated. Experiments based on realistic data set show that the rate of successful verification can achieve 70\% to 80\% at rush hours. △ Less

Submitted 14 November, 2022; originally announced November 2022.

Comments: IEEE Consumer Communications & Networking Conference (CCNC) 2023

Segmentation-free Direct Iris Localization Networks

Authors: Takahiro Toizumi, Koichi Takahashi, Masato Tsukada

Abstract: This paper proposes an efficient iris localization method without using iris segmentation and circle fitting. Conventional iris localization methods first extract iris regions by using semantic segmentation methods such as U-Net. Afterward, the inner and outer iris circles are localized using the traditional circle fitting algorithm. However, this approach requires high-resolution encoder-decoder… ▽ More This paper proposes an efficient iris localization method without using iris segmentation and circle fitting. Conventional iris localization methods first extract iris regions by using semantic segmentation methods such as U-Net. Afterward, the inner and outer iris circles are localized using the traditional circle fitting algorithm. However, this approach requires high-resolution encoder-decoder networks for iris segmentation, so it causes computational costs to be high. In addition, traditional circle fitting tends to be sensitive to noise in input images and fitting parameters, causing the iris recognition performance to be poor. To solve these problems, we propose an iris localization network (ILN), that can directly localize pupil and iris circles with eyelid points from a low-resolution iris image. We also introduce a pupil refinement network (PRN) to improve the accuracy of pupil localization. Experimental results show that the combination of ILN and PRN works in 34.5 ms for one iris image on a CPU, and its localization performance outperforms conventional iris segmentation methods. In addition, generalized evaluation results show that the proposed method has higher robustness for datasets in different domain than other segmentation methods. Furthermore, we also confirm that the proposed ILN and PRN improve the iris recognition accuracy. △ Less

Submitted 19 October, 2022; originally announced October 2022.

Comments: Accepted by WACV 2023

Fast Eye Detector Using Siamese Network for NIR Partial Face Images

Authors: Yuka Ogino, Yuho Shoji, Takahiro Toizumi, Ryoma Oami, Masato Tsukada

Abstract: This paper proposes a fast eye detection method that is based on a Siamese network for near infrared (NIR) partial face images. NIR partial face images do not include the whole face of a subject since they are captured using iris recognition systems with the constraint of frame rate and resolution. The iris recognition systems such as the iris on the move (IOTM) system require fast and accurate ey… ▽ More This paper proposes a fast eye detection method that is based on a Siamese network for near infrared (NIR) partial face images. NIR partial face images do not include the whole face of a subject since they are captured using iris recognition systems with the constraint of frame rate and resolution. The iris recognition systems such as the iris on the move (IOTM) system require fast and accurate eye detection as a pre-process. Our goal is to design eye detection with high speed, high discrimination performance between left and right eyes, and high positional accuracy of eye center. Our method adopts a Siamese network and coarse to fine position estimation with a fast lightweight CNN backbone. The network outputs features of images and the similarity map indicating coarse position of an eye. A regression on a portion of a feature with high similarity refines the coarse position of the eye to obtain the fine position with high accuracy. We demonstrate the effectiveness of the proposed method by comparing it with conventional methods, including SOTA, in terms of the positional accuracy, the discrimination performance, and the processing speed. Our method achieves superior performance in speed. △ Less

Submitted 4 January, 2023; v1 submitted 22 February, 2022; originally announced February 2022.

WebRTC-based measurement tool for peer-to-peer applications and preliminary findings with real users

Authors: Kosuke Nakagawa, Manabu Tsukada, Keiichi Shima, Hiroshi Esaki

Abstract: Direct peer-to-peer (P2P) communication is often used to minimize the end-to-end latency for real-time applications that require accurate synchronization, such as remote musical ensembles. However, there are few studies on the performance of P2P communication between home network environments, thus hindering the deployment of services that require synchronization. In this study, we developed a P2P… ▽ More Direct peer-to-peer (P2P) communication is often used to minimize the end-to-end latency for real-time applications that require accurate synchronization, such as remote musical ensembles. However, there are few studies on the performance of P2P communication between home network environments, thus hindering the deployment of services that require synchronization. In this study, we developed a P2P performance measurement tool using the Web Real-Time Communication (WebRTC) statistics application programming interface. Using this tool, we can easily measure P2P performance between home network environments on a web browser without downloading client applications. We also verified the reliability of round-trip time (RTT) measurements using WebRTC and confirmed that our system could provide the necessary measurement accuracy for RTT and jitter measurements for real-time applications. In addition, we measured the performance of a full mesh topology connection with 10 users in an actual environment in Japan. Consequently, we found that only 66% of the peer connections had a latency of 30 ms or less, which is the minimum requirement for high synchronization applications, such as musical ensembles. △ Less

Submitted 3 December, 2021; originally announced December 2021.

Comments: Accepted to 16th Asian Internet Engineering Conference (AINTEC), 2021

MAC address randomization tolerant crowd monitoring system using Wi-Fi packets

Authors: Yuyi Cai, Manabu Tsukada, Hideya Ochiai, Hiroshi Esaki

Abstract: Media access control (MAC) addresses inside Wi-Fi packets can be used for beneficial activities such as crowdedness estimation, marketing, and hazard maps. However, the MAC address randomization systems introduced around 2014 make all conventional MAC-address-based crowd monitoring systems count the same device more than once. Therefore, there is a need to create a new crowd monitoring system tole… ▽ More Media access control (MAC) addresses inside Wi-Fi packets can be used for beneficial activities such as crowdedness estimation, marketing, and hazard maps. However, the MAC address randomization systems introduced around 2014 make all conventional MAC-address-based crowd monitoring systems count the same device more than once. Therefore, there is a need to create a new crowd monitoring system tolerant to MAC address randomization to estimate the number of devices accurately. In this paper, Vision and TrueSight, two new crowd monitoring algorithms that estimate the number of devices, are proposed to prove that MAC-address-based crowd monitoring is still possible. In addition to probe requests, Vision uses data packets and beacon packets to mitigate the influence of randomization. Moreover, TrueSight uses sequence numbers and hierarchical clustering to estimate the number of devices. The experimental results of this study show that even without installing any special software, Vision can gather 440 randomly generated MAC addresses into one group and count only once, and TrueSight can estimate the number of devices with an accuracy of more than 75% with an acceptable error range of 1. △ Less

Submitted 3 December, 2021; originally announced December 2021.

Comments: Accepted to 16th Asian Internet Engineering Conference (AINTEC), 2021

Misbehavior Detection Using Collective Perception under Privacy Considerations

Authors: Manabu Tsukada, Shimpei Arii, Hideya Ochiai, Hiroshi Esaki

Abstract: In cooperative ITS, security and privacy protection are essential. Cooperative Awareness Message (CAM) is a basic V2V message standard, and misbehavior detection is critical for protection against attacking CAMs from the inside system, in addition to node authentication by Public Key Infrastructure (PKI). On the contrary, pseudonym IDs, which have been introduced to protect privacy from tracking,… ▽ More In cooperative ITS, security and privacy protection are essential. Cooperative Awareness Message (CAM) is a basic V2V message standard, and misbehavior detection is critical for protection against attacking CAMs from the inside system, in addition to node authentication by Public Key Infrastructure (PKI). On the contrary, pseudonym IDs, which have been introduced to protect privacy from tracking, make it challenging to perform misbehavior detection. In this study, we improve the performance of misbehavior detection using observation data of other vehicles. This is referred to as collective perception message (CPM), which is becoming the new standard in European countries. We have experimented using realistic traffic scenarios and succeeded in reducing the rate of rejecting valid CAMs (false positive) by approximately 15 percentage points while maintaining the rate of correctly detecting attacks (true positive). △ Less

Submitted 2 November, 2021; originally announced November 2021.

Comments: Accepted to 2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC)

Reinforcement Learning Based Optimal Camera Placement for Depth Observation of Indoor Scenes

Authors: Yichuan Chen, Manabu Tsukada, Hiroshi Esaki

Abstract: Exploring the most task-friendly camera setting -- optimal camera placement (OCP) problem -- in tasks that use multiple cameras is of great importance. However, few existing OCP solutions specialize in depth observation of indoor scenes, and most versatile solutions work offline. To this problem, an OCP online solution to depth observation of indoor scenes based on reinforcement learning is propos… ▽ More Exploring the most task-friendly camera setting -- optimal camera placement (OCP) problem -- in tasks that use multiple cameras is of great importance. However, few existing OCP solutions specialize in depth observation of indoor scenes, and most versatile solutions work offline. To this problem, an OCP online solution to depth observation of indoor scenes based on reinforcement learning is proposed in this paper. The proposed solution comprises a simulation environment that implements scene observation and reward estimation using shadow maps and an agent network containing a soft actor-critic (SAC)-based reinforcement learning backbone and a feature extractor to extract features from the observed point cloud layer-by-layer. Comparative experiments with two state-of-the-art optimization-based offline methods are conducted. The experimental results indicate that the proposed system outperforms seven out of ten test scenes in obtaining lower depth observation error. The total error in all test scenes is also less than 90% of the baseline ones. Therefore, the proposed system is more competent for depth camera placement in scenarios where there is no prior knowledge of the scenes or where a lower depth observation error is the main objective. △ Less

Submitted 21 October, 2021; originally announced October 2021.

Comments: Accepted to IEEE International Conference on Networking, Sensing and Control (ICNSC) 2021

Roadside-assisted Cooperative Planning using Future Path Sharing for Autonomous Driving

Authors: Mai Hirata, Manabu Tsukada, Keisuke Okumura, Yasumasa Tamura, Hideya Ochiai, Xavier Défago

Abstract: Cooperative intelligent transportation systems (ITS) are used by autonomous vehicles to communicate with surrounding autonomous vehicles and roadside units (RSU). Current C-ITS applications focus primarily on real-time information sharing, such as cooperative perception. In addition to real-time information sharing, self-driving cars need to coordinate their action plans to achieve higher safety a… ▽ More Cooperative intelligent transportation systems (ITS) are used by autonomous vehicles to communicate with surrounding autonomous vehicles and roadside units (RSU). Current C-ITS applications focus primarily on real-time information sharing, such as cooperative perception. In addition to real-time information sharing, self-driving cars need to coordinate their action plans to achieve higher safety and efficiency. For this reason, this study defines a vehicle's future action plan/path and designs a cooperative path-planning model at intersections using future path sharing based on the future path information of multiple vehicles. The notion is that when the RSU detects a potential conflict of vehicle paths or an acceleration opportunity according to the shared future paths, it will generate a coordinated path update that adjusts the speeds of the vehicles. We implemented the proposed method using the open-source Autoware autonomous driving software and evaluated it with the LGSVL autonomous vehicle simulator. We conducted simulation experiments with two vehicles at a blind intersection scenario, finding that each car can travel safely and more efficiently by planning a path that reflects the action plans of all vehicles involved. The time consumed by introducing the RSU is 23.0 % and 28.1 % shorter than that of the stand-alone autonomous driving case at the intersection. △ Less

Submitted 10 August, 2021; originally announced August 2021.

Comments: Accepted to IEEE 94th Vehicular Technology Conference (VTC2021-Fall)

AutoMCM: Maneuver Coordination Service with Abstracted Functions for Autonomous Driving

Authors: Masaya Mizutani, Manabu Tsukada, Hiroshi Esaki

Abstract: A cooperative intelligent transport system (C-ITS) uses vehicle-to-everything (V2X) technology to make self-driving vehicles safer and more efficient. Current C-ITS applications have mainly focused on real-time information sharing, such as for cooperative perception. In addition to better real-time perception, self-driving vehicles need to achieve higher safety and efficiency by coordinating actio… ▽ More A cooperative intelligent transport system (C-ITS) uses vehicle-to-everything (V2X) technology to make self-driving vehicles safer and more efficient. Current C-ITS applications have mainly focused on real-time information sharing, such as for cooperative perception. In addition to better real-time perception, self-driving vehicles need to achieve higher safety and efficiency by coordinating action plans. This study designs a maneuver coordination (MC) protocol that uses seven messages to cover various scenarios and an abstracted MC support service. We implement our proposal as AutoMCM by extending two open-source software tools: Autoware for autonomous driving and OpenC2X for C-ITS. The results show that our system effectively reduces the communication bandwidth by limiting message exchange in an event-driven manner. Furthermore, it shows that the vehicles run 15% faster when the vehicle speed is 30 km/h and 28% faster when the vehicle speed is 50 km/h using our scheme. Our system shows robustness against packet loss in experiments when the message timeout parameters are appropriately set. △ Less

Submitted 14 July, 2021; originally announced July 2021.

Comments: Accepted to 24th IEEE International Conference on Intelligent Transportation (ITSC) 2021