Showing 1–50 of 140 results for author: Henriques, J

AI Research Preference Models

Authors: Thomas Simon Foster, Bassel Al Omari, Tingchen Fu, Thomas Mann, Carl Domond, Lucia Cipolina-Kun, Bhavul Gauri, Muna Aghamelu, Alexander D. Goldie, Eryk Helenowski, Jean-Christophe Gagnon-Audet, Alberto Pepe, Saba Nazir, Daniel Izcovich, Noam Levi, Rishi Hazra, Karen Hambardzumyan, Nicolas Baldwin, Xian Li, Martin Josifoski, Paris Giampouras, Masoud Jalili Sabet, Anya Sims, Hela Momand, Tatiana Shavrina , et al. (8 additional authors not shown)

Abstract: AI research agents (AIRA) can now propose, implement, and evaluate their own machine learning experiments, but progress on frontier tasks is throttled by cost: a candidate solution can be written in minutes, whereas evaluating it can take hours to days of GPU time. An agent can therefore propose far more candidates than it can afford to run, and its progress depends on its research preference: how… ▽ More AI research agents (AIRA) can now propose, implement, and evaluate their own machine learning experiments, but progress on frontier tasks is throttled by cost: a candidate solution can be written in minutes, whereas evaluating it can take hours to days of GPU time. An agent can therefore propose far more candidates than it can afford to run, and its progress depends on its research preference: how it allocates a fixed execution budget across many candidates. We introduce AI Research Preference Models (RPMs) that predict which of multiple candidate solutions are most worth executing, without paying the cost of executing them all. We build RPMs from frozen pretrained language models (with no task-specific training), in two forms: an inference-only model that reasons over candidate plans, code, and prior executed solutions, and an agentic model that additionally runs small-scale pilot experiments before deciding. We integrate both into the AIRA-dojo search agent and evaluate on AIRS-Bench, a recent benchmark of machine learning research tasks for AI research agents. The two variants raise the average normalized score from 0.684 to 0.711 and 0.729 respectively, and reach the unguided agent's 24-hour performance in roughly 15 hours, using less than two-thirds of its execution budget. Our best RPMs also yield new state-of-the-art results on two AIRS-Bench tasks. △ Less

Submitted 14 August, 2026; originally announced August 2026.

Comments: 34 pages, 17 figures, 6 tables

HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams

Authors: Shivani Mall, Swarnim Jain, Joao F. Henriques

Abstract: Much of the recent progress in image and video recognition has come at the cost of memory: larger models, increased resolution, and longer temporal contexts. An inevitable component is the quadratic (or larger) growth of memory and compute based on image resolution, which is a property of the grid sampling used in convolutional networks and vision transformers. In this work we study residual netwo… ▽ More Much of the recent progress in image and video recognition has come at the cost of memory: larger models, increased resolution, and longer temporal contexts. An inevitable component is the quadratic (or larger) growth of memory and compute based on image resolution, which is a property of the grid sampling used in convolutional networks and vision transformers. In this work we study residual networks whose convolutional blocks have logarithmic-square growth instead, enabling them to process very high-resolution video quickly. The key insight is to use a residual architecture's residual stream as a high-resolution buffer, to which convolutional blocks only read and write via log-polar image warp operations. Layers adaptively focus on different parts of each frame, with very high resolution only near the focus point. A complete high-resolution representation is built up in the residual stream, analogous to eye saccades creating a complete picture in biological vision, and a theoretical construction is presented that eliminates the quadratic dependency of the residual stream resolution. Experiments demonstrate that our proposed HiResNets learn to foveate around scenes similarly to human vision, and have superior performance in difficult egocentric video recognition tasks, especially egocentric video with small objects and fine-grained recognition. △ Less

Submitted 5 August, 2026; v1 submitted 3 August, 2026; originally announced August 2026.

Finding the Time to Think: Learning Planning Budgets in Real-Time RL

Authors: Aneesh Muppidi, Firas Darwish, Dylan Cope, João F. Henriques, Jakob Nicolaus Foerster

Abstract: Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the ag… ▽ More Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs. △ Less

Submitted 27 June, 2026; v1 submitted 24 June, 2026; originally announced June 2026.

Rendering Novel Views of MRI Using 3D Gaussian Splatting

Authors: Robin Y. Park, Mark C. Eid, Rhydian Windsor, Amir Jamaludin, Ana I. L. Namburete, João F. Henriques, Andrew Zisserman

Abstract: The objective of this paper is to improve radiological gradings measured on MRIs of spines, by resampling scans so that the new view planes are better aligned with the target anatomy than the original sparse images. To this end, we adapt 3D Gaussian Splatting to form a volumetric reconstruction starting from sparse anisotropic MRIs, and imaging planes aligned with the anatomy relevant for clinical… ▽ More The objective of this paper is to improve radiological gradings measured on MRIs of spines, by resampling scans so that the new view planes are better aligned with the target anatomy than the original sparse images. To this end, we adapt 3D Gaussian Splatting to form a volumetric reconstruction starting from sparse anisotropic MRIs, and imaging planes aligned with the anatomy relevant for clinical evaluation are then sampled and rendered. The novel view plane is optimal for diagnostic radiological grading of the target anatomy, whereas the original MRI is not. The resampled scans are then used to predict ordinal severity grades of localised stenosis conditions in spinal MRIs. We compare our method against Voxel Interpolation resampling, which takes the average of inverse-distance weighted nearest neighbour intensities for each target coordinate. Experiments show that across all stenosis conditions, resampled scans using Gaussian Splatting produce more accurate stenosis gradings compared to the raw scans which do not include the complete anatomy in-plane, as well as images resampled using Voxel Interpolation. △ Less

Submitted 24 June, 2026; originally announced June 2026.

PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models

Authors: Lorenza Prospero, Orest Kupyn, Ostap Viniavskyi, João F. Henriques, Christian Rupprecht

Abstract: Acquiring labeled datasets for 3D human mesh estimation is challenging due to depth ambiguities and the inherent difficulty of annotating 3D geometry from monocular images. Existing datasets are either real, with manually annotated 3D geometry and limited scale, or synthetic, rendered from 3D engines that provide precise labels but suffer from limited photorealism, low diversity, and high producti… ▽ More Acquiring labeled datasets for 3D human mesh estimation is challenging due to depth ambiguities and the inherent difficulty of annotating 3D geometry from monocular images. Existing datasets are either real, with manually annotated 3D geometry and limited scale, or synthetic, rendered from 3D engines that provide precise labels but suffer from limited photorealism, low diversity, and high production costs. In this work, we explore a third path: generated data. We introduce PoseDreamer, a novel pipeline that leverages diffusion models to generate large-scale synthetic datasets with 3D mesh annotations. Our approach combines controllable image generation with Direct Preference Optimization for control alignment, curriculum-based hard sample mining, and multi-stage quality filtering. Together, these components naturally maintain correspondence between 3D labels and generated images, while prioritizing challenging samples to maximize dataset utility. Using PoseDreamer, we generate more than 500,000 high-quality synthetic samples, achieving a 76% improvement in image-quality metrics compared to rendering-based datasets. Models trained on PoseDreamer achieve performance comparable to or superior to those trained on real-world and traditional synthetic datasets. In addition, combining PoseDreamer with synthetic datasets results in better performance than combining real-world and synthetic datasets, demonstrating the complementary nature of our dataset. We will release the full dataset and generation code. △ Less

Submitted 30 March, 2026; originally announced March 2026.

Prediction of room-temperature two-dimensional $π$-electron half-metallic ferrimagnets

Authors: J. Phillips, J. C. G. Henriques, J. Fernández-Rossier, A. T. Costa

Abstract: We propose a strategy to obtain conducting organic materials with fully spin-polarized Fermi surface, lying at a singular flat band, with antiferromagnetically coupled magnetic moments that reside in pi-orbitals of nanographenes. We consider a honeycomb crystal whose unit cell combines two different molecules with S=1/2: an Aza-3-Triangulene, a molecule with orbital degeneracy, and a 2-Triangulene… ▽ More We propose a strategy to obtain conducting organic materials with fully spin-polarized Fermi surface, lying at a singular flat band, with antiferromagnetically coupled magnetic moments that reside in pi-orbitals of nanographenes. We consider a honeycomb crystal whose unit cell combines two different molecules with S=1/2: an Aza-3-Triangulene, a molecule with orbital degeneracy, and a 2-Triangulene. The analyzed system is half-metallic with a ferrimagnetic order, presenting a zero net total magnetic moment per unit cell. We combine density functional theory calculations with a Hubbard model Hamiltonian to compute the magnetic interactions, the bands, the intrinsic Anomalous Hall effect, and the collective spin excitations. We obtain very large intermolecular exchange couplings, in the range of 50 meV, which ensures room temperature stability. When the magnetization is off-plane, intrinsic spin orbit coupling in graphene opens up a topological gap that, despite being very small, leads to a quantized Hall conductance in the tens of mK range. Above 1 Kelvin, the system will behave like a half-metal with fully compensated magnetic moments, thereby combining two characteristics that make it ideal for spintronics applications. △ Less

Submitted 16 April, 2026; v1 submitted 19 February, 2026; originally announced February 2026.

Comments: 9 pages, 7 figures

On the Dangers of Bootstrapping Generation for Continual Learning and Beyond

Authors: Daniil Zverev, A. Sophia Koepke, Joao F. Henriques

Abstract: The use of synthetically generated data for training models is becoming a common practice. While generated data can augment the training data, repeated training on synthetic data raises concerns about distribution drift and degradation of performance due to contamination of the dataset. We investigate the consequences of this bootstrapping process through the lens of continual learning, drawing a… ▽ More The use of synthetically generated data for training models is becoming a common practice. While generated data can augment the training data, repeated training on synthetic data raises concerns about distribution drift and degradation of performance due to contamination of the dataset. We investigate the consequences of this bootstrapping process through the lens of continual learning, drawing a connection to Generative Experience Replay (GER) methods. We present a statistical analysis showing that synthetic data introduces significant bias and variance into training objectives, weakening the reliability of maximum likelihood estimation. We provide empirical evidence showing that popular generative models collapse under repeated training with synthetic data. We quantify this degradation and show that state-of-the-art GER methods fail to maintain alignment in the latent space. Our findings raise critical concerns about the use of synthetic data in continual learning. △ Less

Submitted 5 December, 2025; originally announced December 2025.

Comments: DAGM German Conference on Pattern Recognition, 2025

Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language

Authors: Yan Xia, Letian Shi, Yilin Di, Joao F. Henriques, Daniel Cremers

Abstract: We tackle the problem of localizing 3D point cloud submaps using complex and diverse natural language descriptions, and present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds in a coarse-to-fine localization pipeline. To support benchmarking, we introduce a new city-scale dataset covering both color and non-color point clouds from… ▽ More We tackle the problem of localizing 3D point cloud submaps using complex and diverse natural language descriptions, and present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds in a coarse-to-fine localization pipeline. To support benchmarking, we introduce a new city-scale dataset covering both color and non-color point clouds from diverse urban scenes, and organize location descriptions into three levels of linguistic complexity. In the global place recognition stage, Text2Loc++ combines a pretrained language model with a Hierarchical Transformer with Max pooling (HTM) for sentence-level semantics, and employs an attention-based point cloud encoder for spatial understanding. We further propose Masked Instance Training (MIT) to filter out non-aligned objects and improve multimodal robustness. To enhance the embedding space, we introduce Modality-aware Hierarchical Contrastive Learning (MHCL), incorporating cross-modal, submap-, text-, and instance-level losses. In the fine localization stage, we completely remove explicit text-instance matching and design a lightweight yet powerful framework based on Prototype-based Map Cloning (PMC) and a Cascaded Cross-Attention Transformer (CCAT). Extensive experiments on the KITTI360Pose dataset show that Text2Loc++ outperforms existing methods by up to 15%. In addition, the proposed model exhibits robust generalization when evaluated on the new dataset, effectively handling complex linguistic expressions and a wide variety of urban environments. The code and dataset will be made publicly available. △ Less

Submitted 19 November, 2025; originally announced November 2025.

Comments: This paper builds upon and extends our earlier conference paper Text2Loc presented at CVPR 2024

Ultra-long-range spin coupling in graphene revealed by atomically resolved spin excitations

Authors: Beatriz Viña-Bausá, Antonio. T. Costa, Joao Henriques, Eva Cortés-del Río, Roberto Carrasco, Pierre Mallet, Jean-Yves Veuillen, Joaquín Fernández-Rossier, Iván Brihuega

Abstract: Magnetic interactions between localized spins-1/2 play a central role in quantum magnetism, spin-based quantum computing, and quantum simulation. The range and strength of these interactions are key figures of merit. Here, we probe exchange interactions in pairs and trimers of spins-1/2 introduced by chemisorption of individual hydrogen atoms on graphene. Using scanning tunneling microscopy and in… ▽ More Magnetic interactions between localized spins-1/2 play a central role in quantum magnetism, spin-based quantum computing, and quantum simulation. The range and strength of these interactions are key figures of merit. Here, we probe exchange interactions in pairs and trimers of spins-1/2 introduced by chemisorption of individual hydrogen atoms on graphene. Using scanning tunneling microscopy and inelastic electron tunneling spectroscopy, supported by large-scale mean-field Hubbard calculations, we demonstrate 3 meV exchange couplings at separations beyond 10 nm, surpassing all prior systems. The couplings can be ferro- or antiferromagnetic depending on the relative sublattice arrangement. Real-space mapping of spin excitation amplitudes enables characterization with atomic-resolution. Through atomic manipulation we extend this control to spin trimers, revealing collective spin excitations when pairwise exchange couplings are comparable. △ Less

Submitted 10 November, 2025; originally announced November 2025.

Two topological phases in exchange alternating spin-1 nanographene chains

Authors: João C. G. Henriques, Yelko del Castillo, Ricardo Segundo, Jan Phillips, Joaquín Fernández-Rossier

Abstract: Magnetic nanographenes are emerging as versatile building blocks for artificial spin lattices, enabling the exploration of flagship one-dimensional quantum-magnetism models with unprecedented control. The spin-1 Heisenberg model, including bilinear and biquadratic exchange, was first realized using [3]-triangulenes, revealing the Haldane phase. More recently, Clar's goblets enabled the spin-1/2 He… ▽ More Magnetic nanographenes are emerging as versatile building blocks for artificial spin lattices, enabling the exploration of flagship one-dimensional quantum-magnetism models with unprecedented control. The spin-1 Heisenberg model, including bilinear and biquadratic exchange, was first realized using [3]-triangulenes, revealing the Haldane phase. More recently, Clar's goblets enabled the spin-1/2 Heisenberg model with exchange alternation, uncovering additional topological phases. Here we show that spin-1 nanographenes can be used to explore bond-alternating chains both in the Haldane phase and beyond it, in a dimerized phase with emergent edge spin-1. We use density matrix renormalization group (DMRG) to analyze how biquadratic exchange, which is known to be large in spin-1 nanographenes, determines the phase transition boundary. Combining multiconfigurational and first-principles calculations, we identify two realistic candidates to realize these two different phases: the recently synthesized extended Clar's goblet and a passivated [4]-triangulene. We demonstrate how to distinguish these phases experimentally using inelastic electron tunneling spectroscopy, paving the way for their observation. △ Less

Submitted 4 May, 2026; v1 submitted 27 October, 2025; originally announced October 2025.

Comments: 21 pages, 14 figures, 3 tables

Journal ref: SciPost Phys. 21, 009 (2026)

CRAM: Large-scale Video Continual Learning with Bootstrapped Compression

Authors: Shivani Mall, Joao F. Henriques

Abstract: Continual learning (CL) promises to allow neural networks to learn from continuous streams of inputs, instead of IID (independent and identically distributed) sampling, which requires random access to a full dataset. This would allow for much smaller storage requirements and self-sufficiency of deployed systems that cope with natural distribution shifts, similarly to biological learning. We focus… ▽ More Continual learning (CL) promises to allow neural networks to learn from continuous streams of inputs, instead of IID (independent and identically distributed) sampling, which requires random access to a full dataset. This would allow for much smaller storage requirements and self-sufficiency of deployed systems that cope with natural distribution shifts, similarly to biological learning. We focus on video CL employing a rehearsal-based approach, which reinforces past samples from a memory buffer. We posit that part of the reason why practical video CL is challenging is the high memory requirements of video, further exacerbated by long-videos and continual streams, which are at odds with the common rehearsal-buffer size constraints. To address this, we propose to use compressed vision, i.e. store video codes (embeddings) instead of raw inputs, and train a video classifier by IID sampling from this rolling buffer. Training a video compressor online (so not depending on any pre-trained networks) means that it is also subject to catastrophic forgetting. We propose a scheme to deal with this forgetting by refreshing video codes, which requires careful decompression with a previous version of the network and recompression with a new one. We name our method Continually Refreshed Amodal Memory (CRAM). We expand current video CL benchmarks to large-scale settings, namely EpicKitchens-100 and Kinetics-700, storing thousands of relatively long videos in under 2 GB, and demonstrate empirically that our video CL method outperforms prior art with a significantly reduced memory footprint. △ Less

Submitted 6 August, 2025; originally announced August 2025.

Journal ref: International Conference on Computer Vision, ICCV 2025

TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs

Authors: Felipe Nuti, Tim Franzmeyer, João Henriques

Abstract: Past work has studied the effects of fine-tuning on large language models' (LLMs) overall performance on certain tasks. However, a quantitative and systematic method for analyzing its effect on individual outputs is still lacking. Here, we propose a new method for measuring the contribution that fine-tuning makes to individual LLM responses, assuming access to the original pre-trained model. Our m… ▽ More Past work has studied the effects of fine-tuning on large language models' (LLMs) overall performance on certain tasks. However, a quantitative and systematic method for analyzing its effect on individual outputs is still lacking. Here, we propose a new method for measuring the contribution that fine-tuning makes to individual LLM responses, assuming access to the original pre-trained model. Our method tracks the model's intermediate hidden states, providing a more fine-grained insight into the effects of fine-tuning than a simple comparison of final outputs from pre-trained and fine-tuned models. We introduce and theoretically analyze an exact decomposition of any fine-tuned LLM into a pre-training component and a fine-tuning component. Empirically, we find that model behavior and performance can be steered by up- or down-scaling the fine-tuning component during the forward pass. Motivated by this finding and our theoretical analysis, we define the Tuning Contribution (TuCo) as the ratio of the magnitudes of the fine-tuning component to the pre-training component. We observe that three prominent adversarial attacks on LLMs circumvent safety measures in a way that reduces TuCo, and that TuCo is consistently lower on prompts where these attacks succeed compared to those where they do not. This suggests that attenuating the effect of fine-tuning on model outputs plays a role in the success of such attacks. In summary, TuCo enables the quantitative study of how fine-tuning influences model behavior and safety, and vice versa. △ Less

Submitted 29 June, 2025; originally announced June 2025.

Comments: ICML 2025

UltraGauss: Ultrafast Gaussian Reconstruction of 3D Ultrasound Volumes

Authors: Mark C. Eid, Ana I. L. Namburete, João F. Henriques

Abstract: Ultrasound imaging is widely used due to its safety, affordability, and real-time capabilities, but its 2D interpretation is highly operator-dependent, leading to variability and increased cognitive demand. 2D-to-3D reconstruction mitigates these challenges by providing standardized volumetric views, yet existing methods are often computationally expensive, memory-intensive, or incompatible with u… ▽ More Ultrasound imaging is widely used due to its safety, affordability, and real-time capabilities, but its 2D interpretation is highly operator-dependent, leading to variability and increased cognitive demand. 2D-to-3D reconstruction mitigates these challenges by providing standardized volumetric views, yet existing methods are often computationally expensive, memory-intensive, or incompatible with ultrasound physics. We introduce UltraGauss: the first ultrasound-specific Gaussian Splatting framework, extending view synthesis techniques to ultrasound wave propagation. Unlike conventional perspective-based splatting, UltraGauss models probe-plane intersections in 3D, aligning with acoustic image formation. We derive an efficient rasterization boundary formulation for GPU parallelization and introduce a numerically stable covariance parametrization, improving computational efficiency and reconstruction accuracy. On real clinical ultrasound data, UltraGauss achieves state-of-the-art reconstructions in 5 minutes, and reaching 0.99 SSIM within 20 minutes on a single GPU. A survey of expert clinicians confirms UltraGauss' reconstructions are the most realistic among competing methods. Our CUDA implementation will be released upon publication. △ Less

Submitted 8 May, 2025; originally announced May 2025.

Characterizing $S=3/2$ AKLT Hamiltonian with Scanning Tunneling Spectroscopy

Authors: M. Ferri-Cortés, J. C. G. Henriques, J. Fernández-Rossier

Abstract: The AKLT Hamiltonian is a particular instance of a general class of model Hamiltonians defined in lattices with coordination $z$ where each site hosts a spins $S=z/2$, interacting both with linear and non-linear exchange couplings. In two dimensions, the AKLT model features a gap in the spectrum, and its ground state is a valence bond solid state; that is an universal resource for measurement base… ▽ More The AKLT Hamiltonian is a particular instance of a general class of model Hamiltonians defined in lattices with coordination $z$ where each site hosts a spins $S=z/2$, interacting both with linear and non-linear exchange couplings. In two dimensions, the AKLT model features a gap in the spectrum, and its ground state is a valence bond solid state; that is an universal resource for measurement based quantum computing, motivating the quest of physical systems that realize this Hamiltonian. Given a finite-size system described with a specific instance of this general class of models, we address the question of how to asses if such system is a realization of the AKLT model using inelastic tunnel spectroscopy implemented with scanning tunnel microscopy (IETS-STM). We propose two approaches. First, in the case of a dimer, we show how to leverage non-equilibrium IETS-STM to obtain the energies of all excited states, and determine thereby the magnitude of both linear and non-linear exchange interactions. Second, we explore how IETS can probe the in-gap excitations associated to edge spins. In the AKLT limit, spins $S=3/2$ at the edge of the lattice have coordination 2, giving rise to $S=1/2$ dangling spins that can be probed with IETS. We propose a $S=1/2$ effective Hamiltonian to describe the interactions between these dangling spins in the neighborhood of the AKLT point, where their degeneracy lifted. △ Less

Submitted 6 March, 2025; originally announced March 2025.

On determining the energy dispersion of spin excitations with scanning tunneling spectroscopy

Authors: J. C. G. Henriques, Chenxiao Zhao, G. Catarina, Pascal Ruffieux, Roman Fasel, J. Fernández-Rossier

Abstract: Conventional methods to measure the dispersion relations of collective spin excitations involve probing bulk samples with particles such as neutrons, photons or electrons, which carry a well-defined momentum. Open-ended finite-size spin chains, on the contrary, do not have a well-defined momentum due to the lack of translation symmetry, and their spin excitations are measured with an eminently loc… ▽ More Conventional methods to measure the dispersion relations of collective spin excitations involve probing bulk samples with particles such as neutrons, photons or electrons, which carry a well-defined momentum. Open-ended finite-size spin chains, on the contrary, do not have a well-defined momentum due to the lack of translation symmetry, and their spin excitations are measured with an eminently local probe, using inelastic electron tunneling spectroscopy (IETS) with a scanning tunneling microscope (STM). Here we discuss under what conditions STM-IETS spectra can be Fourier-transformed to yield dispersion relations in these systems. We relate the success of this approach to the degree to which spin excitations form standing waves. We show that STM-IETS can reveal the energy dispersion of magnons in ferromagnets and triplons in valence bond crystals, but not that of spinons, the spin excitations in Heisenberg spin-1/2 chains. We compare our theoretical predictions with state-of-the-art measurements on nanographene chains that realize the relevant spin Hamiltonians. △ Less

Submitted 19 February, 2025; originally announced February 2025.

Comments: 6 pages, 4 figures

TG-OT: Topology-guided CCTA-IVUS registration via optimal transport matching

Authors: R. L. M. van Herten, José P. Henriques, R. Nils Planken, Joost Daemen, Eline M. J. Hartman, Jolanda J. Wentzel, Johannes C. Paetzold, Ivana Išgum

Abstract: Registering coronary CT angiography (CCTA) and intravascular ultrasound (IVUS) enables comprehensive coronary analysis that neither modality can provide alone, yet their fusion remains challenging due to differences in imaging geometry, resolution, and artifact profiles. Existing methods depend on pre-computed lumen or vessel wall segmentations that are unreliable under IVUS acoustic shadowing fro… ▽ More Registering coronary CT angiography (CCTA) and intravascular ultrasound (IVUS) enables comprehensive coronary analysis that neither modality can provide alone, yet their fusion remains challenging due to differences in imaging geometry, resolution, and artifact profiles. Existing methods depend on pre-computed lumen or vessel wall segmentations that are unreliable under IVUS acoustic shadowing from calcifications, limiting their clinical applicability. We propose TG-OT, a fully automatic CCTA-IVUS registration framework that eliminates this dependency by integrating trained feature detectors directly into the registration pipeline. Lightweight CNNs are trained to predict calcifications, bifurcations, and lumen radii on the topological $(θ, z)$ cylinder, encouraging topologically coherent detections without requiring explicit segmentation. Registration is formulated as an optimization over centerline warping parameters, driven by an unbalanced Sinkhorn optimal transport loss on the cylindrical geometry that provides spatially informative gradients even for spatially disjoint predictions, complemented by a lumen matching term. Evaluated on $N{=}47$ paired CCTA-IVUS cases in a 5-fold cross-validation setup, TG-OT achieves strong longitudinal ($\overline{\text{Dice}}_\text{ctl}{=}0.99$), rotational ($\overline{S}_c{=}0.96$), and lumen alignment ($\overline{\text{Dice}}_\text{L}{=}0.69$) without manual interaction or prior segmentation, marking a meaningful step toward clinical integration of automatic CCTA-IVUS fusion. △ Less

Submitted 25 June, 2026; v1 submitted 22 December, 2024; originally announced December 2024.

Comments: Submitted to MICCAI 2026

UniLoc: Towards Universal Place Recognition Using Any Single Modality

Authors: Yan Xia, Zhendong Li, Yun-Jin Li, Letian Shi, Hu Cao, João F. Henriques, Daniel Cremers

Abstract: To date, most place recognition methods focus on single-modality retrieval. While they perform well in specific environments, cross-modal methods offer greater flexibility by allowing seamless switching between map and query sources. It also promises to reduce computation requirements by having a unified model, and achieving greater sample efficiency by sharing parameters. In this work, we develop… ▽ More To date, most place recognition methods focus on single-modality retrieval. While they perform well in specific environments, cross-modal methods offer greater flexibility by allowing seamless switching between map and query sources. It also promises to reduce computation requirements by having a unified model, and achieving greater sample efficiency by sharing parameters. In this work, we develop a universal solution to place recognition, UniLoc, that works with any single query modality (natural language, image, or point cloud). UniLoc leverages recent advances in large-scale contrastive learning, and learns by matching hierarchically at two levels: instance-level matching and scene-level matching. Specifically, we propose a novel Self-Attention based Pooling (SAP) module to evaluate the importance of instance descriptors when aggregated into a place-level descriptor. Experiments on the KITTI-360 dataset demonstrate the benefits of cross-modality for place recognition, achieving superior performance in cross-modal settings and competitive results also for uni-modal scenarios. Our project page is publicly available at https://yan-xia.github.io/projects/UniLoc/. △ Less

Submitted 16 December, 2024; originally announced December 2024.

Comments: 14 pages, 10 figures

TrafficLoc: Localizing Traffic Surveillance Cameras in 3D Scenes

Authors: Yan Xia, Yunxiang Lu, Rui Song, Oussema Dhaouadi, João F. Henriques, Daniel Cremers

Abstract: We tackle the problem of localizing traffic cameras within a 3D reference map and propose a novel image-to-point cloud registration (I2P) method, TrafficLoc, in a coarse-tofine matching fashion. To overcome the lack of large-scale real-world intersection datasets, we first introduce Carla Intersection, a new simulated dataset with 75 urban and rural intersections in Carla. We find that current I2P… ▽ More We tackle the problem of localizing traffic cameras within a 3D reference map and propose a novel image-to-point cloud registration (I2P) method, TrafficLoc, in a coarse-tofine matching fashion. To overcome the lack of large-scale real-world intersection datasets, we first introduce Carla Intersection, a new simulated dataset with 75 urban and rural intersections in Carla. We find that current I2P methods struggle with cross-modal matching under large viewpoint differences, especially at traffic intersections. TrafficLoc thus employs a novel Geometry-guided Attention Loss (GAL) to focus only on the corresponding geometric regions under different viewpoints during 2D-3D feature fusion. To address feature inconsistency in paired image patch-point groups, we further propose Inter-intra Contrastive Learning (ICL) to enhance separating 2D patch/3D group features within each intra-modality and introduce Dense Training Alignment (DTA) with soft-argmax for improving position regression. Extensive experiments show our TrafficLoc greatly improves the performance over the SOTA I2P methods (up to 86%) on Carla Intersection and generalizes well to real-world data. TrafficLoc also achieves new SOTA performance on KITTI and NuScenes datasets, demonstrating the superiority across both in-vehicle and traffic cameras. Our project page is publicly available at https://tum-luk.github.io/projects/trafficloc/. △ Less

Submitted 25 March, 2025; v1 submitted 13 December, 2024; originally announced December 2024.

Electrically Tunable Interband Collective Excitations in Biased Bilayer and Trilayer Graphene

Authors: Tomer Eini, M. F. C. Martins Quintela, J. C. G. Henriques, R. M. Ribeiro, Yarden Mazor, N. M. R. Peres, Itai Epstein

Abstract: Collective excitations of charged particles under the influence of an electromagnetic field give rise to a rich variety of hybrid light-matter quasiparticles with unique properties. In metals, intraband collective response manifested by negative permittivity leads to plasmon-polaritons with extreme field confinement, wavelength squeezing, and potentially low propagation losses. In contrast, photon… ▽ More Collective excitations of charged particles under the influence of an electromagnetic field give rise to a rich variety of hybrid light-matter quasiparticles with unique properties. In metals, intraband collective response manifested by negative permittivity leads to plasmon-polaritons with extreme field confinement, wavelength squeezing, and potentially low propagation losses. In contrast, photons in semiconductors commonly couple to interband collective response in the form of exciton polaritons, which give rise to completely different polaritonic properties, described by a superposition of the photon and exciton and an anti-crossing of the eigenstates. In this work, we identify the existence of plasmon-like collective excitations originating from the interband excitonic response of biased bilayer and trilayer graphene, in the form of graphene-exciton-polaritons (GEPs). We find that GEPs possess electrically tunable polaritonic properties and discover that such excitations follow a universal dispersion law for all surface polaritons in 2D excitonic systems. Accounting for nonlocal corrections to the excitonic response, we find that the GEPs exhibit confinement factors that can exceed those of graphene plasmons, and with moderate losses. These predictions of plasmon-like interband collective excitations in biased graphene systems open up new research avenues for tunable polaritonic phenomena based on excitonic systems, and the ability to control and manipulate such phenomena at the atomic scale. △ Less

Submitted 27 February, 2025; v1 submitted 4 December, 2024; originally announced December 2024.

VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning

Authors: Yichao Liang, Nishanth Kumar, Hao Tang, Adrian Weller, Joshua B. Tenenbaum, Tom Silver, João F. Henriques, Kevin Ellis

Abstract: Broadly intelligent agents should form task-specific abstractions that selectively expose the essential elements of a task, while abstracting away the complexity of the raw sensorimotor space. In this work, we present Neuro-Symbolic Predicates, a first-order abstraction language that combines the strengths of symbolic and neural knowledge representations. We outline an online algorithm for inventi… ▽ More Broadly intelligent agents should form task-specific abstractions that selectively expose the essential elements of a task, while abstracting away the complexity of the raw sensorimotor space. In this work, we present Neuro-Symbolic Predicates, a first-order abstraction language that combines the strengths of symbolic and neural knowledge representations. We outline an online algorithm for inventing such predicates and learning abstract world models. We compare our approach to hierarchical reinforcement learning, vision-language model planning, and symbolic predicate invention approaches, on both in- and out-of-distribution tasks across five simulated robotic domains. Results show that our approach offers better sample complexity, stronger out-of-distribution generalization, and improved interpretability. △ Less

Submitted 28 February, 2025; v1 submitted 30 October, 2024; originally announced October 2024.

Comments: ICLR 2025 (Spotlight)

Interpretable Representation Learning from Videos using Nonlinear Priors

Authors: Marian Longa, João F. Henriques

Abstract: Learning interpretable representations of visual data is an important challenge, to make machines' decisions understandable to humans and to improve generalisation outside of the training distribution. To this end, we propose a deep learning framework where one can specify nonlinear priors for videos (e.g. of Newtonian physics) that allow the model to learn interpretable latent variables and use t… ▽ More Learning interpretable representations of visual data is an important challenge, to make machines' decisions understandable to humans and to improve generalisation outside of the training distribution. To this end, we propose a deep learning framework where one can specify nonlinear priors for videos (e.g. of Newtonian physics) that allow the model to learn interpretable latent variables and use these to generate videos of hypothetical scenarios not observed at training time. We do this by extending the Variational Auto-Encoder (VAE) prior from a simple isotropic Gaussian to an arbitrary nonlinear temporal Additive Noise Model (ANM), which can describe a large number of processes (e.g. Newtonian physics). We propose a novel linearization method that constructs a Gaussian Mixture Model (GMM) approximating the prior, and derive a numerically stable Monte Carlo estimate of the KL divergence between the posterior and prior GMMs. We validate the method on different real-world physics videos including a pendulum, a mass on a spring, a falling object and a pulsar (rotating neutron star). We specify a physical prior for each experiment and show that the correct variables are learned. Once a model is trained, we intervene on it to change different physical variables (such as oscillation amplitude or adding air drag) to generate physically correct videos of hypothetical scenarios that were not observed previously. △ Less

Submitted 24 October, 2024; originally announced October 2024.

Comments: Accepted to BMVC 2024 (Oral)

World of Forms: Deformable Geometric Templates for One-Shot Surface Meshing in Coronary CT Angiography

Authors: Rudolf L. M. van Herten, Ioannis Lagogiannis, Jelmer M. Wolterink, Steffen Bruns, Eva R. Meulendijks, Damini Dey, Joris R. de Groot, José P. Henriques, R. Nils Planken, Simone Saitta, Ivana Išgum

Abstract: Deep learning-based medical image segmentation and surface mesh generation typically involve a sequential pipeline from image to segmentation to meshes, often requiring large training datasets while making limited use of prior geometric knowledge. This may lead to topological inconsistencies and suboptimal performance in low-data regimes. To address these challenges, we propose a data-efficient de… ▽ More Deep learning-based medical image segmentation and surface mesh generation typically involve a sequential pipeline from image to segmentation to meshes, often requiring large training datasets while making limited use of prior geometric knowledge. This may lead to topological inconsistencies and suboptimal performance in low-data regimes. To address these challenges, we propose a data-efficient deep learning method for direct 3D anatomical object surface meshing using geometric priors. Our approach employs a multi-resolution graph neural network that operates on a prior geometric template which is deformed to fit object boundaries of interest. We show how different templates may be used for the different surface meshing targets, and introduce a novel masked autoencoder pretraining strategy for 3D spherical data. The proposed method outperforms nnUNet in a one-shot setting for segmentation of the pericardium, left ventricle (LV) cavity and the LV myocardium. Similarly, the method outperforms other lumen segmentation operating on multi-planar reformatted images. Results further indicate that mesh quality is on par with or improves upon marching cubes post-processing of voxel mask predictions, while remaining flexible in the choice of mesh triangulation prior, thus paving the way for more accurate and topologically consistent 3D medical object surface meshing. △ Less

Submitted 21 February, 2025; v1 submitted 18 September, 2024; originally announced September 2024.

Comments: Submitted to Medical Image Analysis

GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers

Authors: Lorenza Prospero, Abdullah Hamdi, Joao F. Henriques, Christian Rupprecht

Abstract: Reconstructing posed 3D human models from monocular images has important applications in the sports industry, including performance tracking, injury prevention and virtual training. In this work, we combine 3D human pose and shape estimation with 3D Gaussian Splatting (3DGS), a representation of the scene composed of a mixture of Gaussians. This allows training or fine-tuning a human model predict… ▽ More Reconstructing posed 3D human models from monocular images has important applications in the sports industry, including performance tracking, injury prevention and virtual training. In this work, we combine 3D human pose and shape estimation with 3D Gaussian Splatting (3DGS), a representation of the scene composed of a mixture of Gaussians. This allows training or fine-tuning a human model predictor on multi-view images alone, without 3D ground truth. Predicting such mixtures for a human from a single input image is challenging due to self-occlusions and dependence on articulations, while also needing to retain enough flexibility to accommodate a variety of clothes and poses. Our key observation is that the vertices of standardized human meshes (such as SMPL) can provide an adequate spatial density and approximate initial position for the Gaussians. We can then train a transformer model to jointly predict comparatively small adjustments to these positions, as well as the other 3DGS attributes and the SMPL parameters. We show empirically that this combination (using only multi-view supervision) can achieve near real-time inference of 3D human models from a single image without expensive diffusion models or 3D points supervision, thus making it ideal for the sport industry at any level. More importantly, rendering is an effective auxiliary objective to refine 3D pose estimation by accounting for clothes and other geometric variations. The code is available at https://github.com/prosperolo/GST. △ Less

Submitted 16 April, 2025; v1 submitted 6 September, 2024; originally announced September 2024.

Comments: Camera ready for CVSports workshop at CVPR 2025

Dissecting Temporal Understanding in Text-to-Audio Retrieval

Authors: Andreea-Maria Oncescu, João F. Henriques, A. Sophia Koepke

Abstract: Recent advancements in machine learning have fueled research on multimodal tasks, such as for instance text-to-video and text-to-audio retrieval. These tasks require models to understand the semantic content of video and audio data, including objects, and characters. The models also need to learn spatial arrangements and temporal relationships. In this work, we analyse the temporal ordering of sou… ▽ More Recent advancements in machine learning have fueled research on multimodal tasks, such as for instance text-to-video and text-to-audio retrieval. These tasks require models to understand the semantic content of video and audio data, including objects, and characters. The models also need to learn spatial arrangements and temporal relationships. In this work, we analyse the temporal ordering of sounds, which is an understudied problem in the context of text-to-audio retrieval. In particular, we dissect the temporal understanding capabilities of a state-of-the-art model for text-to-audio retrieval on the AudioCaps and Clotho datasets. Additionally, we introduce a synthetic text-audio dataset that provides a controlled setting for evaluating temporal capabilities of recent models. Lastly, we present a loss function that encourages text-audio models to focus on the temporal ordering of events. Code and data are available at https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/dtu/. △ Less

Submitted 1 September, 2024; originally announced September 2024.

Comments: 9 pages, 5 figures, ACM Multimedia 2024, https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/dtu/

ALMA Memo 628 -- High-cadence observations of the Sun

Authors: Sven Wedemeyer, Mikolaj Szydlarski, M. Carmen Toribio, Tobia Carozzi, Daniel Jakobsson, Juan Camilo Guevara Gomez, Henrik Eklund, Vasco M. J. Henriques, Shahin Jafarzadeh, Jaime de la Cruz Rodriguez

Abstract: The Atacama Large Millimeter/submillimeter Array (ALMA) offers new diagnostic capabilities for studying the Sun, providing complementary insights through high spatial and temporal resolution at millimeter wavelengths. ALMA acts as a linear thermometer for atmospheric gas, aiding in understanding the solar atmosphere's structure, dynamics, and energy balance. Given the Sun's complex emission patter… ▽ More The Atacama Large Millimeter/submillimeter Array (ALMA) offers new diagnostic capabilities for studying the Sun, providing complementary insights through high spatial and temporal resolution at millimeter wavelengths. ALMA acts as a linear thermometer for atmospheric gas, aiding in understanding the solar atmosphere's structure, dynamics, and energy balance. Given the Sun's complex emission patterns and rapid evolution, high-cadence imaging is essential for solar observations. Snapshot imaging is required, though it limits available visibility data, making full exploitation of ALMA's capabilities non-trivial. Challenges in processing solar ALMA data highlight the need for revising and enhancing the solar observing mode. The ALMA development study High-Cadence Imaging of the Sun demonstrated the potential benefits of high cadence observations through a forward modelling approach. The resulting report provides initial recommendations for improved post-processing solar ALMA data and explores increasing the observing cadence to sub-second intervals to improve image reliability. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Comments: ALMA Memo 628 summarising the results of the ESO-funded ALMA development study "High-cadence Imaging of the Sun", concluded in 2023 (74 pages, 53 figures)

Report number: ALMA Memo 628

Gapless spin excitations in nanographene-based antiferromagnetic spin-1/2 Heisenberg chains

Authors: Chenxiao Zhao, Lin Yang, João C. G. Henriques, Mar Ferri-Cortés, Gonçalo Catarina, Carlo A. Pignedoli, Ji Ma, Xinliang Feng, Pascal Ruffieux, Joaquín Fernández-Rossier, Roman Fasel

Abstract: Haldane's seminal work established two fundamentally different types of excitation spectra for antiferromagnetic Heisenberg quantum spin chains: gapped excitations in integer-spin chains and gapless excitations in half-integer-spin chains. In finite-length half-integer spin chains, quantization, however, induces a gap in the excitation spectrum, with the upper bound given by the Lieb-Schulz-Mattis… ▽ More Haldane's seminal work established two fundamentally different types of excitation spectra for antiferromagnetic Heisenberg quantum spin chains: gapped excitations in integer-spin chains and gapless excitations in half-integer-spin chains. In finite-length half-integer spin chains, quantization, however, induces a gap in the excitation spectrum, with the upper bound given by the Lieb-Schulz-Mattis (LSM) theorem. Here, we investigate the length-dependent excitations in spin-1/2 Heisenberg chains obtained by covalently linking olympicenes--Olympic rings shaped nanographenes carrying spin-1/2--into one-dimensional chains. The large exchange interaction (J~38 mV) between olympicenes and the negligible magnetic anisotropy in these nanographenes make them an ideal platform for studying quantum spin excitations, which we directly measure using inelastic electron tunneling spectroscopy. We observe a power-law decay of the lowest excitation energy with increasing chain length L, remaining below the LSM boundary. In a long chain with L = 50, a nearly V-shaped excitation continuum is observed, reinforcing the system's gapless nature in the thermodynamic limit. Finally, we visualize the standing wave of a single spinon confined in odd-numbered chains using low-bias current maps. Our results provide compelling evidence for the realization of a one-dimensional analog of a gapless spin liquid. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Comments: 5 figures. arXiv admin note: text overlap with arXiv:2402.13590

3D-Aware Instance Segmentation and Tracking in Egocentric Videos

Authors: Yash Bhalgat, Vadim Tschernezki, Iro Laina, João F. Henriques, Andrea Vedaldi, Andrew Zisserman

Abstract: Egocentric videos present unique challenges for 3D scene understanding due to rapid camera motion, frequent object occlusions, and limited object visibility. This paper introduces a novel approach to instance segmentation and tracking in first-person video that leverages 3D awareness to overcome these obstacles. Our method integrates scene geometry, 3D object centroid tracking, and instance segmen… ▽ More Egocentric videos present unique challenges for 3D scene understanding due to rapid camera motion, frequent object occlusions, and limited object visibility. This paper introduces a novel approach to instance segmentation and tracking in first-person video that leverages 3D awareness to overcome these obstacles. Our method integrates scene geometry, 3D object centroid tracking, and instance segmentation to create a robust framework for analyzing dynamic egocentric scenes. By incorporating spatial and temporal cues, we achieve superior performance compared to state-of-the-art 2D approaches. Extensive evaluations on the challenging EPIC Fields dataset demonstrate significant improvements across a range of tracking and segmentation consistency metrics. Specifically, our method outperforms the next best performing approach by $7$ points in Association Accuracy (AssA) and $4.5$ points in IDF1 score, while reducing the number of ID switches by $73\%$ to $80\%$ across various object categories. Leveraging our tracked instance segmentations, we showcase downstream applications in 3D object reconstruction and amodal video object segmentation in these egocentric settings. △ Less

Submitted 20 November, 2024; v1 submitted 19 August, 2024; originally announced August 2024.

Comments: Camera-ready for ACCV 2024. More experiments added

Building spin-1/2 antiferromagnetic Heisenberg chains with diaza-nanographenes

Authors: Xiaoshuai Fu, Li Huang, Kun Liu, João C. G. Henriques, Yixuan Gao, Xianghe Han, Hui Chen, Yan Wang, Carlos-Andres Palma, Zhihai Cheng, Xiao Lin, Shixuan Du, Ji Ma, Joaquín Fernández-Rossier, Xinliang Feng, Hong-Jun Gao

Abstract: Understanding and engineering the coupling of spins in nanomaterials is of central importance for designing novel devices. Graphene nanostructures with π-magnetism offer a chemically tunable platform to explore quantum magnetic interactions. However, realizing spin chains bearing controlled odd-even effects with suitable nanographene systems is challenging. Here, we demonstrate the successful on-s… ▽ More Understanding and engineering the coupling of spins in nanomaterials is of central importance for designing novel devices. Graphene nanostructures with π-magnetism offer a chemically tunable platform to explore quantum magnetic interactions. However, realizing spin chains bearing controlled odd-even effects with suitable nanographene systems is challenging. Here, we demonstrate the successful on-surface synthesis of spin-1/2 antiferromagnetic Heisenberg chains with parity-dependent magnetization based on antiaromatic diaza-hexa-peri-hexabenzocoronene (diaza-HBC) units. Using distinct synthetic strategies, two types of spin chains with different terminals were synthesized, both exhibiting a robust odd-even effect on the spin coupling along the chain. Combined investigations using scanning tunneling microscopy, non-contact atomic force microscopy, density functional theory calculations, and quantum spin models confirmed the structures of the diaza-HBC chains and revealed their magnetic properties, which has an S = 1/2 spin per unit through electron donation from the diaza-HBC core to the Au(111) substrate. Gapped excitations were observed in even-numbered chains, while enhanced Kondo resonance emerged in odd-numbered units of odd-numbered chains due to the redistribution of the unpaired spin along the chain. Our findings provide an effective strategy to construct nanographene spin chains and unveil the odd-even effect in their magnetic properties, offering potential applications in nanoscale spintronics. △ Less

Submitted 29 July, 2024; originally announced July 2024.

Journal ref: Nature Synthesis (2025)

SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments

Authors: Shu Ishida, João F. Henriques

Abstract: This work compares ways of extending Reinforcement Learning algorithms to Partially Observed Markov Decision Processes (POMDPs) with options. One view of options is as temporally extended action, which can be realized as a memory that allows the agent to retain historical information beyond the policy's context window. While option assignment could be handled using heuristics and hand-crafted obje… ▽ More This work compares ways of extending Reinforcement Learning algorithms to Partially Observed Markov Decision Processes (POMDPs) with options. One view of options is as temporally extended action, which can be realized as a memory that allows the agent to retain historical information beyond the policy's context window. While option assignment could be handled using heuristics and hand-crafted objectives, learning temporally consistent options and associated sub-policies without explicit supervision is a challenge. Two algorithms, PPOEM and SOAP, are proposed and studied in depth to address this problem. PPOEM applies the forward-backward algorithm (for Hidden Markov Models) to optimize the expected returns for an option-augmented policy. However, this learning approach is unstable during on-policy rollouts. It is also unsuited for learning causal policies without the knowledge of future trajectories, since option assignments are optimized for offline sequences where the entire episode is available. As an alternative approach, SOAP evaluates the policy gradient for an optimal option assignment. It extends the concept of the generalized advantage estimation (GAE) to propagate option advantages through time, which is an analytical equivalent to performing temporal back-propagation of option policy gradients. This option policy is only conditional on the history of the agent, not future actions. Evaluated against competing baselines, SOAP exhibited the most robust performance, correctly discovering options for POMDP corridor environments, as well as on standard benchmarks including Atari and MuJoCo, outperforming PPOEM, as well as LSTM and Option-Critic baselines. The open-sourced code is available at https://github.com/shuishida/SoapRL. △ Less

Submitted 11 October, 2024; v1 submitted 26 July, 2024; originally announced July 2024.

Unsupervised Object Detection with Theoretical Guarantees

Authors: Marian Longa, João F. Henriques

Abstract: Unsupervised object detection using deep neural networks is typically a difficult problem with few to no guarantees about the learned representation. In this work we present the first unsupervised object detection method that is theoretically guaranteed to recover the true object positions up to quantifiable small shifts. We develop an unsupervised object detection architecture and prove that the… ▽ More Unsupervised object detection using deep neural networks is typically a difficult problem with few to no guarantees about the learned representation. In this work we present the first unsupervised object detection method that is theoretically guaranteed to recover the true object positions up to quantifiable small shifts. We develop an unsupervised object detection architecture and prove that the learned variables correspond to the true object positions up to small shifts related to the encoder and decoder receptive field sizes, the object sizes, and the widths of the Gaussians used in the rendering process. We perform detailed analysis of how the error depends on each of these variables and perform synthetic experiments validating our theoretical predictions up to a precision of individual pixels. We also perform experiments on CLEVR-based data and show that, unlike current SOTA object detection methods (SAM, CutLER), our method's prediction errors always lie within our theoretical bounds. We hope that this work helps open up an avenue of research into object detection methods with theoretical guarantees. △ Less

Submitted 24 October, 2024; v1 submitted 11 June, 2024; originally announced June 2024.

Comments: Accepted to NeurIPS 2024

Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Image

Authors: Stanislaw Szymanowicz, Eldar Insafutdinov, Chuanxia Zheng, Dylan Campbell, João F. Henriques, Christian Rupprecht, Andrea Vedaldi

Abstract: We propose Flash3D, a method for scene reconstruction and novel view synthesis from a single image which is both very generalisable and efficient. For generalisability, we start from a "foundation" model for monocular depth estimation and extend it to a full 3D shape and appearance reconstructor. For efficiency, we base this extension on feed-forward Gaussian Splatting. Specifically, we predict a… ▽ More We propose Flash3D, a method for scene reconstruction and novel view synthesis from a single image which is both very generalisable and efficient. For generalisability, we start from a "foundation" model for monocular depth estimation and extend it to a full 3D shape and appearance reconstructor. For efficiency, we base this extension on feed-forward Gaussian Splatting. Specifically, we predict a first layer of 3D Gaussians at the predicted depth, and then add additional layers of Gaussians that are offset in space, allowing the model to complete the reconstruction behind occlusions and truncations. Flash3D is very efficient, trainable on a single GPU in a day, and thus accessible to most researchers. It achieves state-of-the-art results when trained and tested on RealEstate10k. When transferred to unseen datasets like NYU it outperforms competitors by a large margin. More impressively, when transferred to KITTI, Flash3D achieves better PSNR than methods trained specifically on that dataset. In some instances, it even outperforms recent methods that use multiple views as input. Code, models, demo, and more results are available at https://www.robots.ox.ac.uk/~vgg/research/flash3d/. △ Less

Submitted 1 June, 2025; v1 submitted 6 June, 2024; originally announced June 2024.

Comments: Project page: https://www.robots.ox.ac.uk/~vgg/research/flash3d/

HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits

Authors: Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, João F. Henriques, Jakob N. Foerster

Abstract: Benchmarks have been essential for driving progress in machine learning. A better understanding of LLM capabilities on real world tasks is vital for safe development. Designing adequate LLM benchmarks is challenging: Data from real-world tasks is hard to collect, public availability of static evaluation data results in test data contamination and benchmark overfitting, and periodically generating… ▽ More Benchmarks have been essential for driving progress in machine learning. A better understanding of LLM capabilities on real world tasks is vital for safe development. Designing adequate LLM benchmarks is challenging: Data from real-world tasks is hard to collect, public availability of static evaluation data results in test data contamination and benchmark overfitting, and periodically generating new evaluation data is tedious and may result in temporally inconsistent results. We introduce HelloFresh, based on continuous streams of real-world data generated by intrinsically motivated human labelers. It covers recent events from X (formerly Twitter) community notes and edits of Wikipedia pages, mitigating the risk of test data contamination and benchmark overfitting. Any X user can propose an X note to add additional context to a misleading post (formerly tweet); if the community classifies it as helpful, it is shown with the post. Similarly, Wikipedia relies on community-based consensus, allowing users to edit articles or revert edits made by other users. Verifying whether an X note is helpful or whether a Wikipedia edit should be accepted are hard tasks that require grounding by querying the web. We backtest state-of-the-art LLMs supplemented with simple web search access and find that HelloFresh yields a temporally consistent ranking. To enable continuous evaluation on HelloFresh, we host a public leaderboard and periodically updated evaluation data at https://tinyurl.com/hello-fresh-LLM. △ Less

Submitted 5 June, 2024; originally announced June 2024.

Comments: ACL 2024 Findings

Giant spatial anisotropy of magnon lifetime in altermagnets

Authors: A. T. Costa, J. C. G. Henriques, J. Fernández-Rossier

Abstract: Altermagnets are a new class of magnetic materials with zero net magnetization (like antiferromagnets) but spin-split electronic bands (like ferromagnets) over a fraction of reciprocal space. As in antiferromagnets, magnons in altermagnets come in two flavours, that either add one or remove one unit of spin to the $S=0$ ground state. However, in altermagnets these two magnon modes are non-degenera… ▽ More Altermagnets are a new class of magnetic materials with zero net magnetization (like antiferromagnets) but spin-split electronic bands (like ferromagnets) over a fraction of reciprocal space. As in antiferromagnets, magnons in altermagnets come in two flavours, that either add one or remove one unit of spin to the $S=0$ ground state. However, in altermagnets these two magnon modes are non-degenerate along some directions in reciprocal space. Here we show that the lifetime of altermagnetic magnons has a very strong dependence on both flavour and direction. Strikingly, coupling to Stoner modes leads to a complete suppression of magnon propagation along selected spatial directions. This giant anisotropy will impact electronic, spin, and energy transport properties and may be exploited in spintronic applications. △ Less

Submitted 21 May, 2024; originally announced May 2024.

Comments: 9 pages, 11 figures (including the appendices)

Journal ref: SciPost Phys. 18, 125 (2025), published 11 April 2025

Select to Perfect: Imitating desired behavior from large multi-agent data

Authors: Tim Franzmeyer, Edith Elkind, Philip Torr, Jakob Foerster, Joao Henriques

Abstract: AI agents are commonly trained with large datasets of demonstrations of human behavior. However, not all behaviors are equally safe or desirable. Desired characteristics for an AI agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories. For example, in a dataset of vehicle interactions, these scores might rela… ▽ More AI agents are commonly trained with large datasets of demonstrations of human behavior. However, not all behaviors are equally safe or desirable. Desired characteristics for an AI agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories. For example, in a dataset of vehicle interactions, these scores might relate to the number of incidents that occurred. We first assess the effect of each individual agent's behavior on the collective desirability score, e.g., assessing how likely an agent is to cause incidents. This allows us to selectively imitate agents with a positive effect, e.g., only imitating agents that are unlikely to cause incidents. To enable this, we propose the concept of an agent's Exchange Value, which quantifies an individual agent's contribution to the collective desirability score. The Exchange Value is the expected change in desirability score when substituting the agent for a randomly selected agent. We propose additional methods for estimating Exchange Values from real-world datasets, enabling us to learn desired imitation policies that outperform relevant baselines. The project website can be found at https://tinyurl.com/select-to-perfect. △ Less

Submitted 6 May, 2024; originally announced May 2024.

Comments: ICLR 2024

RapidVol: Rapid Reconstruction of 3D Ultrasound Volumes from Sensorless 2D Scans

Authors: Mark C. Eid, Pak-Hei Yeung, Madeleine K. Wyburd, João F. Henriques, Ana I. L. Namburete

Abstract: Two-dimensional (2D) freehand ultrasonography is one of the most commonly used medical imaging modalities, particularly in obstetrics and gynaecology. However, it only captures 2D cross-sectional views of inherently 3D anatomies, losing valuable contextual information. As an alternative to requiring costly and complex 3D ultrasound scanners, 3D volumes can be constructed from 2D scans using machin… ▽ More Two-dimensional (2D) freehand ultrasonography is one of the most commonly used medical imaging modalities, particularly in obstetrics and gynaecology. However, it only captures 2D cross-sectional views of inherently 3D anatomies, losing valuable contextual information. As an alternative to requiring costly and complex 3D ultrasound scanners, 3D volumes can be constructed from 2D scans using machine learning. However this usually requires long computational time. Here, we propose RapidVol: a neural representation framework to speed up slice-to-volume ultrasound reconstruction. We use tensor-rank decomposition, to decompose the typical 3D volume into sets of tri-planes, and store those instead, as well as a small neural network. A set of 2D ultrasound scans, with their ground truth (or estimated) 3D position and orientation (pose) is all that is required to form a complete 3D reconstruction. Reconstructions are formed from real fetal brain scans, and then evaluated by requesting novel cross-sectional views. When compared to prior approaches based on fully implicit representation (e.g. neural radiance fields), our method is over 3x quicker, 46% more accurate, and if given inaccurate poses is more robust. Further speed-up is also possible by reconstructing from a structural prior rather than from scratch. △ Less

Submitted 16 April, 2024; originally announced April 2024.

Stale Diffusion: Hyper-realistic 5D Movie Generation Using Old-school Methods

Authors: Joao F. Henriques, Dylan Campbell, Tengda Han

Abstract: Two years ago, Stable Diffusion achieved super-human performance at generating images with super-human numbers of fingers. Following the steady decline of its technical novelty, we propose Stale Diffusion, a method that solidifies and ossifies Stable Diffusion in a maximum-entropy state. Stable Diffusion works analogously to a barn (the Stable) from which an infinite set of horses have escaped (th… ▽ More Two years ago, Stable Diffusion achieved super-human performance at generating images with super-human numbers of fingers. Following the steady decline of its technical novelty, we propose Stale Diffusion, a method that solidifies and ossifies Stable Diffusion in a maximum-entropy state. Stable Diffusion works analogously to a barn (the Stable) from which an infinite set of horses have escaped (the Diffusion). As the horses have long left the barn, our proposal may be seen as antiquated and irrelevant. Nevertheless, we vigorously defend our claim of novelty by identifying as early adopters of the Slow Science Movement, which will produce extremely important pearls of wisdom in the future. Our speed of contributions can also be seen as a quasi-static implementation of the recent call to pause AI experiments, which we wholeheartedly support. As a result of a careful archaeological expedition to 18-months-old Git commit histories, we found that naturally-accumulating errors have produced a novel entropy-maximising Stale Diffusion method, that can produce sleep-inducing hyper-realistic 5D video that is as good as one's imagination. △ Less

Submitted 1 April, 2024; originally announced April 2024.

Comments: SIGBOVIK 2024

Small-scale magnetic flux emergence preceding a chain of energetic solar atmospheric events

Authors: D. Nóbrega-Siverio, I. Cabello, S. Bose, L. H. M. Rouppe van der Voort, R. Joshi, C. Froment, V. M. J. Henriques

Abstract: Advancements in instrumentation have revealed a multitude of small-scale EUV events in the solar atmosphere. Our aim is to employ high-resolution magnetograms to gain a detailed understanding of the magnetic origin of such phenomena. We have used coordinated observations from SST, IRIS, and SDO to analyze an ephemeral magnetic flux emergence episode and the following chain of small-scale energetic… ▽ More Advancements in instrumentation have revealed a multitude of small-scale EUV events in the solar atmosphere. Our aim is to employ high-resolution magnetograms to gain a detailed understanding of the magnetic origin of such phenomena. We have used coordinated observations from SST, IRIS, and SDO to analyze an ephemeral magnetic flux emergence episode and the following chain of small-scale energetic events. These unique observations clearly link these phenomena together. The high-resolution (0."057/pixel) magnetograms obtained with SST/CRISP allows us to reliably measure the magnetic field at the photosphere and detect the emerging bipole that causes the subsequent eruptive atmospheric events. Notably, this small-scale emergence episode remains indiscernible in the lower resolution SDO/HMI magnetograms (0."5/pixel). We report the appearance of a dark bubble in Ca II K related to the emerging bipole, a sign of the canonical expanding magnetic dome predicted in flux emergence simulations. Evidences of reconnection are also found: first through an Ellerman bomb, and later by the launch of a surge next to a UV burst. The UV burst exhibits a weak EUV counterpart in the coronal SDO/AIA channels. By calculating DEM, its plasma is shown to reach a temperature beyond 1 MK and have densities between the upper chromosphere and transition region. Our study showcases the importance of high-resolution magnetograms to unveil the mechanisms triggering phenomena such as EBs, UV bursts, and surges. This could hold implications for small-scale events akin to those recently reported in EUV using Solar Orbiter. The finding of temperatures beyond 1 MK in the UV burst plasma strongly suggests that we are examining analogous features. Therefore, we signal caution regarding drawing conclusions from full-disk magnetograms that lack the necessary resolution to reveal their true magnetic origin. △ Less

Submitted 18 March, 2024; originally announced March 2024.

Comments: Accepted in A&A, 11 pages, 7 figures, 5 movies

Journal ref: A&A 686, A218 (2024)

N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields

Authors: Yash Bhalgat, Iro Laina, João F. Henriques, Andrew Zisserman, Andrea Vedaldi

Abstract: Understanding complex scenes at multiple levels of abstraction remains a formidable challenge in computer vision. To address this, we introduce Nested Neural Feature Fields (N2F2), a novel approach that employs hierarchical supervision to learn a single feature field, wherein different dimensions within the same high-dimensional feature encode scene properties at varying granularities. Our method… ▽ More Understanding complex scenes at multiple levels of abstraction remains a formidable challenge in computer vision. To address this, we introduce Nested Neural Feature Fields (N2F2), a novel approach that employs hierarchical supervision to learn a single feature field, wherein different dimensions within the same high-dimensional feature encode scene properties at varying granularities. Our method allows for a flexible definition of hierarchies, tailored to either the physical dimensions or semantics or both, thereby enabling a comprehensive and nuanced understanding of scenes. We leverage a 2D class-agnostic segmentation model to provide semantically meaningful pixel groupings at arbitrary scales in the image space, and query the CLIP vision-encoder to obtain language-aligned embeddings for each of these segments. Our proposed hierarchical supervision method then assigns different nested dimensions of the feature field to distill the CLIP embeddings using deferred volumetric rendering at varying physical scales, creating a coarse-to-fine representation. Extensive experiments show that our approach outperforms the state-of-the-art feature field distillation methods on tasks such as open-vocabulary 3D segmentation and localization, demonstrating the effectiveness of the learned nested feature field. △ Less

Submitted 28 July, 2024; v1 submitted 16 March, 2024; originally announced March 2024.

Comments: ECCV 2024

Multi-level Product Category Prediction through Text Classification

Authors: Wesley Ferreira Maia, Angelo Carmignani, Gabriel Bortoli, Lucas Maretti, David Luz, Daniel Camilo Fuentes Guzman, Marcos Jardel Henriques, Francisco Louzada Neto

Abstract: This article investigates applying advanced machine learning models, specifically LSTM and BERT, for text classification to predict multiple categories in the retail sector. The study demonstrates how applying data augmentation techniques and the focal loss function can significantly enhance accuracy in classifying products into multiple categories using a robust Brazilian retail dataset. The LSTM… ▽ More This article investigates applying advanced machine learning models, specifically LSTM and BERT, for text classification to predict multiple categories in the retail sector. The study demonstrates how applying data augmentation techniques and the focal loss function can significantly enhance accuracy in classifying products into multiple categories using a robust Brazilian retail dataset. The LSTM model, enriched with Brazilian word embedding, and BERT, known for its effectiveness in understanding complex contexts, were adapted and optimized for this specific task. The results showed that the BERT model, with an F1 Macro Score of up to $99\%$ for segments, $96\%$ for categories and subcategories and $93\%$ for name products, outperformed LSTM in more detailed categories. However, LSTM also achieved high performance, especially after applying data augmentation and focal loss techniques. These results underscore the effectiveness of NLP techniques in retail and highlight the importance of the careful selection of modelling and preprocessing strategies. This work contributes significantly to the field of NLP in retail, providing valuable insights for future research and practical applications. △ Less

Submitted 3 March, 2024; originally announced March 2024.

A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval

Authors: Andreea-Maria Oncescu, João F. Henriques, Andrew Zisserman, Samuel Albanie, A. Sophia Koepke

Abstract: Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, treating visual descriptions as audio descriptions is far from optimal. Even if audio class labels are present, they commonly are not very detailed, making them unsuited for text-audio retrieval. To exploit relevant audio in… ▽ More Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, treating visual descriptions as audio descriptions is far from optimal. Even if audio class labels are present, they commonly are not very detailed, making them unsuited for text-audio retrieval. To exploit relevant audio information from video-text datasets, we introduce a methodology for generating audio-centric descriptions using Large Language Models (LLMs). In this work, we consider the egocentric video setting and propose three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset. Our approach for obtaining audio-centric descriptions gives significantly higher zero-shot performance than using the original visual-centric descriptions. Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset. Finally, we confirm that LLMs can be used to determine the difficulty of identifying the action associated with a sound. △ Less

Submitted 29 February, 2024; originally announced February 2024.

Comments: 9 pages, 2 figures, 9 tables, Accepted at ICASSP 2024

Tunable topological phases in nanographene-based spin-1/2 alternating-exchange Heisenberg chains

Authors: Chenxiao Zhao, Gonçalo Catarina, Jin-Jiang Zhang, João C. G. Henriques, Lin Yang, Ji Ma, Xinliang Feng, Oliver Gröning, Pascal Ruffieux, Joaquín Fernández-Rossier, Roman Fasel

Abstract: Unlocking the potential of topological order within many-body spin systems has long been a central pursuit in the realm of quantum materials. Despite extensive efforts, the quest for a versatile platform enabling site-selective spin manipulation, essential for tuning and probing diverse topological phases, has persisted. Here, we utilize on-surface synthesis to construct spin-1/2 alternating-excha… ▽ More Unlocking the potential of topological order within many-body spin systems has long been a central pursuit in the realm of quantum materials. Despite extensive efforts, the quest for a versatile platform enabling site-selective spin manipulation, essential for tuning and probing diverse topological phases, has persisted. Here, we utilize on-surface synthesis to construct spin-1/2 alternating-exchange Heisenberg (AH) chains[1] with antiferromagnetic couplings $J_1$ and $J_2$ by covalently linking Clar's goblets -- nanographenes each hosting two antiferromagnetically-coupled unpaired electrons[2]. Utilizing scanning tunneling microscopy, we exert atomic-scale control over the spin chain lengths, parities and exchange-coupling terminations, and probe their magnetic response by means of inelastic tunneling spectroscopy. Our investigation confirms the gapped nature of bulk excitations in the chains, known as triplons[3]. Besides, the triplon dispersion relation is successfully extracted from the spatial variation of tunneling spectral amplitudes. Furthermore, depending on the parity and termination of chains, we observe varying numbers of in-gap $S=1/2$ edge spins, enabling the determination of the degeneracy of distinct topological ground states in the thermodynamic limit-either 1, 2, or 4. By monitoring interactions between these edge spins, we identify the exponential decay of spin correlations. Our experimental findings, corroborated by theoretical calculations, present a phase-controlled many-body platform, opening promising avenues toward the development of spin-based quantum devices. △ Less

Submitted 21 February, 2024; originally announced February 2024.

SCENES: Subpixel Correspondence Estimation With Epipolar Supervision

Authors: Dominik A. Kloepfer, João F. Henriques, Dylan Campbell

Abstract: Extracting point correspondences from two or more views of a scene is a fundamental computer vision problem with particular importance for relative camera pose estimation and structure-from-motion. Existing local feature matching approaches, trained with correspondence supervision on large-scale datasets, obtain highly-accurate matches on the test sets. However, they do not generalise well to new… ▽ More Extracting point correspondences from two or more views of a scene is a fundamental computer vision problem with particular importance for relative camera pose estimation and structure-from-motion. Existing local feature matching approaches, trained with correspondence supervision on large-scale datasets, obtain highly-accurate matches on the test sets. However, they do not generalise well to new datasets with different characteristics to those they were trained on, unlike classic feature extractors. Instead, they require finetuning, which assumes that ground-truth correspondences or ground-truth camera poses and 3D structure are available. We relax this assumption by removing the requirement of 3D structure, e.g., depth maps or point clouds, and only require camera pose information, which can be obtained from odometry. We do so by replacing correspondence losses with epipolar losses, which encourage putative matches to lie on the associated epipolar line. While weaker than correspondence supervision, we observe that this cue is sufficient for finetuning existing models on new data. We then further relax the assumption of known camera poses by using pose estimates in a novel bootstrapping approach. We evaluate on highly challenging datasets, including an indoor drone dataset and an outdoor smartphone camera dataset, and obtain state-of-the-art results without strong supervision. △ Less

Submitted 19 January, 2024; originally announced January 2024.

LangProp: A code optimization framework using Large Language Models applied to driving

Authors: Shu Ishida, Gianluca Corrado, George Fedoseev, Hudson Yeo, Lloyd Russell, Jamie Shotton, João F. Henriques, Anthony Hu

Abstract: We propose LangProp, a framework for iteratively optimizing code generated by large language models (LLMs), in both supervised and reinforcement learning settings. While LLMs can generate sensible coding solutions zero-shot, they are often sub-optimal. Especially for code generation tasks, it is likely that the initial code will fail on certain edge cases. LangProp automatically evaluates the code… ▽ More We propose LangProp, a framework for iteratively optimizing code generated by large language models (LLMs), in both supervised and reinforcement learning settings. While LLMs can generate sensible coding solutions zero-shot, they are often sub-optimal. Especially for code generation tasks, it is likely that the initial code will fail on certain edge cases. LangProp automatically evaluates the code performance on a dataset of input-output pairs, catches any exceptions, and feeds the results back to the LLM in the training loop, so that the LLM can iteratively improve the code it generates. By adopting a metric- and data-driven training paradigm for this code optimization procedure, one could easily adapt findings from traditional machine learning techniques such as imitation learning, DAgger, and reinforcement learning. We show LangProp's applicability to general domains such as Sudoku and CartPole, as well as demonstrate the first proof of concept of automated code optimization for autonomous driving in CARLA. We show that LangProp can generate interpretable and transparent policies that can be verified and improved in a metric- and data-driven way. Our code is available at https://github.com/shuishida/LangProp. △ Less

Submitted 3 May, 2024; v1 submitted 18 January, 2024; originally announced January 2024.

Beyond spin models in orbitally-degenerate open-shell nanographenes

Authors: J. C. G. Henriques, D. Jacob, A. Molina-Sánchez, G. Catarina, A. T. Costa, J. Fernández-Rossier

Abstract: The study of open-shell nanographenes has relied on a paradigm where spins are the only low-energy degrees of freedom. Here we show that some nanographenes can host low-energy excitations that include strongly coupled spin and orbital degrees of freedom. The key ingredient is the existence of orbital degeneracy, as a consequence of leaving the benzenoid/half-filling scenario. We analyze the case o… ▽ More The study of open-shell nanographenes has relied on a paradigm where spins are the only low-energy degrees of freedom. Here we show that some nanographenes can host low-energy excitations that include strongly coupled spin and orbital degrees of freedom. The key ingredient is the existence of orbital degeneracy, as a consequence of leaving the benzenoid/half-filling scenario. We analyze the case of nitrogen-doped triangulenes, using both density-functional theory and Hubbard model multiconfigurational and random-phase approximation calculations. We find a rich interplay between orbital and spin degrees of freedom that confirms the need to go beyond the spin-only paradigm, opening a new venue in this field of research. △ Less

Submitted 8 December, 2023; originally announced December 2023.

Comments: 6 pages, 4 figures

Rapid Motor Adaptation for Robotic Manipulator Arms

Authors: Yichao Liang, Kevin Ellis, João Henriques

Abstract: Developing generalizable manipulation skills is a core challenge in embodied AI. This includes generalization across diverse task configurations, encompassing variations in object shape, density, friction coefficient, and external disturbances such as forces applied to the robot. Rapid Motor Adaptation (RMA) offers a promising solution to this challenge. It posits that essential hidden variables i… ▽ More Developing generalizable manipulation skills is a core challenge in embodied AI. This includes generalization across diverse task configurations, encompassing variations in object shape, density, friction coefficient, and external disturbances such as forces applied to the robot. Rapid Motor Adaptation (RMA) offers a promising solution to this challenge. It posits that essential hidden variables influencing an agent's task performance, such as object mass and shape, can be effectively inferred from the agent's action and proprioceptive history. Drawing inspiration from RMA in locomotion and in-hand rotation, we use depth perception to develop agents tailored for rapid motor adaptation in a variety of manipulation tasks. We evaluated our agents on four challenging tasks from the Maniskill2 benchmark, namely pick-and-place operations with hundreds of objects from the YCB and EGAD datasets, peg insertion with precise position and orientation, and operating a variety of faucets and handles, with customized environment variations. Empirical results demonstrate that our agents surpass state-of-the-art methods like automatic domain randomization and vision-based policies, obtaining better generalization performance and sample efficiency. △ Less

Submitted 29 March, 2024; v1 submitted 7 December, 2023; originally announced December 2023.

Comments: Accepted at CVPR 2024. 12 pages

Designer spin models in tunable two-dimensional nanographene lattices

Authors: J. C. G. Henriques, Mar Ferri-Cortés, J. Fernández-Rossier

Abstract: Motivated by recent experimental breakthroughs, we propose a strategy to design two-dimensional spin lattices with competing interactions that lead to non-trivial emergent quantum states. We consider $S=1/2$ nanographenes with $C_3$ symmetry as building blocks, and we leverage the potential to control both the sign and the strength of exchange with first neighbours to build a family of spin models… ▽ More Motivated by recent experimental breakthroughs, we propose a strategy to design two-dimensional spin lattices with competing interactions that lead to non-trivial emergent quantum states. We consider $S=1/2$ nanographenes with $C_3$ symmetry as building blocks, and we leverage the potential to control both the sign and the strength of exchange with first neighbours to build a family of spin models. Specifically, we consider the case of a Heisenberg model in a triangle-decorated honeycomb lattice with competing ferromagnetic and antiferromagnetic interactions whose ratio can be varied in a wide range. Based on exact diagonalization of both fermionic and spin models we predict a quantum phase transition between a valence bond crystal of spin singlets with triplon excitations living in a Kagomé lattice and a Néel phase of effective $S=3/2$ in the limit of dominant ferromagnetic interactions. △ Less

Submitted 6 February, 2024; v1 submitted 4 December, 2023; originally announced December 2023.

Comments: 6 pages, 4 figures

Text2Loc: 3D Point Cloud Localization from Natural Language

Authors: Yan Xia, Letian Shi, Zifeng Ding, João F. Henriques, Daniel Cremers

Abstract: We tackle the problem of 3D point cloud localization based on a few natural linguistic descriptions and introduce a novel neural network, Text2Loc, that fully interprets the semantic relationship between points and text. Text2Loc follows a coarse-to-fine localization pipeline: text-submap global place recognition, followed by fine localization. In global place recognition, relational dynamics amon… ▽ More We tackle the problem of 3D point cloud localization based on a few natural linguistic descriptions and introduce a novel neural network, Text2Loc, that fully interprets the semantic relationship between points and text. Text2Loc follows a coarse-to-fine localization pipeline: text-submap global place recognition, followed by fine localization. In global place recognition, relational dynamics among each textual hint are captured in a hierarchical transformer with max-pooling (HTM), whereas a balance between positive and negative pairs is maintained using text-submap contrastive learning. Moreover, we propose a novel matching-free fine localization method to further refine the location predictions, which completely removes the need for complicated text-instance matching and is lighter, faster, and more accurate than previous methods. Extensive experiments show that Text2Loc improves the localization accuracy by up to $2\times$ over the state-of-the-art on the KITTI360Pose dataset. Our project page is publicly available at \url{https://yan-xia.github.io/projects/text2loc/}. △ Less

Submitted 28 March, 2024; v1 submitted 27 November, 2023; originally announced November 2023.

Comments: Accepted by CVPR 2024

Automatic Coronary Artery Plaque Quantification and CAD-RADS Prediction using Mesh Priors

Authors: Rudolf L. M. van Herten, Nils Hampe, Richard A. P. Takx, Klaas Jan Franssen, Yining Wang, Dominika Suchá, José P. Henriques, Tim Leiner, R. Nils Planken, Ivana Išgum

Abstract: Coronary artery disease (CAD) remains the leading cause of death worldwide. Patients with suspected CAD undergo coronary CT angiography (CCTA) to evaluate the risk of cardiovascular events and determine the treatment. Clinical analysis of coronary arteries in CCTA comprises the identification of atherosclerotic plaque, as well as the grading of any coronary artery stenosis typically obtained throu… ▽ More Coronary artery disease (CAD) remains the leading cause of death worldwide. Patients with suspected CAD undergo coronary CT angiography (CCTA) to evaluate the risk of cardiovascular events and determine the treatment. Clinical analysis of coronary arteries in CCTA comprises the identification of atherosclerotic plaque, as well as the grading of any coronary artery stenosis typically obtained through the CAD-Reporting and Data System (CAD-RADS). This requires analysis of the coronary lumen and plaque. While voxel-wise segmentation is a commonly used approach in various segmentation tasks, it does not guarantee topologically plausible shapes. To address this, in this work, we propose to directly infer surface meshes for coronary artery lumen and plaque based on a centerline prior and use it in the downstream task of CAD-RADS scoring. The method is developed and evaluated using a total of 2407 CCTA scans. Our method achieved lesion-wise volume intraclass correlation coefficients of 0.98, 0.79, and 0.85 for calcified, non-calcified, and total plaque volume respectively. Patient-level CAD-RADS categorization was evaluated on a representative hold-out test set of 300 scans, for which the achieved linearly weighted kappa ($κ$) was 0.75. CAD-RADS categorization on the set of 658 scans from another hospital and scanner led to a $κ$ of 0.71. The results demonstrate that direct inference of coronary artery meshes for lumen and plaque is feasible, and allows for the automated prediction of routinely performed CAD-RADS categorization. △ Less

Submitted 17 October, 2023; originally announced October 2023.

Comments: 12 pages, 6 figures, accepted in IEEE Transactions on Medical Imaging

LoCUS: Learning Multiscale 3D-consistent Features from Posed Images

Authors: Dominik A. Kloepfer, Dylan Campbell, João F. Henriques

Abstract: An important challenge for autonomous agents such as robots is to maintain a spatially and temporally consistent model of the world. It must be maintained through occlusions, previously-unseen views, and long time horizons (e.g., loop closure and re-identification). It is still an open question how to train such a versatile neural representation without supervision. We start from the idea that the… ▽ More An important challenge for autonomous agents such as robots is to maintain a spatially and temporally consistent model of the world. It must be maintained through occlusions, previously-unseen views, and long time horizons (e.g., loop closure and re-identification). It is still an open question how to train such a versatile neural representation without supervision. We start from the idea that the training objective can be framed as a patch retrieval problem: given an image patch in one view of a scene, we would like to retrieve (with high precision and recall) all patches in other views that map to the same real-world location. One drawback is that this objective does not promote reusability of features: by being unique to a scene (achieving perfect precision/recall), a representation will not be useful in the context of other scenes. We find that it is possible to balance retrieval and reusability by constructing the retrieval set carefully, leaving out patches that map to far-away locations. Similarly, we can easily regulate the scale of the learned features (e.g., points, objects, or rooms) by adjusting the spatial tolerance for considering a retrieval to be positive. We optimize for (smooth) Average Precision (AP), in a single unified ranking-based objective. This objective also doubles as a criterion for choosing landmarks or keypoints, as patches with high AP. We show results creating sparse, multi-scale, semantic spatial maps composed of highly identifiable landmarks, with applications in landmark retrieval, localization, semantic segmentation and instance segmentation. △ Less

Submitted 2 October, 2023; originally announced October 2023.

Journal ref: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2023, pages 16634-16644

fakenewsbr: A Fake News Detection Platform for Brazilian Portuguese

Authors: Luiz Giordani, Gilsiley Darú, Rhenan Queiroz, Vitor Buzinaro, Davi Keglevich Neiva, Daniel Camilo Fuentes Guzmán, Marcos Jardel Henriques, Oilson Alberto Gonzatto Junior, Francisco Louzada

Abstract: The proliferation of fake news has become a significant concern in recent times due to its potential to spread misinformation and manipulate public opinion. This paper presents a comprehensive study on detecting fake news in Brazilian Portuguese, focusing on journalistic-type news. We propose a machine learning-based approach that leverages natural language processing techniques, including TF-IDF… ▽ More The proliferation of fake news has become a significant concern in recent times due to its potential to spread misinformation and manipulate public opinion. This paper presents a comprehensive study on detecting fake news in Brazilian Portuguese, focusing on journalistic-type news. We propose a machine learning-based approach that leverages natural language processing techniques, including TF-IDF and Word2Vec, to extract features from textual data. We evaluate the performance of various classification algorithms, such as logistic regression, support vector machine, random forest, AdaBoost, and LightGBM, on a dataset containing both true and fake news articles. The proposed approach achieves high accuracy and F1-Score, demonstrating its effectiveness in identifying fake news. Additionally, we developed a user-friendly web platform, fakenewsbr.com, to facilitate the verification of news articles' veracity. Our platform provides real-time analysis, allowing users to assess the likelihood of fake news articles. Through empirical analysis and comparative studies, we demonstrate the potential of our approach to contribute to the fight against the spread of fake news and promote more informed media consumption. △ Less

Submitted 20 September, 2023; v1 submitted 20 September, 2023; originally announced September 2023.