Showing 1–46 of 46 results for author: Key

Advancing Open and Reproducible Relational Learning: RelArena-$α$, TabPFN-Rel and RPI

Authors: Adrian Hayler, Klemens Flöge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum, Jonas Kübler, Siyuan Guo, Oscar Key, Jan Hendrik Metzen, Rylee Grace, David Salinas, Arthur Cahu, Simon Bing, Benjamin Jäger, Tuana Çelik, Mihir Manium, Vitor Monteiro, Jake Robertson, Jerry Chen, Eliott Kalfon , et al. (22 additional authors not shown)

Abstract: This first release of Prior Labs in relational learning shows our continued commitment to open science. We open-source three pieces of software that we expect to accelerate research in the field towards meaningful real-world impact. We aim to steer further development based on feedback from, and in collaboration with, the community. Given the early stage of development, our $α$-release targets res… ▽ More This first release of Prior Labs in relational learning shows our continued commitment to open science. We open-source three pieces of software that we expect to accelerate research in the field towards meaningful real-world impact. We aim to steer further development based on feedback from, and in collaboration with, the community. Given the early stage of development, our $α$-release targets researchers and early-adopting practitioners. Over the past years, a variety of datasets and tasks for relational learning have emerged, but the community has not converged on a reliable, reproducible way to compare different methods on these tasks. Our $α$-release, RelArena-$α$, provides a unified framework for running and comparing baselines on RelBench v1 by standardizing data loading, evaluation protocols, tuning regimes, and support for systems with custom tuning, inspired by established tabular benchmarks such as TabArena. We plan to work with the research community to further develop RelArena-$α$ into a catalyst for progress in the relational learning community. We release the initial version of TabPFN-Rel, a purpose-built relational harness for TabPFN-3. Currently ranked first among models on RelArena-$α$, TabPFN-Rel makes key improvements upon RDBLearn. Beyond its ranking, TabPFN-Rel serves as a strong baseline, adding to the growing evidence that flattening a relational database into a single table remains competitive with specialized relational architectures on real-world tasks. To facilitate adoption of relational learning methods in research and industry, we release an initial $α$-version of our Relational Predictive Interface, RPI, an open-source, model-agnostic interface that enables early adopters to easily define problems on new databases and apply any model implemented in RelArena-$α$, including TabPFN-Rel, to these problems. △ Less

Submitted 17 August, 2026; originally announced August 2026.

Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

Authors: Hanzuo Liu, Xuan Qi, Chunyu Liu, Haotian Zhong, Yulong Wang, Rayying, Key, Alex Lamb, Mingyu Gao

Abstract: Transformer depth is not used uniformly: lower and middle layers build semantic representations, while upper layers increasingly specialize them for prediction. We turn this division of labor into CoMem (Comprehension Memory), which writes each context chunk only through an intermediate layer, retrieves a fixed number of cached residual states, and recomputes the query-conditioned upper layers ove… ▽ More Transformer depth is not used uniformly: lower and middle layers build semantic representations, while upper layers increasingly specialize them for prediction. We turn this division of labor into CoMem (Comprehension Memory), which writes each context chunk only through an intermediate layer, retrieves a fixed number of cached residual states, and recomputes the query-conditioned upper layers over the resulting pack. For a fixed retrieval budget, model-side read compute and memory are independent of stored-context length. We evaluate a continued-trained Qwen3-8B base LM under a unified chat-template-free protocol. The backbone is frozen; the flagship trains only a rank-32 self-distillation LoRA on plain PG19, and we report an adapter-free arm separately. CoMem reaches 97.05 on RULER and 38.27 on LoCoMo versus 34.59 for full-context KV-Direct; the dialogue-memory advantage survives conversation-cluster resampling and an independent judge. Results on additional long-context and long-document tasks expose both the benefits of bounded retrieval and its in-window compression tax. Controlled depth sweeps show that deeper caching lowers per-query recomputation but incurs a fidelity loss that self-distillation substantially repairs. In a separate adapter-free efficiency control on an NVIDIA H20 at 128k, CoMem uses 18.26 GB rather than 89.36 GB and achieves a 7.83x prefill speedup. These results show that long-context memory can be organized along the layer axis, not only the token axis. △ Less

Submitted 30 July, 2026; originally announced July 2026.

Comments: 19 pages, 4 figures, 27 tables. Submitted to ACL Rolling Review

ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters

Authors: Jiasheng Zhou, Longbin Zeng, Clavis Chen, Ruiming Lu, Qinwei Yang, Leyi Ye, Ray Ying, Key Zhang

Abstract: Large-scale LLM training requires always-on, fine-grained observability for effective performance diagnosis at scale. Coarse resource monitors alone cannot localize root causes, and fine-grained profilers incur prohibitive (5%-30%) overheads and massive trace volumes, making always-on deployment impractical in large production clusters. We propose ARGUS, a low-overhead, fine-grained, always-on t… ▽ More Large-scale LLM training requires always-on, fine-grained observability for effective performance diagnosis at scale. Coarse resource monitors alone cannot localize root causes, and fine-grained profilers incur prohibitive (5%-30%) overheads and massive trace volumes, making always-on deployment impractical in large production clusters. We propose ARGUS, a low-overhead, fine-grained, always-on tracing and real-time analysis system for training workloads in 10,000+ GPU-scale production clusters. ARGUS decomposes observation along the training call hierarchy into CPU call stacks, framework semantics, and GPU kernel execution, with always-on collection under a combined overhead of less than 2%. It builds a unified data pipeline and compresses raw kernel events by approximately 3,700x from 10 MB to 2.7 KB per rank per step. Its progressive diagnosis framework automatically isolates anomalous windows, straggler ranks, and degraded kernels through iteration-time, phase-level, and kernel-level analysis. Deployed for over six months on a 10,000+ GPU production cluster, ARGUS has supported continuous fail-slow detection and performance optimization. Our case studies further demonstrate its effectiveness across representative anomalies, including compute stragglers, link degradation, pipeline-bubble amplification, FlashAttention JIT stalls, and compute stragglers masked by communication symptoms. △ Less

Submitted 8 July, 2026; v1 submitted 18 June, 2026; originally announced June 2026.

TabPFN-3: Technical Report

Authors: Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, Shi Bin Hoo, Magnus Bühler, Anurag Garg, Dominik Safaric, Jake Robertson, Benjamin Jäger, Simone Alessi, Adrian Hayler, Vladyslav Moroshan, Lennart Purucker, Philipp Singer, Alan Arazi, Julien Siems, Jan Hendrik Metzen, Georg Grab, Nick Erickson, Siyuan Guo, Eliott Kalfon , et al. (16 additional authors not shown)

Abstract: Tabular data underpins most high-value prediction problems in science and industry, and TabPFN has driven the foundation model revolution for this modality. Designed with feedback from our users, TabPFN-3 builds on this foundation to scale state-of-the-art performance to datasets with 1M training rows and substantially reduce training and inference time. Pretrained exclusively on synthetic data fr… ▽ More Tabular data underpins most high-value prediction problems in science and industry, and TabPFN has driven the foundation model revolution for this modality. Designed with feedback from our users, TabPFN-3 builds on this foundation to scale state-of-the-art performance to datasets with 1M training rows and substantially reduce training and inference time. Pretrained exclusively on synthetic data from our prior, TabPFN-3 dramatically pushes the frontier of tabular prediction and brings substantial gains on time series, relational, and tabular-text data. On the standard tabular benchmark TabArena, a forward pass of TabPFN-3 outperforms all other models, including tuned and ensembled baselines, by a significant margin, and pareto-dominates the speed/performance frontier. On more diverse datasets, TabPFN-3 ranks first on datasets with many classes, and beats 8-hour-tuned gradient-boosted-tree baselines on datasets up to 1M training rows and 200 features. TabPFN-3 introduces test-time compute scaling to tabular foundation models. Our API offering TabPFN-3-Plus (Thinking) exploits this to beat all non-TabPFN models by over 200 Elo on TabArena, rising to 420 Elo on the largest data subset, and outperforms AutoGluon 1.5 extreme while being 10x faster, without using LLMs, real data, internet search or any other model besides TabPFN. TabPFN-3 extends the capabilities of our models, enabling SOTA prediction on relational data (new SOTA foundation model on RelBenchV1) and tabular-text data (SOTA on TabSTAR via TabPFN-3-Plus); and improves existing integrations: a specialized checkpoint, TabPFN-TS-3, ranks 2nd on the time-series benchmark fev-bench, and SHAP-value computation is up to 120x faster. TabPFN-3 achieves this performance while being up to 20x faster than TabPFN-2.5. In addition, a reduced KV cache and row-chunking scale to 1M rows on one H100 with fast inference speed. △ Less

Submitted 28 May, 2026; v1 submitted 13 May, 2026; originally announced May 2026.

SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask

Authors: Liu Hanzuo, Chaofan Lin, Weixuan Sun, Yulong Wang, Key, Rayying, Mingyu Gao

Abstract: Semi-structured sparsity provides a practical path to accelerate large language models (LLMs) with native hardware support, but post-training semi-structured pruning often suffers from substantial quality degradation due to strong structural coupling. Existing methods rely on large-scale sparse retraining to recover accuracy, resulting in high computational cost. We propose SparseForge, a post-t… ▽ More Semi-structured sparsity provides a practical path to accelerate large language models (LLMs) with native hardware support, but post-training semi-structured pruning often suffers from substantial quality degradation due to strong structural coupling. Existing methods rely on large-scale sparse retraining to recover accuracy, resulting in high computational cost. We propose SparseForge, a post-training framework that improves recovery efficiency by directly optimizing the sparsity mask rather than scaling up retraining tokens. SparseForge combines Hessian-aware importance estimation with progressive annealing of soft masks into hardware-executable structured sparsity, enabling stable and efficient sparse recovery. On LLaMA-2-7B under 2:4 sparsity, SparseForge achieves 57.27% average zero-shot accuracy with only $\textbf{5B}$ retraining tokens, surpassing the dense model's 56.43% accuracy and approaching the 57.52% result of a state-of-the-art method using $\textbf{40B}$ tokens. Such improvements on the accuracy-efficiency trade-off from SparseForge are shown to be consistent across model families. △ Less

Submitted 7 May, 2026; originally announced May 2026.

A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models

Authors: Avish Vijayaraghavan, Jaskaran Singh Kawatra, Sebin Sabu, Jonny Sheldon, Will Poulett, Alex Eze, Daniel Key, John Booth, Shiren Patel, Jonny Pearson, Dan Schofield, Jonathan Hope, Pavithra Rajendran, Neil Sebire

Abstract: Electronic Patient Record (EPR) systems contain valuable clinical information, but much of it is trapped in unstructured text, limiting its use for research and decision-making. Large language models can extract such information but require substantial computational resources to run locally, and sending sensitive clinical data to cloud-based services, even when deidentified, raises significant pat… ▽ More Electronic Patient Record (EPR) systems contain valuable clinical information, but much of it is trapped in unstructured text, limiting its use for research and decision-making. Large language models can extract such information but require substantial computational resources to run locally, and sending sensitive clinical data to cloud-based services, even when deidentified, raises significant patient privacy concerns. In this study, we develop a resource-efficient semi-automated annotation workflow using small language models (SLMs) to extract structured information from unstructured EPR data, focusing on paediatric histopathology reports. As a proof-of-concept, we apply the workflow to paediatric renal biopsy reports, a domain chosen for its constrained diagnostic scope and well-defined underlying biology. We develop the workflow iteratively with clinical oversight across three meetings, manually annotating 400 reports from a dataset of 2,111 at Great Ormond Street Hospital as a gold standard, while developing an automated information extraction approach using SLMs. We frame extraction as a Question-Answering task grounded by clinician-guided entity guidelines and few-shot examples, evaluating five instruction-tuned SLMs with a disagreement modelling framework to prioritise reports for clinical review. Gemma 2 2B achieves the highest accuracy at 84.3%, outperforming off-the-shelf models including spaCy (74.3%), BioBERT-SQuAD (62.3%), RoBERTa-SQuAD (59.7%), and GLiNER (60.2%). Entity guidelines improved performance by 7-19% over the zero-shot baseline, and few-shot examples by 6-38%, though their benefits do not compound when combined. These results demonstrate that SLMs can extract structured information from specialised clinical domains on CPU-only infrastructure with minimal clinician involvement. Our code is available at https://github.com/gosh-dre/nlp_renal_biopsy. △ Less

Submitted 7 April, 2026; v1 submitted 5 April, 2026; originally announced April 2026.

Comments: 36 pages, includes supplementary information

Adaptive Encoding Strategy for Quantum Annealing in Mixed-Variable Engineering Optimization

Authors: Fabian Key, Lukas Freinberger, Mayu Muramatsu, Norbert Hosters

Abstract: Mixed discrete-continuous optimization is central to engineering design, where discrete choices interact with continuous fields. These problems are difficult due to high-dimensional, complex search spaces. To tackle them, Quantum Annealing (QA) is promising, yet its native binary nature supports only discrete variables, making accurate and efficient encodings of continuous quantities a central cha… ▽ More Mixed discrete-continuous optimization is central to engineering design, where discrete choices interact with continuous fields. These problems are difficult due to high-dimensional, complex search spaces. To tackle them, Quantum Annealing (QA) is promising, yet its native binary nature supports only discrete variables, making accurate and efficient encodings of continuous quantities a central challenge. Existing approaches either split the coupled problem, mapping discrete decisions to QA while solving continuous fields classically, or use fixed-bit-depth encodings. The former compromises QA's global search advantages; the latter can underrepresent dynamic range or inflate the number of binary variables. We show that simply increasing bit depth can even degrade performance on current QA hardware, underscoring the need for alternative encodings. In response, we introduce an adaptive encoding strategy for continuous variables in QA that enables efficient treatment of coupled mixed-variable problems. We propose an update strategy for the representable ranges of the continuous variables and demonstrate its utility by integrating it into the minimum complementary energy formulation for structural design optimization, which provides a single, coupled constrained problem. We apply a quadratic penalty method where we update the representation of the continuous variables while targeting the full original objective, preserving QA's global search capability. On a published benchmark, the size optimization of a composite rod, our adaptive encoding improves solution quality under a fixed binary variable budget, demonstrating a superior precision-resource trade-off. Since the framework generalizes beyond structural design, it offers practical guidance for encoding continuous variables for QA and indicates that adaptive representations can enhance precision on current hardware. △ Less

Submitted 10 July, 2026; v1 submitted 18 March, 2026; originally announced March 2026.

Real-Time American Sign Language Recognition Using 3D Convolutional Neural Networks and LSTM: Architecture, Training, and Deployment

Authors: Dawnena Key

Abstract: This paper presents a real-time American Sign Language (ASL) recognition system utilizing a hybrid deep learning architecture combining 3D Convolutional Neural Networks (3D CNN) with Long Short-Term Memory (LSTM) networks. The system processes webcam video streams to recognize word-level ASL signs, addressing communication barriers for over 70 million deaf and hard-of-hearing individuals worldwide… ▽ More This paper presents a real-time American Sign Language (ASL) recognition system utilizing a hybrid deep learning architecture combining 3D Convolutional Neural Networks (3D CNN) with Long Short-Term Memory (LSTM) networks. The system processes webcam video streams to recognize word-level ASL signs, addressing communication barriers for over 70 million deaf and hard-of-hearing individuals worldwide. Our architecture leverages 3D convolutions to capture spatial-temporal features from video frames, followed by LSTM layers that model sequential dependencies inherent in sign language gestures. Trained on the WLASL dataset (2,000 common words), ASL-LEX lexical database (~2,700 signs), and a curated set of 100 expert-annotated ASL signs, the system achieves F1-scores ranging from 0.71 to 0.99 across sign classes. The model is deployed on AWS infrastructure with edge deployment capability on OAK-D cameras for real-time inference. We discuss the architecture design, training methodology, evaluation metrics, and deployment considerations for practical accessibility applications. △ Less

Submitted 18 December, 2025; originally announced December 2025.

Comments: 10 pages, 1 figure, 2 tables. Patent pending (US 63/918,518). Code available at https://github.com/dawnenakey/spokhandSLR

TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models

Authors: Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rosen Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann , et al. (1 additional authors not shown)

Abstract: The first tabular foundation model, TabPFN, and its successor TabPFNv2 have impacted tabular AI substantially, with dozens of methods building on it and hundreds of applications across different use cases. This report introduces TabPFN-2.5, the next generation of our tabular foundation model, built for datasets with up to 50,000 data points and 2,000 features, a 20x increase in data cells compared… ▽ More The first tabular foundation model, TabPFN, and its successor TabPFNv2 have impacted tabular AI substantially, with dozens of methods building on it and hundreds of applications across different use cases. This report introduces TabPFN-2.5, the next generation of our tabular foundation model, built for datasets with up to 50,000 data points and 2,000 features, a 20x increase in data cells compared to TabPFNv2. TabPFN-2.5 is now the leading method for the industry standard benchmark TabArena (which contains datasets with up to 100,000 training data points), substantially outperforming tuned tree-based models and matching the accuracy of AutoGluon 1.4, a complex four-hour tuned ensemble that even includes the previous TabPFNv2. Remarkably, default TabPFN-2.5 has a 100% win rate against default XGBoost on small to medium-sized classification datasets (<=10,000 data points, 500 features) and a 87% win rate on larger datasets up to 100K samples and 2K features (85% for regression). For production use cases, we introduce a new distillation engine that converts TabPFN-2.5 into a compact MLP or tree ensemble, preserving most of its accuracy while delivering orders-of-magnitude lower latency and plug-and-play deployment. This new release will immediately strengthen the performance of the many applications and methods already built on the TabPFN ecosystem. △ Less

Submitted 4 February, 2026; v1 submitted 11 November, 2025; originally announced November 2025.

On design, analysis, and hybrid manufacturing of microstructured blade-like geometries

Authors: Pablo Antolin, Michael Barton, Georges-Pierre Bonneau, Annalisa Buffa, Amaia Calleja-Ochoa, Gershon Elber, Stefanie Elgeti, Gaizka Gómez Escudero, Alicia Gonzalez, Haizea González Barrio, Stefanie Hahmann, Thibaut Hirschler, Q Youn Honga, Konstantin Key, Myung-Soo Kim, Michael Kofler, Norberto Lopez de Lacalle, Silvia de la Maza, Kanika Rajain, Jacques Zwar

Abstract: With the evolution of new manufacturing technologies such as multi-material 3D printing, one can think of new type of objects that consist of considerably less, yet heterogeneous, material, consequently being porous, lighter and cheaper, while having the very same functionality as the original object when manufactured from one single solid material. We aim at questioning five decades of traditiona… ▽ More With the evolution of new manufacturing technologies such as multi-material 3D printing, one can think of new type of objects that consist of considerably less, yet heterogeneous, material, consequently being porous, lighter and cheaper, while having the very same functionality as the original object when manufactured from one single solid material. We aim at questioning five decades of traditional paradigms in geometric CAD and focus at new generation of CAD objects that are not solid, but contain heterogeneous free-form internal microstructures. We propose a unified manufacturing pipeline that involves all stages, namely design, optimization, manufacturing, and inspection of microstructured free-form geometries. We demonstrate our pipeline on an industrial test case of a blisk blade that sustains the desired pressure limits, yet requires significantly less material when compared to the solid counterpart. △ Less

Submitted 8 September, 2025; originally announced September 2025.

Comments: 14 pages, 23 figures

A Spline-Based Stress Function Approach for the Principle of Minimum Complementary Energy

Authors: Fabian Key, Lukas Freinberger

Abstract: In computational engineering, ensuring the integrity and safety of structures in fields such as aerospace and civil engineering relies on accurate stress prediction. However, analytical methods are limited to simple test cases, and displacement-based finite element methods (FEMs), while commonly used, require a large number of unknowns to achieve high accuracy; stress-based numerical methods have… ▽ More In computational engineering, ensuring the integrity and safety of structures in fields such as aerospace and civil engineering relies on accurate stress prediction. However, analytical methods are limited to simple test cases, and displacement-based finite element methods (FEMs), while commonly used, require a large number of unknowns to achieve high accuracy; stress-based numerical methods have so far failed to provide a simple and effective alternative. This work aims to develop a novel numerical approach that overcomes these limitations by enabling accurate stress prediction with improved flexibility for complex geometries and boundary conditions and fewer degrees of freedom (DOFs). The proposed method is based on a spline-based stress function formulation for the principle of minimum complementary energy, which we apply to plane, linear elastostatics. The method is first validated against analytical solutions and then tested on two test cases challenging for current state-of-the-art numerical schemes, a bi-layer cantilever with anisotropic material behavior, and a cantilever with a non-prismatic, parabolic-shaped beam geometry. Results demonstrate that our approach, unlike analytical methods, can be easily applied to general geometries and boundary conditions, and achieves stress accuracy comparable to that reported in the literature for displacement-based FEMs, while requiring significantly fewer DOFs. This novel spline-based stress function approach thus provides an efficient and flexible tool for accurate stress prediction, with promising applications in structural analysis and numerical design. △ Less

Submitted 23 September, 2025; v1 submitted 24 June, 2025; originally announced June 2025.

A SAT-centered XAI method for Deep Learning based Video Understanding

Authors: Hojer Key

Abstract: This paper introduces a novel formal SAT-based explanation model for deep learning in video understanding. The proposed method integrates SAT solving techniques with the principles of formal explainable AI to address the limitations of existing XAI techniques in this domain. By encoding deep learning models and video data into a logical framework and formulating explanation queries as satisfiabili… ▽ More This paper introduces a novel formal SAT-based explanation model for deep learning in video understanding. The proposed method integrates SAT solving techniques with the principles of formal explainable AI to address the limitations of existing XAI techniques in this domain. By encoding deep learning models and video data into a logical framework and formulating explanation queries as satisfiability problems, the method aims to generate logic-based explanations with formal guarantees. The paper details the conceptual framework, the process of encoding deep learning models and video data, the formulation of "Why?" and "Why not?" questions, and a novel architecture integrating a SAT solver with a deep learning video understanding model. While challenges related to computational complexity and the representational power of propositional logic remain, the proposed approach offers a promising direction for enhancing the explainability of deep learning in the complex and critical domain of video understanding. △ Less

Submitted 12 May, 2025; v1 submitted 31 March, 2025; originally announced March 2025.

Automated Reasoning in Blockchain: Foundations, Applications, and Frontiers

Authors: Hojer Key

Abstract: Blockchain technology has emerged as a transformative paradigm for decentralized and secure data management across diverse application domains, including healthcare, supply chain management, and the Internet of Things. Its core features, such as decentralization, immutability, and auditability, achieved through distributed consensus algorithms and cryptographic techniques, offer significant advant… ▽ More Blockchain technology has emerged as a transformative paradigm for decentralized and secure data management across diverse application domains, including healthcare, supply chain management, and the Internet of Things. Its core features, such as decentralization, immutability, and auditability, achieved through distributed consensus algorithms and cryptographic techniques, offer significant advantages for multi-stakeholder applications requiring transparency and trust. However, the inherent complexity and security-critical nature of blockchain systems necessitate rigorous analysis and verification to ensure their correctness, reliability, and resilience against potential vulnerabilities. △ Less

Submitted 12 May, 2025; v1 submitted 26 March, 2025; originally announced March 2025.

Approximate Top-$k$ for Increased Parallelism

Authors: Oscar Key, Luka Ribar, Alberto Cattaneo, Luke Hudlass-Galley, Douglas Orr

Abstract: We present an evaluation of bucketed approximate top-$k$ algorithms. Computing top-$k$ exactly suffers from limited parallelism, because the $k$ largest values must be aggregated along the vector, thus is not well suited to computation on highly-parallel machine learning accelerators. By relaxing the requirement that the top-$k$ is exact, bucketed algorithms can dramatically increase the paralleli… ▽ More We present an evaluation of bucketed approximate top-$k$ algorithms. Computing top-$k$ exactly suffers from limited parallelism, because the $k$ largest values must be aggregated along the vector, thus is not well suited to computation on highly-parallel machine learning accelerators. By relaxing the requirement that the top-$k$ is exact, bucketed algorithms can dramatically increase the parallelism available by independently computing many smaller top-$k$ operations. We explore the design choices of this class of algorithms using both theoretical analysis and empirical evaluation on downstream tasks. Our motivating examples are sparsity algorithms for language models, which often use top-$k$ to select the most important parameters or activations. We also release a fast bucketed top-$k$ implementation for PyTorch. △ Less

Submitted 5 December, 2024; originally announced December 2024.

An Ising Machine Formulation for Design Updates in Topology Optimization of Flow Channels

Authors: Yudai Suzuki, Shiori Aoki, Fabian Key, Katsuhiro Endo, Yoshiki Matsuda, Shu Tanaka, Marek Behr, Mayu Muramatsu

Abstract: Topology optimization is an essential tool in computational engineering, for example, to improve the design and efficiency of flow channels. At the same time, Ising machines, including digital or quantum annealers, have been used as efficient solvers for combinatorial optimization problems. Beyond combinatorial optimization, recent works have demonstrated applicability to other engineering tasks b… ▽ More Topology optimization is an essential tool in computational engineering, for example, to improve the design and efficiency of flow channels. At the same time, Ising machines, including digital or quantum annealers, have been used as efficient solvers for combinatorial optimization problems. Beyond combinatorial optimization, recent works have demonstrated applicability to other engineering tasks by tailoring corresponding problem formulations. In this study, we present a novel Ising machine formulation for computing design updates during topology optimization with the goal of minimizing dissipation energy in flow channels. We explore the potential of this approach to improve the efficiency and performance of the optimization process. To this end, we conduct experiments to study the impact of various factors within the novel formulation. Additionally, we compare it to a classical method using the number of optimization steps and the final values of the objective function as indicators of the time intensity of the optimization and the performance of the resulting designs, respectively. Our findings show that the proposed update strategy can accelerate the topology optimization process while producing comparable designs. However, it tends to be less exploratory, which may lead to lower performance of the designs. These results highlight the potential of incorporating Ising formulations for optimization tasks but also show their limitations when used to compute design updates in an iterative optimization process. In conclusion, this work provides an efficient alternative for design updates in topology optimization and enhances the understanding of integrating Ising machine formulations in engineering optimization. △ Less

Submitted 13 November, 2024; originally announced November 2024.

Generative Discrete Event Process Simulation for Hidden Markov Models to Predict Competitor Time-to-Market

Authors: Nandakishore Santhi, Stephan Eidenbenz, Brian Key, George Tompkins

Abstract: We study the challenge of predicting the time at which a competitor product, such as a novel high-capacity EV battery or a new car model, will be available to customers; as new information is obtained, this time-to-market estimate is revised. Our scenario is as follows: We assume that the product is under development at a Firm B, which is a competitor to Firm A; as they are in the same industry, F… ▽ More We study the challenge of predicting the time at which a competitor product, such as a novel high-capacity EV battery or a new car model, will be available to customers; as new information is obtained, this time-to-market estimate is revised. Our scenario is as follows: We assume that the product is under development at a Firm B, which is a competitor to Firm A; as they are in the same industry, Firm A has a relatively good understanding of the processes and steps required to produce the product. While Firm B tries to keep its activities hidden (think of stealth-mode for start-ups), Firm A is nevertheless able to gain periodic insights by observing what type of resources Firm B is using. We show how Firm A can build a model that predicts when Firm B will be ready to sell its product; the model leverages knowledge of the underlying processes and required resources to build a Parallel Discrete Simulation (PDES)-based process model that it then uses as a generative model to train a Hidden Markov Model (HMM). We study the question of how many resource observations Firm A requires in order to accurately assess the current state of development at Firm B. In order to gain general insights into the capabilities of this approach, we study the effect of different process graph densities, different densities of the resource-activity maps, etc., and also scaling properties as we increase the number resource counts. We find that in most cases, the HMM achieves a prediction accuracy of 70 to 80 percent after 20 (daily) observations of a production process that lasts 150 days on average and we characterize the effects of different problem instance densities on this prediction accuracy. Our results give insight into the level of market knowledge required for accurate and early time-to-market prediction. △ Less

Submitted 6 November, 2024; originally announced November 2024.

Report number: LA-UR:24-31512

Advancing EEG-Based Gaze Prediction Using Depthwise Separable Convolution and Enhanced Pre-Processing

Authors: Matthew L Key, Tural Mehtiyev, Xiaodong Qu

Abstract: In the field of EEG-based gaze prediction, the application of deep learning to interpret complex neural data poses significant challenges. This study evaluates the effectiveness of pre-processing techniques and the effect of additional depthwise separable convolution on EEG vision transformers (ViTs) in a pretrained model architecture. We introduce a novel method, the EEG Deeper Clustered Vision T… ▽ More In the field of EEG-based gaze prediction, the application of deep learning to interpret complex neural data poses significant challenges. This study evaluates the effectiveness of pre-processing techniques and the effect of additional depthwise separable convolution on EEG vision transformers (ViTs) in a pretrained model architecture. We introduce a novel method, the EEG Deeper Clustered Vision Transformer (EEG-DCViT), which combines depthwise separable convolutional neural networks (CNNs) with vision transformers, enriched by a pre-processing strategy involving data clustering. The new approach demonstrates superior performance, establishing a new benchmark with a Root Mean Square Error (RMSE) of 51.6 mm. This achievement underscores the impact of pre-processing and model refinement in enhancing EEG-based applications. △ Less

Submitted 6 August, 2024; originally announced August 2024.

Journal ref: International Conference on Human-Computer Interaction (HCII 2024)

Effect of Kernel Size on CNN-Vision-Transformer-Based Gaze Prediction Using Electroencephalography Data

Authors: Chuhui Qiu, Bugao Liang, Matthew L Key

Abstract: In this paper, we present an algorithm of gaze prediction from Electroencephalography (EEG) data. EEG-based gaze prediction is a new research topic that can serve as an alternative to traditional video-based eye-tracking. Compared to the existing state-of-the-art (SOTA) method, we improved the root mean-squared-error of EEG-based gaze prediction to 53.06 millimeters, while reducing the training ti… ▽ More In this paper, we present an algorithm of gaze prediction from Electroencephalography (EEG) data. EEG-based gaze prediction is a new research topic that can serve as an alternative to traditional video-based eye-tracking. Compared to the existing state-of-the-art (SOTA) method, we improved the root mean-squared-error of EEG-based gaze prediction to 53.06 millimeters, while reducing the training time to less than 33% of its original duration. Our source code can be found at https://github.com/AmCh-Q/CSCI6907Project △ Less

Submitted 6 August, 2024; originally announced August 2024.

Comments: International Conference on Human-Computer Interaction (HCII 2024)

Integrating HCI Datasets in Project-Based Machine Learning Courses: A College-Level Review and Case Study

Authors: Xiaodong Qu, Matthew Key, Eric Luo, Chuhui Qiu

Abstract: This study explores the integration of real-world machine learning (ML) projects using human-computer interfaces (HCI) datasets in college-level courses to enhance both teaching and learning experiences. Employing a comprehensive literature review, course websites analysis, and a detailed case study, the research identifies best practices for incorporating HCI datasets into project-based ML educat… ▽ More This study explores the integration of real-world machine learning (ML) projects using human-computer interfaces (HCI) datasets in college-level courses to enhance both teaching and learning experiences. Employing a comprehensive literature review, course websites analysis, and a detailed case study, the research identifies best practices for incorporating HCI datasets into project-based ML education. Key f indings demonstrate increased student engagement, motivation, and skill development through hands-on projects, while instructors benefit from effective tools for teaching complex concepts. The study also addresses challenges such as data complexity and resource allocation, offering recommendations for future improvements. These insights provide a valuable framework for educators aiming to bridge the gap between △ Less

Submitted 6 August, 2024; originally announced August 2024.

Journal ref: International Conference on Human-Computer Interaction (HCII 2024)

HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

Authors: Andy He, Darren Key, Mason Bulling, Andrew Chang, Skyler Shapiro, Everett Lee

Abstract: Graphics Processing Units (GPUs) have become the leading hardware accelerator for deep learning applications and are used widely in training and inference of transformers; transformers have achieved state-of-the-art performance in many areas of machine learning and are especially used in most modern Large Language Models (LLMs). However, GPUs require large amounts of energy, which poses environmen… ▽ More Graphics Processing Units (GPUs) have become the leading hardware accelerator for deep learning applications and are used widely in training and inference of transformers; transformers have achieved state-of-the-art performance in many areas of machine learning and are especially used in most modern Large Language Models (LLMs). However, GPUs require large amounts of energy, which poses environmental concerns, demands high operational costs, and causes GPUs to be unsuitable for edge computing. We develop an accelerator for transformers, namely, Llama 2, an open-source state-of-the-art LLM, using high level synthesis (HLS) on Field Programmable Gate Arrays (FPGAs). HLS allows us to rapidly prototype FPGA designs without writing code at the register-transfer level (RTL). We name our method HLSTransform, and the FPGA designs we synthesize with HLS achieve up to a 12.75x reduction and 8.25x reduction in energy used per token on the Xilinx Virtex UltraScale+ VU9P FPGA compared to an Intel Xeon Broadwell E5-2686 v4 CPU and NVIDIA RTX 3090 GPU respectively, while increasing inference speeds by up to 2.46x compared to CPU and maintaining 0.53x the speed of an RTX 3090 GPU despite the GPU's 4 times higher base clock rate. With the lack of existing open-source FPGA accelerators for transformers, we open-source our code and document our steps for synthesis. We hope this work will serve as a step in democratizing the use of FPGAs in transformer inference and inspire research into energy-efficient inference methods as a whole. The code can be found on https://github.com/HLSTransform/submission. △ Less

Submitted 29 April, 2024; originally announced May 2024.

Comments: 7 pages, 2 figures

Scalable Data Assimilation with Message Passing

Authors: Oscar Key, So Takao, Daniel Giles, Marc Peter Deisenroth

Abstract: Data assimilation is a core component of numerical weather prediction systems. The large quantity of data processed during assimilation requires the computation to be distributed across increasingly many compute nodes, yet existing approaches suffer from synchronisation overhead in this setting. In this paper, we exploit the formulation of data assimilation as a Bayesian inference problem and appl… ▽ More Data assimilation is a core component of numerical weather prediction systems. The large quantity of data processed during assimilation requires the computation to be distributed across increasingly many compute nodes, yet existing approaches suffer from synchronisation overhead in this setting. In this paper, we exploit the formulation of data assimilation as a Bayesian inference problem and apply a message-passing algorithm to solve the spatial inference problem. Since message passing is inherently based on local computations, this approach lends itself to parallel and distributed computation. In combination with a GPU-accelerated implementation, we can scale the algorithm to very large grid sizes while retaining good accuracy and compute and memory requirements. △ Less

Submitted 1 October, 2024; v1 submitted 19 April, 2024; originally announced April 2024.

Journal ref: Environ. Data Science 4 (2025) e1

WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment

Authors: Hao Tang, Darren Key, Kevin Ellis

Abstract: We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment. The world model tries to explain its interactions, while also being optimistic about what reward it can achieve. We define this optimism as a logical constraint between a program and a planner. We study our agent on gridworlds, and on task planning, findi… ▽ More We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment. The world model tries to explain its interactions, while also being optimistic about what reward it can achieve. We define this optimism as a logical constraint between a program and a planner. We study our agent on gridworlds, and on task planning, finding our approach is more sample-efficient compared to deep RL, more compute-efficient compared to ReAct-style agents, and that it can transfer its knowledge across environments by editing its code. △ Less

Submitted 20 September, 2024; v1 submitted 19 February, 2024; originally announced February 2024.

Eulerian Formulation of the Tensor-Based Morphology Equations for Strain-Based Blood Damage Modeling

Authors: Nico Dirkes, Fabian Key, Marek Behr

Abstract: The development of blood-handling medical devices, such as ventricular assist devices, requires the analysis of their biocompatibility. Among other aspects, this includes hemolysis, i.e., red blood cell damage. For this purpose, computational fluid dynamics (CFD) methods are employed to predict blood flow in prototypes. The most basic hemolysis models directly estimate red blood cell damage from f… ▽ More The development of blood-handling medical devices, such as ventricular assist devices, requires the analysis of their biocompatibility. Among other aspects, this includes hemolysis, i.e., red blood cell damage. For this purpose, computational fluid dynamics (CFD) methods are employed to predict blood flow in prototypes. The most basic hemolysis models directly estimate red blood cell damage from fluid stress in the resulting flow field. More advanced models explicitly resolve cell deformation. On the downside, these models are typically written in a Lagrangian formulation, i.e., they require pathline tracking. We present a new Eulerian description of cell deformation, enabling the evaluation of the solution across the whole domain. The resulting hemolysis model can be applied to any converged CFD simulation due to one-way coupling with the fluid velocity field. We discuss the efficient numerical treatment of the model equations in a stabilized finite element context. We verify the model by comparison to the original Lagrangian formulation in selected benchmark flows. Two more complex test cases demonstrate the method's capabilities in real-world applications. The results highlight the advantages over previous hemolysis models. In conclusion, the model holds great potential for the design process of future generations of medical devices. △ Less

Submitted 11 April, 2024; v1 submitted 14 February, 2024; originally announced February 2024.

Comments: 39 pages, 11 figures

Journal ref: Computer Methods in Applied Mechanics and Engineering, Volume 426, 2024, 116979

A Formulation of Structural Design Optimization Problems for Quantum Annealing

Authors: Fabian Key, Lukas Freinberger

Abstract: We present a novel formulation of structural design optimization problems specifically tailored to be solved by quantum annealing (QA). Structural design optimization aims to find the best, i.e., material-efficient yet high-performance, configuration of a structure. To this end, computational optimization strategies can be employed, where a recently evolving strategy based on quantum mechanical ef… ▽ More We present a novel formulation of structural design optimization problems specifically tailored to be solved by quantum annealing (QA). Structural design optimization aims to find the best, i.e., material-efficient yet high-performance, configuration of a structure. To this end, computational optimization strategies can be employed, where a recently evolving strategy based on quantum mechanical effects is QA. This approach requires the optimization problem to be present, e.g., as a quadratic unconstrained binary optimization (QUBO) model. Thus, we develop a novel formulation of the optimization problem. The latter typically involves an analysis model for the component. Here, we use energy minimization principles that govern the behavior of structures under applied loads. This allows us to state the optimization problem as one overall minimization problem. Next, we map this to a QUBO problem that can be immediately solved by QA. We validate the proposed approach using a size optimization problem of a compound rod under self-weight loading. To this end, we develop strategies to account for the limitations of currently available hardware and find that the presented formulation is suitable for solving structural design optimization problems through QA and, for small-scale problems, already works on today's hardware. △ Less

Submitted 30 November, 2023; originally announced November 2023.

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

Authors: Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, Matt J. Kusner

Abstract: The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch sel… ▽ More The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain. △ Less

Submitted 14 November, 2023; v1 submitted 12 July, 2023; originally announced July 2023.

Comments: NeurIPS 2023

Toward Trustworthy Neural Program Synthesis

Authors: Darren Key, Wen-Ding Li, Kevin Ellis

Abstract: We develop an approach to estimate the probability that a program sampled from a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave. This allows learning a model that forms a well-calibrated probabilistic prediction of program correctness.… ▽ More We develop an approach to estimate the probability that a program sampled from a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave. This allows learning a model that forms a well-calibrated probabilistic prediction of program correctness. Our system also infers which predicates are useful to explain the behavior of the generated code, and humans preferred these in a human study over raw language model outputs. Our method is simple, easy to implement, and maintains state of the art generation accuracy results. △ Less

Submitted 9 October, 2023; v1 submitted 29 September, 2022; originally announced October 2022.

Comments: 9 pages, 8 figures

Towards Healing the Blindness of Score Matching

Authors: Mingtian Zhang, Oscar Key, Peter Hayes, David Barber, Brooks Paige, François-Xavier Briol

Abstract: Score-based divergences have been widely used in machine learning and statistics applications. Despite their empirical success, a blindness problem has been observed when using these for multi-modal distributions. In this work, we discuss the blindness problem and propose a new family of divergences that can mitigate the blindness problem. We illustrate our proposed divergence in the context of de… ▽ More Score-based divergences have been widely used in machine learning and statistics applications. Despite their empirical success, a blindness problem has been observed when using these for multi-modal distributions. In this work, we discuss the blindness problem and propose a new family of divergences that can mitigate the blindness problem. We illustrate our proposed divergence in the context of density estimation and report improved performance compared to traditional approaches. △ Less

Submitted 22 November, 2025; v1 submitted 15 September, 2022; originally announced September 2022.

Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift

Authors: Liwen Ouyang, Aaron Key

Abstract: Covariate shifts are a common problem in predictive modeling on real-world problems. This paper proposes addressing the covariate shift problem by minimizing Maximum Mean Discrepancy (MMD) statistics between the training and test sets in either feature input space, feature representation space, or both. We designed three techniques that we call MMD Representation, MMD Mask, and MMD Hybrid to deal… ▽ More Covariate shifts are a common problem in predictive modeling on real-world problems. This paper proposes addressing the covariate shift problem by minimizing Maximum Mean Discrepancy (MMD) statistics between the training and test sets in either feature input space, feature representation space, or both. We designed three techniques that we call MMD Representation, MMD Mask, and MMD Hybrid to deal with the scenarios where only a distribution shift exists, only a missingness shift exists, or both types of shift exist, respectively. We find that integrating an MMD loss component helps models use the best features for generalization and avoid dangerous extrapolation as much as possible for each test sample. Models treated with this MMD approach show better performance, calibration, and extrapolation on the test set. △ Less

Submitted 1 March, 2022; v1 submitted 19 November, 2021; originally announced November 2021.

Comments: a short version accepted by NeurIPS DistShift Workshop 2021

Composite Goodness-of-fit Tests with Kernels

Authors: Oscar Key, Arthur Gretton, François-Xavier Briol, Tamara Fernandez

Abstract: Model misspecification can create significant challenges for the implementation of probabilistic models, and this has led to development of a range of robust methods which directly account for this issue. However, whether these more involved methods are required will depend on whether the model is really misspecified, and there is a lack of generally applicable methods to answer this question. In… ▽ More Model misspecification can create significant challenges for the implementation of probabilistic models, and this has led to development of a range of robust methods which directly account for this issue. However, whether these more involved methods are required will depend on whether the model is really misspecified, and there is a lack of generally applicable methods to answer this question. In this paper, we propose one such method. More precisely, we propose kernel-based hypothesis tests for the challenging composite testing problem, where we are interested in whether the data comes from any distribution in some parametric family. Our tests make use of minimum distance estimators based on the maximum mean discrepancy and the kernel Stein discrepancy. They are widely applicable, including whenever the density of the parametric model is known up to normalisation constant, or if the model takes the form of a simulator. As our main result, we show that we are able to estimate the parameter and conduct our test on the same data (without data splitting), while maintaining a correct test level. Our approach is illustrated on a range of problems, including testing for goodness-of-fit of an unnormalised non-parametric density model, and an intractable generative model of a biological cellular network. △ Less

Submitted 19 April, 2025; v1 submitted 19 November, 2021; originally announced November 2021.

Journal ref: Journal of Machine Learning Research 26(51):1-60 2025

Estimation of Corporate Greenhouse Gas Emissions via Machine Learning

Authors: You Han, Achintya Gopal, Liwen Ouyang, Aaron Key

Abstract: As an important step to fulfill the Paris Agreement and achieve net-zero emissions by 2050, the European Commission adopted the most ambitious package of climate impact measures in April 2021 to improve the flow of capital towards sustainable activities. For these and other international measures to be successful, reliable data is key. The ability to see the carbon footprint of companies around th… ▽ More As an important step to fulfill the Paris Agreement and achieve net-zero emissions by 2050, the European Commission adopted the most ambitious package of climate impact measures in April 2021 to improve the flow of capital towards sustainable activities. For these and other international measures to be successful, reliable data is key. The ability to see the carbon footprint of companies around the world will be critical for investors to comply with the measures. However, with only a small portion of companies volunteering to disclose their greenhouse gas (GHG) emissions, it is nearly impossible for investors to align their investment strategies with the measures. By training a machine learning model on disclosed GHG emissions, we are able to estimate the emissions of other companies globally who do not disclose their emissions. In this paper, we show that our model provides accurate estimates of corporate GHG emissions to investors such that they are able to align their investments with the regulatory measures and achieve net-zero goals. △ Less

Submitted 9 September, 2021; originally announced September 2021.

Comments: Accepted for the Tackling Climate Change with Machine Learning Workshop at ICML 2021

Signal Processing Challenges and Examples for {\it in-situ} Transmission Electron Microscopy

Authors: Josh Kacher, Yao Xie, Sven P. Voigt, Shixiang Zhu, Henry Yuchi, Jordan Key, Surya R. Kalidindi

Abstract: Transmission Electron Microscopy (TEM) is a powerful tool for imaging material structure and characterizing material chemistry. Recent advances in data collection technology for TEM have enabled high-volume and high-resolution data collection at a microsecond frame rate. Taking advantage of these advances in data collection rates requires the development and application of data processing tools, i… ▽ More Transmission Electron Microscopy (TEM) is a powerful tool for imaging material structure and characterizing material chemistry. Recent advances in data collection technology for TEM have enabled high-volume and high-resolution data collection at a microsecond frame rate. Taking advantage of these advances in data collection rates requires the development and application of data processing tools, including image analysis, feature extraction, and streaming data processing techniques. In this paper, we highlight a few areas in materials science that have benefited from combining signal processing and statistical analysis with data collection capabilities in TEM and present a future outlook on opportunities of integrating signal processing with automated TEM data analysis. △ Less

Submitted 20 August, 2021; v1 submitted 17 April, 2021; originally announced April 2021.

Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties

Authors: Lisa Schut, Oscar Key, Rory McGrath, Luca Costabello, Bogdan Sacaleanu, Medb Corcoran, Yarin Gal

Abstract: Counterfactual explanations (CEs) are a practical tool for demonstrating why machine learning classifiers make particular decisions. For CEs to be useful, it is important that they are easy for users to interpret. Existing methods for generating interpretable CEs rely on auxiliary generative models, which may not be suitable for complex datasets, and incur engineering overhead. We introduce a simp… ▽ More Counterfactual explanations (CEs) are a practical tool for demonstrating why machine learning classifiers make particular decisions. For CEs to be useful, it is important that they are easy for users to interpret. Existing methods for generating interpretable CEs rely on auxiliary generative models, which may not be suitable for complex datasets, and incur engineering overhead. We introduce a simple and fast method for generating interpretable CEs in a white-box setting without an auxiliary model, by using the predictive uncertainty of the classifier. Our experiments show that our proposed algorithm generates more interpretable CEs, according to IM1 scores, than existing methods. Additionally, our approach allows us to estimate the uncertainty of a CE, which may be important in safety-critical applications, such as those in the medical domain. △ Less

Submitted 16 March, 2021; originally announced March 2021.

Comments: 21 pages, 13 Figures

Journal ref: Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021

On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty

Authors: Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, Yarin Gal

Abstract: Inducing point Gaussian process approximations are often considered a gold standard in uncertainty estimation since they retain many of the properties of the exact GP and scale to large datasets. A major drawback is that they have difficulty scaling to high dimensional inputs. Deep Kernel Learning (DKL) promises a solution: a deep feature extractor transforms the inputs over which an inducing poin… ▽ More Inducing point Gaussian process approximations are often considered a gold standard in uncertainty estimation since they retain many of the properties of the exact GP and scale to large datasets. A major drawback is that they have difficulty scaling to high dimensional inputs. Deep Kernel Learning (DKL) promises a solution: a deep feature extractor transforms the inputs over which an inducing point Gaussian process is defined. However, DKL has been shown to provide unreliable uncertainty estimates in practice. We study why, and show that with no constraints, the DKL objective pushes "far-away" data points to be mapped to the same features as those of training-set points. With this insight we propose to constrain DKL's feature extractor to approximately preserve distances through a bi-Lipschitz constraint, resulting in a feature space favorable to DKL. We obtain a model, DUE, which demonstrates uncertainty quality outperforming previous DKL and other single forward pass uncertainty methods, while maintaining the speed and accuracy of standard neural networks. △ Less

Submitted 7 March, 2022; v1 submitted 22 February, 2021; originally announced February 2021.

A novel approach to fluid-structure interaction simulations involving large translation and contact

Authors: Daniel Hilger, Norbert Hosters, Fabian Key, Stefanie Elgeti, Marek Behr

Abstract: In this work, we present a novel method for the mesh update in flow problems with moving boundaries, the phantom domain deformation mesh update method (PD-DMUM). The PD-DMUM is designed to avoid remeshing; even in the event of large, unidirectional displacements of boundaries. The method combines the concept of two mesh adaptation approaches: (1) The virtual ring shear-slip mesh updatemethod (VR-S… ▽ More In this work, we present a novel method for the mesh update in flow problems with moving boundaries, the phantom domain deformation mesh update method (PD-DMUM). The PD-DMUM is designed to avoid remeshing; even in the event of large, unidirectional displacements of boundaries. The method combines the concept of two mesh adaptation approaches: (1) The virtual ring shear-slip mesh updatemethod (VR-SSMUM); and (2) the elastic mesh update method (EMUM). As in the VR-SSMUM, the PD-DMUMextends the fluid domain by a phantom domain; the PD-DMUM can thus locally adapt the element density. Combined with the EMUM, the PD-DMUMallows the consideration of arbitrary boundary movements. In this work, we apply the PD-DMUM in two test cases. Within the first test case, we validate the PD-DMUM in a 2D Poiseuille flow on a moving background mesh. Subsequently the fluid-structure interaction (FSI) problem in the second test case serves as a proof of concept. More, we stress the advantages of the novel method with regard to conventional mesh update approaches. △ Less

Submitted 13 January, 2021; originally announced January 2021.

On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes

Authors: Tim G. J. Rudner, Oscar Key, Yarin Gal, Tom Rainforth

Abstract: We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. Specifically, we show both theoretically and via an extensive empirical evaluation that the SNR of the gradient estimates for the latent variable's variational parameters decreases as the number of importance sampl… ▽ More We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. Specifically, we show both theoretically and via an extensive empirical evaluation that the SNR of the gradient estimates for the latent variable's variational parameters decreases as the number of importance samples increases. As a result, these gradient estimates degrade to pure noise if the number of importance samples is too large. To address this pathology, we show how doubly reparameterized gradient estimators, originally proposed for training variational autoencoders, can be adapted to the DGP setting and that the resultant estimators completely remedy the SNR issue, thereby providing more reliable training. Finally, we demonstrate that our fix can lead to consistent improvements in the predictive performance of DGP models. △ Less

Submitted 21 July, 2021; v1 submitted 1 November, 2020; originally announced November 2020.

Comments: Published in Proceedings of the 38th International Conference on Machine Learning (ICML 2021)

Interlocking Backpropagation: Improving depthwise model-parallelism

Authors: Aidan N. Gomez, Oscar Key, Kuba Perlin, Stephen Gou, Nick Frosst, Jeff Dean, Yarin Gal

Abstract: The number of parameters in state of the art neural networks has drastically increased in recent years. This surge of interest in large scale neural networks has motivated the development of new distributed training strategies enabling such models. One such strategy is model-parallel distributed training. Unfortunately, model-parallelism can suffer from poor resource utilisation, which leads to wa… ▽ More The number of parameters in state of the art neural networks has drastically increased in recent years. This surge of interest in large scale neural networks has motivated the development of new distributed training strategies enabling such models. One such strategy is model-parallel distributed training. Unfortunately, model-parallelism can suffer from poor resource utilisation, which leads to wasted resources. In this work, we improve upon recent developments in an idealised model-parallel optimisation setting: local learning. Motivated by poor resource utilisation in the global setting and poor task performance in the local setting, we introduce a class of intermediary strategies between local and global learning referred to as interlocking backpropagation. These strategies preserve many of the compute-efficiency advantages of local optimisation, while recovering much of the task performance achieved by global optimisation. We assess our strategies on both image classification ResNets and Transformer language models, finding that our strategy consistently out-performs local learning in terms of task performance, and out-performs global learning in training efficiency. △ Less

Submitted 7 July, 2022; v1 submitted 8 October, 2020; originally announced October 2020.

SplineLib: A Modern Multi-Purpose C++ Spline Library

Authors: Markus Frings, Norbert Hosters, Corinna Müller, Max Spahn, Christoph Susen, Konstantin Key, Stefanie Elgeti

Abstract: This paper provides the description of a novel, multi-purpose spline library. In accordance with the increasingly diverse modes of usage of splines, it is multi-purpose in the sense that it supports geometry representation, finite element analysis, and optimization. The library features reading and writing for various file formats and a wide range of spline manipulation algorithms. Further, a new… ▽ More This paper provides the description of a novel, multi-purpose spline library. In accordance with the increasingly diverse modes of usage of splines, it is multi-purpose in the sense that it supports geometry representation, finite element analysis, and optimization. The library features reading and writing for various file formats and a wide range of spline manipulation algorithms. Further, a new efficient and objective-oriented algorithm for B-spline basis function evaluation is included. All features are available by a spline-type independent interface. The library is written in modern C++ with CMake as build system. This enables it for usage in typical scientific applications. It is provided as open-source library. △ Less

Submitted 27 February, 2020; originally announced February 2020.

Comments: 16 pages, 4 figures, submitted to Advances in Engineering Software

ACM Class: G.1.1; D.1.5; D.3.3; J.6

The Virtual Ring Shear-Slip Mesh Update Method

Authors: Fabian Key, Lutz Pauli, Stefanie Elgeti

Abstract: A novel method - the Virtual Ring Shear-Slip Mesh Update Method (VR-SSMUM) - for the efficient and accurate modeling of moving boundary or interface problems in the context of the numerical analysis of fluid flow is presented. We focus on cases with periodic straight-line translation including object entry and exit. The periodic character of the motion is reflected in the method via a mapping of t… ▽ More A novel method - the Virtual Ring Shear-Slip Mesh Update Method (VR-SSMUM) - for the efficient and accurate modeling of moving boundary or interface problems in the context of the numerical analysis of fluid flow is presented. We focus on cases with periodic straight-line translation including object entry and exit. The periodic character of the motion is reflected in the method via a mapping of the physical domain onto a closed virtual ring. Therefore, we use an extended mesh, where unneeded portions are deactivated to control the computational overhead. We provide a validation case as well as examples for the applicability of the method to 2D and 3D models of packaging machines. △ Less

Submitted 10 April, 2018; originally announced April 2018.

Automated software vulnerability detection with machine learning

Authors: Jacob A. Harer, Louis Y. Kim, Rebecca L. Russell, Onur Ozdemir, Leonard R. Kosta, Akshay Rangamani, Lei H. Hamilton, Gabriel I. Centeno, Jonathan R. Key, Paul M. Ellingwood, Erik Antelman, Alan Mackay, Marc W. McConley, Jeffrey M. Opper, Peter Chin, Tomo Lazovich

Abstract: Thousands of security vulnerabilities are discovered in production software each year, either reported publicly to the Common Vulnerabilities and Exposures database or discovered internally in proprietary code. Vulnerabilities often manifest themselves in subtle ways that are not obvious to code reviewers or the developers themselves. With the wealth of open source code available for analysis, the… ▽ More Thousands of security vulnerabilities are discovered in production software each year, either reported publicly to the Common Vulnerabilities and Exposures database or discovered internally in proprietary code. Vulnerabilities often manifest themselves in subtle ways that are not obvious to code reviewers or the developers themselves. With the wealth of open source code available for analysis, there is an opportunity to learn the patterns of bugs that can lead to security vulnerabilities directly from data. In this paper, we present a data-driven approach to vulnerability detection using machine learning, specifically applied to C and C++ programs. We first compile a large dataset of hundreds of thousands of open-source functions labeled with the outputs of a static analyzer. We then compare methods applied directly to source code with methods applied to artifacts extracted from the build process, finding that source-based models perform better. We also compare the application of deep neural network models with more traditional models such as random forests and find the best performance comes from combining features learned by deep models with tree-based models. Ultimately, our highest performing model achieves an area under the precision-recall curve of 0.49 and an area under the ROC curve of 0.87. △ Less

Submitted 2 August, 2018; v1 submitted 14 February, 2018; originally announced March 2018.

Simple Pricing Schemes for the Cloud

Authors: Ian A. Kash, Peter Key, Warut Suksompong

Abstract: The problem of pricing the cloud has attracted much recent attention due to the widespread use of cloud computing and cloud services. From a theoretical perspective, several mechanisms that provide strong efficiency or fairness guarantees and desirable incentive properties have been designed. However, these mechanisms often rely on a rigid model, with several parameters needing to be precisely kno… ▽ More The problem of pricing the cloud has attracted much recent attention due to the widespread use of cloud computing and cloud services. From a theoretical perspective, several mechanisms that provide strong efficiency or fairness guarantees and desirable incentive properties have been designed. However, these mechanisms often rely on a rigid model, with several parameters needing to be precisely known in order for the guarantees to hold. In this paper, we consider a stochastic model and show that it is possible to obtain good welfare and revenue guarantees with simple mechanisms that do not make use of the information on some of these parameters. In particular, we prove that a mechanism that sets the same price per time step for jobs of any length achieves at least 50% of the welfare and revenue obtained by a mechanism that can set different prices for jobs of different lengths, and the ratio can be improved if we have more specific knowledge of some parameters. Similarly, a mechanism that sets the same price for all servers even though the servers may receive different kinds of jobs can provide a reasonable welfare and revenue approximation compared to a mechanism that is allowed to set different prices for different servers. △ Less

Submitted 12 November, 2017; v1 submitted 23 May, 2017; originally announced May 2017.

Comments: To appear in the 13th Conference on Web and Internet Economics (WINE), 2017. A preliminary version was presented at the 12th Workshop on the Economics of Networks, Systems and Computation (NetEcon), 2017

Journal ref: ACM Transactions on Economics and Computation, 7(2):7 (2019)

The architecture of innovation: Tracking face-to-face interactions with ubicomp technologies

Authors: Chloë Brown, Christos Efstratiou, Ilias Leontiadis, Daniele Quercia, Cecilia Mascolo, James Scott, Peter Key

Abstract: The layouts of the buildings we live in shape our everyday lives. In office environments, building spaces affect employees' communication, which is crucial for productivity and innovation. However, accurate measurement of how spatial layouts affect interactions is a major challenge and traditional techniques may not give an objective view.We measure the impact of building spaces on social interact… ▽ More The layouts of the buildings we live in shape our everyday lives. In office environments, building spaces affect employees' communication, which is crucial for productivity and innovation. However, accurate measurement of how spatial layouts affect interactions is a major challenge and traditional techniques may not give an objective view.We measure the impact of building spaces on social interactions using wearable sensing devices. We study a single organization that moved between two different buildings, affording a unique opportunity to examine how space alone can affect interactions. The analysis is based on two large scale deployments of wireless sensing technologies: short-range, lightweight RFID tags capable of detecting face-to-face interactions. We analyze the traces to study the impact of the building change on social behavior, which represents a first example of using ubiquitous sensing technology to study how the physical design of two workplaces combines with organizational structure to shape contact patterns. △ Less

Submitted 26 June, 2014; originally announced June 2014.

Comments: 14 pages, 9 figures. To appear in ACM International Joint Conference on Pervasive and Ubiquitous Computing (Ubicomp 2014)

Optimising Trade-offs Among Stakeholders in Ad Auctions

Authors: Yoram Bachrach, Sofia Ceppi, Ian A. Kash, Peter Key, David Kurokawa

Abstract: We examine trade-offs among stakeholders in ad auctions. Our metrics are the revenue for the utility of the auctioneer, the number of clicks for the utility of the users and the welfare for the utility of the advertisers. We show how to optimize linear combinations of the stakeholder utilities, showing that these can be tackled through a GSP auction with a per-click reserve price. We then examine… ▽ More We examine trade-offs among stakeholders in ad auctions. Our metrics are the revenue for the utility of the auctioneer, the number of clicks for the utility of the users and the welfare for the utility of the advertisers. We show how to optimize linear combinations of the stakeholder utilities, showing that these can be tackled through a GSP auction with a per-click reserve price. We then examine constrained optimization of stakeholder utilities. We use simulations and analysis of real-world sponsored search auction data to demonstrate the feasible trade-offs, examining the effect of changing the allowed number of ads on the utilities of the stakeholders. We investigate both short term effects, when the players do not have the time to modify their behavior, and long term equilibrium conditions. Finally, we examine a combinatorially richer constrained optimization problem, where there are several possible allowed configurations (templates) of ad formats. This model captures richer ad formats, which allow using the available screen real estate in various ways. We show that two natural generalizations of the GSP auction rules to this domain are poorly behaved, resulting in not having a symmetric Nash equilibrium or having one with poor welfare. We also provide positive results for restricted cases. △ Less

Submitted 21 April, 2014; originally announced April 2014.

Comments: 18 pages, 10 figures, ACM Conference on Economics and Computation 2014

Efficient Advert Assignment

Authors: Frank Kelly, Peter Key, Neil Walton

Abstract: We develop a framework for the analysis of large-scale Ad-auctions where adverts are assigned over a continuum of search types. For this pay-per-click market, we provide an efficient mechanism that maximizes social welfare. In particular, we show that the social welfare optimization can be solved in separate optimizations conducted on the time-scales relevant to the search platform and advertisers… ▽ More We develop a framework for the analysis of large-scale Ad-auctions where adverts are assigned over a continuum of search types. For this pay-per-click market, we provide an efficient mechanism that maximizes social welfare. In particular, we show that the social welfare optimization can be solved in separate optimizations conducted on the time-scales relevant to the search platform and advertisers. Here, on each search occurrence, the platform solves an assignment problem and, on a slower time-scale, each advertiser submits a bid which matches its demand for click-throughs with supply. Importantly, knowledge of global parameters, such as the distribution of search terms, is not required when separating the problem in this way. Exploiting the information asymmetry between the platform and advertiser, we describe a simple mechanism which incentivizes truthful bidding and has a unique Nash equilibrium that is socially optimal, and thus implements our decomposition. Further, we consider models where advertisers adapt their bids smoothly over time, and prove convergence to the solution that maximizes social welfare. Finally, we describe several extensions which illustrate the flexibility and tractability of our framework. △ Less

Submitted 2 June, 2016; v1 submitted 10 April, 2014; originally announced April 2014.

Ranking and Tradeoffs in Sponsored Search Auctions

Authors: Ben Roberts, Dinan Gunawardena, Ian A. Kash, Peter Key

Abstract: In a sponsored search auction, decisions about how to rank ads impose tradeoffs between objectives such as revenue and welfare. In this paper, we examine how these tradeoffs should be made. We begin by arguing that the most natural solution concept to evaluate these tradeoffs is the lowest symmetric Nash equilibrium (SNE). As part of this argument, we generalise the well known connection between t… ▽ More In a sponsored search auction, decisions about how to rank ads impose tradeoffs between objectives such as revenue and welfare. In this paper, we examine how these tradeoffs should be made. We begin by arguing that the most natural solution concept to evaluate these tradeoffs is the lowest symmetric Nash equilibrium (SNE). As part of this argument, we generalise the well known connection between the lowest SNE and the VCG outcome. We then propose a new ranking algorithm, loosely based on the revenue-optimal auction, that uses a reserve price to order the ads (not just to filter them) and give conditions under which it raises more revenue than simply applying that reserve price. Finally, we conduct extensive simulations examining the tradeoffs enabled by different ranking algorithms and show that our proposed algorithm enables superior operating points by a variety of metrics. △ Less

Submitted 29 April, 2013; originally announced April 2013.

Comments: To appear in Proceedings of the 14th ACM Conference on Electronic Commerce (EC '13)

ACM Class: J.4

Budget Optimization for Sponsored Search: Censored Learning in MDPs

Authors: Kareem Amin, Michael Kearns, Peter Key, Anton Schwaighofer

Abstract: We consider the budget optimization problem faced by an advertiser participating in repeated sponsored search auctions, seeking to maximize the number of clicks attained under that budget. We cast the budget optimization problem as a Markov Decision Process (MDP) with censored observations, and propose a learning algorithm based on the wellknown Kaplan-Meier or product-limit estimator. We validate… ▽ More We consider the budget optimization problem faced by an advertiser participating in repeated sponsored search auctions, seeking to maximize the number of clicks attained under that budget. We cast the budget optimization problem as a Markov Decision Process (MDP) with censored observations, and propose a learning algorithm based on the wellknown Kaplan-Meier or product-limit estimator. We validate the performance of this algorithm by comparing it to several others on a large set of search auction data from Microsoft adCenter, demonstrating fast convergence to optimal performance. △ Less

Submitted 16 October, 2012; originally announced October 2012.

Comments: Appears in Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence (UAI2012)

Report number: UAI-P-2012-PG-54-63

Fixed and Market Pricing for Cloud Services

Authors: Vineet Abhishek, Ian A. Kash, Peter Key

Abstract: We study a model of congestible resources, where pricing and scheduling are intertwined. Motivated by the problem of pricing cloud instances, we model a cloud computing service as linked $GI/GI/\cdot$ queuing systems where the provider chooses to offer a fixed pricing service, a dynamic market based service, or a hybrid of both, where jobs can be preempted in the market-based service. Users (jobs)… ▽ More We study a model of congestible resources, where pricing and scheduling are intertwined. Motivated by the problem of pricing cloud instances, we model a cloud computing service as linked $GI/GI/\cdot$ queuing systems where the provider chooses to offer a fixed pricing service, a dynamic market based service, or a hybrid of both, where jobs can be preempted in the market-based service. Users (jobs), who are heterogeneous in both the value they place on service and their cost for waiting, then choose between the services offered. Combining insights from auction theory with queuing theory we are able to characterize user equilibrium behavior, and show its insensitivity to the precise market design mechanism used. We then provide theoretical and simulation based evidence suggesting that a fixed price typically, though not always, generates a higher expected revenue than the hybrid system for the provider. △ Less

Submitted 6 January, 2017; v1 submitted 26 January, 2012; originally announced January 2012.

Comments: 26 pages and 7 figures. An early version of this work appeared in the 7th Workshop on the Economics of Networks, Systems, and Computation (NetEcon) 2012