Showing 1–44 of 44 results for author: Seto, S

A Sharp Diameter-Dependent Lower Bound for the First Nonzero Neumann Eigenvalue of Geodesic Triangles in Space Forms

Authors: Shoo Seto, Guofang Wei, Yusen Xia

Abstract: We prove a sharp lower bound for the first nonzero Neumann eigenvalue of geodesic triangles of given diameter in two-dimensional space forms. The bound is given by the first positive radial Neumann eigenvalue of an one dimensional model; it is approached by degenerating isosceles triangles. When \(K>0\) and \(D=π/(2\sqrt K)\), equality is attained precisely by birectangular triangles. We also prov… ▽ More We prove a sharp lower bound for the first nonzero Neumann eigenvalue of geodesic triangles of given diameter in two-dimensional space forms. The bound is given by the first positive radial Neumann eigenvalue of an one dimensional model; it is approached by degenerating isosceles triangles. When \(K>0\) and \(D=π/(2\sqrt K)\), equality is attained precisely by birectangular triangles. We also prove a hot-spots theorem for non-acute spherical triangles of diameter at most \(π/2\), and establish antisymmetry and eigenvalue monotonicity for isosceles spherical triangles of diameter \(π/2\). △ Less

Submitted 17 August, 2026; originally announced August 2026.

MSC Class: 53c21 (Primary) 35P15; 58J50 (Secondary)

Normalized Rewards for Preference Optimization

Authors: Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, Katherine Metcalf

Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences. However, DAAs have been observed to over-optimize their implicit reward model and decrease the likelihood of preferred responses. This results in a decrease in the total likelihood assigned to responses seen in the preference dataset, potentially resulting in undesirable beh… ▽ More Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences. However, DAAs have been observed to over-optimize their implicit reward model and decrease the likelihood of preferred responses. This results in a decrease in the total likelihood assigned to responses seen in the preference dataset, potentially resulting in undesirable behavior. To counteract this undesired side-effect of DAAs, we examine the effect of using objectives that add a regularization term to maintain the total length-normalized probabilities of the chosen and rejected responses. To better understand over-optimization, we investigate how response likelihood changes are distributed over the tokens with and without regularization. We find that a significant portion of the likelihood changes are due to a small set of outlier tokens, which explains how DAAs improve generation quality despite decreasing the likelihoods of chosen responses. We apply the proposed regularization to reference-based (DPO) and reference-free (SimPO) methods and find (1) improved trade-offs between generation quality and general benchmark capability and (2) improvements in reward modeling across datasets. For example, on Llama-3.1-8B-Instruct, we see both a >20% relative increase in AlpacaEval2 scores and >9% relative performance gains on general benchmarks. Additionally, we find that the added regularization term effectively mitigates the amount of displacement within preferred responses overall, and for the outlier tokens specifically, by utilizing low-likelihood tokens. △ Less

Submitted 25 June, 2026; originally announced July 2026.

Comments: ICML 2026

Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions

Authors: Anastasiia Sedova, Natalie Schluter, Skyler Seto, Maartje ter Hoeve

Abstract: Cross-lingual knowledge transfer is critical for building high-performing multilingual language models for languages with insufficient training data. When target language data is scarce, the knowledge required for many downstream tasks involving scientific reasoning, commonsense inference, and world knowledge must be acquired primarily from the high-resource language, making effective knowledge tr… ▽ More Cross-lingual knowledge transfer is critical for building high-performing multilingual language models for languages with insufficient training data. When target language data is scarce, the knowledge required for many downstream tasks involving scientific reasoning, commonsense inference, and world knowledge must be acquired primarily from the high-resource language, making effective knowledge transfer essential. Existing methods for improving such cross-lingual knowledge transfer require large amounts of parallel data, translation systems, auxiliary models, or additional training stages that are largely unavailable for many languages. We propose LINK - a data-level intervention method that improves knowledge transfer during model pretraining through lexical substitutions in high-resource part of pretraining data using bilingual vocabularies. For a given replacement ratio, randomly selected words in a portion of the high-resource (English) training corpus are swapped with their word-level translations, requiring no additional model training and only a bilingual vocabulary, which can be obtained at near-zero cost for virtually any language. Evaluation on eight languages across five model sizes shows notable improvements on downstream tasks in the target language, with up to a 2x speedup in training to reach equivalent performance. △ Less

Submitted 22 May, 2026; originally announced May 2026.

Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Authors: Paul Jeha, Anastasiia Sedova, Louis Béthune, Skyler Seto, Jes Frellsen, Pierre Ablin, Natalie Schluter

Abstract: For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularize… ▽ More For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularizes the model by shrinking weights to restrict network capacity, auxiliary data mixing uses a tunable mixing ratio to expand the training distribution and diversify the training signal with new knowledge. Both offer a principled way to improve training in a data-constrained domain. We compare these levers systematically across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Three findings emerge. First, mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Second, we quantify how much mixing helps: it boosts performance by an amount equivalent to 2--3$\times$ the unique target data on validation loss and 2--13$\times$ on downstream task accuracy, with the gain scaling steeply with model size. Third, this divergence reveals that target-language validation loss systematically underestimates mixing's value. Mixing regularizes by diversifying the training signal and contributes knowledge the repeated target corpus cannot supply; validation loss captures only the first effect. Our practical recommendations are: mix in a high-resource language, prioritize the mixing ratio over hyperparameter tuning, and transfer hyperparameters from a small proxy model via $μ$P. △ Less

Submitted 13 May, 2026; originally announced May 2026.

Scaling Laws for Mixture Pretraining Under Data Constraints

Authors: Anastasiia Sedova, Skyler Seto, Natalie Schluter, Pierre Ablin

Abstract: As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain,… ▽ More As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain-specific, and quality-filtered mixtures. Across all settings, we find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 15-20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints. △ Less

Submitted 15 May, 2026; v1 submitted 12 May, 2026; originally announced May 2026.

Optimal Splitting of Language Models from Mixtures to Specialized Domains

Authors: Skyler Seto, Pierre Ablin, Anastasiia Filippova, Jiayuan Ye, Louis Bethune, Angelos Katharopoulos, David Grangier

Abstract: Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involv… ▽ More Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred to as split model training. We propose a method for pretraining multiple models independently over a general pretraining corpus, and determining the optimal compute allocation between pretraining and continued pretraining using scaling laws. Our approach accurately predicts the loss of a model of size N with D pretraining and D' specialization tokens, and extrapolates to larger model sizes and number of tokens. Applied to language model training, our approach improves performance consistently across common sense knowledge and reasoning benchmarks across different model sizes and compute budgets. △ Less

Submitted 19 March, 2026; originally announced March 2026.

Comments: 26 pages, 11 tables, 17 figures

Which Data Matter? Embedding-Based Data Selection for Speech Recognition

Authors: Zakaria Aldeneh, Skyler Seto, Maureen de Seyssel, Jie Chi, Zijin Gu, Takuya Higuchi, Jee-weon Jung, Shinji Watanabe, David Grangier, Barry-John Theobald, Tatiana Likhomanenko

Abstract: Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mism… ▽ More Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mismatch between training and test conditions. In this work, we study targeted data selection as a strategy to address these challenges, selecting relevant subsets from 100k hours of in-the-wild training data to optimize performance on target domains. We represent speech samples using embeddings that capture complementary characteristic--speaker attributes, phonetic content, and semantic meaning--and analyze how relevance and diversity along these axes when performing data selection affect downstream ASR performance. Our experiments with CTC-based Conformer models show that training on a strategically selected 5% subset can exceed the performance of models trained on the full dataset by up to 36.8% relative WER reduction on target domains. △ Less

Submitted 12 March, 2026; v1 submitted 5 March, 2026; originally announced March 2026.

Navigating the Sociotechnical Imaginaries of Brazilian Tech Workers

Authors: Kenzo Soares Seto

Abstract: This chapter examines the sociotechnical imaginaries of Brazilian tech workers, a group often overlooked in digital labor research despite their role in designing the digital systems that shape everyday life. Grounded in the idea of sociotechnical imaginaries as collectively constructed visions that guide technology development and governance, the chapter argues that looking from the Global South… ▽ More This chapter examines the sociotechnical imaginaries of Brazilian tech workers, a group often overlooked in digital labor research despite their role in designing the digital systems that shape everyday life. Grounded in the idea of sociotechnical imaginaries as collectively constructed visions that guide technology development and governance, the chapter argues that looking from the Global South helps challenge data universalism and foregrounds locally situated values, constraints, and futures. Drawing on semi-structured interviews with 26 Brazilian professionals conducted between July and December 2023, it maps how workers make sense of responsibility, bias, and power in AI and platform development. The findings highlight recurring tensions between academic and industry discourse on algorithmic bias, the limits of corporate accountability regarding user harm and surveillance, and the contested meanings of digital sovereignty, including grassroots initiatives that seek alternative technological futures aligned with marginalized communities needs. △ Less

Submitted 9 January, 2026; originally announced January 2026.

Journal ref: Ergin Bulut, Julie Chen, Rafael Grohmann, and Kylie Jarrett (eds.) (2025). SAGE Handbook of Digital Labour. SAGE: 314-324.9781529669831

Sharp Spectral Gap Estimates on Manifolds under Integral Ricci Curvature Bounds

Authors: Xavier Ramos Olivé, Shoo Seto, Malik Tuerkoen

Abstract: We prove sharp spectral gap estimates on compact manifolds with integral curvature bounds. We generalize the results of Kröger (Kröger '92) as well as of Bakry and Qian (Bakry-Qian '00) to the case of integral curvature and confirm the conjecture in (Ramos et al. '20). We prove sharp spectral gap estimates on compact manifolds with integral curvature bounds. We generalize the results of Kröger (Kröger '92) as well as of Bakry and Qian (Bakry-Qian '00) to the case of integral curvature and confirm the conjecture in (Ramos et al. '20). △ Less

Submitted 26 May, 2026; v1 submitted 30 October, 2025; originally announced October 2025.

Comments: 15 pages

MSC Class: 58C40; 35P15

Closing the Gap Between Text and Speech Understanding in LLMs

Authors: Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, Zakaria Aldeneh

Abstract: Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to… ▽ More Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on this analysis, we introduce SALAD--Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation--which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with a strong open-weight model across broad-domain benchmarks in knowledge, language understanding, and reasoning, while training on over an order of magnitude less speech data from public corpora. △ Less

Submitted 23 February, 2026; v1 submitted 15 October, 2025; originally announced October 2025.

Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

Authors: Eileen Pan, Anna Seo Gyeong Choi, Maartje ter Hoeve, Skyler Seto, Allison Koenecke

Abstract: Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additio… ▽ More Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures. △ Less

Submitted 1 October, 2025; originally announced October 2025.

Comments: EMNLP Findings 2025, 12 pages, 11 tables, 3 figures

Assessing the Role of Data Quality in Training Bilingual Language Models

Authors: Skyler Seto, Maartje ter Hoeve, Maureen de Seyssel, David Grangier

Abstract: Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate… ▽ More Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate causes of these inconsistencies by comparing bilingual and monolingual language models. Our analysis reveals that unequal data quality, not just data quantity, is a major driver of performance degradation in bilingual settings. We propose a simple yet effective data filtering strategy to select higher-quality bilingual training data with only high quality English data. Applied to French, German, and Chinese, our approach improves monolingual performance by 2-4% and reduces bilingual model performance gaps to 1%. These results highlight the overlooked importance of data quality in multilingual pretraining and offer a practical recipe for balancing performance. △ Less

Submitted 15 June, 2025; originally announced June 2025.

Comments: 26 pages, 18 figures, 25 tables

Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting

Authors: Chen Huang, Skyler Seto, Hadi Pouransari, Mehrdad Farajtabar, Raviteja Vemulapalli, Fartash Faghri, Oncel Tuzel, Barry-John Theobald, Josh Susskind

Abstract: Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuning foundation models on one task often leads to the issue of concept forgetting on other tasks. Recent methods of robust fine-tuning aim to mitigate forgetting of prior knowledge without affecting the fine-tuning perform… ▽ More Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuning foundation models on one task often leads to the issue of concept forgetting on other tasks. Recent methods of robust fine-tuning aim to mitigate forgetting of prior knowledge without affecting the fine-tuning performance. Knowledge is often preserved by matching the original and fine-tuned model weights or feature pairs. However, such point-wise matching can be too strong, without explicit awareness of the feature neighborhood structures that encode rich knowledge as well. We propose a novel regularization method Proxy-FDA that explicitly preserves the structural knowledge in feature space. Proxy-FDA performs Feature Distribution Alignment (using nearest neighbor graphs) between the pre-trained and fine-tuned feature spaces, and the alignment is further improved by informative proxies that are generated dynamically to increase data diversity. Experiments show that Proxy-FDA significantly reduces concept forgetting during fine-tuning, and we find a strong correlation between forgetting and a distributional distance metric (in comparison to L2 distance). We further demonstrate Proxy-FDA's benefits in various fine-tuning settings (end-to-end, few-shot and continual tuning) and across different tasks like image classification, captioning and VQA. △ Less

Submitted 29 May, 2025; originally announced May 2025.

Comments: ICML 2025

Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks

Authors: Maureen de Seyssel, Jie Chi, Skyler Seto, Maartje ter Hoeve, Masha Fedzechkina, Natalie Schluter

Abstract: We introduce a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning). Inspired from speech processing, these zero-shot tasks measure whether minimal differences in representation can be reliably detected. This offers a flexible and interpretable alternative to probing. Applied to XLM-R (Con… ▽ More We introduce a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning). Inspired from speech processing, these zero-shot tasks measure whether minimal differences in representation can be reliably detected. This offers a flexible and interpretable alternative to probing. Applied to XLM-R (Conneau et al, 2020) across pretraining checkpoints and layers, we find that language discrimination declines over training and becomes concentrated in lower layers, while meaning discrimination strengthens over time and stabilizes in deeper layers. We then explore probing tasks, showing some alignment between our metrics and linguistic learning performance. Our results position ABX tasks as a lightweight framework for analyzing the structure of multilingual representations. △ Less

Submitted 19 November, 2025; v1 submitted 23 May, 2025; originally announced May 2025.

Comments: Comments: Published in EMNLP 2025. https://aclanthology.org/2025.emnlp-main.1210.pdf

Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models

Authors: Anirudh Sundar, Sinead Williamson, Katherine Metcalf, Barry-John Theobald, Skyler Seto, Masha Fedzechkina

Abstract: Aligned representations across languages is a desired property in multilingual large language models (mLLMs), as alignment can improve performance in cross-lingual tasks. Typically alignment requires fine-tuning a model, which is computationally expensive, and sizable language data, which often may not be available. A data-efficient alternative to fine-tuning is model interventions -- a method for… ▽ More Aligned representations across languages is a desired property in multilingual large language models (mLLMs), as alignment can improve performance in cross-lingual tasks. Typically alignment requires fine-tuning a model, which is computationally expensive, and sizable language data, which often may not be available. A data-efficient alternative to fine-tuning is model interventions -- a method for manipulating model activations to steer generation into the desired direction. We analyze the effect of a popular intervention (finding experts) on the alignment of cross-lingual representations in mLLMs. We identify the neurons to manipulate for a given language and introspect the embedding space of mLLMs pre- and post-manipulation. We show that modifying the mLLM's activations changes its embedding space such that cross-lingual alignment is enhanced. Further, we show that the changes to the embedding space translate into improved downstream performance on retrieval tasks, with up to 2x improvements in top-1 accuracy on cross-lingual retrieval. △ Less

Submitted 21 July, 2025; v1 submitted 21 February, 2025; originally announced February 2025.

Comments: 34 pages

ExpertLens: Activation steering features are highly interpretable

Authors: Masha Fedzechkina, Eleonora Gualdoni, Sinead Williamson, Katherine Metcalf, Skyler Seto, Barry-John Theobald

Abstract: Activation steering methods in large language models (LLMs) have emerged as an effective way to perform targeted updates to enhance generated language without requiring large amounts of adaptation data. We ask whether the features discovered by activation steering methods are interpretable. We identify neurons responsible for specific concepts (e.g., ``cat'') using the ``finding experts'' method f… ▽ More Activation steering methods in large language models (LLMs) have emerged as an effective way to perform targeted updates to enhance generated language without requiring large amounts of adaptation data. We ask whether the features discovered by activation steering methods are interpretable. We identify neurons responsible for specific concepts (e.g., ``cat'') using the ``finding experts'' method from research on activation steering and show that the ExpertLens, i.e., inspection of these neurons provides insights about model representation. We find that ExpertLens representations are stable across models and datasets and closely align with human representations inferred from behavioral data, matching inter-human alignment levels. ExpertLens significantly outperforms the alignment captured by word/sentence embeddings. By reconstructing human concept organization through ExpertLens, we show that it enables a granular view of LLM concept representation. Our findings suggest that ExpertLens is a flexible and lightweight approach for capturing and analyzing model representations. △ Less

Submitted 3 November, 2025; v1 submitted 20 February, 2025; originally announced February 2025.

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

Authors: Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier

Abstract: Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters,… ▽ More Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear combination coefficients as a function of the input domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a model size constraint. △ Less

Submitted 3 February, 2025; originally announced February 2025.

Training Bilingual LMs with Data Constraints in the Targeted Language

Authors: Skyler Seto, Maartje ter Hoeve, Richard He Bai, Natalie Schluter, David Grangier

Abstract: Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretraining data. For most other languages, however, such high quality pretraining data is unavailable. In this work, we study how to boost pretrained model performance in a target language with insufficient pretraining data fo… ▽ More Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretraining data. For most other languages, however, such high quality pretraining data is unavailable. In this work, we study how to boost pretrained model performance in a target language with insufficient pretraining data for training a high performing language model, by enlisting data from an auxiliary language for which high quality data is available. We study this by quantifying the performance gap between training with data in a data-rich auxiliary language compared with training in the target language, exploring the benefits of translation systems, studying the limitations of model scaling when data is limited in the target languages, and proposing new methods for upsampling data from the auxiliary language. Our results show that stronger auxiliary datasets result in performance gains without modification to the model or training objective for close languages, and, in particular, that performance gains due to the development of more information-rich English pretraining datasets can extend to targeted language settings with limited data. △ Less

Submitted 5 February, 2025; v1 submitted 19 November, 2024; originally announced November 2024.

Comments: 26 pages, 22 figures, 15 tables

Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP

Authors: Chen Huang, Skyler Seto, Samira Abnar, David Grangier, Navdeep Jaitly, Josh Susskind

Abstract: Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are unseen or under-represented during pretraining. Prompt learning offers a parameter-efficient finetuning framework that can adapt CLIP to downstream tasks ev… ▽ More Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are unseen or under-represented during pretraining. Prompt learning offers a parameter-efficient finetuning framework that can adapt CLIP to downstream tasks even when limited annotation data are available. In this paper, we improve prompt learning by distilling the textual knowledge from natural language prompts (either human- or LLM-generated) to provide rich priors for those under-represented concepts. We first obtain a prompt ``summary'' aligned to each input image via a learned prompt aggregator. Then we jointly train a prompt generator, optimized to produce a prompt embedding that stays close to the aggregated summary while minimizing task loss at the same time. We dub such prompt embedding as Aggregate-and-Adapted Prompt Embedding (AAPE). AAPE is shown to be able to generalize to different downstream data distributions and tasks, including vision-language understanding tasks (e.g., few-shot classification, VQA) and generation tasks (image captioning) where AAPE achieves competitive performance. We also show AAPE is particularly helpful to handle non-canonical and OOD examples. Furthermore, AAPE learning eliminates LLM-based inference cost as required by baselines, and scales better with data and LLM model size. △ Less

Submitted 31 October, 2024; originally announced October 2024.

Comments: NeurIPS 2024

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling

Authors: David Grangier, Simin Fan, Skyler Seto, Pierre Ablin

Abstract: Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most tasks. In this work, we build specialist models from large generalist training sets instead. We propose a novel method, ClusteRed Importance SamPling (CRISP). CRI… ▽ More Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most tasks. In this work, we build specialist models from large generalist training sets instead. We propose a novel method, ClusteRed Importance SamPling (CRISP). CRISP clusters the generalist dataset and samples from these clusters based on their frequencies in the smaller specialist dataset. It is scalable, suitable for both pretraining and continued pretraining, and works well in multi-task settings. CRISP performs favorably compared to other methods that adjust the training distribution of the generalist data with guidance from the limited domain-specific data. Our findings demonstrate improvements across different domains in terms of language modeling perplexity and accuracy on multiple-choice question tasks. We also present ablation studies that examine the impact of dataset sizes, clustering configurations, and model sizes. △ Less

Submitted 10 March, 2025; v1 submitted 30 September, 2024; originally announced October 2024.

Comments: 23 pages, presented at the International Conference on Learning Representation (ICLR), 2025

Learning Spatially-Aware Language and Audio Embeddings

Authors: Bhavika Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, Elena Menyaylenko, Barry-John Theobald, Jonathan Sheaffer, Miguel Sarabia

Abstract: Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension, the machine must know what a lion is (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of ling… ▽ More Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension, the machine must know what a lion is (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of linguistic information align with the semantic and spatial attributes of the sound (what a roar sounds like when its coming from behind). State-of-the-art audio foundation models which learn to map between audio scenes and natural textual descriptions, are trained on non-spatial audio and text pairs, and hence lack spatial awareness. In contrast, sound event localization and detection models are limited to recognizing sounds from a fixed number of classes, and they localize the source to absolute position (e.g., 0.2m) rather than a position described using natural language (e.g., "next to me"). To address these gaps, we present ELSA a spatially aware-audio and text embedding model trained using multimodal contrastive learning. ELSA supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound. To train ELSA: (a) we spatially augment the audio and captions of three open-source audio datasets totaling 4,738 hours of audio, and (b) we design an encoder to capture the semantics of non-spatial audio, and the semantics and spatial attributes of spatial audio using contrastive learning. ELSA is competitive with state-of-the-art for both semantic retrieval and 3D source localization. In particular, ELSA achieves +2.8% mean audio-to-text and text-to-audio R@1 above the baseline, and outperforms by -11.6° mean-absolute-error in 3D source localization over the baseline. △ Less

Submitted 26 November, 2024; v1 submitted 17 September, 2024; originally announced September 2024.

Comments: 26 pages, 7 figures, accepted at NeurIPS 2024

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Authors: Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, Tong Zhang

Abstract: Reinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1) training an EXplicit Reward Model (EXRM) as in RLHF, and 2) using an implicit reward learned from preference data through methods such as Dire… ▽ More Reinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1) training an EXplicit Reward Model (EXRM) as in RLHF, and 2) using an implicit reward learned from preference data through methods such as Direct Preference Optimization (DPO). Prior work has shown that the implicit reward model of DPO (denoted as DPORM) can approximate an EXRM in the limit. DPORM's effectiveness directly implies the optimality of the learned policy, and also has practical implication for LLM alignment methods including iterative DPO. However, it is unclear how well DPORM empirically matches the performance of EXRM. This work studies the accuracy at distinguishing preferred and rejected answers for both DPORM and EXRM. Our findings indicate that even though DPORM fits the training dataset comparably, it generalizes less effectively than EXRM, especially when the validation datasets contain distribution shifts. Across five out-of-distribution settings, DPORM has a mean drop in accuracy of 3% and a maximum drop of 7%. These findings highlight that DPORM has limited generalization ability and substantiates the integration of an explicit reward model in iterative DPO approaches. △ Less

Submitted 3 October, 2024; v1 submitted 5 September, 2024; originally announced September 2024.

Comments: 12 pages, 8 tables, 3 figures; Paper Accepted at EMNLP Findings 2024

Can you Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?

Authors: Zakaria Aldeneh, Takuya Higuchi, Jee-weon Jung, Skyler Seto, Tatiana Likhomanenko, Stephen Shum, Ahmed Hussen Abdelaziz, Shinji Watanabe, Barry-John Theobald

Abstract: Self-supervised features are typically used in place of filter-bank features in speaker verification models. However, these models were originally designed to ingest filter-bank features as inputs, and thus, training them on top of self-supervised features assumes that both feature types require the same amount of learning for the task. In this work, we observe that pre-trained self-supervised spe… ▽ More Self-supervised features are typically used in place of filter-bank features in speaker verification models. However, these models were originally designed to ingest filter-bank features as inputs, and thus, training them on top of self-supervised features assumes that both feature types require the same amount of learning for the task. In this work, we observe that pre-trained self-supervised speech features inherently include information required for downstream speaker verification task, and therefore, we can simplify the downstream model without sacrificing performance. To this end, we revisit the design of the downstream model for speaker verification using self-supervised features. We show that we can simplify the model to use 97.51% fewer parameters while achieving a 29.93% average improvement in performance on SUPERB. Consequently, we show that the simplified downstream model is more data efficient compared to baseline--it achieves better performance with only 60% of the training data. △ Less

Submitted 13 June, 2024; v1 submitted 1 February, 2024; originally announced February 2024.

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Authors: Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly

Abstract: Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending s… ▽ More Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training ($\textbf{WRAP}$) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by $\sim3x$. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data. △ Less

Submitted 29 January, 2024; originally announced January 2024.

REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptation

Authors: Skyler Seto, Barry-John Theobald, Federico Danieli, Navdeep Jaitly, Dan Busbridge

Abstract: Fully-test-time adaptation (F-TTA) can mitigate performance loss due to distribution shifts between train and test data (1) without access to the training data, and (2) without knowledge of the model training procedure. In online F-TTA, a pre-trained model is adapted using a stream of test samples by minimizing a self-supervised objective, such as entropy minimization. However, models adapted with… ▽ More Fully-test-time adaptation (F-TTA) can mitigate performance loss due to distribution shifts between train and test data (1) without access to the training data, and (2) without knowledge of the model training procedure. In online F-TTA, a pre-trained model is adapted using a stream of test samples by minimizing a self-supervised objective, such as entropy minimization. However, models adapted with online using entropy minimization, are unstable especially in single sample settings, leading to degenerate solutions, and limiting the adoption of TTA inference strategies. Prior works identify noisy, or unreliable, samples as a cause of failure in online F-TTA. One solution is to ignore these samples, which can lead to bias in the update procedure, slow adaptation, and poor generalization. In this work, we present a general framework for improving robustness of F-TTA to these noisy samples, inspired by self-paced learning and robust loss functions. Our proposed approach, Robust Entropy Adaptive Loss Minimization (REALM), achieves better adaptation accuracy than previous approaches throughout the adaptation process on corruptions of CIFAR-10 and ImageNet-1K, demonstrating its effectiveness. △ Less

Submitted 7 September, 2023; originally announced September 2023.

Comments: Accepted at WACV 2024, 17 pages, 7 figures, 11 tables

Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning

Authors: Miguel Sarabia, Elena Menyaylenko, Alessandro Toso, Skyler Seto, Zakaria Aldeneh, Shadi Pirhosseinloo, Luca Zappella, Barry-John Theobald, Nicholas Apostoloff, Jonathan Sheaffer

Abstract: We present Spatial LibriSpeech, a spatial audio dataset with over 650 hours of 19-channel audio, first-order ambisonics, and optional distractor noise. Spatial LibriSpeech is designed for machine learning model training, and it includes labels for source position, speaking direction, room acoustics and geometry. Spatial LibriSpeech is generated by augmenting LibriSpeech samples with 200k+ simulate… ▽ More We present Spatial LibriSpeech, a spatial audio dataset with over 650 hours of 19-channel audio, first-order ambisonics, and optional distractor noise. Spatial LibriSpeech is designed for machine learning model training, and it includes labels for source position, speaking direction, room acoustics and geometry. Spatial LibriSpeech is generated by augmenting LibriSpeech samples with 200k+ simulated acoustic conditions across 8k+ synthetic rooms. To demonstrate the utility of our dataset, we train models on four spatial audio tasks, resulting in a median absolute error of 6.60° on 3D source localization, 0.43m on distance, 90.66ms on T30, and 2.74dB on DRR estimation. We show that the same models generalize well to widely-used evaluation datasets, e.g., obtaining a median absolute error of 12.43° on 3D source localization on TUT Sound Events 2018, and 157.32ms on T30 estimation on ACE Challenge. △ Less

Submitted 18 August, 2023; originally announced August 2023.

Journal ref: Proceedings of INTERSPEECH (2023), pp. 3724-3728

Understanding the Robustness of Multi-Exit Models under Common Corruptions

Authors: Akshay Mehra, Skyler Seto, Navdeep Jaitly, Barry-John Theobald

Abstract: Multi-Exit models (MEMs) use an early-exit strategy to improve the accuracy and efficiency of deep neural networks (DNNs) by allowing samples to exit the network before the last layer. However, the effectiveness of MEMs in the presence of distribution shifts remains largely unexplored. Our work examines how distribution shifts generated by common image corruptions affect the accuracy/efficiency of… ▽ More Multi-Exit models (MEMs) use an early-exit strategy to improve the accuracy and efficiency of deep neural networks (DNNs) by allowing samples to exit the network before the last layer. However, the effectiveness of MEMs in the presence of distribution shifts remains largely unexplored. Our work examines how distribution shifts generated by common image corruptions affect the accuracy/efficiency of MEMs. We find that under common corruptions, early-exiting at the first correct exit reduces the inference cost and provides a significant boost in accuracy ( 10%) over exiting at the last layer. However, with realistic early-exit strategies, which do not assume knowledge about the correct exits, MEMs still reduce inference cost but provide a marginal improvement in accuracy (1%) compared to exiting at the last layer. Moreover, the presence of distribution shift widens the gap between an MEM's maximum classification accuracy and realistic early-exit strategies by 5% on average compared with the gap on in-distribution data. Our empirical analysis shows that the lack of calibration due to a distribution shift increases the susceptibility of such early-exit strategies to exit early and increases misclassification rates. Furthermore, the lack of calibration increases the inconsistency in the predictions of the model across exits, leading to both inefficient inference and more misclassifications compared with evaluation on in-distribution data. Finally, we propose two metrics, underthinking and overthinking, that quantify the different behavior of practical early-exit strategy under distribution shifts, and provide insights into improving the practical utility of MEMs. △ Less

Submitted 3 December, 2022; originally announced December 2022.

Comments: 16 pages, 22 figures

Explicit lower bound of the first eigenvalue of the Laplacian on Kähler manifolds

Authors: Benjamin Rutkowski, Shoo Seto

Abstract: We establish an explicit lower bound of the first eigenvalue of the Laplacian on Kähler manifolds based off the comparison results of Li and Wang. The lower bound will depend on the diameter, dimension, holomorphic sectional curvature and orthogonal Ricci curvature. We establish an explicit lower bound of the first eigenvalue of the Laplacian on Kähler manifolds based off the comparison results of Li and Wang. The lower bound will depend on the diameter, dimension, holomorphic sectional curvature and orthogonal Ricci curvature. △ Less

Submitted 22 July, 2022; originally announced July 2022.

Comments: Accepted for publication to Involve, a Journal of Mathematics

World Robot Challenge 2020 -- Partner Robot: A Data-Driven Approach for Room Tidying with Mobile Manipulator

Authors: Tatsuya Matsushima, Yuki Noguchi, Jumpei Arima, Toshiki Aoki, Yuki Okita, Yuya Ikeda, Koki Ishimoto, Shohei Taniguchi, Yuki Yamashita, Shoichi Seto, Shixiang Shane Gu, Yusuke Iwasawa, Yutaka Matsuo

Abstract: Tidying up a household environment using a mobile manipulator poses various challenges in robotics, such as adaptation to large real-world environmental variations, and safe and robust deployment in the presence of humans.The Partner Robot Challenge in World Robot Challenge (WRC) 2020, a global competition held in September 2021, benchmarked tidying tasks in the real home environments, and importa… ▽ More Tidying up a household environment using a mobile manipulator poses various challenges in robotics, such as adaptation to large real-world environmental variations, and safe and robust deployment in the presence of humans.The Partner Robot Challenge in World Robot Challenge (WRC) 2020, a global competition held in September 2021, benchmarked tidying tasks in the real home environments, and importantly, tested for full system performances.For this challenge, we developed an entire household service robot system, which leverages a data-driven approach to adapt to numerous edge cases that occur during the execution, instead of classical manual pre-programmed solutions. In this paper, we describe the core ingredients of the proposed robot system, including visual recognition, object manipulation, and motion planning. Our robot system won the second prize, verifying the effectiveness and potential of data-driven robot systems for mobile manipulation in home environments. △ Less

Submitted 21 July, 2022; v1 submitted 20 July, 2022; originally announced July 2022.

On the role of Lip Articulation in Visual Speech Perception

Authors: Zakaria Aldeneh, Masha Fedzechkina, Skyler Seto, Katherine Metcalf, Miguel Sarabia, Nicholas Apostoloff, Barry-John Theobald

Abstract: Generating realistic lip motion from audio to simulate speech production is critical for driving natural character animation. Previous research has shown that traditional metrics used to optimize and assess models for generating lip motion from speech are not a good indicator of subjective opinion of animation quality. Devising metrics that align with subjective opinion first requires understandin… ▽ More Generating realistic lip motion from audio to simulate speech production is critical for driving natural character animation. Previous research has shown that traditional metrics used to optimize and assess models for generating lip motion from speech are not a good indicator of subjective opinion of animation quality. Devising metrics that align with subjective opinion first requires understanding what impacts human perception of quality. In this work, we focus on the degree of articulation and run a series of experiments to study how articulation strength impacts human perception of lip motion accompanying speech. Specifically, we study how increasing under-articulated (dampened) and over-articulated (exaggerated) lip motion affects human perception of quality. We examine the impact of articulation strength on human perception when considering only lip motion, where viewers are presented with talking faces represented by landmarks, and in the context of embodied characters, where viewers are presented with photo-realistic videos. Our results show that viewers prefer over-articulated lip motion consistently more than under-articulated lip motion and that this preference generalizes across different speakers and embodiments. △ Less

Submitted 10 November, 2022; v1 submitted 18 March, 2022; originally announced March 2022.

Comments: Submitted to ICASSP 2023

FORML: Learning to Reweight Data for Fairness

Authors: Bobby Yan, Skyler Seto, Nicholas Apostoloff

Abstract: Machine learning models are trained to minimize the mean loss for a single metric, and thus typically do not consider fairness and robustness. Neglecting such metrics in training can make these models prone to fairness violations when training data are imbalanced or test distributions differ. This work introduces Fairness Optimized Reweighting via Meta-Learning (FORML), a training algorithm that b… ▽ More Machine learning models are trained to minimize the mean loss for a single metric, and thus typically do not consider fairness and robustness. Neglecting such metrics in training can make these models prone to fairness violations when training data are imbalanced or test distributions differ. This work introduces Fairness Optimized Reweighting via Meta-Learning (FORML), a training algorithm that balances fairness and robustness with accuracy by jointly learning training sample weights and neural network parameters. The approach increases model fairness by learning to balance the contributions from both over- and under-represented sub-groups through dynamic reweighting of the data learned from a user-specified held-out set representative of the distribution under which fairness is desired. FORML improves equality of opportunity fairness criteria on image classification tasks, reduces bias of corrupted labels, and facilitates building more fair datasets via data condensation. These improvements are achieved without pre-processing data or post-processing model outputs, without learning an additional weighting function, without changing model architecture, and while maintaining accuracy on the original predictive metric. △ Less

Submitted 19 July, 2022; v1 submitted 3 February, 2022; originally announced February 2022.

Comments: 9 pages, 2 figures, Presented at ICML 2022 DataPerf Workshop

Fundamental gaps of spherical triangles

Authors: Shoo Seto, Guofang Wei, Xuwen Zhu

Abstract: We compute Dirichlet eigenvalues and eigenfunctions explicitly for spherical lunes and the spherical triangles which are half the lunes, and show that the fundamental gap goes to infinity when the angle of the lune goes to zero. Then we show the spherical equilateral triangle of diameter $\fracπ{2}$ is a strict local minimizer of the fundamental gap on the space of the spherical triangles with dia… ▽ More We compute Dirichlet eigenvalues and eigenfunctions explicitly for spherical lunes and the spherical triangles which are half the lunes, and show that the fundamental gap goes to infinity when the angle of the lune goes to zero. Then we show the spherical equilateral triangle of diameter $\fracπ{2}$ is a strict local minimizer of the fundamental gap on the space of the spherical triangles with diameter $\fracπ{2}$, which partially extends Lu-Rowlett's result from the plane to the sphere. △ Less

Submitted 1 September, 2020; originally announced September 2020.

Comments: 18 pages, 3 figures, all comments welcome

HALO: Learning to Prune Neural Networks with Shrinkage

Authors: Skyler Seto, Martin T. Wells, Wenyu Zhang

Abstract: Deep neural networks achieve state-of-the-art performance in a variety of tasks by extracting a rich set of features from unstructured data, however this performance is closely tied to model size. Modern techniques for inducing sparsity and reducing model size are (1) network pruning, (2) training with a sparsity inducing penalty, and (3) training a binary mask jointly with the weights of the netw… ▽ More Deep neural networks achieve state-of-the-art performance in a variety of tasks by extracting a rich set of features from unstructured data, however this performance is closely tied to model size. Modern techniques for inducing sparsity and reducing model size are (1) network pruning, (2) training with a sparsity inducing penalty, and (3) training a binary mask jointly with the weights of the network. We study different sparsity inducing penalties from the perspective of Bayesian hierarchical models and present a novel penalty called Hierarchical Adaptive Lasso (HALO) which learns to adaptively sparsify weights of a given network via trainable parameters. When used to train over-parametrized networks, our penalty yields small subnetworks with high accuracy without fine-tuning. Empirically, on image recognition tasks, we find that HALO is able to learn highly sparse network (only 5% of the parameters) with significant gains in performance over state-of-the-art magnitude pruning methods at the same level of sparsity. Code is available at https://github.com/skyler120/sparsity-halo. △ Less

Submitted 27 February, 2021; v1 submitted 24 August, 2020; originally announced August 2020.

Comments: Accepted at SDM 2021

CAZSL: Zero-Shot Regression for Pushing Models by Generalizing Through Context

Authors: Wenyu Zhang, Skyler Seto, Devesh K. Jha

Abstract: Learning accurate models of the physical world is required for a lot of robotic manipulation tasks. However, during manipulation, robots are expected to interact with unknown workpieces so that building predictive models which can generalize over a number of these objects is highly desirable. In this paper, we study the problem of designing deep learning agents which can generalize their models of… ▽ More Learning accurate models of the physical world is required for a lot of robotic manipulation tasks. However, during manipulation, robots are expected to interact with unknown workpieces so that building predictive models which can generalize over a number of these objects is highly desirable. In this paper, we study the problem of designing deep learning agents which can generalize their models of the physical world by building context-aware learning models. The purpose of these agents is to quickly adapt and/or generalize their notion of physics of interaction in the real world based on certain features about the interacting objects that provide different contexts to the predictive models. With this motivation, we present context-aware zero shot learning (CAZSL, pronounced as casual) models, an approach utilizing a Siamese network architecture, embedding space masking and regularization based on context variables which allows us to learn a model that can generalize to different parameters or features of the interacting objects. We test our proposed learning algorithm on the recently released Omnipush datatset that allows testing of meta-learning capabilities using low-dimensional data. Codes for CAZSL are available at https://www.merl.com/research/license/CAZSL. △ Less

Submitted 1 November, 2020; v1 submitted 25 March, 2020; originally announced March 2020.

Comments: Accepted at IROS 2020

The first nonzero eigenvalue of the $p$-Laplacian on Differential forms

Authors: Shoo Seto

Abstract: we introduce a generalization of the $p$-Laplace operator to act on differential forms and generalize an estimate of Gallot-Meyer for the first nonzero eigenvalue on closed Riemannian manifolds. we introduce a generalization of the $p$-Laplace operator to act on differential forms and generalize an estimate of Gallot-Meyer for the first nonzero eigenvalue on closed Riemannian manifolds. △ Less

Submitted 24 March, 2019; v1 submitted 14 March, 2019; originally announced March 2019.

Comments: comments welcome

Journal ref: Pacific J. Math. 309 (2020) 213-222

Zhong-Yang type eigenvalue estimate with integral curvature condition

Authors: Xavier Ramos Olivé, Shoo Seto, Guofang Wei, Qi S. Zhang

Abstract: We prove a sharp Zhong-Yang type eigenvalue lower bound for closed Riemannian manifolds with control on integral Ricci curvature. We prove a sharp Zhong-Yang type eigenvalue lower bound for closed Riemannian manifolds with control on integral Ricci curvature. △ Less

Submitted 24 March, 2019; v1 submitted 3 December, 2018; originally announced December 2018.

Comments: 18 pages; comments are welcome, typo fixed

First eigenvalue of the $p$-Laplacian on Kähler manifolds

Authors: Casey Blacker, Shoo Seto

Abstract: We prove a Lichnerowicz type lower bound for the first nontrivial eigenvalue of the $p$-Laplacian on Kähler manifolds. Parallel to the $p = 2$ case, the first eigenvalue lower bound is improved by using a decomposition of the Hessian on Kähler manifolds with positive Ricci curvature. We prove a Lichnerowicz type lower bound for the first nontrivial eigenvalue of the $p$-Laplacian on Kähler manifolds. Parallel to the $p = 2$ case, the first eigenvalue lower bound is improved by using a decomposition of the Hessian on Kähler manifolds with positive Ricci curvature. △ Less

Submitted 10 September, 2018; v1 submitted 29 April, 2018; originally announced April 2018.

Comments: comments welcome

Fundamental gap estimate for convex domains on sphere -- the case $n=2$

Authors: Xianzhe Dai, Shoo Seto, Guofang Wei

Abstract: In [SWW16, HW17] it is shown that the difference of the first two eigenvalues of the Laplacian with Dirichlet boundary condition on convex domain with diameter $D$ of sphere $\mathbb S^n$ is $\geq 3 \frac{π^2}{D^2}$ when $n \geq 3$. We prove the same result when $n=2$. In fact our proof works for all dimension. We also give an asymptotic expansion of the first and second Dirichlet eigenvalues of t… ▽ More In [SWW16, HW17] it is shown that the difference of the first two eigenvalues of the Laplacian with Dirichlet boundary condition on convex domain with diameter $D$ of sphere $\mathbb S^n$ is $\geq 3 \frac{π^2}{D^2}$ when $n \geq 3$. We prove the same result when $n=2$. In fact our proof works for all dimension. We also give an asymptotic expansion of the first and second Dirichlet eigenvalues of the model in [SWW16]. △ Less

Submitted 3 March, 2018; originally announced March 2018.

Comments: 19 pages

MSC Class: 35P15; 53C35

A Double Parametric Bootstrap Test for Topic Models

Authors: Skyler Seto, Sarah Tan, Giles Hooker, Martin T. Wells

Abstract: Non-negative matrix factorization (NMF) is a technique for finding latent representations of data. The method has been applied to corpora to construct topic models. However, NMF has likelihood assumptions which are often violated by real document corpora. We present a double parametric bootstrap test for evaluating the fit of an NMF-based topic model based on the duality of the KL divergence and P… ▽ More Non-negative matrix factorization (NMF) is a technique for finding latent representations of data. The method has been applied to corpora to construct topic models. However, NMF has likelihood assumptions which are often violated by real document corpora. We present a double parametric bootstrap test for evaluating the fit of an NMF-based topic model based on the duality of the KL divergence and Poisson maximum likelihood estimation. The test correctly identifies whether a topic model based on an NMF approach yields reliable results in simulated and real data. △ Less

Submitted 20 November, 2017; v1 submitted 19 November, 2017; originally announced November 2017.

Comments: Presented at NIPS 2017 Symposium on Interpretable Machine Learning

First eigenvalue of the $p$-Laplacian under integral curvature condition

Authors: Shoo Seto, Guofang Wei

Abstract: We give various estimates of the first eigenvalue of the $p$-Laplace operator on closed Riemannian manifold with integral curvature conditions. We give various estimates of the first eigenvalue of the $p$-Laplace operator on closed Riemannian manifold with integral curvature conditions. △ Less

Submitted 15 July, 2017; originally announced July 2017.

Comments: 11 pages, to appear in Journal of Nonlinear Analysis

MSC Class: 58C40

Sharp Fundamental Gap Estimate on Convex Domains of Sphere

Authors: Shoo Seto, Lili Wang, Guofang Wei

Abstract: In their celebrated work, B. Andrews and J. Clutterbuck proved the fundamental gap (the difference between the first two eigenvalues) conjecture for convex domains in the Euclidean space and conjectured similar results holds for spaces with constant sectional curvature. We prove the conjecture for the sphere. Namely when $D$, the diameter of a convex domain in the unit $S^n$ sphere, is… ▽ More In their celebrated work, B. Andrews and J. Clutterbuck proved the fundamental gap (the difference between the first two eigenvalues) conjecture for convex domains in the Euclidean space and conjectured similar results holds for spaces with constant sectional curvature. We prove the conjecture for the sphere. Namely when $D$, the diameter of a convex domain in the unit $S^n$ sphere, is $\le \fracπ{2}$, the gap is greater than the gap of the corresponding $1$-dim sphere model. We also prove the gap is $\ge 3\frac{π^2}{D^2}$ when $n \ge 3$, giving a sharp bound. As in Andrews-Clutterbuck's proof of the fundamental gap, the key is to prove a super log-concavity of the first eigenfunction. △ Less

Submitted 14 March, 2017; v1 submitted 3 June, 2016; originally announced June 2016.

Comments: An error and typos corrected, modified the proof of Theorem 1.5, Theorem 3.2, and Appendix B added

Longitudinal spin-relaxation of donor-bound electrons in direct bandgap semiconductors

Authors: Todd Karin, Xiayu Linpeng, M. V. Durnev, Russell Barbour, M. M. Glazov, E. Ya. Sherman, Simon Watkins, Satoru Seto, Kai-Mei C. Fu

Abstract: We measure the donor-bound electron longitudinal spin-relaxation time ($T_1$) as a function of magnetic field ($B$) in three high-purity direct-bandgap semiconductors: GaAs, InP, and CdTe, observing a maximum $T_1$ of $1.4~\text{ms}$, $0.4~\text{ms}$ and $1.2~\text{ms}$, respectively. In GaAs and InP at low magnetic field, up to $\sim2~\text{T}$, the spin-relaxation mechanism is strongly density a… ▽ More We measure the donor-bound electron longitudinal spin-relaxation time ($T_1$) as a function of magnetic field ($B$) in three high-purity direct-bandgap semiconductors: GaAs, InP, and CdTe, observing a maximum $T_1$ of $1.4~\text{ms}$, $0.4~\text{ms}$ and $1.2~\text{ms}$, respectively. In GaAs and InP at low magnetic field, up to $\sim2~\text{T}$, the spin-relaxation mechanism is strongly density and temperature dependent and is attributed to the random precession of the electron spin in hyperfine fields caused by the lattice nuclear spins. In all three semiconductors at high magnetic field, we observe a power-law dependence ${T_1 \propto B^{-ν}}$ with ${3\lesssim ν\lesssim 4}$. Our theory predicts that the direct spin-phonon interaction is important in all three materials in this regime in contrast to quantum dot structures. In addition, the "admixture" mechanism caused by Dresselhaus spin-orbit coupling combined with single-phonon processes has a comparable contribution in GaAs. We find excellent agreement between high-field theory and experiment for GaAs and CdTe with no free parameters, however a significant discrepancy exists for InP. △ Less

Submitted 19 May, 2016; originally announced May 2016.

Comments: 20 pages, 12 figures

Journal ref: Phys. Rev. B 94, 125401 (2016)

Multivariate Time Series Classification Using Dynamic Time Warping Template Selection for Human Activity Recognition

Authors: Skyler Seto, Wenyu Zhang, Yichen Zhou

Abstract: Accurate and computationally efficient means for classifying human activities have been the subject of extensive research efforts. Most current research focuses on extracting complex features to achieve high classification accuracy. We propose a template selection approach based on Dynamic Time Warping, such that complex feature extraction and domain knowledge is avoided. We demonstrate the predic… ▽ More Accurate and computationally efficient means for classifying human activities have been the subject of extensive research efforts. Most current research focuses on extracting complex features to achieve high classification accuracy. We propose a template selection approach based on Dynamic Time Warping, such that complex feature extraction and domain knowledge is avoided. We demonstrate the predictive capability of the algorithm on both simulated and real smartphone data. △ Less

Submitted 21 December, 2015; originally announced December 2015.

Asymptotic expansion of the Bergman kernel via perturbation of the Bargmann-Fock model

Authors: Hamid Hezari, Casey Lynn Kelleher, Shoo Seto, Hang Xu

Abstract: We give an alternate proof of the existence of the asymptotic expansion of the Bergman kernel associated to the $k$-th tensor powers of a positive line bundle $L$ in a $\frac{1}{\sqrt{k}}$-neighborhood of the diagonal using elementary methods. We use the observation that after rescaling the Kähler potential $k\varphi$ in a $\frac{1}{\sqrt{k}}$-neighborhood of a given point, the potential becomes a… ▽ More We give an alternate proof of the existence of the asymptotic expansion of the Bergman kernel associated to the $k$-th tensor powers of a positive line bundle $L$ in a $\frac{1}{\sqrt{k}}$-neighborhood of the diagonal using elementary methods. We use the observation that after rescaling the Kähler potential $k\varphi$ in a $\frac{1}{\sqrt{k}}$-neighborhood of a given point, the potential becomes an asymptotic perturbation of the Bargmann-Fock metric. We then prove that the Bergman kernel is also an asymptotic perturbation of the Bargmann-Fock Bergman kernel. △ Less

Submitted 14 December, 2014; v1 submitted 26 November, 2014; originally announced November 2014.

Comments: 28 pages