Showing 1–32 of 32 results for author: Ju, D

Frucht's theorem and other set-theoretic principles below the axiom of choice and the axiom of foundation

Authors: Junhong Chen, Daheng Ju

Abstract: We take the first step toward the study of set-theoretic principles below the axiom of choice $\mathsf{AC}$ and the axiom of foundation $\mathsf{AF}$ by studying Frucht's theorem, an ordinary mathematical theorem which is provable with either $\mathsf{AC}$ or $\mathsf{AF}$ but not provable without both, and its variants. Specifically, we propose a number of such principles, study the relations bet… ▽ More We take the first step toward the study of set-theoretic principles below the axiom of choice $\mathsf{AC}$ and the axiom of foundation $\mathsf{AF}$ by studying Frucht's theorem, an ordinary mathematical theorem which is provable with either $\mathsf{AC}$ or $\mathsf{AF}$ but not provable without both, and its variants. Specifically, we propose a number of such principles, study the relations between these principles and the standard axioms, and prove provability and unprovability results using (infinite) graph-theoretic constructions and permutation models, which draw a preliminary map of this new area of set theory. △ Less

Submitted 5 August, 2026; v1 submitted 26 July, 2026; originally announced July 2026.

Anatomically Consistent TMJ Disc Segmentation via Semantic Anchoring and Clinical Priors

Authors: Dayun Ju, Chanyoung Kim, Sunyoung Jung, Hyo-Jung Jung, Chena Lee, Younjung Park, Seong Jae Hwang

Abstract: Segmenting the temporomandibular joint (TMJ) disc from MRI is essential for accurate diagnosis of internal derangement, yet it remains unreliable in practice due to its small size, low contrast, and morphological variability. Existing methods, primarily adapted from general segmentation architectures, often produce fragmented or anatomically inconsistent masks, leading to unstable measurements of… ▽ More Segmenting the temporomandibular joint (TMJ) disc from MRI is essential for accurate diagnosis of internal derangement, yet it remains unreliable in practice due to its small size, low contrast, and morphological variability. Existing methods, primarily adapted from general segmentation architectures, often produce fragmented or anatomically inconsistent masks, leading to unstable measurements of disc position and shape for downstream diagnosis. To address these challenges, we propose TISC, a TMJ disc segmentation framework that integrates semantic anchoring with clinical metadata-guided boundary refinement. The framework first establishes robust disc localization in the foundation model feature space via a Prototypical Semantic Anchoring (PSA) module that aggregates adjacent-slice MedDINOv3 features and derives a prototype-driven similarity map. It then performs targeted boundary refinement through a Clinical-Metadata Point Refinement (C-MPR) module, with point-wise predictions modulated by Mouth Open Limitation (MOL), a clinical indicator associated with disc displacement without reduction. On a large-scale cohort of 2,488 PD MRI volumes from 1,300 patients, our method achieves up to a 4.96 Dice improvement over strong baselines across diverse architectures, delivering more anatomically coherent and clinically reliable TMJ disc segmentation. △ Less

Submitted 19 June, 2026; originally announced June 2026.

Comments: 10 pages, 3 figures

ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting

Authors: Yeonkyung Lee, Dayun Ju, Youngmin Kim, Seil Kang, Seong Jae Hwang

Abstract: Recent advancements in Video Large Language Models (VideoLLMs) have enabled strong performance across diverse multimodal video tasks. To reduce the high computational cost of processing dense video frames, efficiency-oriented methods such as frame selection have been widely adopted. While effective at minimizing redundancy, these methods often cause notable performance drops on tasks requiring tem… ▽ More Recent advancements in Video Large Language Models (VideoLLMs) have enabled strong performance across diverse multimodal video tasks. To reduce the high computational cost of processing dense video frames, efficiency-oriented methods such as frame selection have been widely adopted. While effective at minimizing redundancy, these methods often cause notable performance drops on tasks requiring temporal reasoning. Unlike humans, who can infer event progression from sparse visual cues, VideoLLMs frequently misinterpret temporal relations when intermediate frames are omitted. To address this limitation, we explore visual prompting (VP) as a lightweight yet effective way to enhance temporal understanding in VideoLLMs. Our analysis reveals that simply annotating each frame with explicit ordinal information helps the model perceive temporal continuity. This visual cue also supports frame-level referencing and mitigates positional ambiguity within a sparsely sampled sequence. Building on these insights, we introduce ViKey, a training-free framework that combines VP with a lightweight Keyword-Frame Mapping (KFM) module. KFM leverages frame indices as dictionary-like keys to link textual cues to the most relevant frames, providing explicit temporal anchors during inference. Despite its simplicity, our approach substantially improves temporal reasoning and, on some datasets, preserves dense-frame baseline performance with as few as 20% of frames. △ Less

Submitted 24 March, 2026; originally announced March 2026.

Comments: accepted to CVPR2026

Rare Text Semantics Were Always There in Your Diffusion Transformer

Authors: Seil Kang, Woojung Han, Dayun Ju, Seong Jae Hwang

Abstract: Starting from flow- and diffusion-based transformers, Multi-modal Diffusion Transformers (MM-DiTs) have reshaped text-to-vision generation, gaining acclaim for exceptional visual fidelity. As these models advance, users continually push the boundary with imaginative or rare prompts, which advanced models still falter in generating, since their concepts are often too scarce to leave a strong imprin… ▽ More Starting from flow- and diffusion-based transformers, Multi-modal Diffusion Transformers (MM-DiTs) have reshaped text-to-vision generation, gaining acclaim for exceptional visual fidelity. As these models advance, users continually push the boundary with imaginative or rare prompts, which advanced models still falter in generating, since their concepts are often too scarce to leave a strong imprint during pre-training. In this paper, we propose a simple yet effective intervention that surfaces rare semantics inside MM-DiTs without additional training steps, data, denoising-time optimization, or reliance on external modules (e.g., large language models). In particular, the joint-attention mechanism intrinsic to MM-DiT sequentially updates text embeddings alongside image embeddings throughout transformer blocks. We find that by mathematically expanding representational basins around text token embeddings via variance scale-up before the joint-attention blocks, rare semantics clearly emerge in MM-DiT's outputs. Furthermore, our results generalize effectively across text-to-vision tasks, including text-to-image, text-to-video, and text-driven image editing. Our work invites generative models to reveal the semantics that users intend, once hidden yet ready to surface. △ Less

Submitted 4 October, 2025; originally announced October 2025.

Comments: Accepted to NeurIPS 2025

Domain Regeneration: How well do LLMs match syntactic properties of text domains?

Authors: Da Ju, Hagen Blix, Adina Williams

Abstract: Recent improvement in large language model performance have, in all likelihood, been accompanied by improvement in how well they can approximate the distribution of their training data. In this work, we explore the following question: which properties of text domains do LLMs faithfully approximate, and how well do they do so? Applying observational approaches familiar from corpus linguistics, we p… ▽ More Recent improvement in large language model performance have, in all likelihood, been accompanied by improvement in how well they can approximate the distribution of their training data. In this work, we explore the following question: which properties of text domains do LLMs faithfully approximate, and how well do they do so? Applying observational approaches familiar from corpus linguistics, we prompt a commonly used, opensource LLM to regenerate text from two domains of permissively licensed English text which are often contained in LLM training data -- Wikipedia and news text. This regeneration paradigm allows us to investigate whether LLMs can faithfully match the original human text domains in a fairly semantically-controlled setting. We investigate varying levels of syntactic abstraction, from more simple properties like sentence length, and article readability, to more complex and higher order properties such as dependency tag distribution, parse depth, and parse complexity. We find that the majority of the regenerated distributions show a shifted mean, a lower standard deviation, and a reduction of the long tail, as compared to the human originals. △ Less

Submitted 2 June, 2025; v1 submitted 12 May, 2025; originally announced May 2025.

Pathology-Aware Adaptive Watermarking for Text-Driven Medical Image Synthesis

Authors: Chanyoung Kim, Dayun Ju, Jinyeong Kim, Woojung Han, Roberto Alcover-Couso, Seong Jae Hwang

Abstract: As recent text-conditioned diffusion models have enabled the generation of high-quality images, concerns over their potential misuse have also grown. This issue is critical in the medical domain, where text-conditioned generated medical images could enable insurance fraud or falsified records, highlighting the urgent need for reliable safeguards against unethical use. While watermarking techniques… ▽ More As recent text-conditioned diffusion models have enabled the generation of high-quality images, concerns over their potential misuse have also grown. This issue is critical in the medical domain, where text-conditioned generated medical images could enable insurance fraud or falsified records, highlighting the urgent need for reliable safeguards against unethical use. While watermarking techniques have emerged as a promising solution in general image domains, their direct application to medical imaging presents significant challenges. A key challenge is preserving fine-grained disease manifestations, as even minor distortions from a watermark may lead to clinical misinterpretation, which compromises diagnostic integrity. To overcome this gap, we present MedSign, a deep learning-based watermarking framework specifically designed for text-to-medical image synthesis, which preserves pathologically significant regions by adaptively adjusting watermark strength. Specifically, we generate a pathology localization map using cross-attention between medical text tokens and the diffusion denoising network, aggregating token-wise attention across layers, heads, and time steps. Leveraging this map, we optimize the LDM decoder to incorporate watermarking during image synthesis, ensuring cohesive integration while minimizing interference in diagnostically critical regions. Experimental results show that our MedSign preserves diagnostic integrity while ensuring watermark robustness, achieving state-of-the-art performance in image quality and detection accuracy on MIMIC-CXR and OIA-ODIR datasets. △ Less

Submitted 1 September, 2025; v1 submitted 11 March, 2025; originally announced March 2025.

Comparing Anti-foundation Axioms by Comparing Identity Conditions for Sets

Authors: Daheng Ju, Qihang Jing

Abstract: In non-well-founded set theory, which anti-foundation axiom is philosophically justified, BAFA, FAFA, SAFA, AFA, or some other? In this paper, we investigate a general approach to answering this question: first, consider which identity condition for sets is justified; second, consider which anti-foundation axiom it justifies. Specifically, we study in detail two plausible identity conditions. In non-well-founded set theory, which anti-foundation axiom is philosophically justified, BAFA, FAFA, SAFA, AFA, or some other? In this paper, we investigate a general approach to answering this question: first, consider which identity condition for sets is justified; second, consider which anti-foundation axiom it justifies. Specifically, we study in detail two plausible identity conditions. △ Less

Submitted 8 December, 2024; originally announced December 2024.

Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models

Authors: Da Ju, Adina Williams, Brian Karrer, Maximilian Nickel

Abstract: Large language models have increasingly been proposed as a powerful replacement for classical agent-based models (ABMs) to simulate social dynamics. By using LLMs as a proxy for human behavior, the hope of this new approach is to be able to simulate significantly more complex dynamics than with classical ABMs and gain new insights in fields such as social science, political science, and economics.… ▽ More Large language models have increasingly been proposed as a powerful replacement for classical agent-based models (ABMs) to simulate social dynamics. By using LLMs as a proxy for human behavior, the hope of this new approach is to be able to simulate significantly more complex dynamics than with classical ABMs and gain new insights in fields such as social science, political science, and economics. However, due to the black box nature of LLMs, it is unclear whether LLM agents actually execute the intended semantics that are encoded in their natural language instructions and, if the resulting dynamics of interactions are meaningful. To study this question, we propose a new evaluation framework that grounds LLM simulations within the dynamics of established reference models of social science. By treating LLMs as a black-box function, we evaluate their input-output behavior relative to this reference model, which allows us to evaluate detailed aspects of their behavior. Our results show that, while it is possible to engineer prompts that approximate the intended dynamics, the quality of these simulations is highly sensitive to the particular choice of prompts. Importantly, simulations are even sensitive to arbitrary variations such as minor wording changes and whitespace. This puts into question the usefulness of current versions of LLMs for meaningful simulations, as without a reference model, it is impossible to determine a priori what impact seemingly meaningless changes in prompt will have on the simulation. △ Less

Submitted 6 December, 2024; originally announced December 2024.

Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation

Authors: Chanyoung Kim, Dayun Ju, Woojung Han, Ming-Hsuan Yang, Seong Jae Hwang

Abstract: Open-Vocabulary Semantic Segmentation (OVSS) has advanced with recent vision-language models (VLMs), enabling segmentation beyond predefined categories through various learning schemes. Notably, training-free methods offer scalable, easily deployable solutions for handling unseen data, a key goal of OVSS. Yet, a critical issue persists: lack of object-level context consideration when segmenting co… ▽ More Open-Vocabulary Semantic Segmentation (OVSS) has advanced with recent vision-language models (VLMs), enabling segmentation beyond predefined categories through various learning schemes. Notably, training-free methods offer scalable, easily deployable solutions for handling unseen data, a key goal of OVSS. Yet, a critical issue persists: lack of object-level context consideration when segmenting complex objects in the challenging environment of OVSS based on arbitrary query prompts. This oversight limits models' ability to group semantically consistent elements within object and map them precisely to user-defined arbitrary classes. In this work, we introduce a novel approach that overcomes this limitation by incorporating object-level contextual knowledge within images. Specifically, our model enhances intra-object consistency by distilling spectral-driven features from vision foundation models into the attention mechanism of the visual encoder, enabling semantically coherent components to form a single object mask. Additionally, we refine the text embeddings with zero-shot object presence likelihood to ensure accurate alignment with the specific objects represented in the images. By leveraging object-level contextual knowledge, our proposed approach achieves state-of-the-art performance with strong generalizability across diverse datasets. △ Less

Submitted 7 April, 2025; v1 submitted 26 November, 2024; originally announced November 2024.

Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning

Authors: Song Jiang, Da JU, Andrew Cohen, Sasha Mitts, Aaron Foss, Justine T Kao, Xian Li, Yuandong Tian

Abstract: How are LLM-based agents used in the future? While many of the existing work on agents has focused on improving the performance of a specific family of objective and challenging tasks, in this work, we take a different perspective by thinking about full delegation: agents take over humans' routine decision-making processes and are trusted by humans to find solutions that fit people's personalized… ▽ More How are LLM-based agents used in the future? While many of the existing work on agents has focused on improving the performance of a specific family of objective and challenging tasks, in this work, we take a different perspective by thinking about full delegation: agents take over humans' routine decision-making processes and are trusted by humans to find solutions that fit people's personalized needs and are adaptive to ever-changing context. In order to achieve such a goal, the behavior of the agents, i.e., agentic behaviors, should be evaluated not only on their achievements (i.e., outcome evaluation), but also how they achieved that (i.e., procedure evaluation). For this, we propose APEC Agent Constitution, a list of criteria that an agent should follow for good agentic behaviors, including Accuracy, Proactivity, Efficiency and Credibility. To verify whether APEC aligns with human preferences, we develop APEC-Travel, a travel planning agent that proactively extracts hidden personalized needs via multi-round dialog with travelers. APEC-Travel is constructed purely from synthetic data generated by Llama3.1-405B-Instruct with a diverse set of travelers' persona to simulate rich distribution of dialogs. Iteratively fine-tuned to follow APEC Agent Constitution, APEC-Travel surpasses baselines by 20.7% on rule-based metrics and 9.1% on LLM-as-a-Judge scores across the constitution axes. △ Less

Submitted 21 November, 2024; originally announced November 2024.

To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning

Authors: Da JU, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine T Kao, Maryam Fazel-Zarandi, Yuandong Tian

Abstract: Travel planning is a challenging and time-consuming task that aims to find an itinerary which satisfies multiple, interdependent constraints regarding flights, accommodations, attractions, and other travel arrangements. In this paper, we propose To the Globe (TTG), a real-time demo system that takes natural language requests from users, translates it to symbolic form via a fine-tuned Large Languag… ▽ More Travel planning is a challenging and time-consuming task that aims to find an itinerary which satisfies multiple, interdependent constraints regarding flights, accommodations, attractions, and other travel arrangements. In this paper, we propose To the Globe (TTG), a real-time demo system that takes natural language requests from users, translates it to symbolic form via a fine-tuned Large Language Model, and produces optimal travel itineraries with Mixed Integer Linear Programming solvers. The overall system takes ~5 seconds to reply to the user request with guaranteed itineraries. To train TTG, we develop a synthetic data pipeline that generates user requests, flight and hotel information in symbolic form without human annotations, based on the statistics of real-world datasets, and fine-tune an LLM to translate NL user requests to their symbolic form, which is sent to the symbolic solver to compute optimal itineraries. Our NL-symbolic translation achieves ~91% exact match in a backtranslation metric (i.e., whether the estimated symbolic form of generated natural language matches the groundtruth), and its returned itineraries have a ratio of 0.979 compared to the optimal cost of the ground truth user request. When evaluated by users, TTG achieves consistently high Net Promoter Scores (NPS) of 35-40% on generated itinerary. △ Less

Submitted 21 October, 2024; originally announced October 2024.

Journal ref: EMNLP 2024 Demo Track

Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality

Authors: Da Ju, Karen Ulrich, Adina Williams

Abstract: People tend to use language to mention surprising properties of events: for example, when a banana is blue, we are more likely to mention color than when it is yellow. This fact is taken to suggest that yellowness is somehow a typical feature of bananas, and blueness is exceptional. Similar to how a yellow color is typical of bananas, there may also be genders that are typical of occupations. In t… ▽ More People tend to use language to mention surprising properties of events: for example, when a banana is blue, we are more likely to mention color than when it is yellow. This fact is taken to suggest that yellowness is somehow a typical feature of bananas, and blueness is exceptional. Similar to how a yellow color is typical of bananas, there may also be genders that are typical of occupations. In this work, we explore this question using information theoretic techniques coupled with corpus statistic analysis. In two distinct large corpora, we do not find strong evidence that occupations and gender display the same patterns of mentioning as do bananas and color. Instead, we find that gender mentioning is correlated with femaleness of occupation in particular, suggesting perhaps that woman-dominated occupations are seen as somehow ``more gendered'' than male-dominated ones, and thereby they encourage more gender mentioning overall. △ Less

Submitted 6 August, 2024; originally announced August 2024.

Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning

Authors: Woojung Han, Chanyoung Kim, Dayun Ju, Yumin Shim, Seong Jae Hwang

Abstract: Recent advances in text-conditioned image generation diffusion models have begun paving the way for new opportunities in modern medical domain, in particular, generating Chest X-rays (CXRs) from diagnostic reports. Nonetheless, to further drive the diffusion models to generate CXRs that faithfully reflect the complexity and diversity of real data, it has become evident that a nontrivial learning a… ▽ More Recent advances in text-conditioned image generation diffusion models have begun paving the way for new opportunities in modern medical domain, in particular, generating Chest X-rays (CXRs) from diagnostic reports. Nonetheless, to further drive the diffusion models to generate CXRs that faithfully reflect the complexity and diversity of real data, it has become evident that a nontrivial learning approach is needed. In light of this, we propose CXRL, a framework motivated by the potential of reinforcement learning (RL). Specifically, we integrate a policy gradient RL approach with well-designed multiple distinctive CXR-domain specific reward models. This approach guides the diffusion denoising trajectory, achieving precise CXR posture and pathological details. Here, considering the complex medical image environment, we present "RL with Comparative Feedback" (RLCF) for the reward mechanism, a human-like comparative evaluation that is known to be more effective and reliable in complex scenarios compared to direct evaluation. Our CXRL framework includes jointly optimizing learnable adaptive condition embeddings (ACE) and the image generator, enabling the model to produce more accurate and higher perceptual CXR quality. Our extensive evaluation of the MIMIC-CXR-JPG dataset demonstrates the effectiveness of our RL-based tuning approach. Consequently, our CXRL generates pathologically realistic CXRs, establishing a new standard for generating CXRs with high fidelity to real-world clinical scenarios. △ Less

Submitted 11 March, 2024; originally announced March 2024.

EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation

Authors: Chanyoung Kim, Woojung Han, Dayun Ju, Seong Jae Hwang

Abstract: Semantic segmentation has innately relied on extensive pixel-level annotated data, leading to the emergence of unsupervised methodologies. Among them, leveraging self-supervised Vision Transformers for unsupervised semantic segmentation (USS) has been making steady progress with expressive deep features. Yet, for semantically segmenting images with complex objects, a predominant challenge remains:… ▽ More Semantic segmentation has innately relied on extensive pixel-level annotated data, leading to the emergence of unsupervised methodologies. Among them, leveraging self-supervised Vision Transformers for unsupervised semantic segmentation (USS) has been making steady progress with expressive deep features. Yet, for semantically segmenting images with complex objects, a predominant challenge remains: the lack of explicit object-level semantic encoding in patch-level features. This technical limitation often leads to inadequate segmentation of complex objects with diverse structures. To address this gap, we present a novel approach, EAGLE, which emphasizes object-centric representation learning for unsupervised semantic segmentation. Specifically, we introduce EiCue, a spectral technique providing semantic and structural cues through an eigenbasis derived from the semantic similarity matrix of deep image features and color affinity from an image. Further, by incorporating our object-centric contrastive loss with EiCue, we guide our model to learn object-level representations with intra- and inter-image object-feature consistency, thereby enhancing semantic accuracy. Extensive experiments on COCO-Stuff, Cityscapes, and Potsdam-3 datasets demonstrate the state-of-the-art USS results of EAGLE with accurate and consistent semantic segmentation across complex scenes. △ Less

Submitted 5 April, 2024; v1 submitted 3 March, 2024; originally announced March 2024.

Improving Open Language Models by Learning from Organic Interactions

Authors: Jing Xu, Da Ju, Joshua Lane, Mojtaba Komeili, Eric Michael Smith, Megan Ung, Morteza Behrooz, William Ngan, Rashel Moritz, Sainbayar Sukhbaatar, Y-Lan Boureau, Jason Weston, Kurt Shuster

Abstract: We present BlenderBot 3x, an update on the conversational model BlenderBot 3, which is now trained using organic conversation and feedback data from participating users of the system in order to improve both its skills and safety. We are publicly releasing the participating de-identified interaction data for use by the research community, in order to spur further progress. Training models with org… ▽ More We present BlenderBot 3x, an update on the conversational model BlenderBot 3, which is now trained using organic conversation and feedback data from participating users of the system in order to improve both its skills and safety. We are publicly releasing the participating de-identified interaction data for use by the research community, in order to spur further progress. Training models with organic data is challenging because interactions with people "in the wild" include both high quality conversations and feedback, as well as adversarial and toxic behavior. We study techniques that enable learning from helpful teachers while avoiding learning from people who are trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is both preferred in conversation to BlenderBot 3, and is shown to produce safer responses in challenging situations. While our current models are still far from perfect, we believe further improvement can be achieved by continued use of the techniques explored in this work. △ Less

Submitted 7 June, 2023; originally announced June 2023.

Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls

Authors: Da Ju, Jing Xu, Y-Lan Boureau, Jason Weston

Abstract: The promise of interaction between intelligent conversational agents and humans is that models can learn from such feedback in order to improve. Unfortunately, such exchanges in the wild will not always involve human utterances that are benign or of high quality, and will include a mixture of engaged (helpers) and unengaged or even malicious users (trolls). In this work we study how to perform rob… ▽ More The promise of interaction between intelligent conversational agents and humans is that models can learn from such feedback in order to improve. Unfortunately, such exchanges in the wild will not always involve human utterances that are benign or of high quality, and will include a mixture of engaged (helpers) and unengaged or even malicious users (trolls). In this work we study how to perform robust learning in such an environment. We introduce a benchmark evaluation, SafetyMix, which can evaluate methods that learn safe vs. toxic language in a variety of adversarial settings to test their robustness. We propose and analyze several mitigating learning algorithms that identify trolls either at the example or at the user level. Our main finding is that user-based methods, that take into account that troll users will exhibit adversarial behavior across multiple examples, work best in a variety of settings on our benchmark. We then test these methods in a further real-life setting of conversations collected during deployment, with similar results. △ Less

Submitted 5 August, 2022; originally announced August 2022.

BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage

Authors: Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, Jason Weston

Abstract: We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks. We release both the model weights and code, and have also deployed the model on a public web page to interact with organic users. This technical report describes how the model was built (arc… ▽ More We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks. We release both the model weights and code, and have also deployed the model on a public web page to interact with organic users. This technical report describes how the model was built (architecture, model and training scheme), and details of its deployment, including safety mechanisms. Human evaluations show its superiority to existing open-domain dialogue agents, including its predecessors (Roller et al., 2021; Komeili et al., 2022). Finally, we detail our plan for continual learning using the data collected from deployment, which will also be publicly released. The goal of this research program is thus to enable the community to study ever-improving responsible agents that learn through interaction. △ Less

Submitted 10 August, 2022; v1 submitted 5 August, 2022; originally announced August 2022.

Electric double layer of colloidal particles in salt-free concentrated suspensions including non-uniform size effects and orientational ordering of water dipoles

Authors: Jun-Sik Sin, Kwang-Il Kim, Kuk-Chol Ri, Dok-Yong Ju, Nam-Hyok Kim, Chung-Sik Sin

Abstract: The response of a suspension under a variety of static or alternating external fields strongly depends on the equilibrium electric double layer that surrounds the colloidal particles in the suspension. The theoretical models for salt-free suspensions can be improved by incorporating non-uniform size effects and orientational ordering of water dipoles neglected in previous mean-field approaches, wh… ▽ More The response of a suspension under a variety of static or alternating external fields strongly depends on the equilibrium electric double layer that surrounds the colloidal particles in the suspension. The theoretical models for salt-free suspensions can be improved by incorporating non-uniform size effects and orientational ordering of water dipoles neglected in previous mean-field approaches, which are based on the Poisson-Boltzmann approach. Our model including non-uniform size effects and orientational ordering of water dipoles seems to have quite a promising effect because the model can predict the phenomena like a heavy decrease in relative permittivity of the suspension and counterion stratification near highly charged colloidal particle. In this work we numerically obtain the electric potential, the counterions concentration and the relative permittivity around a charged particle in a concentrated salt-free suspension corrected by non-uniform size effects and orientational ordering of water dipoles. The results show the worth of such corrections for medium to high particle charges at every particle volume fraction. We conclude that non-uniform size effects and orientational ordering of water dipoles are necessary for the development of new theoretical models to study non-equilibrium properties in concentrated colloidal suspensions. △ Less

Submitted 10 February, 2022; originally announced February 2022.

Comments: arXiv admin note: text overlap with arXiv:1102.0255 by other authors

Journal ref: Phys. Chem. Chem. Phys., 2016, 18, 234-243

Staircase Attention for Recurrent Processing of Sequences

Authors: Da Ju, Stephen Roller, Sainbayar Sukhbaatar, Jason Weston

Abstract: Attention mechanisms have become a standard tool for sequence modeling tasks, in particular by stacking self-attention layers over the entire input sequence as in the Transformer architecture. In this work we introduce a novel attention procedure called staircase attention that, unlike self-attention, operates across the sequence (in time) recurrently processing the input by adding another step of… ▽ More Attention mechanisms have become a standard tool for sequence modeling tasks, in particular by stacking self-attention layers over the entire input sequence as in the Transformer architecture. In this work we introduce a novel attention procedure called staircase attention that, unlike self-attention, operates across the sequence (in time) recurrently processing the input by adding another step of processing. A step in the staircase comprises of backward tokens (encoding the sequence so far seen) and forward tokens (ingesting a new part of the sequence), or an extreme Ladder version with a forward step of zero that simply repeats the Transformer on each step of the ladder, sharing the weights. We thus describe a family of such models that can trade off performance and compute, by either increasing the amount of recurrence through time, the amount of sequential processing via recurrence in depth, or both. Staircase attention is shown to be able to solve tasks that involve tracking that conventional Transformers cannot, due to this recurrence. Further, it is shown to provide improved modeling power for the same size model (number of parameters) compared to self-attentive Transformers on large language modeling and dialogue tasks, yielding significant perplexity gains. △ Less

Submitted 8 June, 2021; originally announced June 2021.

The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

Authors: Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzman, Angela Fan

Abstract: One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benc… ▽ More One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond. △ Less

Submitted 6 June, 2021; originally announced June 2021.

Not All Memories are Created Equal: Learning to Forget by Expiring

Authors: Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, Angela Fan

Abstract: Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of preserving and storing memories. However, not all content in the past is equally important to remember. We propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant info… ▽ More Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of preserving and storing memories. However, not all content in the past is equally important to remember. We propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant information. This forgetting of memories enables Transformers to scale to attend over tens of thousands of previous timesteps efficiently, as not all states from previous timesteps are preserved. We demonstrate that Expire-Span can help models identify and retain critical information and show it can achieve strong performance on reinforcement learning tasks specifically designed to challenge this functionality. Next, we show that Expire-Span can scale to memories that are tens of thousands in size, setting a new state of the art on incredibly long context tasks such as character-level language modeling and a frame-by-frame moving objects task. Finally, we analyze the efficiency of Expire-Span compared to existing approaches and demonstrate that it trains faster and uses less memory. △ Less

Submitted 13 June, 2021; v1 submitted 13 May, 2021; originally announced May 2021.

Recipes for Safety in Open-domain Chatbots

Authors: Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, Emily Dinan

Abstract: Models trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior and unwanted biases. We investigate a variety of methods to mitigate these issues in the context of open-domain generative dialogue models. We introduce a new human-and-model-in-the-loop framework for both training safer models and for… ▽ More Models trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior and unwanted biases. We investigate a variety of methods to mitigate these issues in the context of open-domain generative dialogue models. We introduce a new human-and-model-in-the-loop framework for both training safer models and for evaluating them, as well as a novel method to distill safety considerations inside generative models without the use of an external classifier at deployment time. We conduct experiments comparing these methods and find our new techniques are (i) safer than existing models as measured by automatic and human evaluations while (ii) maintaining usability metrics such as engagingness relative to the state of the art. We then discuss the limitations of this work by analyzing failure cases of our models. △ Less

Submitted 4 August, 2021; v1 submitted 14 October, 2020; originally announced October 2020.

Multi-Modal Open-Domain Dialogue

Authors: Kurt Shuster, Eric Michael Smith, Da Ju, Jason Weston

Abstract: Recent work in open-domain conversational agents has demonstrated that significant improvements in model engagingness and humanness metrics can be achieved via massive scaling in both pre-training data and model size (Adiwardana et al., 2020; Roller et al., 2020). However, if we want to build agents with human-like abilities, we must expand beyond handling just text. A particularly important topic… ▽ More Recent work in open-domain conversational agents has demonstrated that significant improvements in model engagingness and humanness metrics can be achieved via massive scaling in both pre-training data and model size (Adiwardana et al., 2020; Roller et al., 2020). However, if we want to build agents with human-like abilities, we must expand beyond handling just text. A particularly important topic is the ability to see images and communicate about what is perceived. With the goal of engaging humans in multi-modal dialogue, we investigate combining components from state-of-the-art open-domain dialogue agents with those from state-of-the-art vision models. We study incorporating different image fusion schemes and domain-adaptive pre-training and fine-tuning strategies, and show that our best resulting model outperforms strong existing models in multi-modal dialogue while simultaneously performing as well as its predecessor (text-only) BlenderBot (Roller et al., 2020) in text-based conversation. We additionally investigate and incorporate safety components in our final model, and show that such efforts do not diminish model performance with respect to engagingness metrics. △ Less

Submitted 2 October, 2020; originally announced October 2020.

Dynamics of the sub-ambient gelation and shearing of solutions of P3HT incorporated with a non-fullerene acceptor o-IDTBR towards active layer formation in bulk heterojunction organic solar cells

Authors: Li Quan, Dongrun Ju, Stephanie Lee, Dilhan M. Kalyon

Abstract: Organic solar cells (OSCs) containing an active layer consisting of a nanostructured blend of a conjugated polymer like poly(3-hexylthiophene) (P3HT) and an electron acceptor molecule have the potential of competing against silicon-based photovoltaic panels. However, this potential is unfulfilled primarily due to interrelated production and stability issues. The generally employed spin coating pro… ▽ More Organic solar cells (OSCs) containing an active layer consisting of a nanostructured blend of a conjugated polymer like poly(3-hexylthiophene) (P3HT) and an electron acceptor molecule have the potential of competing against silicon-based photovoltaic panels. However, this potential is unfulfilled primarily due to interrelated production and stability issues. The generally employed spin coating process for fabricating organic solar cells cannot be scaled up. Recently, He et al., have reported that the gelation of P3HT with [6,6]-phenyl-C61-butyric acid methyl ester (PC60BM) under sub-ambient conditions can provide a continuous extrusion/coating based route to the processing of organic solar cells and that increases in power conversion efficiencies (PCEs) of the P3HT/PC60BM active layer are possible under certain shearing and thermal histories of the P3HT/PC60BM gels. Here oscillatory and steady torsional flows were used to investigate the gel formation dynamics of P3HT with a recently proposed non-fullerene o-IDTBR under sub-ambient conditions. The gel strengths defined on the basis of linear viscoelastic material functions as determined via small-amplitude oscillatory shear were observed to be functions of the P3HT and o-IDTBR concentrations, the solvent used and the shearing conditions. Overall, the gels which formed upon quenching to sub-zero temperatures were found to be stable during small-amplitude oscillatory shear (linear viscoelastic range) but broke down even at the relatively low shear rates associated with steady torsional flows, suggesting that the shearing conditions used during the processing of gels of P3HT with small molecule acceptor blends can alter the gel structure and possibly affect the resulting active layer performance. △ Less

Submitted 23 September, 2020; originally announced September 2020.

Comments: 27 manuscript pages and 19 figures

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions

Authors: Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, Pratik Ringshia, Kurt Shuster, Eric Michael Smith, Arthur Szlam, Jack Urbanek, Mary Williamson

Abstract: We present our view of what is necessary to build an engaging open-domain conversational agent: covering the qualities of such an agent, the pieces of the puzzle that have been built so far, and the gaping holes we have not filled yet. We present a biased view, focusing on work done by our own group, while citing related work in each area. In particular, we discuss in detail the properties of cont… ▽ More We present our view of what is necessary to build an engaging open-domain conversational agent: covering the qualities of such an agent, the pieces of the puzzle that have been built so far, and the gaping holes we have not filled yet. We present a biased view, focusing on work done by our own group, while citing related work in each area. In particular, we discuss in detail the properties of continual learning, providing engaging content, and being well-behaved -- and how to measure success in providing them. We end with a discussion of our experience and learnings, and our recommendations to the community. △ Less

Submitted 13 July, 2020; v1 submitted 22 June, 2020; originally announced June 2020.

Recipes for building an open-domain chatbot

Authors: Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston

Abstract: Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a… ▽ More Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models. △ Less

Submitted 30 April, 2020; v1 submitted 28 April, 2020; originally announced April 2020.

Direct Measurement of the Electronic Structure and band gap nature of atomic-layer-thick 2H-MoTe2

Authors: Wenjuan Zhao, Xieyu Zhou, Dayu Yan, Yuan Huang, Cong Li, Qiang Gao, Hongtao Rong, Yongqing Cai, Eike F. Schwier, Dianxing Ju, Cheng Shen, Yang Wang, Yu Xu, Wei Ji, Youguo Shi, Lin Zhao, Lihong Bao, Qingyan Wang, Kenya Shimada, Xutang Tao, Hongjun Gao, Zuyan Xu, Xingjiang Zhou, Guodong Liu

Abstract: The millimeter sized monolayer and bilayer 2H-MoTe2 single crystal samples are prepared by a new mechanical exfoliation method. Based on such high-quality samples, we report the first direct electronic structure study on them, using standard high resolution angle-resolved photoemission spectroscopy (ARPES). A direct band gap of 0.924eV is found at K in the rubidium-doped monolayer MoTe2. Similar v… ▽ More The millimeter sized monolayer and bilayer 2H-MoTe2 single crystal samples are prepared by a new mechanical exfoliation method. Based on such high-quality samples, we report the first direct electronic structure study on them, using standard high resolution angle-resolved photoemission spectroscopy (ARPES). A direct band gap of 0.924eV is found at K in the rubidium-doped monolayer MoTe2. Similar valence band alignment is also observed in bilayer MoTe2,supporting an assumption of a analogous direct gap semiconductor on it. Our measurements indicate a rather large band splitting of 212meV at the valence band maximum (VBM) in monolayer MoTe2, and the splitting is systematically enlarged with layer stacking, from monolayer to bilayer and to bulk. Meanwhile, our PBE band calculation on these materials show excellent agreement with ARPES results. Some fundamental electronic parameters are derived from the experimental and calculated electronic structures. Our findings lay a foundation for further application-related study on monolayer and bilayer MoTe2. △ Less

Submitted 16 January, 2020; originally announced January 2020.

Comments: 12 pages, 4 figures

All-in-One Image-Grounded Conversational Agents

Authors: Da Ju, Kurt Shuster, Y-Lan Boureau, Jason Weston

Abstract: As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an arch… ▽ More As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an architecture that combines state-of-the-art Transformer and ResNeXt modules fed into a novel attentive multimodal module to produce a combined model trained on many tasks. We provide a thorough analysis of the components of the model, and transfer performance when training on one, some, or all of the tasks. Our final models provide a single system that obtains good results on all vision and language tasks considered, and improves the state-of-the-art in image-grounded conversational applications. △ Less

Submitted 15 January, 2020; v1 submitted 27 December, 2019; originally announced December 2019.

The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents

Authors: Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, Jason Weston

Abstract: We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producin… ▽ More We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pre-trained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the fine-tune and task transfer settings. We obtain state-of-the-art results on many of the tasks, providing a strong baseline for this challenge. △ Less

Submitted 28 April, 2020; v1 submitted 9 November, 2019; originally announced November 2019.

Comments: ACL 2020

High-Level Strategy Selection under Partial Observability in StarCraft: Brood War

Authors: Jonas Gehring, Da Ju, Vegard Mella, Daniel Gant, Nicolas Usunier, Gabriel Synnaeve

Abstract: We consider the problem of high-level strategy selection in the adversarial setting of real-time strategy games from a reinforcement learning perspective, where taking an action corresponds to switching to the respective strategy. Here, a good strategy successfully counters the opponent's current and possible future strategies which can only be estimated using partial observations. We investigate… ▽ More We consider the problem of high-level strategy selection in the adversarial setting of real-time strategy games from a reinforcement learning perspective, where taking an action corresponds to switching to the respective strategy. Here, a good strategy successfully counters the opponent's current and possible future strategies which can only be estimated using partial observations. We investigate whether we can utilize the full game state information during training time (in the form of an auxiliary prediction task) to increase performance. Experiments carried out within a StarCraft: Brood War bot against strong community bots show substantial win rate improvements over a fixed-strategy baseline and encouraging results when learning with the auxiliary task. △ Less

Submitted 20 November, 2018; originally announced November 2018.

Aging effect in the BESIII drift chamber

Authors: M. Y. Dong, Q. L. Xiu, L. H. Wu, Z. Wu, Z. H. Qin, P. Shen, F. F. An, X. D. Ju, Y. Liu, K. Zhu, Q. Ouyang, Y. B. Chen

Abstract: As the main tracking detector of BESIII, the drift chamber works for accurate measurements of the tracking and the momentum of the charged particles decayed from the reaction of BEPCII e+ and e-. After operation six years, the drift chamber is suffering from aging problems due to huge beam related background. The gains of the cells in the first ten layers experience an obvious decrease, reaching a… ▽ More As the main tracking detector of BESIII, the drift chamber works for accurate measurements of the tracking and the momentum of the charged particles decayed from the reaction of BEPCII e+ and e-. After operation six years, the drift chamber is suffering from aging problems due to huge beam related background. The gains of the cells in the first ten layers experience an obvious decrease, reaching a maximum of about 29% for the first layer cells. Two calculation methods for the gains change (Bhabha events and accumulated charges with 0.3% aging ratio for inner chamber cells) get almost the same results. For the Malter effect encountered by the inner drift chamber in Jan., 2012, about 0.2% water vapor was added to MDC gas mixture to solve this cathode aging problem. These results provide an important reference for MDC operation high voltage setting and the upgrade of the inner drift chamber. △ Less

Submitted 17 August, 2015; v1 submitted 18 April, 2015; originally announced April 2015.

Comments: 5 pages, 6 figures

ACM Class: J.2

A new inner drift chamber for BESIII MDC

Authors: M. Y. Dong, Z. H. Qin, X. Y. Ma, J. Zhang, J. Dong, W. Xie, Q. L. Xiu, X. D. Ju, R. G. Liu, Q. Ouyang

Abstract: Due to the beam related background, the inner chamber of BESIII MDC has aging effect after 5 years running. The gains of the inner chamber cells decrease obviously, and the max gain decrease is about 26% for the first layer cells. A new inner drift chamber with eight stereo sense wire layers as a backup for MDC is under construction, which is almost the same as the current one but using stepped en… ▽ More Due to the beam related background, the inner chamber of BESIII MDC has aging effect after 5 years running. The gains of the inner chamber cells decrease obviously, and the max gain decrease is about 26% for the first layer cells. A new inner drift chamber with eight stereo sense wire layers as a backup for MDC is under construction, which is almost the same as the current one but using stepped endplates to shorten the wire length beyond the effective solid angle. This new structure will be of benefit to reducing the counting rate of single cell. The manufacture of each component is going smoothly, and the new inner drift chamber will be finished by the end of April 2014. △ Less

Submitted 7 March, 2014; originally announced March 2014.

Comments: 4 pages, 4 figures, proceedings for the XXXIII international symposium on Physics in Collision (PIC2013)