-
Fast MoE Inference via Predictive Prefetching and Expert Replication
Authors:
Ankit Jyothish,
Ali Jannesari,
Aishwarya Sarkar,
Joseph Zuber
Abstract:
The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple toke…
▽ More
The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple tokens waiting on the same experts for their computation which arises from sparsity of expert activation. To address these challenges, we propose a dynamic expert replication strategy that predicts which experts are likely to be overloaded and replicates them for upcoming batches of tokens. The replicated experts process batch tokens concurrently across layers, which leads to improved parallelism, shorter GPU idle time, and significantly faster inference. Experimental evaluations conducted on large-scale MoE models, including Switch-base-128 and Switch-base-256, demonstrate that our method achieves near-complete GPU utilization (approx 100%), leading to upto 3x improvement in inference speed while preserving approximately 90-95% of the performance of baseline architectures
△ Less
Submitted 12 May, 2026;
originally announced May 2026.
-
Enhanced Soups for Graph Neural Networks
Authors:
Joseph Zuber,
Aishwarya Sarkar,
Joseph Jennings,
Ali Jannesari
Abstract:
Graph Neural Networks (GNN) have demonstrated state-of-the-art performance in numerous scientific and high-performance computing (HPC) applications. Recent work suggests that "souping" (combining) individually trained GNNs into a single model can improve performance without increasing compute and memory costs during inference. However, existing souping algorithms are often slow and memory-intensiv…
▽ More
Graph Neural Networks (GNN) have demonstrated state-of-the-art performance in numerous scientific and high-performance computing (HPC) applications. Recent work suggests that "souping" (combining) individually trained GNNs into a single model can improve performance without increasing compute and memory costs during inference. However, existing souping algorithms are often slow and memory-intensive, which limits their scalability.
We introduce Learned Souping for GNNs, a gradient-descent-based souping strategy that substantially reduces time and memory overhead compared to existing methods. Our approach is evaluated across multiple Open Graph Benchmark (OGB) datasets and GNN architectures, achieving up to 1.2% accuracy improvement and 2.1X speedup. Additionally, we propose Partition Learned Souping, a novel partition-based variant of learned souping that significantly reduces memory usage. On the ogbn-products dataset with GraphSAGE, partition learned souping achieves a 24.5X speedup and a 76% memory reduction without compromising accuracy.
△ Less
Submitted 14 March, 2025;
originally announced March 2025.
-
Differentiable Folding for Nearest Neighbor Model Optimization
Authors:
Ryan K. Krueger,
Sharon Aviran,
David H. Mathews,
Jeffrey Zuber,
Max Ward
Abstract:
The Nearest Neighbor model is the $\textit{de facto}$ thermodynamic model of RNA secondary structure formation and is a cornerstone of RNA structure prediction and sequence design. The current functional form (Turner 2004) contains $\approx13,000$ underlying thermodynamic parameters, and fitting these to both experimental and structural data is computationally challenging. Here, we leverage recent…
▽ More
The Nearest Neighbor model is the $\textit{de facto}$ thermodynamic model of RNA secondary structure formation and is a cornerstone of RNA structure prediction and sequence design. The current functional form (Turner 2004) contains $\approx13,000$ underlying thermodynamic parameters, and fitting these to both experimental and structural data is computationally challenging. Here, we leverage recent advances in $\textit{differentiable folding}$, a method for directly computing gradients of the RNA folding algorithms, to devise an efficient, scalable, and flexible means of parameter optimization that uses known RNA structures and thermodynamic experiments. Our method yields a significantly improved parameter set that outperforms existing baselines on all metrics, including an increase in the average predicted probability of ground-truth sequence-structure pairs for a single RNA family by over 23 orders of magnitude. Our framework provides a path towards drastically improved RNA models, enabling the flexible incorporation of new experimental data, definition of novel loss terms, large training sets, and even treatment as a module in larger deep learning pipelines. We make available a new database, RNAometer, with experimentally-determined stabilities for small RNA model systems.
△ Less
Submitted 12 May, 2025; v1 submitted 12 March, 2025;
originally announced March 2025.
-
SATViz: Real-Time Visualization of Clausal Proofs
Authors:
Tim Holzenkamp,
Kevin Kuryshev,
Thomas Oltmann,
Lucas Wäldele,
Johann Zuber,
Tobias Heuer,
Ashlin Iser
Abstract:
Visual layouts of graphs representing SAT instances can highlight the community structure of SAT instances. The community structure of SAT instances has been associated with both instance hardness and known clause quality heuristics. Our tool SATViz visualizes CNF formulas using the variable interaction graph and a force-directed layout algorithm. With SATViz, clause proofs can be animated to cont…
▽ More
Visual layouts of graphs representing SAT instances can highlight the community structure of SAT instances. The community structure of SAT instances has been associated with both instance hardness and known clause quality heuristics. Our tool SATViz visualizes CNF formulas using the variable interaction graph and a force-directed layout algorithm. With SATViz, clause proofs can be animated to continuously highlight variables that occur in a moving window of recently learned clauses. If needed, SATViz can also create new layouts of the variable interaction graph with the adjusted edge weights. In this paper, we describe the structure and feature set of SATViz. We also present some interesting visualizations created with SATViz.
△ Less
Submitted 31 July, 2026; v1 submitted 13 September, 2022;
originally announced September 2022.
-
Agent-based simulations for protecting nursing homes with prevention and vaccination strategies
Authors:
Jana Lasser,
Johannes Zuber,
Johannes Sorger,
Elma Dervic,
Katharina Ledebur,
Simon David Lindner,
Elisabeth Klager,
Maria Kletečka-Pulker,
Harald Willschke,
Katrin Stangl,
Sarah Stadtmann,
Christian Haslinger,
Peter Klimek,
Thomas Wochele-Thoma
Abstract:
Due to its high lethality amongst the elderly, the safety of nursing homes has been of central importance during the COVID-19 pandemic. With test procedures becoming available at scale, such as antigen or RT-LAMP tests, and increasing availability of vaccinations, nursing homes might be able to safely relax prohibitory measures while controlling the spread of infections (meaning an average of one…
▽ More
Due to its high lethality amongst the elderly, the safety of nursing homes has been of central importance during the COVID-19 pandemic. With test procedures becoming available at scale, such as antigen or RT-LAMP tests, and increasing availability of vaccinations, nursing homes might be able to safely relax prohibitory measures while controlling the spread of infections (meaning an average of one or less secondary infections per index case). Here, we develop a detailed agent-based epidemiological model for the spread of SARS-CoV-2 in nursing homes to identify optimal prevention strategies. The model is microscopically calibrated to high-resolution data from nursing homes in Austria, including detailed social contact networks and information on past outbreaks. We find that the effectiveness of mitigation testing depends critically on the timespan between test and test result, the detection threshold of the viral load for the test to give a positive result, and the screening frequencies of residents and employees. Under realistic conditions and in absence of an effective vaccine, we find that preventive screening of employees only might be sufficient to control outbreaks in nursing homes, provided that turnover times and detection thresholds of the tests are low enough. If vaccines that are moderately effective against infection and transmission are available, control is achieved if 80% or more of the inhabitants are vaccinated, even if no preventive testing is in place and residents are allowed to have visitors. Since these results strongly depend on vaccine efficacy against infection, retention of testing infrastructures, regular voluntary screening and sequencing of virus genomes is advised to enable early identification of new variants of concern.
△ Less
Submitted 14 June, 2021; v1 submitted 16 November, 2020;
originally announced April 2021.
-
Don't Rock the Boat: Algorithms for Balanced Dynamic Loading and Unloading
Authors:
Sándor P. Fekete,
Sven von Höveling,
Joseph S. B. Mitchell,
Christian Rieck,
Christian Scheffer,
Arne Schmidt,
James R. Zuber
Abstract:
We consider dynamic loading and unloading problems for heavy geometric objects. The challenge is to maintain balanced configurations at all times: minimize the maximal motion of the overall center of gravity. While this problem has been studied from an algorithmic point of view, previous work only focuses on balancing the final center of gravity; we give a variety of results for computing balanced…
▽ More
We consider dynamic loading and unloading problems for heavy geometric objects. The challenge is to maintain balanced configurations at all times: minimize the maximal motion of the overall center of gravity. While this problem has been studied from an algorithmic point of view, previous work only focuses on balancing the final center of gravity; we give a variety of results for computing balanced loading and unloading schemes that minimize the maximal motion of the center of gravity during the entire process. In particular, we consider the one-dimensional case and distinguish between loading and unloading. In the unloading variant, the positions of the intervals are given, and we search for an optimal unloading order of the intervals. We prove that the unloading variant is NP-complete and give a 2.7-approximation algorithm. In the loading variant, we have to compute both the positions of the intervals and their loading order. We give optimal approaches for several variants that model different loading scenarios that may arise, e.g., in the loading of a transport ship with containers.
△ Less
Submitted 17 January, 2018; v1 submitted 18 December, 2017;
originally announced December 2017.