Showing 1–15 of 15 results for author: Wilson, J D

Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

Authors: Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, Indranil Das, Junyi Cao, Yufeng Du, Jiabin Yu, Peixue Wu, Jinchen He, Yifan Su, Yikun Jiang, Yujie Zhang, Chang Liu, Ze-Min Huang, Weizhen Jia, Yunkai Wang, Farshid Jafarpour, Yong Zhao , et al. (39 additional authors not shown)

Abstract: While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integr… ▽ More While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools. △ Less

Submitted 20 November, 2025; v1 submitted 30 September, 2025; originally announced September 2025.

Comments: 39 pages, 6 figures, 6 tables

Reductive Quantum Phase Estimation

Authors: Nicholas J. C. Papadopoulos, Jarrod T. Reilly, John Drew Wilson, Murray J. Holland

Abstract: Estimating a quantum phase is a necessary task in a wide range of fields of quantum science. To accomplish this task, two well-known methods have been developed in distinct contexts, namely, Ramsey interferometry (RI) in atomic and molecular physics and quantum phase estimation (QPE) in quantum computing. We demonstrate that these canonical examples are instances of a larger class of phase estimat… ▽ More Estimating a quantum phase is a necessary task in a wide range of fields of quantum science. To accomplish this task, two well-known methods have been developed in distinct contexts, namely, Ramsey interferometry (RI) in atomic and molecular physics and quantum phase estimation (QPE) in quantum computing. We demonstrate that these canonical examples are instances of a larger class of phase estimation protocols, which we call reductive quantum phase estimation (RQPE) circuits. Here we present an explicit algorithm that allows one to create an RQPE circuit. This circuit distinguishes an arbitrary set of phases with a fewer number of qubits and unitary applications, thereby solving a general class of quantum hypothesis testing to which RI and QPE belong. We further demonstrate a trade-off between measurement precision and phase distinguishability, which allows one to tune the circuit to be optimal for a specific application. △ Less

Submitted 11 July, 2024; v1 submitted 6 February, 2024; originally announced February 2024.

Comments: 12 pages, 6 figures

ECoHeN: A Hypothesis Testing Framework for Extracting Communities from Heterogeneous Networks

Authors: Connor P. Gibbs, Bailey K. Fosdick, James D. Wilson

Abstract: Community discovery is the general process of attaining assortative communities from a network: collections of nodes that are densely connected within yet sparsely connected to the rest of the network. While community discovery has been well studied, few such techniques exist for heterogeneous networks, which contain different types of nodes and possibly different connectivity patterns between the… ▽ More Community discovery is the general process of attaining assortative communities from a network: collections of nodes that are densely connected within yet sparsely connected to the rest of the network. While community discovery has been well studied, few such techniques exist for heterogeneous networks, which contain different types of nodes and possibly different connectivity patterns between the node types. In this paper, we introduce a framework called ECoHeN, which \textbf{e}xtracts \textbf{co}mmunities from a \textbf{he}terogeneous \textbf{n}etwork in a statistically meaningful way. Using a heterogeneous configuration model as a reference distribution, ECoHeN identifies communities that are significantly more densely connected than expected given the node types and connectivity of its membership. Specifically, the ECoHeN algorithm extracts communities one at a time through a dynamic set of iterative updating rules, is guaranteed to converge, and imposes no constraints on the type composition of extracted communities. To our knowledge this is the first discovery method that distinguishes and identifies both homogeneous and heterogeneous, possibly overlapping, community structure in a network. We demonstrate the performance of ECoHeN through simulation and in application to a political blogs network to identify collections of blogs which reference one another more than expected considering the ideology of its' members. △ Less

Submitted 20 December, 2022; originally announced December 2022.

Comments: 31 pages, 8 figures, 1 table, 1 algorithm

Interpretable Network Representation Learning with Principal Component Analysis

Authors: James D. Wilson, Jihui Lee

Abstract: We consider the problem of interpretable network representation learning for samples of network-valued data. We propose the Principal Component Analysis for Networks (PCAN) algorithm to identify statistically meaningful low-dimensional representations of a network sample via subgraph count statistics. The PCAN procedure provides an interpretable framework for which one can readily visualize, explo… ▽ More We consider the problem of interpretable network representation learning for samples of network-valued data. We propose the Principal Component Analysis for Networks (PCAN) algorithm to identify statistically meaningful low-dimensional representations of a network sample via subgraph count statistics. The PCAN procedure provides an interpretable framework for which one can readily visualize, explore, and formulate predictive models for network samples. We furthermore introduce a fast sampling-based algorithm, sPCAN, which is significantly more computationally efficient than its counterpart, but still enjoys advantages of interpretability. We investigate the relationship between these two methods and analyze their large-sample properties under the common regime where the sample of networks is a collection of kernel-based random graphs. We show that under this regime, the embeddings of the sPCAN method enjoy a central limit theorem and moreover that the population level embeddings of PCAN and sPCAN are equivalent. We assess PCAN's ability to visualize, cluster, and classify observations in network samples arising in nature, including functional connectivity network samples and dynamic networks describing the political co-voting habits of the U.S. Senate. Our analyses reveal that our proposed algorithm provides informative and discriminatory features describing the networks in each sample. The PCAN and sPCAN methods build on the current literature of network representation learning and set the stage for a new line of research in interpretable learning on network-valued data. Publicly available software for the PCAN and sPCAN methods are available at https://www.github.com/jihuilee/. △ Less

Submitted 27 June, 2021; originally announced June 2021.

Comments: 33 pages. Submitted and currently under review

Nonparametric Feature Impact and Importance

Authors: Terence Parr, James D. Wilson, Jeff Hamrick

Abstract: Practitioners use feature importance to rank and eliminate weak predictors during model development in an effort to simplify models and improve generality. Unfortunately, they also routinely conflate such feature importance measures with feature impact, the isolated effect of an explanatory variable on the response variable. This can lead to real-world consequences when importance is inappropriate… ▽ More Practitioners use feature importance to rank and eliminate weak predictors during model development in an effort to simplify models and improve generality. Unfortunately, they also routinely conflate such feature importance measures with feature impact, the isolated effect of an explanatory variable on the response variable. This can lead to real-world consequences when importance is inappropriately interpreted as impact for business or medical insight purposes. The dominant approach for computing importances is through interrogation of a fitted model, which works well for feature selection, but gives distorted measures of feature impact. The same method applied to the same data set can yield different feature importances, depending on the model, leading us to conclude that impact should be computed directly from the data. While there are nonparametric feature selection algorithms, they typically provide feature rankings, rather than measures of impact or importance. They also typically focus on single-variable associations with the response. In this paper, we give mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data. To assess quality, we show that features ranked by these definitions are competitive with existing feature selection techniques using three real data sets for predictive tasks. △ Less

Submitted 8 June, 2020; originally announced June 2020.

Technical Report: Partial Dependence through Stratification

Authors: Terence Parr, James D. Wilson

Abstract: Partial dependence curves (FPD) introduced by Friedman, are an important model interpretation tool, but are often not accessible to business analysts and scientists who typically lack the skills to choose, tune, and assess machine learning models. It is also common for the same partial dependence algorithm on the same data to give meaningfully different curves for different models, which calls int… ▽ More Partial dependence curves (FPD) introduced by Friedman, are an important model interpretation tool, but are often not accessible to business analysts and scientists who typically lack the skills to choose, tune, and assess machine learning models. It is also common for the same partial dependence algorithm on the same data to give meaningfully different curves for different models, which calls into question their precision. Expertise is required to distinguish between model artifacts and true relationships in the data. In this paper, we contribute methods for computing partial dependence curves, for both numerical (StratPD) and categorical explanatory variables (CatStratPD), that work directly from training data rather than predictions of a model. Our methods provide a direct estimate of partial dependence, and rely on approximating the partial derivative of an unknown regression function without first fitting a model and then approximating its partial derivative. We investigate settings where contemporary partial dependence methods---including FPD, ALE, and SHAP methods---give biased results. Furthermore, we demonstrate that our approach works correctly on synthetic and plausibly on real data sets. Our goal is not to argue that model-based techniques are not useful. Rather, we hope to open a new line of inquiry into nonparametric partial dependence. △ Less

Submitted 24 April, 2020; v1 submitted 15 July, 2019; originally announced July 2019.

Comments: Tweaks/clarifications, added ntrials optional hyper parameter, corrected interpretation of Integrated Gradients technique. For code, see repo https://github.com/parrt/stratx

Monitoring dynamic networks: a simulation-based strategy for comparing monitoring methods and a comparative study

Authors: Lisha Yu, Inez M. Zwetsloot, Nathaniel T. Stevens, James D. Wilson, Kwok Leung Tsui

Abstract: Recently there has been a lot of interest in monitoring and identifying changes in dynamic networks, which has led to the development of a variety of monitoring methods. Unfortunately, these methods have not been systematically compared; moreover, new methods are often designed for a specialized use case. In light of this, we propose the use of simulation to compare the performance of network moni… ▽ More Recently there has been a lot of interest in monitoring and identifying changes in dynamic networks, which has led to the development of a variety of monitoring methods. Unfortunately, these methods have not been systematically compared; moreover, new methods are often designed for a specialized use case. In light of this, we propose the use of simulation to compare the performance of network monitoring methods over a variety of dynamic network changes. Using our family of simulated dynamic networks, we compare the performance of several state-of-the-art social network monitoring methods in the literature. We compare their performance over a variety of types of change; we consider both increases in communication levels, node propensity change as well as changes in community structure. We show that there does not exist one method that is uniformly superior to the others; the best method depends on the context and the type of change one wishes to detect. As such, we conclude that a variety of methods is needed for network monitoring and that it is important to understand in which scenarios a given method is appropriate. △ Less

Submitted 24 May, 2019; originally announced May 2019.

Comments: 37 pages, 13 figures, 4 tables. Submitted for publication

Analysis of Population Functional Connectivity Data via Multilayer Network Embeddings

Authors: James D. Wilson, Melanie Baybay, Rishi Sankar, Paul Stillman, Abbie M. Popa

Abstract: Population analyses of functional connectivity have provided a rich understanding of how brain function differs across time, individual, and cognitive task. An important but challenging task in such population analyses is the identification of reliable features that describe the function of the brain, while accounting for individual heterogeneity. Our work is motivated by two particularly importan… ▽ More Population analyses of functional connectivity have provided a rich understanding of how brain function differs across time, individual, and cognitive task. An important but challenging task in such population analyses is the identification of reliable features that describe the function of the brain, while accounting for individual heterogeneity. Our work is motivated by two particularly important challenges in this area: first, how can one analyze functional connectivity data over populations of individuals, and second, how can one use these analyses to infer group similarities and differences. Motivated by these challenges, we model population connectivity data as a multilayer network and develop the multi-node2vec algorithm, an efficient and scalable embedding method that automatically learns continuous node feature representations from multilayer networks. We use multi-node2vec to analyze resting state fMRI scans over a group of 74 healthy individuals and 60 patients with schizophrenia. We demonstrate how multilayer network embeddings can be used to visualize, cluster, and classify functional regions of the brain for these individuals. We furthermore compare the multilayer network embeddings of the two groups. We identify significant differences between the groups in the default mode network and salience network - findings that are supported by the triple network model theory of cognitive organization. Our findings reveal that multi-node2vec is a powerful and reliable method for analyzing multilayer networks. △ Less

Submitted 17 August, 2020; v1 submitted 17 September, 2018; originally announced September 2018.

Comments: 5 figures, 3 tables. Accepted at Network Science. Data and code available at https://github.com/jdwilson4/multi-node2vec

The power of A/B testing under interference

Authors: James D. Wilson, David T. Uminsky

Abstract: In this paper, we address the fundamental statistical question: how can you assess the power of an A/B test when the units in the study are exposed to interference? This question is germane to many scientific and industrial practitioners that rely on A/B testing in environments where control over interference is limited. We begin by proving that interference has a measurable effect on its sensitiv… ▽ More In this paper, we address the fundamental statistical question: how can you assess the power of an A/B test when the units in the study are exposed to interference? This question is germane to many scientific and industrial practitioners that rely on A/B testing in environments where control over interference is limited. We begin by proving that interference has a measurable effect on its sensitivity, or power. We quantify the power of an A/B test of equality of means as a function of the number of exposed individuals under any interference mechanism. We further derive a central limit theorem for the number of exposed individuals under a simple Bernoulli switching interference mechanism. Based on these results, we develop a strategy to estimate the power of an A/B test when actors experience interference according to an observed network model. We demonstrate how to leverage this theory to estimate the power of an A/B test on units sharing any network relationship, and highlight the utility of our method on two applications - a Facebook friendship network as well as a large Twitter follower network. These results yield, for the first time, the capacity to understand how to design an A/B test to detect, with a specified confidence, a fixed measurable treatment effect when the A/B test is conducted under interference driven by networks. △ Less

Submitted 10 October, 2017; originally announced October 2017.

Comments: 14 pages

Topic supervised non-negative matrix factorization

Authors: Kelsey MacMillan, James D. Wilson

Abstract: Topic models have been extensively used to organize and interpret the contents of large, unstructured corpora of text documents. Although topic models often perform well on traditional training vs. test set evaluations, it is often the case that the results of a topic model do not align with human interpretation. This interpretability fallacy is largely due to the unsupervised nature of topic mode… ▽ More Topic models have been extensively used to organize and interpret the contents of large, unstructured corpora of text documents. Although topic models often perform well on traditional training vs. test set evaluations, it is often the case that the results of a topic model do not align with human interpretation. This interpretability fallacy is largely due to the unsupervised nature of topic models, which prohibits any user guidance on the results of a model. In this paper, we introduce a semi-supervised method called topic supervised non-negative matrix factorization (TS-NMF) that enables the user to provide labeled example documents to promote the discovery of more meaningful semantic structure of a corpus. In this way, the results of TS-NMF better match the intuition and desired labeling of the user. The core of TS-NMF relies on solving a non-convex optimization problem for which we derive an iterative algorithm that is shown to be monotonic and convergent to a local optimum. We demonstrate the practical utility of TS-NMF on the Reuters and PubMed corpora, and find that TS-NMF is especially useful for conceptual or broad topics, where topic key terms are not well understood. Although identifying an optimal latent structure for the data is not a primary objective of the proposed approach, we find that TS-NMF achieves higher weighted Jaccard similarity scores than the contemporary methods, (unsupervised) NMF and latent Dirichlet allocation, at supervision rates as low as 10% to 20%. △ Less

Submitted 2 July, 2017; v1 submitted 12 June, 2017; originally announced June 2017.

Community extraction in multilayer networks with heterogeneous community structure

Authors: James D. Wilson, John Palowitch, Shankar Bhamidi, Andrew B. Nobel

Abstract: Multilayer networks are a useful way to capture and model multiple, binary or weighted relationships among a fixed group of objects. While community detection has proven to be a useful exploratory technique for the analysis of single-layer networks, the development of community detection methods for multilayer networks is still in its infancy. We propose and investigate a procedure, called Multila… ▽ More Multilayer networks are a useful way to capture and model multiple, binary or weighted relationships among a fixed group of objects. While community detection has proven to be a useful exploratory technique for the analysis of single-layer networks, the development of community detection methods for multilayer networks is still in its infancy. We propose and investigate a procedure, called Multilayer Extraction, that identifies densely connected vertex-layer sets in multilayer networks. Multilayer Extraction makes use of a significance based score that quantifies the connectivity of an observed vertex-layer set through comparison with a fixed degree random graph model. Multilayer Extraction directly handles networks with heterogeneous layers where community structure may be different from layer to layer. The procedure can capture overlapping communities, as well as background vertex-layer pairs that do not belong to any community. We establish consistency of the vertex-layer set optimizer of our proposed multilayer score under the multilayer stochastic block model. We investigate the performance of Multilayer Extraction on three applications and a test bed of simulations. Our theoretical and numerical evaluations suggest that Multilayer Extraction is an effective exploratory tool for analyzing complex multilayer networks. Publicly available R software for Multilayer Extraction is available at https://github.com/jdwilson4/MultilayerExtraction. △ Less

Submitted 7 November, 2017; v1 submitted 20 October, 2016; originally announced October 2016.

Comments: 46 pages. Accepted at the Journal of Machine Learning Research (11/17)

Monitoring communication outbreaks among an unknown team of actors in dynamic networks

Authors: Ross Sparks, James D. Wilson

Abstract: This paper investigates the detection of communication outbreaks among a small team of actors in time-varying networks. We propose monitoring plans for known and unknown teams based on generalizations of the exponentially weighted moving average (EWMA) statistic. For unknown teams, we propose an efficient neighborhood-based search to estimate a collection of candidate teams. This procedure dramati… ▽ More This paper investigates the detection of communication outbreaks among a small team of actors in time-varying networks. We propose monitoring plans for known and unknown teams based on generalizations of the exponentially weighted moving average (EWMA) statistic. For unknown teams, we propose an efficient neighborhood-based search to estimate a collection of candidate teams. This procedure dramatically reduces the computational complexity of an exhaustive search. Our procedure consists of two steps: communication counts between actors are first smoothed using a multivariate EWMA strategy. Densely connected teams are identified as candidates using a neighborhood search approach. These candidate teams are then monitored using a surveillance plan derived from a generalized EWMA statistic. Monitoring plans are established for collaborative teams, teams with a dominant leader, as well as for global outbreaks. We consider weighted heterogeneous dynamic networks, where the expected communication count between each pair of actors is potentially different across pairs and time, as well as homogeneous networks, where the expected communication count is constant across time and actors. Our monitoring plans are evaluated on a test bed of simulated networks as well as on the U.S. Senate co-voting network, which models the Senate voting patterns from 1857 to 2015. Our analysis suggests that our surveillance strategies can efficiently detect relevant and significant changes in dynamic networks. △ Less

Submitted 29 June, 2016; originally announced June 2016.

Comments: 23 pages, 2 figures, Submitted

Modeling and detecting change in temporal networks via a dynamic degree corrected stochastic block model

Authors: James D. Wilson, Nathaniel T. Stevens, William H. Woodall

Abstract: In many applications it is of interest to identify anomalous behavior within a dynamic interacting system. Such anomalous interactions are reflected by structural changes in the network representation of the system. We propose and investigate the use of a dynamic version of the degree corrected stochastic block model (DCSBM) to model and monitor dynamic networks that undergo a significant structur… ▽ More In many applications it is of interest to identify anomalous behavior within a dynamic interacting system. Such anomalous interactions are reflected by structural changes in the network representation of the system. We propose and investigate the use of a dynamic version of the degree corrected stochastic block model (DCSBM) to model and monitor dynamic networks that undergo a significant structural change. We apply statistical process monitoring techniques to the estimated parameters of the DCSBM to identify significant structural changes in the network. Application of our surveillance strategy to the dynamic U.S. Senate co-voting network reveals that we are able to detect significant changes in the network that reflect both times of cohesion and times of polarization among Republican and Democratic party members. These findings provide valuable insight about the evolution of the bipartisan political system in the United States. Our analysis demonstrates that the dynamic DCSBM monitoring procedure effectively detects local and global structural changes in dynamic networks. The DCSBM approach is an example of a more general framework that combines parametric random graph models and statistical process monitoring techniques for network surveillance. △ Less

Submitted 30 November, 2016; v1 submitted 13 May, 2016; originally announced May 2016.

Comments: 27 pages, 7 figures, submitted

An overview and perspective on social network monitoring

Authors: William H. Woodall, Meng J. Zhao, Kamran Paynabar, Ross Sparks, James D. Wilson

Abstract: In this expository paper we give an overview of some statistical methods for the monitoring of social networks. We discuss the advantages and limitations of various methods as well as some relevant issues. One of our primary contributions is to give the relationships between network monitoring methods and monitoring methods in engineering statistics and public health surveillance. We encourage res… ▽ More In this expository paper we give an overview of some statistical methods for the monitoring of social networks. We discuss the advantages and limitations of various methods as well as some relevant issues. One of our primary contributions is to give the relationships between network monitoring methods and monitoring methods in engineering statistics and public health surveillance. We encourage researchers in the industrial process monitoring area to work on developing and comparing the performance of social network monitoring methods. We also discuss some of the issues in social network monitoring and give a number of research ideas. △ Less

Submitted 31 March, 2016; originally announced March 2016.

Comments: To appear, IIE Transactions

A testing based extraction algorithm for identifying significant communities in networks

Authors: James D. Wilson, Simi Wang, Peter J. Mucha, Shankar Bhamidi, Andrew B. Nobel

Abstract: A common and important problem arising in the study of networks is how to divide the vertices of a given network into one or more groups, called communities, in such a way that vertices of the same community are more interconnected than vertices belonging to different ones. We propose and investigate a testing based community detection procedure called Extraction of Statistically Significant Commu… ▽ More A common and important problem arising in the study of networks is how to divide the vertices of a given network into one or more groups, called communities, in such a way that vertices of the same community are more interconnected than vertices belonging to different ones. We propose and investigate a testing based community detection procedure called Extraction of Statistically Significant Communities (ESSC). The ESSC procedure is based on $p$-values for the strength of connection between a single vertex and a set of vertices under a reference distribution derived from a conditional configuration network model. The procedure automatically selects both the number of communities in the network and their size. Moreover, ESSC can handle overlapping communities and, unlike the majority of existing methods, identifies "background" vertices that do not belong to a well-defined community. The method has only one parameter, which controls the stringency of the hypothesis tests. We investigate the performance and potential use of ESSC and compare it with a number of existing methods, through a validation study using four real network data sets. In addition, we carry out a simulation study to assess the effectiveness of ESSC in networks with various types of community structure, including networks with overlapping communities and those with background vertices. These results suggest that ESSC is an effective exploratory tool for the discovery of relevant community structure in complex network systems. Data and software are available at \urlhttp://www.unc.edu/~jameswd/research.html. △ Less

Submitted 3 December, 2014; v1 submitted 4 August, 2013; originally announced August 2013.

Comments: Published in at http://dx.doi.org/10.1214/14-AOAS760 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOAS-AOAS760

Journal ref: Annals of Applied Statistics 2014, Vol. 8, No. 3, 1853-1891