-
A generalisation of the signal-to-noise ratio using proper scoring rules
Authors:
Jochen Bröcker,
Eviatar Bach
Abstract:
A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of thes…
▽ More
A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts.
The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The latter have previously been interpreted as having a signal-to-noise "paradox", or anomalous signal-to-noise ratio, using the RPC statistic.
For the synthetic data, the RPC statistic as well as the scoring rule-based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.
△ Less
Submitted 27 March, 2026; v1 submitted 23 October, 2025;
originally announced October 2025.
-
Learning Enhanced Ensemble Filters
Authors:
Eviatar Bach,
Ricardo Baptista,
Edoardo Calvello,
Bohan Chen,
Andrew Stuart
Abstract:
The filtering distribution in hidden Markov models evolves according to the law of a mean-field model in state-observation space. The ensemble Kalman filter (EnKF) approximates this mean-field model with an ensemble of interacting particles, employing a Gaussian ansatz for the joint distribution of the state and observation at each observation time. These methods are robust, but the Gaussian ansat…
▽ More
The filtering distribution in hidden Markov models evolves according to the law of a mean-field model in state-observation space. The ensemble Kalman filter (EnKF) approximates this mean-field model with an ensemble of interacting particles, employing a Gaussian ansatz for the joint distribution of the state and observation at each observation time. These methods are robust, but the Gaussian ansatz limits accuracy. Here this shortcoming is addressed by using machine learning to map the joint predicted state and observation to the updated state estimate. The derivation of methods from a mean field formulation of the true filtering distribution suggests a single parametrization of the algorithm that can be deployed at different ensemble sizes. And we use a mean field formulation of the ensemble Kalman filter as an inductive bias for our architecture.
To develop this perspective, in which the mean-field limit of the algorithm and finite interacting ensemble particle approximations share a common set of parameters, a novel form of neural operator is introduced, taking probability distributions as input: a measure neural mapping (MNM). A MNM is used to design a novel approach to filtering, the MNM-enhanced ensemble filter (MNMEF), which is defined in both the mean-field limit and for interacting ensemble particle approximations. The ensemble approach uses empirical measures as input to the MNM and is implemented using the set transformer, which is invariant to ensemble permutation and allows for different ensemble sizes. In practice fine-tuning of a small number of parameters, for specific ensemble sizes, further enhances the accuracy of the scheme. The promise of the approach is demonstrated by its superior root-mean-square-error performance relative to leading methods in filtering the Lorenz '96 and Kuramoto-Sivashinsky models.
△ Less
Submitted 23 December, 2025; v1 submitted 24 April, 2025;
originally announced April 2025.
-
Forecast error growth: A dynamic-stochastic model
Authors:
Eviatar Bach,
Dan Crisan,
Michael Ghil
Abstract:
There is a history of simple forecast error growth models designed to capture the key properties of error growth in operational numerical weather prediction (NWP) models. We propose here such a scalar model that relies on the previous ones and incorporates multiplicative noise in a nonlinear stochastic differential equation (SDE). We analyze the properties of this SDE, including the shape of the e…
▽ More
There is a history of simple forecast error growth models designed to capture the key properties of error growth in operational numerical weather prediction (NWP) models. We propose here such a scalar model that relies on the previous ones and incorporates multiplicative noise in a nonlinear stochastic differential equation (SDE). We analyze the properties of this SDE, including the shape of the error growth curve for small times and its stationary distribution, and prove well-posedness and positivity of solutions. Next, we fit this model to operational NWP error growth curves, and show good agreement with both the mean and probabilistic features of the error growth. These results suggest that the dynamic-stochastic error growth model proposed herein and similar ones could play a role in many other areas of the sciences that involve prediction.
△ Less
Submitted 12 March, 2025; v1 submitted 10 November, 2024;
originally announced November 2024.
-
Evolution of the electrical characteristics of the ATLAS18 ITk strip sensors with HL-LHC radiation exposure range
Authors:
J. Fernandez-Tejero,
E. Bach,
V. Cindro,
V. Fadeyev,
P. Federicova,
C. Fleta,
S. Hirose,
J. Kroll,
I. Mandic,
K. Maeyama,
M. Mikestikova,
L. Poley,
B. Stelzer,
P. Tuma,
M. Ullan,
Y. Unno
Abstract:
The objective of the study is to evaluate the evolution of the performance of the new ATLAS Inner-Tracker (ITk) strip sensors as a function of radiation exposure, to ensure the proper operation of the upgraded detector during the lifetime of the High-Luminosity Large Hadron Collider (HL-LHC). Full-size ATLAS ITk Barrel Short-Strip (SS) sensors with final layout design, ATLAS18SS, have been irradia…
▽ More
The objective of the study is to evaluate the evolution of the performance of the new ATLAS Inner-Tracker (ITk) strip sensors as a function of radiation exposure, to ensure the proper operation of the upgraded detector during the lifetime of the High-Luminosity Large Hadron Collider (HL-LHC). Full-size ATLAS ITk Barrel Short-Strip (SS) sensors with final layout design, ATLAS18SS, have been irradiated with neutrons and gammas, to confirm the results obtained with prototypes and miniature sensors during the development phase. The irradiations cover a wide range of fluences and doses that ITk will experience, going from 1e13 neq/cm2 and 0.49 Mrad, to 1.6e15 neq/cm2 and 80 Mrad. The split irradiation enables a proper combination of fluence and dose values of the HL-LHC, including a 1.5 safety factor. A complete electrical characterization of the key sensor parameters before and after irradiation is presented, studying the leakage current, bulk capacitance, single-strip and inter-strip characteristics. The results confirm the fulfilment of the ATLAS specifications throughout the whole experiment. The study of a wide range of fluences and doses also allows to obtain detailed results, such as the frequency dependence of the bulk capacitance measurements for highly irradiated sensors, or the evolution of the punch-through protection and inter-strip resistance with radiation.
△ Less
Submitted 3 October, 2024;
originally announced October 2024.
-
Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems
Authors:
Ashesh Chattopadhyay,
Ebrahim Nabizadeh,
Eviatar Bach,
Pedram Hassanzadeh
Abstract:
Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. Howev…
▽ More
Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. However, EnKF requires evolving large ensembles of forecasts using the dynamical model of the system. This often becomes computationally intractable, especially when the number of states of the system is very large, e.g., for weather prediction. With small ensembles, the estimated background error covariance matrix in the EnKF algorithm suffers from sampling error, leading to an erroneous estimate of the analysis state (initial condition for the next forecast cycle). In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case. This framework utilizes a pre-trained deep learning-based data-driven surrogate that inexpensively generates and evolves a large data-driven ensemble of the states of the system to accurately compute the background error covariance matrix with less sampling error. The H-EnKF framework estimates a better initial condition without the need for any ad-hoc localization strategies. H-EnKF can be extended to any ensemble-based DA algorithm, e.g., particle filters, which are currently difficult to use for high dimensional systems.
△ Less
Submitted 9 June, 2022;
originally announced June 2022.
-
A multi-model ensemble Kalman filter for data assimilation and forecasting
Authors:
Eviatar Bach,
Michael Ghil
Abstract:
Data assimilation (DA) aims to optimally combine model forecasts and observations that are both partial and noisy. Multi-model DA generalizes the variational or Bayesian formulation of the Kalman filter, and we prove that it is also the minimum variance linear unbiased estimator. Here, we formulate and implement a multi-model ensemble Kalman filter (MM-EnKF) based on this framework. The MM-EnKF ca…
▽ More
Data assimilation (DA) aims to optimally combine model forecasts and observations that are both partial and noisy. Multi-model DA generalizes the variational or Bayesian formulation of the Kalman filter, and we prove that it is also the minimum variance linear unbiased estimator. Here, we formulate and implement a multi-model ensemble Kalman filter (MM-EnKF) based on this framework. The MM-EnKF can combine multiple model ensembles for both DA and forecasting in a flow-dependent manner; it uses adaptive model error estimation to provide matrix-valued weights for the separate models and the observations. We apply this methodology to various situations using the Lorenz96 model for illustration purposes. Our numerical experiments include multiple models with parametric error, different resolved scales, and different fidelities. The MM-EnKF results in significant error reductions compared to the best model, as well as to an unweighted multi-model ensemble, with respect to both probabilistic and deterministic error metrics.
△ Less
Submitted 18 January, 2023; v1 submitted 4 February, 2022;
originally announced February 2022.
-
Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers
Authors:
Ashesh Chattopadhyay,
Mustafa Mustafa,
Pedram Hassanzadeh,
Eviatar Bach,
Karthik Kashinath
Abstract:
There is growing interest in data-driven weather prediction (DDWP), for example using convolutional neural networks such as U-NETs that are trained on data from models or reanalysis. Here, we propose 3 components to integrate with commonly used DDWP models in order to improve their physical consistency and forecast accuracy. These components are 1) a deep spatial transformer added to the latent sp…
▽ More
There is growing interest in data-driven weather prediction (DDWP), for example using convolutional neural networks such as U-NETs that are trained on data from models or reanalysis. Here, we propose 3 components to integrate with commonly used DDWP models in order to improve their physical consistency and forecast accuracy. These components are 1) a deep spatial transformer added to the latent space of the U-NETs to preserve a property called equivariance, which is related to correctly capturing rotations and scalings of features in spatio-temporal data, 2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and 3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit/feasibility of each component, we use geopotential height at 500~hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the equivariance-preserving networks (U-STNs) clearly outperform the U-NETs, for example improving the forecast skill by $45\%$. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. The DDWP+DA framework substantially benefits from large ($O(1000)$) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promises, e.g., it reduces the average error by factors of 2-3.
△ Less
Submitted 16 March, 2021;
originally announced March 2021.
-
Proof that the Kalman gain minimizes the generalized variance
Authors:
Eviatar Bach
Abstract:
The optimal gain matrix of the Kalman filter is often derived by minimizing the trace of the posterior covariance matrix. Here, I show that the Kalman gain also minimizes the determinant of the covariance matrix, a quantity known as the generalized variance. When the error distributions are Gaussian, the differential entropy is also minimized.
The optimal gain matrix of the Kalman filter is often derived by minimizing the trace of the posterior covariance matrix. Here, I show that the Kalman gain also minimizes the determinant of the covariance matrix, a quantity known as the generalized variance. When the error distributions are Gaussian, the differential entropy is also minimized.
△ Less
Submitted 10 March, 2021;
originally announced March 2021.
-
Power law scaling for the adiabatic algorithm for search engine ranking
Authors:
Adam Frees,
John King Gamble,
Kenneth Rudinger,
Eric Bach,
Mark Friesen,
Robert Joynt,
S. N. Coppersmith
Abstract:
An important method for search engine result ranking works by finding the principal eigenvector of the "Google matrix." Recently, a quantum algorithm for preparing this eigenvector and evidence of an exponential speedup for some scale-free networks were presented. Here, we show that the run-time depends on features of the graphs other than the degree distribution, and can be altered sufficiently t…
▽ More
An important method for search engine result ranking works by finding the principal eigenvector of the "Google matrix." Recently, a quantum algorithm for preparing this eigenvector and evidence of an exponential speedup for some scale-free networks were presented. Here, we show that the run-time depends on features of the graphs other than the degree distribution, and can be altered sufficiently to rule out a general exponential speedup. For a sample of graphs with degree distributions that more closely resemble the Web than in previous work, the proposed algorithm for eigenvector preparation does not appear to run exponentially faster than the classical case.
△ Less
Submitted 5 December, 2012; v1 submitted 9 November, 2012;
originally announced November 2012.