Showing 1–50 of 77 results for author: Krahmer, F

KV Cache Compression Through the Lens of Transform Coding

Authors: Hannah Laus, Claudio Mayrink Verdun, Hao Wang, Flavio du Pin Calmon, Felix Krahmer

Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attentio… ▽ More The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attention mechanisms. We prove that, under a white-noise quantization model, the expected attention-aware distortion decomposes into additive key and value contributions that factor across tokens and channels. Building on transform coding and reverse water-filling, which are classical tools from signal processing and rate-distortion theory, we introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion. On Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, evaluated across LongBench, RULER, GSM8K, MMLU-Pro, and MATH-500, our method achieves near-lossless accuracy at approximately $5.8\times$ compression, whereas each baseline degrades in at least some settings. △ Less

Submitted 14 August, 2026; originally announced August 2026.

Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier

Authors: Berk Hayta, Hannah Laus, Simon Mittermaier, Felix Krahmer

Abstract: Real-world sensor-based learning systems require uncertainty estimation that is both reliable and computationally efficient. Evidential Deep Learning (EDL) provides single-pass uncertainty estimation by modeling the class probabilities via Dirichlet distributions, where the Dirichlet parameters are predicted by a learned neural network mapping. However, this approach can lead to computational chal… ▽ More Real-world sensor-based learning systems require uncertainty estimation that is both reliable and computationally efficient. Evidential Deep Learning (EDL) provides single-pass uncertainty estimation by modeling the class probabilities via Dirichlet distributions, where the Dirichlet parameters are predicted by a learned neural network mapping. However, this approach can lead to computational challenges, as Dirichlet expected objectives are more complex than standard supervised learning losses, complicating their analysis and implementation. We address this issue by approximating the objective of the first-order empirical risk minimization problem induced by EDL with a plug-in loss evaluated at the Dirichlet mean and show that, under mild assumptions, the approximation error decays with growing evidence for a broad class of loss functions, including mean-squared error and cross-entropy loss. As a special case, our analysis provides justification for the use of softmax in the context of uncertainty estimation, since under a particular evidence-to-Dirichlet mapping, our framework includes the standard softmax classifier. We validate the proposed simplified objectives on the Google Speech Commands dataset and show that they achieve predictive accuracy and selective prediction performance comparable to classical EDL, while being simpler to implement using standard deep learning losses and training pipelines. To the best of our knowledge, this empirical analysis is the first to obtain coverage-accuracy trade-offs for speech recognition tasks through EDL. △ Less

Submitted 21 May, 2026; originally announced May 2026.

On Trajectory-Based Stability Analysis for $1$-bit Sigma-Delta Quantization and its Application to the Second-Order Case

Authors: Rohan Joy, Felix Krahmer, Alessandro Lupoli

Abstract: A state-of-the-art strategy for digitally representing a bandlimited signal $f$ is $ΣΔ$ quantization. $ΣΔ$ quantization schemes choose a bit sequence $(q_n)$ representing the samples $(y_n)$ of $f$ sequentially based on a state sequence $(u_n)$ defined via a recurrence relation of the form \begin{equation*} u_n = (h*u)_n + y_n - q_n, \end{equation*} where $h_j = 0$ for $j\le 0.$ The effectivenes… ▽ More A state-of-the-art strategy for digitally representing a bandlimited signal $f$ is $ΣΔ$ quantization. $ΣΔ$ quantization schemes choose a bit sequence $(q_n)$ representing the samples $(y_n)$ of $f$ sequentially based on a state sequence $(u_n)$ defined via a recurrence relation of the form \begin{equation*} u_n = (h*u)_n + y_n - q_n, \end{equation*} where $h_j = 0$ for $j\le 0.$ The effectiveness of a quantization scheme crucially depends on the fact that it is stable, i.e. , the state variable remains uniformly bounded in a given class of signals. Thus, a common strategy is to choose $$q_n = \operatorname{sign}((h*u)_n + y_n).$$ It is well known that a sufficient condition for this quantization rule to induce stability is that $$ \|h\|_{\ell^1}+\|f\|_{\infty}\le 2.$$ At the same time, one empirically observes that this condition is conservative and stability holds significantly beyond this bound. In this paper, we address this gap by establishing the first stability guarantees beyond first order that outperform the $\ell^1$ based stability condition. In contrast to many previous approaches, our analysis describes the trajectories of the state variables rather than characterizing the invariant set, an approach that had previously been performed only in some specific example cases. This viewpoint has the main advantage that it makes it possible to treat longer filters, which are difficult to handle through invariant-set analysis because of the resulting high dimensionality. We apply our technique to second-order $ΣΔ$ schemes with sparse feedback filters as proposed by Günturk \cite{gunturk2003one}, showing that the filter length required to guarantee stability significantly improves from the length $O\left(\frac{1}{1-\|f\|_{\infty}}\right)$ needed to apply the $\ell^1$ based criterion to $O\left(\frac{1}{\sqrt{1-\|f\|_{\infty}}}\right)$. △ Less

Submitted 16 May, 2026; originally announced May 2026.

Low-Bit Quantization of Bandlimited Graph Signals via Iterative Methods

Authors: Felix Krahmer, He Lyu, Rayan Saab, Jinna Qian, Anna Veselovska, Rongrong Wang

Abstract: We study the quantization of real-valued bandlimited signals on graphs, focusing on low-bit representations. We propose iterative noise-shaping algorithms for quantization, including sampling approaches with and without vertex replacement. The methods leverage the spectral properties of the graph Laplacian and exploit graph incoherence to achieve high-fidelity approximations. Theoretical guarantee… ▽ More We study the quantization of real-valued bandlimited signals on graphs, focusing on low-bit representations. We propose iterative noise-shaping algorithms for quantization, including sampling approaches with and without vertex replacement. The methods leverage the spectral properties of the graph Laplacian and exploit graph incoherence to achieve high-fidelity approximations. Theoretical guarantees are provided for the random sampling method, and extensive numerical experiments on synthetic and real-world graphs illustrate the efficiency and robustness of the proposed schemes. △ Less

Submitted 26 January, 2026; originally announced January 2026.

Comments: 17 pages, 5 figures

MSC Class: 42C15; 94A12; 05C50; 94A29 ACM Class: G.1.2; E.4; G.2.2

Conformal Prediction for Multi-Source Detection on a Network

Authors: Xingchao Jian, Purui Zhang, Lan Tian, Feng Ji, Wenfei Liang, Wee Peng Tay, Bihan Wen, Felix Krahmer

Abstract: Detecting the origin of information or infection spread in networks is a fundamental challenge with applications in misinformation tracking, epidemiology, and beyond. We study the multi-source detection problem: given snapshot observations of node infection status on a graph, estimate the set of source nodes that initiated the propagation. Existing methods either lack statistical guarantees or are… ▽ More Detecting the origin of information or infection spread in networks is a fundamental challenge with applications in misinformation tracking, epidemiology, and beyond. We study the multi-source detection problem: given snapshot observations of node infection status on a graph, estimate the set of source nodes that initiated the propagation. Existing methods either lack statistical guarantees or are limited to specific diffusion models and assumptions. We propose a novel conformal prediction framework that provides statistically valid recall guarantees for source set detection, independent of the underlying diffusion process or data distribution. Our approach introduces principled score functions to quantify the alignment between predicted probabilities and true sources, and leverages a calibration set to construct prediction sets with user-specified recall and coverage levels. The method is applicable to both single- and multi-source scenarios, supports general network diffusion dynamics, and is computationally efficient for large graphs. Empirical results demonstrate that our method achieves rigorous coverage with competitive accuracy, outperforming existing baselines in both reliability and scalability.The code is available online. △ Less

Submitted 30 November, 2025; v1 submitted 11 November, 2025; originally announced November 2025.

Fast One-Pass Sparse Approximation of the Top Eigenvectors of Huge Approximately Low-Rank Matrices? Yes, $MAM^*$!

Authors: Edem Boahen, Simone Brugiapaglia, Hung-Hsu Chou, Mark Iwen, Felix Krahmer

Abstract: Motivated by applications such as sparse PCA, in this paper we present provably-accurate one-pass algorithms for the sparse approximation of the top eigenvectors of extremely massive matrices based on a single compact linear sketch. The resulting compressive-sensing-based approaches can approximate the leading eigenvectors of huge approximately low-rank matrices that are too large to store in memo… ▽ More Motivated by applications such as sparse PCA, in this paper we present provably-accurate one-pass algorithms for the sparse approximation of the top eigenvectors of extremely massive matrices based on a single compact linear sketch. The resulting compressive-sensing-based approaches can approximate the leading eigenvectors of huge approximately low-rank matrices that are too large to store in memory based on a single pass over its entries while utilizing a total memory footprint on the order of the much smaller desired sparse eigenvector approximations. Finally, the compressive sensing recovery algorithm itself (which takes the gathered compressive matrix measurements as input, and then outputs sparse approximations of its top eigenvectors) can also be formulated to run in a time which principally depends on the size of the sought sparse approximations, making its runtime sublinear in the size of the large matrix whose eigenvectors one aims to approximate. Preliminary experiments on huge matrices having $\sim 10^{16}$ entries illustrate the developed theory and demonstrate the practical potential of the proposed approach. △ Less

Submitted 4 May, 2026; v1 submitted 22 July, 2025; originally announced July 2025.

Comments: 42 pages, 12 figures. added new experimental section

Phasebook: A Survey of Selected Open Problems in Phase Retrieval

Authors: Marc Allain, Selin Aslan, Wim Coene, Sjoerd Dirksen, Jonathan Dong, Julien Flamant, Mark Iwen, Felix Krahmer, Tristan van Leeuwen, Oleh Melnyk, Andreas Menzel, Allard P. Mosk, Viktor Nikitin, Gerlind Plonka, Palina Salanevich, Matthias Wellershoff

Abstract: Phase retrieval is an inverse problem that, on one hand, is crucial in many applications across imaging and physics, and, on the other hand, leads to deep research questions in theoretical signal processing and applied harmonic analysis. This survey paper is an outcome of the recent workshop Phase Retrieval in Mathematics and Applications (PRiMA) (held on August 5--9 2024 at the Lorentz Center in… ▽ More Phase retrieval is an inverse problem that, on one hand, is crucial in many applications across imaging and physics, and, on the other hand, leads to deep research questions in theoretical signal processing and applied harmonic analysis. This survey paper is an outcome of the recent workshop Phase Retrieval in Mathematics and Applications (PRiMA) (held on August 5--9 2024 at the Lorentz Center in Leiden, The Netherlands) that brought together experts working on theoretical and practical aspects of the phase retrieval problem with the purpose to formulate and explore essential open problems in the field. △ Less

Submitted 21 May, 2025; originally announced May 2025.

Solving Inverse Problems with Deep Linear Neural Networks: Global Convergence Guarantees for Gradient Descent with Weight Decay

Authors: Hannah Laus, Suzanna Parkinson, Vasileios Charisopoulos, Felix Krahmer, Rebecca Willett

Abstract: Machine learning methods are commonly used to solve inverse problems, wherein an unknown signal must be estimated from few indirect measurements generated via a known acquisition procedure. In particular, neural networks perform well empirically but have limited theoretical guarantees. In this work, we study an underdetermined linear inverse problem that admits several possible solution operators… ▽ More Machine learning methods are commonly used to solve inverse problems, wherein an unknown signal must be estimated from few indirect measurements generated via a known acquisition procedure. In particular, neural networks perform well empirically but have limited theoretical guarantees. In this work, we study an underdetermined linear inverse problem that admits several possible solution operators that map measurements to estimates of the target signal. A standard remedy (e.g., in compressed sensing) for establishing the uniqueness of the solution mapping is to assume the existence of a latent low-dimensional structure in the source signal. We ask the following question: do deep linear neural networks adapt to unknown low-dimensional structure when trained by gradient descent with weight decay regularization? We prove that mildly overparameterized deep linear networks trained in this manner converge to an approximate solution mapping that accurately solves the inverse problem while implicitly encoding latent subspace structure. We show rigorously that deep linear networks trained with weight decay automatically adapt to latent subspace structure in the data under practical stepsize and weight initialization schemes. Our work highlights that regularization and overparameterization improve generalization, while overparameterization also accelerates convergence during training. △ Less

Submitted 4 December, 2025; v1 submitted 21 February, 2025; originally announced February 2025.

Spectral method for low-dose Poisson and Bernoulli phase retrieval

Authors: Sjoerd Dirksen, Felix Krahmer, Patricia Römer, Palina Salanevich

Abstract: We consider the problem of phaseless reconstruction from measurements with Poisson or Bernoulli distributed noise. This is of particular interest in biological imaging experiments where a low dose of radiation has to be used to mitigate potential damage of the specimen, resulting in low observed particle counts. We derive recovery guarantees for the spectral method for these noise models in the ca… ▽ More We consider the problem of phaseless reconstruction from measurements with Poisson or Bernoulli distributed noise. This is of particular interest in biological imaging experiments where a low dose of radiation has to be used to mitigate potential damage of the specimen, resulting in low observed particle counts. We derive recovery guarantees for the spectral method for these noise models in the case of Gaussian measurements. Our results give a quantitative insight in the trade-off between the employed radiation dose per measurement and the overall sampling complexity. △ Less

Submitted 18 February, 2025; originally announced February 2025.

Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent

Authors: Santhosh Karnik, Anna Veselovska, Mark Iwen, Felix Krahmer

Abstract: We provide a rigorous analysis of implicit regularization in an overparametrized tensor factorization problem beyond the lazy training regime. For matrix factorization problems, this phenomenon has been studied in a number of works. A particular challenge has been to design universal initialization strategies which provably lead to implicit regularization in gradient-descent methods. At the same t… ▽ More We provide a rigorous analysis of implicit regularization in an overparametrized tensor factorization problem beyond the lazy training regime. For matrix factorization problems, this phenomenon has been studied in a number of works. A particular challenge has been to design universal initialization strategies which provably lead to implicit regularization in gradient-descent methods. At the same time, it has been argued by Cohen et. al. 2016 that more general classes of neural networks can be captured by considering tensor factorizations. However, in the tensor case, implicit regularization has only been rigorously established for gradient flow or in the lazy training regime. In this paper, we prove the first tensor result of its kind for gradient descent rather than gradient flow. We focus on the tubal tensor product and the associated notion of low tubal rank, encouraged by the relevance of this model for image data. We establish that gradient descent in an overparametrized tensor factorization model with a small random initialization exhibits an implicit bias towards solutions of low tubal rank. Our theoretical findings are illustrated in an extensive set of numerical simulations show-casing the dynamics predicted by our theory as well as the crucial role of using a small random initialization. △ Less

Submitted 21 October, 2024; originally announced October 2024.

Comments: 58 pages, 4 figures

High-Dimensional Confidence Regions in Sparse MRI

Authors: Frederik Hoppe, Felix Krahmer, Claudio Mayrink Verdun, Marion Menzel, Holger Rauhut

Abstract: One of the most promising solutions for uncertainty quantification in high-dimensional statistics is the debiased LASSO that relies on unconstrained $\ell_1$-minimization. The initial works focused on real Gaussian designs as a toy model for this problem. However, in medical imaging applications, such as compressive sensing for MRI, the measurement system is represented by a (subsampled) complex F… ▽ More One of the most promising solutions for uncertainty quantification in high-dimensional statistics is the debiased LASSO that relies on unconstrained $\ell_1$-minimization. The initial works focused on real Gaussian designs as a toy model for this problem. However, in medical imaging applications, such as compressive sensing for MRI, the measurement system is represented by a (subsampled) complex Fourier matrix. The purpose of this work is to extend the method to the MRI case in order to construct confidence intervals for each pixel of an MR image. We show that a sufficient amount of data is $n \gtrsim \max\{ s_0\log^2 s_0\log p, s_0 \log^2 p \}$. △ Less

Submitted 18 July, 2024; originally announced July 2024.

Comments: Recognized with Best Student Paper Award at ICASSP 2023. arXiv admin note: substantial text overlap with arXiv:2212.14864

Non-Asymptotic Uncertainty Quantification in High-Dimensional Learning

Authors: Frederik Hoppe, Claudio Mayrink Verdun, Hannah Laus, Felix Krahmer, Holger Rauhut

Abstract: Uncertainty quantification (UQ) is a crucial but challenging task in many high-dimensional regression or learning problems to increase the confidence of a given predictor. We develop a new data-driven approach for UQ in regression that applies both to classical regression approaches such as the LASSO as well as to neural networks. One of the most notable UQ techniques is the debiased LASSO, which… ▽ More Uncertainty quantification (UQ) is a crucial but challenging task in many high-dimensional regression or learning problems to increase the confidence of a given predictor. We develop a new data-driven approach for UQ in regression that applies both to classical regression approaches such as the LASSO as well as to neural networks. One of the most notable UQ techniques is the debiased LASSO, which modifies the LASSO to allow for the construction of asymptotic confidence intervals by decomposing the estimation error into a Gaussian and an asymptotically vanishing bias component. However, in real-world problems with finite-dimensional data, the bias term is often too significant to be neglected, resulting in overly narrow confidence intervals. Our work rigorously addresses this issue and derives a data-driven adjustment that corrects the confidence intervals for a large class of predictors by estimating the means and variances of the bias terms from training data, exploiting high-dimensional concentration phenomena. This gives rise to non-asymptotic confidence intervals, which can help avoid overestimating uncertainty in critical applications such as MRI diagnosis. Importantly, our analysis extends beyond sparse regression to data-driven predictors like neural networks, enhancing the reliability of model-based deep learning. Our findings bridge the gap between established theory and the practical applicability of such debiased methods. △ Less

Submitted 18 July, 2024; originally announced July 2024.

With or Without Replacement? Improving Confidence in Fourier Imaging

Authors: Frederik Hoppe, Claudio Mayrink Verdun, Felix Krahmer, Marion I. Menzel, Holger Rauhut

Abstract: Over the last few years, debiased estimators have been proposed in order to establish rigorous confidence intervals for high-dimensional problems in machine learning and data science. The core argument is that the error of these estimators with respect to the ground truth can be expressed as a Gaussian variable plus a remainder term that vanishes as long as the dimension of the problem is sufficie… ▽ More Over the last few years, debiased estimators have been proposed in order to establish rigorous confidence intervals for high-dimensional problems in machine learning and data science. The core argument is that the error of these estimators with respect to the ground truth can be expressed as a Gaussian variable plus a remainder term that vanishes as long as the dimension of the problem is sufficiently high. Thus, uncertainty quantification (UQ) can be performed exploiting the Gaussian model. Empirically, however, the remainder term cannot be neglected in many realistic situations of moderately-sized dimensions, in particular in certain structured measurement scenarios such as Magnetic Resonance Imaging (MRI). This, in turn, can downgrade the advantage of the UQ methods as compared to non-UQ approaches such as the standard LASSO. In this paper, we present a method to improve the debiased estimator by sampling without replacement. Our approach leverages recent results of ours on the structure of the random nature of certain sampling schemes showing how a transition between sampling with and without replacement can lead to a weighted reconstruction scheme with improved performance for the standard LASSO. In this paper, we illustrate how this reweighted sampling idea can also improve the debiased estimator and, consequently, provide a better method for UQ in Fourier imaging. △ Less

Submitted 18 July, 2024; originally announced July 2024.

Comments: Accepted at Cosera 2024

The Mathematics of Dots and Pixels: On the Theoretical Foundations of Image Halftoning

Authors: Felix Krahmer, Anna Veselovska

Abstract: The evolution of image halftoning, from its analog roots to contemporary digital methodologies, encapsulates a fascinating journey marked by technological advancements and creative innovations. Yet the theoretical understanding of halftoning is much more recent. In this article, we explore various approaches towards shedding light on the design of halftoning approaches and why they work. We disc… ▽ More The evolution of image halftoning, from its analog roots to contemporary digital methodologies, encapsulates a fascinating journey marked by technological advancements and creative innovations. Yet the theoretical understanding of halftoning is much more recent. In this article, we explore various approaches towards shedding light on the design of halftoning approaches and why they work. We discuss both halftoning in a continuous domain and on a pixel grid. We start by reviewing the mathematical foundation of the so-called electrostatic halftoning method, which departed from the heuristic of considering the back dots of the halftoned image as charged particles attracted by the grey values of the image in combination with mutual repulsion. Such an attraction-repulsion model can be mathematically represented via an energy functional in a reproducing kernel Hilbert space allowing for a rigorous analysis of the resulting optimization problem as well as a convergence analysis in a suitable topology. A second class of methods that we discuss in detail is the class of error diffusion schemes, arguably among the most popular halftoning techniques due to their ability to work directly on a pixel grid and their ease of application. The main idea of these schemes is to choose the locations of the black pixels via a recurrence relation designed to agree with the image in terms of the local averages. We discuss some recent mathematical understanding of these methods that is based on a connection to Sigma-Delta quantizers, a popular class of algorithms for analog-to-digital conversion. △ Less

Submitted 18 June, 2024; originally announced June 2024.

Comments: 27 pages, 6 figures

MSC Class: 94A08; 62H35

Uncertainty quantification for learned ISTA

Authors: Frederik Hoppe, Claudio Mayrink Verdun, Felix Krahmer, Hannah Laus, Holger Rauhut

Abstract: Model-based deep learning solutions to inverse problems have attracted increasing attention in recent years as they bridge state-of-the-art numerical performance with interpretability. In addition, the incorporated prior domain knowledge can make the training more efficient as the smaller number of parameters allows the training step to be executed with smaller datasets. Algorithm unrolling scheme… ▽ More Model-based deep learning solutions to inverse problems have attracted increasing attention in recent years as they bridge state-of-the-art numerical performance with interpretability. In addition, the incorporated prior domain knowledge can make the training more efficient as the smaller number of parameters allows the training step to be executed with smaller datasets. Algorithm unrolling schemes stand out among these model-based learning techniques. Despite their rapid advancement and their close connection to traditional high-dimensional statistical methods, they lack certainty estimates and a theory for uncertainty quantification is still elusive. This work provides a step towards closing this gap proposing a rigorous way to obtain confidence intervals for the LISTA estimator. △ Less

Submitted 14 September, 2023; originally announced September 2023.

Comments: to appear at the 33rd IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2023)

Approximating Positive Homogeneous Functions with Scale Invariant Neural Networks

Authors: Stefan Bamberger, Reinhard Heckel, Felix Krahmer

Abstract: We investigate to what extent it is possible to solve linear inverse problems with $ReLu$ networks. Due to the scaling invariance arising from the linearity, an optimal reconstruction function $f$ for such a problem is positive homogeneous, i.e., satisfies $f(λx) = λf(x)$ for all non-negative $λ$. In a $ReLu$ network, this condition translates to considering networks without bias terms. We first c… ▽ More We investigate to what extent it is possible to solve linear inverse problems with $ReLu$ networks. Due to the scaling invariance arising from the linearity, an optimal reconstruction function $f$ for such a problem is positive homogeneous, i.e., satisfies $f(λx) = λf(x)$ for all non-negative $λ$. In a $ReLu$ network, this condition translates to considering networks without bias terms. We first consider recovery of sparse vectors from few linear measurements. We prove that $ReLu$- networks with only one hidden layer cannot even recover $1$-sparse vectors, not even approximately, and regardless of the width of the network. However, with two hidden layers, approximate recovery with arbitrary precision and arbitrary sparsity level $s$ is possible in a stable way. We then extend our results to a wider class of recovery problems including low-rank matrix recovery and phase retrieval. Furthermore, we also consider the approximation of general positive homogeneous functions with neural networks. Extending previous work, we establish new results explaining under which conditions such functions can be approximated with neural networks. Our results also shed some light on the seeming contradiction between previous works showing that neural networks for inverse problems typically have very large Lipschitz constants, but still perform very well also for adversarial noise. Namely, the error bounds in our expressivity results include a combination of a small constant term and a term that is linear in the noise level, indicating that robustness issues may occur only for very small noise levels. △ Less

Submitted 5 August, 2023; originally announced August 2023.

Comments: 31 pages

MSC Class: 41A30; 68T07

On the reconstruction of bandlimited signals from random samples quantized via noise-shaping

Authors: Rohan Joy, Felix Krahmer, Alessandro Lupoli, Radha Ramakrishnan

Abstract: Noise-shaping quantization techniques are widely used for converting bandlimited signals from the analog to the digital domain. They work by ``shaping" the quantization noise so that it falls close to the reconstruction operator's null space. We investigate the compatibility of two such schemes, specifically $ΣΔ$ quantization and distributed noise-shaping quantization, with random samples of bandl… ▽ More Noise-shaping quantization techniques are widely used for converting bandlimited signals from the analog to the digital domain. They work by ``shaping" the quantization noise so that it falls close to the reconstruction operator's null space. We investigate the compatibility of two such schemes, specifically $ΣΔ$ quantization and distributed noise-shaping quantization, with random samples of bandlimited functions. Suppose $R>1$ is a real number and assume that $\{x_i\}_{i=1}^m$ is a sequence of i.i.d random variables uniformly distributed on $[-\tilde{R},\tilde{R}]$, where $\tilde{R}>R$ is appropriately chosen. We show that by using a noise-shaping quantizer to quantize the (randomly sign flipped) values of a real-valued $π$-bandlimited function $f$ at $\{x_i\}_{i=1}^m$, a function $f^{\sharp}$ can be reconstructed from these quantized values such that $\|f-f^{\sharp}\|_{L^2[-R, R]}$ decays with high probability as $m$ and $\tilde{R}$ increase. This decay holds uniformly over all bandlimited $f$. We emphasize that the sample points $\{x_i\}_{i=1}^m$ are completely random, that is, they have no predefined structure, which makes our findings the first of their kind. △ Less

Submitted 20 May, 2026; v1 submitted 27 June, 2023; originally announced June 2023.

MSC Class: 94A20; 94A12; 42C15; 41A29

How robust is randomized blind deconvolution via nuclear norm minimization against adversarial noise?

Authors: Julia Kostin, Felix Krahmer, Dominik Stöger

Abstract: In this paper, we study the problem of recovering two unknown signals from their convolution, which is commonly referred to as blind deconvolution. Reformulation of blind deconvolution as a low-rank recovery problem has led to multiple theoretical recovery guarantees in the past decade due to the success of the nuclear norm minimization heuristic. In particular, in the absence of noise, exact reco… ▽ More In this paper, we study the problem of recovering two unknown signals from their convolution, which is commonly referred to as blind deconvolution. Reformulation of blind deconvolution as a low-rank recovery problem has led to multiple theoretical recovery guarantees in the past decade due to the success of the nuclear norm minimization heuristic. In particular, in the absence of noise, exact recovery has been established for sufficiently incoherent signals contained in lower-dimensional subspaces. However, if the convolution is corrupted by additive bounded noise, the stability of the recovery problem remains much less understood. In particular, existing reconstruction bounds involve large dimension factors and therefore fail to explain the empirical evidence for dimension-independent robustness of nuclear norm minimization. Recently, theoretical evidence has emerged for ill-posed behavior of low-rank matrix recovery for sufficiently small noise levels. In this work, we develop improved recovery guarantees for blind deconvolution with adversarial noise which exhibit square-root scaling in the noise level. Hence, our results are consistent with existing counterexamples which speak against linear scaling in the noise level as demonstrated for related low-rank matrix recovery problems. △ Less

Submitted 17 March, 2023; originally announced March 2023.

Uncertainty quantification for sparse Fourier recovery

Authors: Frederik Hoppe, Felix Krahmer, Claudio Mayrink Verdun, Marion I. Menzel, Holger Rauhut

Abstract: One of the most prominent methods for uncertainty quantification in high-dimen-sional statistics is the desparsified LASSO that relies on unconstrained $\ell_1$-minimization. The majority of initial works focused on real (sub-)Gaussian designs. However, in many applications, such as magnetic resonance imaging (MRI), the measurement process possesses a certain structure due to the nature of the pro… ▽ More One of the most prominent methods for uncertainty quantification in high-dimen-sional statistics is the desparsified LASSO that relies on unconstrained $\ell_1$-minimization. The majority of initial works focused on real (sub-)Gaussian designs. However, in many applications, such as magnetic resonance imaging (MRI), the measurement process possesses a certain structure due to the nature of the problem. The measurement operator in MRI can be described by a subsampled Fourier matrix. The purpose of this work is to extend the uncertainty quantification process using the desparsified LASSO to design matrices originating from a bounded orthonormal system, which naturally generalizes the subsampled Fourier case and also allows for the treatment of the case where the sparsity basis is not the standard basis. In particular we construct honest confidence intervals for every pixel of an MR image that is sparse in the standard basis provided the number of measurements satisfies $n \gtrsim\max\{ s\log^2 s\log p, s \log^2 p \}$ or that is sparse with respect to the Haar Wavelet basis provided a slightly larger number of measurements. △ Less

Submitted 13 September, 2023; v1 submitted 30 December, 2022; originally announced December 2022.

Non-intrusive surrogate modelling using sparse random features with applications in crashworthiness analysis

Authors: Maternus Herold, Anna Veselovska, Jonas Jehle, Felix Krahmer

Abstract: Efficient surrogate modelling is a key requirement for uncertainty quantification in data-driven scenarios. In this work, a novel approach of using Sparse Random Features for surrogate modelling in combination with self-supervised dimensionality reduction is described. The method is compared to other methods on synthetic and real data obtained from crashworthiness analyses. The results show a supe… ▽ More Efficient surrogate modelling is a key requirement for uncertainty quantification in data-driven scenarios. In this work, a novel approach of using Sparse Random Features for surrogate modelling in combination with self-supervised dimensionality reduction is described. The method is compared to other methods on synthetic and real data obtained from crashworthiness analyses. The results show a superiority of the here described approach over state of the art surrogate modelling techniques, Polynomial Chaos Expansions and Neural Networks. △ Less

Submitted 29 December, 2022; originally announced December 2022.

Comments: 19 pages, 7 figures

Enhanced Digital Halftoning via Weighted Sigma-Delta Modulation

Authors: Felix Krahmer, Anna Veselovska

Abstract: In this paper, we study error diffusion techniques for digital halftoning from the perspective of 1-bit Sigma-Delta quantization. We introduce a method to generate Sigma-Delta schemes for two-dimensional signals as a weighted combination of its one-dimensional counterparts and show that various error diffusion schemes proposed in the literature can be represented in this framework via Sigma-Delta… ▽ More In this paper, we study error diffusion techniques for digital halftoning from the perspective of 1-bit Sigma-Delta quantization. We introduce a method to generate Sigma-Delta schemes for two-dimensional signals as a weighted combination of its one-dimensional counterparts and show that various error diffusion schemes proposed in the literature can be represented in this framework via Sigma-Delta schemes of first order. Under the model of two-dimensional bandlimited signals, which is motivated by a mathematical model of human visual perception, we derive quantitative error bounds for such weighted Sigma-Delta schemes. We see these bounds as a step towards a mathematical understanding of the good empirical performance of error diffusion, even though they are formulated in the supremum norm, which is known to not fully capture the visual similarity of images. Motivated by the correspondence between existing error diffusion algorithms and first-order Sigma-Delta schemes, we study the performance of the analogous weighted combinations of second-order Sigma-Delta schemes and show that they exhibit a superior performance in terms of guaranteed error decay for two-dimensional bandlimited signals. In extensive numerical simulations for real world images, we demonstrate that with some modifications to enhance stability this superior performance also translates to the problem of digital halftoning. More concretely, we find that certain second-order weighted Sigma-Delta schemes exhibit competitive performance for digital halftoning of real world images in terms of the Feature Similarity Index (FSIM), a state-of-the-art measure for image quality assessment. △ Less

Submitted 15 February, 2022; v1 submitted 10 February, 2022; originally announced February 2022.

Comments: 34 pages, 23 figures

The Surprising Benefits of Hysteresis in Unlimited Sampling: Theory, Algorithms and Experiments

Authors: Dorian Florescu, Felix Krahmer, Ayush Bhandari

Abstract: The Unlimited Sensing Framework (USF) was recently introduced to overcome the sensor saturation bottleneck in conventional digital acquisition systems. At its core, the USF allows for high-dynamic-range (HDR) signal reconstruction by converting a continuous-time signal into folded, low-dynamic-range (LDR), modulo samples. HDR reconstruction is then carried out by algorithmic unfolding of the folde… ▽ More The Unlimited Sensing Framework (USF) was recently introduced to overcome the sensor saturation bottleneck in conventional digital acquisition systems. At its core, the USF allows for high-dynamic-range (HDR) signal reconstruction by converting a continuous-time signal into folded, low-dynamic-range (LDR), modulo samples. HDR reconstruction is then carried out by algorithmic unfolding of the folded samples. In hardware, however, implementing an ideal modulo folding requires careful calibration, analog design and high precision. At the interface of theory and practice, this paper explores a computational sampling strategy that relaxes strict hardware requirements by compensating them via a novel, mathematically guaranteed recovery method. Our starting point is a generalized model for USF. The generalization relies on two new parameters modeling hysteresis and folding transients} in addition to the modulo threshold. Hysteresis accounts for the mismatch between the reset threshold and the amplitude displacement at the folding time and we refer to a continuous transition period in the implementation of a reset as folding transient. Both these effects are motivated by our hardware experiments and also occur in previous, domain-specific applications. We show that the effect of hysteresis is beneficial for the USF and we leverage it to derive the first recovery guarantees in the context of our generalized USF model. Additionally, we show how the proposed recovery can be directly generalized for the case of lower sampling rates. Our theoretical work is corroborated by hardware experiments that are based on a hysteresis enabled, modulo ADC testbed comprising off-the-shelf electronic components. Thus, by capitalizing on a collaboration between hardware and algorithms, our paper enables an end-to-end pipeline for HDR sampling allowing more flexible hardware implementations. △ Less

Submitted 24 November, 2021; originally announced November 2021.

Comments: 24 pages

Johnson-Lindenstrauss Embeddings with Kronecker Structure

Authors: Stefan Bamberger, Felix Krahmer, Rachel Ward

Abstract: We prove the Johnson-Lindenstrauss property for matrices $ΦD_ξ$ where $Φ$ has the restricted isometry property and $D_ξ$ is a diagonal matrix containing the entries of a Kronecker product $ξ= ξ^{(1)} \otimes \dots \otimes ξ^{(d)}$ of $d$ independent Rademacher vectors. Such embeddings have been proposed in recent works for a number of applications concerning compression of tensor structured data,… ▽ More We prove the Johnson-Lindenstrauss property for matrices $ΦD_ξ$ where $Φ$ has the restricted isometry property and $D_ξ$ is a diagonal matrix containing the entries of a Kronecker product $ξ= ξ^{(1)} \otimes \dots \otimes ξ^{(d)}$ of $d$ independent Rademacher vectors. Such embeddings have been proposed in recent works for a number of applications concerning compression of tensor structured data, including the oblivious sketching procedure by Ahle et al. for approximate tensor computations. For preserving the norms of $p$ points simultaneously, our result requires $Φ$ to have the restricted isometry property for sparsity $C(d) (\log p)^d$. In the case of subsampled Hadamard matrices, this can improve the dependence of the embedding dimension on $p$ to $(\log p)^d$ while the best previously known result required $(\log p)^{d + 1}$. That is, for the case of $d=2$ at the core of the oblivious sketching procedure by Ahle et al., the scaling improves from cubic to quadratic. We provide a counterexample to prove that the scaling established in our result is optimal under mild assumptions. △ Less

Submitted 24 June, 2021; originally announced June 2021.

MSC Class: 15A69; 68Q87

The Hanson-Wright Inequality for Random Tensors

Authors: Stefan Bamberger, Felix Krahmer, Rachel Ward

Abstract: We provide moment bounds for expressions of the type $(X^{(1)} \otimes \dots \otimes X^{(d)})^T A (X^{(1)} \otimes \dots \otimes X^{(d)})$ where $\otimes$ denotes the Kronecker product and $X^{(1)}, \dots, X^{(d)}$ are random vectors with independent, mean 0, variance 1, subgaussian entries. The bounds are tight up to constants depending on $d$ for the case of Gaussian random vectors. Our proof al… ▽ More We provide moment bounds for expressions of the type $(X^{(1)} \otimes \dots \otimes X^{(d)})^T A (X^{(1)} \otimes \dots \otimes X^{(d)})$ where $\otimes$ denotes the Kronecker product and $X^{(1)}, \dots, X^{(d)}$ are random vectors with independent, mean 0, variance 1, subgaussian entries. The bounds are tight up to constants depending on $d$ for the case of Gaussian random vectors. Our proof also provides a decoupling inequality for expressions of this type. Using these bounds, we obtain new, improved concentration inequalities for expressions of the form $\|B (X^{(1)} \otimes \dots \otimes X^{(d)})\|_2$. △ Less

Submitted 24 June, 2021; originally announced June 2021.

Proof methods for robust low-rank matrix recovery

Authors: Tim Fuchs, David Gross, Peter Jung, Felix Krahmer, Richard Kueng, Dominik Stöger

Abstract: Low-rank matrix recovery problems arise naturally as mathematical formulations of various inverse problems, such as matrix completion, blind deconvolution, and phase retrieval. Over the last two decades, a number of works have rigorously analyzed the reconstruction performance for such scenarios, giving rise to a rather general understanding of the potential and the limitations of low-rank matrix… ▽ More Low-rank matrix recovery problems arise naturally as mathematical formulations of various inverse problems, such as matrix completion, blind deconvolution, and phase retrieval. Over the last two decades, a number of works have rigorously analyzed the reconstruction performance for such scenarios, giving rise to a rather general understanding of the potential and the limitations of low-rank matrix models in sensing problems. In this article, we compare the two main proof techniques that have been paving the way to a rigorous analysis, discuss their potential and limitations, and survey their successful applications. On the one hand, we review approaches based on descent cone analysis, showing that they often lead to strong guarantees even in the presence of adversarial noise, but face limitations when it comes to structured observations. On the other hand, we discuss techniques using approximate dual certificates and the golfing scheme, which are often better suited to deal with practical measurement structures, but sometimes lead to weaker guarantees. Lastly, we review recent progress towards analyzing descent cones also for structured scenarios -- exploiting the idea of splitting the cones into multiple parts that are analyzed via different techniques. △ Less

Submitted 8 June, 2021; originally announced June 2021.

Comments: 39 pages, 6 figures

Sketching with Kerdock's crayons: Fast sparsifying transforms for arbitrary linear maps

Authors: Tim Fuchs, David Gross, Felix Krahmer, Richard Kueng, Dustin G. Mixon

Abstract: Given an arbitrary matrix $A\in\mathbb{R}^{n\times n}$, we consider the fundamental problem of computing $Ax$ for any $x\in\mathbb{R}^n$ such that $Ax$ is $s$-sparse. While fast algorithms exist for particular choices of $A$, such as the discrete Fourier transform, there is currently no $o(n^2)$ algorithm that treats the unstructured case. In this paper, we devise a randomized approach to tackle t… ▽ More Given an arbitrary matrix $A\in\mathbb{R}^{n\times n}$, we consider the fundamental problem of computing $Ax$ for any $x\in\mathbb{R}^n$ such that $Ax$ is $s$-sparse. While fast algorithms exist for particular choices of $A$, such as the discrete Fourier transform, there is currently no $o(n^2)$ algorithm that treats the unstructured case. In this paper, we devise a randomized approach to tackle the unstructured case. Our method relies on a representation of $A$ in terms of certain real-valued mutually unbiased bases derived from Kerdock sets. In the preprocessing phase of our algorithm, we compute this representation of $A$ in $O(n^3\log n)$ operations. Next, given any unit vector $x\in\mathbb{R}^n$ such that $Ax$ is $s$-sparse, our randomized fast transform uses this representation of $A$ to compute the entrywise $ε$-hard threshold of $Ax$ with high probability in only $O(sn + ε^{-2}\|A\|_{2\to\infty}^2n\log n)$ operations. In addition to a performance guarantee, we provide numerical results that demonstrate the plausibility of real-world implementation of our algorithm. △ Less

Submitted 12 May, 2021; originally announced May 2021.

Unlimited Sampling from Theory to Practice: Fourier-Prony Recovery and Prototype ADC

Authors: Ayush Bhandari, Felix Krahmer, Thomas Poskitt

Abstract: Following the Unlimited Sampling strategy to alleviate the omnipresent dynamic range barrier, we study the problem of recovering a bandlimited signal from point-wise modulo samples, aiming to connect theoretical guarantees with hardware implementation considerations. Our starting point is a class of non-idealities that we observe in prototyping an unlimited sampling based analog-to-digital convert… ▽ More Following the Unlimited Sampling strategy to alleviate the omnipresent dynamic range barrier, we study the problem of recovering a bandlimited signal from point-wise modulo samples, aiming to connect theoretical guarantees with hardware implementation considerations. Our starting point is a class of non-idealities that we observe in prototyping an unlimited sampling based analog-to-digital converter. To address these non-idealities, we provide a new Fourier domain recovery algorithm. Our approach is validated both in theory and via extensive experiments on our prototype analog-to-digital converter, providing the first demonstration of unlimited sampling for data arising from real hardware, both for the current and previous approaches. Advantages of our algorithm include that it is agnostic to the modulo threshold and it can handle arbitrary folding times. We expect that the end-to-end realization studied in this paper will pave the path for exploring the unlimited sampling methodology in a number of real world applications. △ Less

Submitted 12 May, 2021; originally announced May 2021.

Comments: 10 figures. Manuscript submitted for possible publication. We report a Fourier domain algorithm that is validated on real experiments based on a modulo sampling ADC

The Modulo Radon Transform: Theory, Algorithms and Applications

Authors: Matthias Beckmann, Ayush Bhandari, Felix Krahmer

Abstract: Recently, experiments have been reported where researchers were able to perform high dynamic range (HDR) tomography in a heuristic fashion, by fusing multiple tomographic projections. This approach to HDR tomography has been inspired by HDR photography and inherits the same disadvantages. Taking a computational imaging approach to the HDR tomography problem, we here suggest a new model based on th… ▽ More Recently, experiments have been reported where researchers were able to perform high dynamic range (HDR) tomography in a heuristic fashion, by fusing multiple tomographic projections. This approach to HDR tomography has been inspired by HDR photography and inherits the same disadvantages. Taking a computational imaging approach to the HDR tomography problem, we here suggest a new model based on the Modulo Radon Transform (MRT), which we rigorously introduce and analyze. By harnessing a joint design between hardware and algorithms, we present a single-shot HDR tomography approach, which to our knowledge, is the only approach that is backed by mathematical guarantees. On the hardware front, instead of recording the Radon Transform projections that my potentially saturate, we propose to measure modulo values of the same. This ensures that the HDR measurements are folded into a lower dynamic range. On the algorithmic front, our recovery algorithms reconstruct the HDR images from folded measurements. Beyond mathematical aspects such as injectivity and inversion of the MRT for different scenarios including band-limited and approximately compactly supported images, we also provide a first proof-of-concept demonstration. To do so, we implement MRT by experimentally folding tomographic measurements available as an open source data set using our custom designed modulo hardware. Our reconstruction clearly shows the advantages of our approach for experimental data. In this way, our MRT based solution paves a path for HDR acquisition in a number of related imaging problems. △ Less

Submitted 10 May, 2021; originally announced May 2021.

Comments: 32 pages, submitted for possible publication

Riemannian thresholding methods for row-sparse and low-rank matrix recovery

Authors: Henrik Eisenmann, Felix Krahmer, Max Pfeffer, André Uschmajew

Abstract: In this paper, we present modifications of the iterative hard thresholding (IHT) method for recovery of jointly row-sparse and low-rank matrices. In particular a Riemannian version of IHT is considered which significantly reduces computational cost of the gradient projection in the case of rank-one measurement operators, which have concrete applications in blind deconvolution. Experimental results… ▽ More In this paper, we present modifications of the iterative hard thresholding (IHT) method for recovery of jointly row-sparse and low-rank matrices. In particular a Riemannian version of IHT is considered which significantly reduces computational cost of the gradient projection in the case of rank-one measurement operators, which have concrete applications in blind deconvolution. Experimental results are reported that show near-optimal recovery for Gaussian and rank-one measurements, and that adaptive stepsizes give crucial improvement. A Riemannian proximal gradient method is derived for the special case of unknown sparsity. △ Less

Submitted 30 September, 2022; v1 submitted 3 March, 2021; originally announced March 2021.

On the robustness of noise-blind low-rank recovery from rank-one measurements

Authors: Felix Krahmer, Christian Kümmerle, Oleh Melnyk

Abstract: We prove new results about the robustness of well-known convex noise-blind optimization formulations for the reconstruction of low-rank matrices from underdetermined linear measurements. Our results are applicable for symmetric rank-one measurements as used in a formulation of the phase retrieval problem. We obtain these results by establishing that with high probability rank-one measurement ope… ▽ More We prove new results about the robustness of well-known convex noise-blind optimization formulations for the reconstruction of low-rank matrices from underdetermined linear measurements. Our results are applicable for symmetric rank-one measurements as used in a formulation of the phase retrieval problem. We obtain these results by establishing that with high probability rank-one measurement operators defined by i.i.d. Gaussian vectors exhibit the so-called Schatten-1 quotient property, which corresponds to a lower bound for the inradius of their image of the nuclear norm (Schatten-1) unit ball. We complement our analysis by numerical experiments comparing the solutions of noise-blind and noise-aware formulations. These experiments confirm that noise-blind optimization methods exhibit comparable robustness to noise-aware formulations. Keywords: low-rank matrix recovery, phase retrieval, quotient property, noise-blind, robustness, nuclear norm minimization △ Less

Submitted 23 October, 2020; originally announced October 2020.

Comments: 39 pages, 4 figures

A sample efficient sparse FFT for arbitrary frequency candidate sets in high dimensions

Authors: Lutz Kämmerer, Felix Krahmer, Toni Volkmer

Abstract: In this paper a sublinear time algorithm is presented for the reconstruction of functions that can be represented by just few out of a potentially large candidate set of Fourier basis functions in high spatial dimensions, a so-called high-dimensional sparse fast Fourier transform. In contrast to many other such algorithms, our method works for arbitrary candidate sets and does not make additional… ▽ More In this paper a sublinear time algorithm is presented for the reconstruction of functions that can be represented by just few out of a potentially large candidate set of Fourier basis functions in high spatial dimensions, a so-called high-dimensional sparse fast Fourier transform. In contrast to many other such algorithms, our method works for arbitrary candidate sets and does not make additional structural assumptions on the candidate set. Our transform significantly improves upon the other approaches available for such a general framework in terms of the scaling of the sample complexity. Our algorithm is based on sampling the function along multiple rank-1 lattices with random generators. Combined with a dimension-incremental approach, our method yields a sparse Fourier transform whose computational complexity only grows mildly in the dimension and can hence be efficiently computed even in high dimensions. Our theoretical analysis establishes that any Fourier $s$-sparse function can be accurately reconstructed with high probability. This guarantee is complemented by several numerical tests demonstrating the high efficiency and versatile applicability for the exactly sparse case and also for the compressible case. △ Less

Submitted 23 June, 2020; originally announced June 2020.

MSC Class: 65T; 65T40; 42A10

High order low-bit Sigma-Delta quantization for fusion frames

Authors: Zhen Gao, Felix Krahmer, Alexander M. Powell

Abstract: We construct high order low-bit Sigma-Delta $(ΣΔ)$ quantizers for the vector-valued setting of fusion frames. We prove that these $ΣΔ$ quantizers can be stably implemented to quantize fusion frame measurements on subspaces $W_n$ using $\log_2( {\rm dim}(W_n)+1)$ bits per measurement. Signal reconstruction is performed using a version of Sobolev duals for fusion frames, and numerical experiments ar… ▽ More We construct high order low-bit Sigma-Delta $(ΣΔ)$ quantizers for the vector-valued setting of fusion frames. We prove that these $ΣΔ$ quantizers can be stably implemented to quantize fusion frame measurements on subspaces $W_n$ using $\log_2( {\rm dim}(W_n)+1)$ bits per measurement. Signal reconstruction is performed using a version of Sobolev duals for fusion frames, and numerical experiments are given to validate the overall performance. △ Less

Submitted 3 October, 2020; v1 submitted 17 June, 2020; originally announced June 2020.

Comments: 18 pages, 2 figures

On recovery guarantees for angular synchronization

Authors: Frank Filbir, Felix Krahmer, Oleh Melnyk

Abstract: The angular synchronization problem of estimating a set of unknown angles from their known noisy pairwise differences arises in various applications. It can be reformulated as a optimization problem on graphs involving the graph Laplacian matrix. We consider a general, weighted version of this problem, where the impact of the noise differs between different pairs of entries and some of the differe… ▽ More The angular synchronization problem of estimating a set of unknown angles from their known noisy pairwise differences arises in various applications. It can be reformulated as a optimization problem on graphs involving the graph Laplacian matrix. We consider a general, weighted version of this problem, where the impact of the noise differs between different pairs of entries and some of the differences are erased completely; this version arises for example in ptychography. We study two common approaches for solving this problem, namely eigenvector relaxation and semidefinite convex relaxation. Although some recovery guarantees are available for both methods, their performance is either unsatisfying or restricted to the unweighted graphs. We close this gap, deriving recovery guarantees for the weighted problem that are completely analogous to the unweighted version. △ Less

Submitted 8 September, 2022; v1 submitted 5 May, 2020; originally announced May 2020.

Comments: 20 pages, 5 figures

Journal ref: J Fourier Anal Appl 27, 31 (2021)

Well conditioned ptychograpic imaging via lost subspace completion

Authors: Anton Forstner, Felix Krahmer, Oleh Melnyk, Nada Sissouno

Abstract: Ptychography, a special case of the phase retrieval problem, is a popular method in modern imaging. Its measurements are based on the shifts of a locally supported window function. In general, direct recovery of an object from such measurements is known to be an ill-posed problem. Although for some windows the conditioning can be controlled, for a number of important cases it is not possible, for… ▽ More Ptychography, a special case of the phase retrieval problem, is a popular method in modern imaging. Its measurements are based on the shifts of a locally supported window function. In general, direct recovery of an object from such measurements is known to be an ill-posed problem. Although for some windows the conditioning can be controlled, for a number of important cases it is not possible, for instance for Gaussian windows. In this paper we develop a subspace completion algorithm, which enables stable reconstruction for a much wider choice of windows, including Gaussian windows. The combination with a regularization technique leads to improved conditioning and better noise robustness. △ Less

Submitted 9 April, 2020; originally announced April 2020.

Comments: 21 pages, 4 figures

One-Bit Sigma-Delta modulation on the circle

Authors: Olga Graf, Felix Krahmer, Sara Krause-Solberg

Abstract: Manifold models in data analysis and signal processing have become more prominent in recent years. In this paper, we will look at one of the main tasks of modern signal processing, namely, at analog-to-digital (A/D) conversion in connection with a simple manifold model - the circle. We will focus on Sigma-Delta modulation which is a popular method for A/D conversion of bandlimited signals that emp… ▽ More Manifold models in data analysis and signal processing have become more prominent in recent years. In this paper, we will look at one of the main tasks of modern signal processing, namely, at analog-to-digital (A/D) conversion in connection with a simple manifold model - the circle. We will focus on Sigma-Delta modulation which is a popular method for A/D conversion of bandlimited signals that employs coarse quantization coupled with oversampling. Classical Sigma-Delta schemes provide mismatches and large errors at the initialization point if the signal to be converted is defined on the circle. In this paper, our goal is to get around these problems for Sigma-Delta schemes. Our results show how to design an update for the first and the second order schemes based on the reconstruction error analysis such that for the updated scheme the reconstruction error is improved. △ Less

Submitted 14 November, 2019; originally announced November 2019.

Predicting sparse circle maps from their dynamics

Authors: Felix Krahmer, Christian Kühn, Nada Sissouno

Abstract: The problem of identifying a dynamical system from its dynamics is of great importance for many applications. Recently it has been suggested to impose sparsity models for improved recovery performance. In this paper, we provide recovery guarantees for such a scenario. More precisely, we show that ergodic systems on the circle described by sparse trigonometric polynomials can be recovered from a nu… ▽ More The problem of identifying a dynamical system from its dynamics is of great importance for many applications. Recently it has been suggested to impose sparsity models for improved recovery performance. In this paper, we provide recovery guarantees for such a scenario. More precisely, we show that ergodic systems on the circle described by sparse trigonometric polynomials can be recovered from a number of samples scaling near-linearly in the sparsity. △ Less

Submitted 9 January, 2020; v1 submitted 14 November, 2019; originally announced November 2019.

On the geometry of polytopes generated by heavy-tailed random vectors

Authors: Olivier Guédon, Felix Krahmer, Christian Kümmerle, Shahar Mendelson, Holger Rauhut

Abstract: We study the geometry of centrally-symmetric random polytopes, generated by $N$ independent copies of a random vector $X$ taking values in $\mathbb{R}^n$. We show that under minimal assumptions on $X$, for $N \gtrsim n$ and with high probability, the polytope contains a deterministic set that is naturally associated with the random vector---namely, the polar of a certain floating body. This solves… ▽ More We study the geometry of centrally-symmetric random polytopes, generated by $N$ independent copies of a random vector $X$ taking values in $\mathbb{R}^n$. We show that under minimal assumptions on $X$, for $N \gtrsim n$ and with high probability, the polytope contains a deterministic set that is naturally associated with the random vector---namely, the polar of a certain floating body. This solves the long-standing question on whether such a random polytope contains a canonical body. Moreover, by identifying the floating bodies associated with various random vectors we recover the estimates that have been obtained previously, and thanks to the minimal assumptions on $X$ we derive estimates in cases that had been out of reach, involving random polytopes generated by heavy-tailed random vectors (e.g., when $X$ is $q$-stable or when $X$ has an unconditional structure). Finally, the structural results are used for the study of a fundamental question in compressive sensing---noise blind sparse recovery. △ Less

Submitted 16 July, 2019; originally announced July 2019.

Comments: 23 pages

MSC Class: 52A22; 46B06; 60B20; 65K10; 52A23; 46B09; 15B52

Complex phase retrieval from subgaussian measurements

Authors: Felix Krahmer, Dominik Stöger

Abstract: Phase retrieval refers to the problem of reconstructing an unknown vector $x_0 \in \mathbb{C}^n$ or $x_0 \in \mathbb{R}^n $ from $m$ measurements of the form $y_i = \big\vert \langle ξ^{\left(i\right)}, x_0 \rangle \big\vert^2 $, where $ \left\{ ξ^{\left(i\right)} \right\}^m_{i=1} \subset \mathbb{C}^m $ are known measurement vectors. While Gaussian measurements allow for recovery of arbitrary sign… ▽ More Phase retrieval refers to the problem of reconstructing an unknown vector $x_0 \in \mathbb{C}^n$ or $x_0 \in \mathbb{R}^n $ from $m$ measurements of the form $y_i = \big\vert \langle ξ^{\left(i\right)}, x_0 \rangle \big\vert^2 $, where $ \left\{ ξ^{\left(i\right)} \right\}^m_{i=1} \subset \mathbb{C}^m $ are known measurement vectors. While Gaussian measurements allow for recovery of arbitrary signals provided the number of measurements scales at least linearly in the number of dimensions, it has been shown that ambiguities may arise for certain other classes of measurements $ \left\{ ξ^{\left(i\right)} \right\}^{m}_{i=1}$ such as Bernoulli measurements or Fourier measurements. In this paper, we will prove that even when a subgaussian vector $ ξ^{\left(i\right)} \in \mathbb{C}^m $ does not fulfill a small-ball probability assumption, the PhaseLift method is still able to reconstruct a large class of signals $x_0 \in \mathbb{R}^n$ from the measurements. This extends recent work by Krahmer and Liu from the real-valued to the complex-valued case. However, our proof strategy is quite different and we expect some of the new proof ideas to be useful in several other measurement scenarios as well. We then extend our results $x_0 \in \mathbb{C}^n $ up to an additional assumption which, as we show, is necessary. △ Less

Submitted 17 July, 2020; v1 submitted 19 June, 2019; originally announced June 2019.

Comments: 25 pages

On Unlimited Sampling and Reconstruction

Authors: Ayush Bhandari, Felix Krahmer, Ramesh Raskar

Abstract: Shannon's sampling theorem is one of the cornerstone topics that is well understood and explored, both mathematically and algorithmically. That said, practical realization of this theorem still suffers from a severe bottleneck due to the fundamental assumption that the samples can span an arbitrary range of amplitudes. In practice, the theorem is realized using so-called analog-to-digital converte… ▽ More Shannon's sampling theorem is one of the cornerstone topics that is well understood and explored, both mathematically and algorithmically. That said, practical realization of this theorem still suffers from a severe bottleneck due to the fundamental assumption that the samples can span an arbitrary range of amplitudes. In practice, the theorem is realized using so-called analog-to-digital converters (ADCs) which clip or saturate whenever the signal amplitude exceeds the maximum recordable ADC voltage thus leading to a significant information loss. In this paper, we develop an alternative paradigm for sensing and recovery, called the Unlimited Sampling Framework. It is based on the observation that when a signal is mapped to an appropriate bounded interval via a modulo operation before entering the ADC, the saturation problem no longer exists, but one rather encounters a different type of information loss due to the modulo operation. Such an alternative setup can be implemented, for example, via so-called folding or self-reset ADCs, as they have been proposed in various contexts in the circuit design literature. The key task that we need to accomplish in order to cope with this new type of information loss is to recover a bandlimited signal from its modulo samples. In this paper we derive conditions when perfect recovery is possible and complement them with a stable recovery algorithm. The sampling density required to guarantee recovery is independent of the maximum recordable ADC voltage and depends on the signal bandwidth only. Our recovery guarantees extend to measurements affected by bounded noise, which includes the case of round-off quantization. Numerical experiments validate our approach. Applications of the unlimited sampling paradigm can be found in a number of fields such as signal processing, communication and imaging. △ Less

Submitted 30 November, 2020; v1 submitted 9 May, 2019; originally announced May 2019.

Comments: Accepted to IEEE Trans. on Sig. Proc. (10.1109/TSP.2020.3041955)

On the convex geometry of blind deconvolution and matrix completion

Authors: Felix Krahmer, Dominik Stöger

Abstract: Low-rank matrix recovery from structured measurements has been a topic of intense study in the last decade and many important problems like matrix completion and blind deconvolution have been formulated in this framework. An important benchmark method to solve these problems is to minimize the nuclear norm, a convex proxy for the rank. A common approach to establish recovery guarantees for this co… ▽ More Low-rank matrix recovery from structured measurements has been a topic of intense study in the last decade and many important problems like matrix completion and blind deconvolution have been formulated in this framework. An important benchmark method to solve these problems is to minimize the nuclear norm, a convex proxy for the rank. A common approach to establish recovery guarantees for this convex program relies on the construction of a so-called approximate dual certificate. However, this approach provides only limited insight in various respects. Most prominently, the noise bounds exhibit seemingly suboptimal dimension factors. In this paper we take a novel, more geometric viewpoint to analyze both the matrix completion and the blind deconvolution scenario. We find that for both these applications the dimension factors in the noise bounds are not an artifact of the proof, but the problems are intrinsically badly conditioned. We show, however, that bad conditioning only arises for very small noise levels: Under mild assumptions that include many realistic noise levels we derive near-optimal error estimates for blind deconvolution under adversarial noise. △ Less

Submitted 9 April, 2020; v1 submitted 28 February, 2019; originally announced February 2019.

Sparse Harmonic Transforms: A New Class of Sublinear-time Algorithms for Learning Functions of Many Variables

Authors: Bosu Choi, Mark Iwen, Felix Krahmer

Abstract: We develop fast and memory efficient numerical methods for learning functions of many variables that admit sparse representations in terms of general bounded orthonormal tensor product bases. Such functions appear in many applications including, e.g., various Uncertainty Quantification(UQ) problems involving the solution of parametric PDE that are approximately sparse in Chebyshev or Legendre prod… ▽ More We develop fast and memory efficient numerical methods for learning functions of many variables that admit sparse representations in terms of general bounded orthonormal tensor product bases. Such functions appear in many applications including, e.g., various Uncertainty Quantification(UQ) problems involving the solution of parametric PDE that are approximately sparse in Chebyshev or Legendre product bases. We expect that our results provide a starting point for a new line of research on sublinear-time solution techniques for UQ applications of the type above which will eventually be able to scale to significantly higher-dimensional problems than what are currently computationally feasible. More concretely, let $B$ be a finite Bounded Orthonormal Product Basis (BOPB) of cardinality $|B|=N$. We will develop methods that approximate any function $f$ that is sparse in the BOPB, that is, $f:\mathcal{D}\subset R^D\rightarrow C$ of the form $f(\mathbf{x})=\sum_{b\in S}c_b\cdot b(\mathbf{x})$ with $S\subset B$ of cardinality $|S| =s\ll N$. Our method has a runtime of just $(s\log N)^{O(1)}$, uses only $(s\log N)^{O(1)}$ function evaluations on a fixed and nonadaptive grid, and not more than $(s\log N)^{O(1)}$ bits of memory. For $s\ll N$, the runtime $(s\log N)^{O(1)}$ will be less than what is required to simply enumerate the elements of the basis $B$; thus our method is the first approach applicable in a general BOPB framework that falls into the class referred to as "sublinear-time". This and the similarly reduced sample and memory requirements set our algorithm apart from previous works based on standard compressive sensing algorithms such as basis pursuit which typically store and utilize full intermediate basis representations of size $Ω(N)$. △ Less

Submitted 7 May, 2020; v1 submitted 14 August, 2018; originally announced August 2018.

On Recovery Guarantees for One-Bit Compressed Sensing on Manifolds

Authors: Mark A. Iwen, Felix Krahmer, Sara Krause-Solberg, Johannes Maly

Abstract: This paper studies the problem of recovering a signal from one-bit compressed sensing measurements under a manifold model; that is, assuming that the signal lies on or near a manifold of low intrinsic dimension. We provide a convex recovery method based on the Geometric Multi-Resolution Analysis and prove recovery guarantees with a near-optimal scaling in the intrinsic manifold dimension. Our meth… ▽ More This paper studies the problem of recovering a signal from one-bit compressed sensing measurements under a manifold model; that is, assuming that the signal lies on or near a manifold of low intrinsic dimension. We provide a convex recovery method based on the Geometric Multi-Resolution Analysis and prove recovery guarantees with a near-optimal scaling in the intrinsic manifold dimension. Our method is the first tractable algorithm with such guarantees for this setting. The results are complemented by numerical experiments confirming the validity of our approach. △ Less

Submitted 23 July, 2020; v1 submitted 17 July, 2018; originally announced July 2018.

Are good local minima wide in sparse recovery?

Authors: Michael Moeller, Otmar Loffeld, Juergen Gall, Felix Krahmer

Abstract: The idea of compressed sensing is to exploit representations in suitable (overcomplete) dictionaries that allow to recover signals far beyond the Nyquist rate provided that they admit a sparse representation in the respective dictionary. The latter gives rise to the sparse recovery problem of finding the best sparse linear approximation of given data in a given generating system. In this paper we… ▽ More The idea of compressed sensing is to exploit representations in suitable (overcomplete) dictionaries that allow to recover signals far beyond the Nyquist rate provided that they admit a sparse representation in the respective dictionary. The latter gives rise to the sparse recovery problem of finding the best sparse linear approximation of given data in a given generating system. In this paper we analyze the iterative hard thresholding (IHT) algorithm as one of the most popular greedy methods for solving the sparse recovery problem, and demonstrate that systematically perturbing the IHT algorithm by adding noise to intermediate iterates yields improved results. Further improvements can be obtained by entirely rephrasing the problem as a parametric deep-learning-type of optimization problem. By introducing perturbations via dropout, we demonstrate to significantly outperform the classical IHT algorithm, obtaining $3$ to $6$ times lower average objective errors. △ Less

Submitted 21 June, 2018; originally announced June 2018.

A Quotient Property for Matrices with Heavy-Tailed Entries and its Application to Noise-Blind Compressed Sensing

Authors: Felix Krahmer, Christian Kümmerle, Holger Rauhut

Abstract: For a large class of random matrices $A$ with i.i.d. entries we show that the $\ell_1$-quotient property holds with probability exponentially close to 1. In contrast to previous results, our analysis does not require concentration of the entrywise distributions. We provide a unified proof that recovers corresponding previous results for (sub-)Gaussian and Weibull distributions. Our findings genera… ▽ More For a large class of random matrices $A$ with i.i.d. entries we show that the $\ell_1$-quotient property holds with probability exponentially close to 1. In contrast to previous results, our analysis does not require concentration of the entrywise distributions. We provide a unified proof that recovers corresponding previous results for (sub-)Gaussian and Weibull distributions. Our findings generalize known results on the geometry of random polytopes, providing lower bounds on the size of the largest Euclidean ball contained in the centrally symmetric polytope spanned by the columns of $A$. At the same time, our results establish robustness of noise-blind $\ell_1$-decoders for recovering sparse vectors $x$ from underdetermined, noisy linear measurements $y = Ax + w$ under the weakest possible assumptions on the entrywise distributions that allow for recovery with optimal sample complexity even in the noiseless case. Our analysis predicts superior robustness behavior for measurement matrices with super-Gaussian entries, which we confirm by numerical experiments. △ Less

Submitted 11 June, 2018; originally announced June 2018.

Comments: 22 pages, 2 figures

MSC Class: 46B20; 46B09; 15A52; 65K10; 52A22

Sparse Power Factorization: Balancing peakiness and sample complexity

Authors: Jakob Geppert, Felix Krahmer, Dominik Stöger

Abstract: In many applications, one is faced with an inverse problem, where the known signal depends in a bilinear way on two unknown input vectors. Often at least one of the input vectors is assumed to be sparse, i.e., to have only few non-zero entries. Sparse Power Factorization (SPF), proposed by Lee, Wu, and Bresler, aims to tackle this problem. They have established recovery guarantees for a somewhat r… ▽ More In many applications, one is faced with an inverse problem, where the known signal depends in a bilinear way on two unknown input vectors. Often at least one of the input vectors is assumed to be sparse, i.e., to have only few non-zero entries. Sparse Power Factorization (SPF), proposed by Lee, Wu, and Bresler, aims to tackle this problem. They have established recovery guarantees for a somewhat restrictive class of signals under the assumption that the measurements are random. We generalize these recovery guarantees to a significantly enlarged and more realistic signal class at the expense of a moderately increased number of measurements. △ Less

Submitted 24 April, 2018; originally announced April 2018.

Comments: 18 pages

Optimal Fast Johnson-Lindenstrauss Embeddings for Large Data Sets

Authors: Stefan Bamberger, Felix Krahmer

Abstract: Johnson-Lindenstrauss embeddings are widely used to reduce the dimension and thus the processing time of data. To reduce the total complexity, also fast algorithms for applying these embeddings are necessary. To date, such fast algorithms are only available either for a non-optimal embedding dimension or up to a certain threshold on the number of data points. We address a variant of this problem… ▽ More Johnson-Lindenstrauss embeddings are widely used to reduce the dimension and thus the processing time of data. To reduce the total complexity, also fast algorithms for applying these embeddings are necessary. To date, such fast algorithms are only available either for a non-optimal embedding dimension or up to a certain threshold on the number of data points. We address a variant of this problem where one aims to simultaneously embed larger subsets of the data set. Our method follows an approach by Nelson: A subsampled Hadamard transform maps points into a space of lower, but not optimal dimension. Subsequently, a random matrix with independent entries projects to an optimal embedding dimension. For subsets whose size scales at least polynomially in the ambient dimension, the complexity of this method comes close to the number of operations just to read the data under mild assumptions on the size of the data set that are considerably less restrictive than in previous works. We also prove a lower bound showing that subsampled Hadamard matrices alone cannot reach an optimal embedding dimension. Hence, the second embedding cannot be omitted. △ Less

Submitted 29 April, 2020; v1 submitted 5 December, 2017; originally announced December 2017.

Spectral Methods for Passive Imaging: Non-asymptotic Performance and Robustness

Authors: Kiryung Lee, Felix Krahmer, Justin Romberg

Abstract: We study the problem of passive imaging through convolutive channels. A scene is illuminated with an unknown, unstructured source, and the measured response is the convolution of this source with multiple channel responses, each of which is time-limited. Spectral methods based on the commutativity of convolution, first proposed and analyzed in the 1990s, provide an elegant mathematical framework f… ▽ More We study the problem of passive imaging through convolutive channels. A scene is illuminated with an unknown, unstructured source, and the measured response is the convolution of this source with multiple channel responses, each of which is time-limited. Spectral methods based on the commutativity of convolution, first proposed and analyzed in the 1990s, provide an elegant mathematical framework for attacking this problem. However, these now classical methods are very sensitive to noise, especially when working from relatively small sample sizes. In this paper, we show that a linear subspace model on the coefficients of the impulse responses of the channels can make this problem well-posed. We derive non-asymptotic error bounds for the generic subspace model by analyzing the spectral gap of the cross-correlation matrix of the channels relative to the perturbation introduced by noise. Numerical results show that this modified spectral method offers significant improvements over the classical method and outperforms other competing methods for multichannel blind deconvolution. △ Less

Submitted 23 August, 2017; v1 submitted 14 August, 2017; originally announced August 2017.

On Unlimited Sampling

Authors: Ayush Bhandari, Felix Krahmer, Ramesh Raskar

Abstract: Shannon's sampling theorem provides a link between the continuous and the discrete realms stating that bandlimited signals are uniquely determined by its values on a discrete set. This theorem is realized in practice using so called analog--to--digital converters (ADCs). Unlike Shannon's sampling theorem, the ADCs are limited in dynamic range. Whenever a signal exceeds some preset threshold, the A… ▽ More Shannon's sampling theorem provides a link between the continuous and the discrete realms stating that bandlimited signals are uniquely determined by its values on a discrete set. This theorem is realized in practice using so called analog--to--digital converters (ADCs). Unlike Shannon's sampling theorem, the ADCs are limited in dynamic range. Whenever a signal exceeds some preset threshold, the ADC saturates, resulting in aliasing due to clipping. The goal of this paper is to analyze an alternative approach that does not suffer from these problems. Our work is based on recent developments in ADC design, which allow for ADCs that reset rather than to saturate, thus producing modulo samples. An open problem that remains is: Given such modulo samples of a bandlimited function as well as the dynamic range of the ADC, how can the original signal be recovered and what are the sufficient conditions that guarantee perfect recovery? In this paper, we prove such sufficiency conditions and complement them with a stable recovery algorithm. Our results are not limited to certain amplitude ranges, in fact even the same circuit architecture allows for the recovery of arbitrary large amplitudes as long as some estimate of the signal norm is available when recovering. Numerical experiments that corroborate our theory indeed show that it is possible to perfectly recover function that takes values that are orders of magnitude higher than the ADC's threshold. △ Less

Submitted 8 November, 2017; v1 submitted 19 July, 2017; originally announced July 2017.

Comments: 11 pages, 4 figures, copy of initial version to appear in Proceedings of 12th International Conference on Sampling Theory and Applications (SampTA)

Blind Demixing and Deconvolution at Near-Optimal Rate

Authors: Peter Jung, Felix Krahmer, Dominik Stöger

Abstract: We consider simultaneous blind deconvolution of r source signals from their noisy superposition, a problem also referred to blind demixing and deconvolution. This signal processing problem occurs in the context of the Internet of Things where a massive number of sensors sporadically communicate only short messages over unknown channels. We show that robust recovery of message and channel vectors c… ▽ More We consider simultaneous blind deconvolution of r source signals from their noisy superposition, a problem also referred to blind demixing and deconvolution. This signal processing problem occurs in the context of the Internet of Things where a massive number of sensors sporadically communicate only short messages over unknown channels. We show that robust recovery of message and channel vectors can be achieved via convex optimization when random linear encoding using i.i.d. complex Gaussian matrices is used at the devices and the number of required measurements at the receiver scales with the degrees of freedom of the overall estimation problem. Since the scaling is linear in r our result significantly improves over recent works. △ Less

Submitted 2 May, 2017; v1 submitted 13 April, 2017; originally announced April 2017.

Comments: 49 pages, 1 figure; v2: a few typos removed

Total Variation Minimization in Compressed Sensing

Authors: Felix Krahmer, Christian Kruschel, Michael Sandbichler

Abstract: This chapter gives an overview over recovery guarantees for total variation minimization in compressed sensing for different measurement scenarios. In addition to summarizing the results in the area, we illustrate why an approach that is common for synthesis sparse signals fails and different techniques are necessary. Lastly, we discuss a generalizations of recent results for Gaussian measurements… ▽ More This chapter gives an overview over recovery guarantees for total variation minimization in compressed sensing for different measurement scenarios. In addition to summarizing the results in the area, we illustrate why an approach that is common for synthesis sparse signals fails and different techniques are necessary. Lastly, we discuss a generalizations of recent results for Gaussian measurements to the subgaussian case. △ Less

Submitted 3 November, 2017; v1 submitted 7 April, 2017; originally announced April 2017.

Comments: 23 pages, 2 figures