-
The impact of lunar topography on the 21-cm power spectrum for grid-based arrays : Insights for the Dark-ages EXplorer (DEX)
Authors:
S. Ghosh,
L. V. E. Koopmans,
C. Brinkerink,
A. R. Offringa,
A. J. Boonstra,
S. A. Brackenhoff,
E. Ceccotti,
J. K. Chege,
L. Y. Gao,
B. K. Gehlot,
L. I. Gurvits,
C. Höfer,
F. G. Mertens,
M. Mevius,
S. Munshi,
A. Saxena,
J. A. Tauber,
H. Vedantham,
S. Yatawatta,
S. Zaroubi
Abstract:
The Dark Ages (DA) provides a crucial window into the physics of the infant Universe, with the 21-cm signal offering the only direct probe for mapping out the three-dimensional distribution of matter at this epoch. To measure this cosmological signal, the Dark-ages EXplorer (DEX) has been proposed as a compact, grid-based radio array on the lunar farside. The minimal design consists of a 32…
▽ More
The Dark Ages (DA) provides a crucial window into the physics of the infant Universe, with the 21-cm signal offering the only direct probe for mapping out the three-dimensional distribution of matter at this epoch. To measure this cosmological signal, the Dark-ages EXplorer (DEX) has been proposed as a compact, grid-based radio array on the lunar farside. The minimal design consists of a 32 $\times$ 32 array of 3-m dipole antennas, operating in the $7 - 50$ MHz band. A practical challenge on the lunar surface is that the antennas may get displaced from their intended positions due to deployment imprecisions and non-coplanarity arising from local surface undulations. We present, for the first time, an end-to-end simulation pipeline, called SPADE-21cm, that integrates a sky model with a DA 21-cm signal model simulated in the lunar frame and incorporating lunar topography data. We study the effects of both lateral (xy) and vertical (z) offsets on the two-dimensional power spectra across the $7 - 12$ MHz and $30 - 35$ MHz spectral windows, with tolerance thresholds derived only for the latter. Our results show that positional offsets bias the power spectrum by $10 - 30$ per cent relative to the expected 21-cm power spectrum during DA. Lateral offsets within $σ_{xy}/λ\lesssim 0.027$ (at 32.5 MHz) keep the fraction of Fourier modes with strong contamination (> 50 per cent of the signal) to less than 1 per cent, whereas vertical height offsets affect a larger fraction. This conclusion holds for the 21-cm window with $k_\parallel > 0.5$ $h$ cMpc$^{-1}$ over the range of $k_\perp = 0.003 - 0.009$ $h$ cMpc$^{-1}$.
△ Less
Submitted 9 December, 2025;
originally announced December 2025.
-
Mitigating incoherent excess variance in high-redshift 21 cm observations with multi-output cross-Gaussian process regression
Authors:
S. Munshi,
L. V. E. Koopmans,
F. G. Mertens,
A. R. Offringa,
S. A. Brackenhoff,
E. Ceccotti,
J. K. Chege,
L. Y. Gao,
S. Ghosh,
M. Mevius,
S. Zaroubi
Abstract:
Systematic effects that limit the achievable sensitivity of current low-frequency radio telescopes to the 21 cm signal are among the foremost challenges in observational 21 cm cosmology. The standard approach to retrieving the 21 cm signal from radio interferometric data separates it from bright astrophysical foregrounds by exploiting their spectrally smooth nature, in contrast to the finer spectr…
▽ More
Systematic effects that limit the achievable sensitivity of current low-frequency radio telescopes to the 21 cm signal are among the foremost challenges in observational 21 cm cosmology. The standard approach to retrieving the 21 cm signal from radio interferometric data separates it from bright astrophysical foregrounds by exploiting their spectrally smooth nature, in contrast to the finer spectral structure of the 21 cm signal. Contaminants exhibiting rapid frequency fluctuations, on the other hand, are difficult to separate from the 21 cm signal using standard techniques, and the power from these contaminants contributes to low-level systematics that can limit our ability to detect the 21 cm signal. Many of these low-level systematics are incoherent across multiple nights of observation, resulting in an incoherent excess variance above the thermal noise sensitivity of the instrument. In this paper, we develop a method called cross-GPR (cross covariance Gaussian process regression) that exploits the incoherence of these systematics to separate them from the 21 cm signal, which remains coherent across multiple nights of observation. We first develop and demonstrate the technique on synthetic signals in a general setting, and then apply it to gridded interferometric visibility cubes. We perform realistic simulations of visibility cubes containing foregrounds, 21 cm signal, noise, and incoherent systematics. The simulations show that the method can successfully separate and subtract incoherent contributions to the excess variance, and its advantages over standard techniques become more evident when the spectral behavior of the contaminants resembles that of the 21 cm signal. Simulations performed on a variety of 21 cm signal shapes also reveal that the cross-GPR approach can subtract incoherent contributions to the excess variance, without suppressing the 21 cm signal.
△ Less
Submitted 22 December, 2025; v1 submitted 11 August, 2025;
originally announced August 2025.
-
Improved upper limits on the 21-cm signal power spectrum at $z=17.0$ and $z=20.3$ from an optimal field observed with NenuFAR
Authors:
S. Munshi,
F. G. Mertens,
J. K. Chege,
L. V. E. Koopmans,
A. R. Offringa,
B. Semelin,
R. Barkana,
J. Dhandha,
A. Fialkov,
R. Mériot,
S. Sikder,
A. Bracco,
S. A. Brackenhoff,
E. Ceccotti,
R. Ghara,
S. Ghosh,
I. Hothi,
M. Mevius,
P. Ocvirk,
A. K. Shaw,
S. Yatawatta,
P. Zarka
Abstract:
We report the deepest upper limits to date on the power spectrum of the 21-cm signal during the Cosmic Dawn (redshifts: $z>15$), using four nights of observations with NenuFAR. The limits are derived from two redshift bins, centred at $z=20.3$ and $z=17.0$, with integration times of 26.1 h and 23.6 h, from observations of an optimal target field chosen to minimise sidelobe leakage from bright sour…
▽ More
We report the deepest upper limits to date on the power spectrum of the 21-cm signal during the Cosmic Dawn (redshifts: $z>15$), using four nights of observations with NenuFAR. The limits are derived from two redshift bins, centred at $z=20.3$ and $z=17.0$, with integration times of 26.1 h and 23.6 h, from observations of an optimal target field chosen to minimise sidelobe leakage from bright sources. Our analysis incorporates improvements to the data processing pipeline, particularly in subtracting strong radio sources in the primary beam sidelobes and mitigating low-level radio frequency interference, yielding a 50-fold reduction in the excess variance compared to a previous analysis of the north celestial pole field. At $z=20.3$, we achieve a best $2σ$ upper limit of $Δ^{2}_{21}<4.6 \times 10^5 \, \textrm{mK}^{2}$ at $k=0.038$ $h\, \mathrm{cMpc}^{-1}$, while at $z=17.0$, the best limit is $Δ^{2}_{21}<5.0 \times 10^6 \, \textrm{mK}^{2}$ at $k=0.041$ $h\, \mathrm{cMpc}^{-1}$. These are the strongest constraints on the 21-cm power spectrum at the respective redshifts, with the limit at $z = 20.3$ being deeper by more than an order of magnitude over all previous Cosmic Dawn power spectrum limits. Comparison against simulated exotic 21-cm signals shows that while the $z=20.3$ limits begin to exclude the most extreme models predicting signals stronger than the EDGES detection, an order-of-magnitude improvement would constrain signals compatible with EDGES. A coherence analysis reveals that the excess variance is largely incoherent across nights for the $z=20.3$ redshift bin, suggesting that deeper integrations could yield significantly stronger constraints on the 21-cm signal from the Cosmic Dawn.
△ Less
Submitted 25 August, 2025; v1 submitted 14 July, 2025;
originally announced July 2025.
-
Constraints on the state of the IGM at $z\sim 8-10$ using redshifted 21-cm observations with LOFAR
Authors:
R. Ghara,
S. Zaroubi,
B. Ciardi,
G. Mellema,
S. K. Giri,
F. G. Mertens,
M. Mevius,
L. V. E. Koopmans,
I. T. Iliev,
A. Acharya,
S. A. Brackenhoff,
E. Ceccotti,
K. Chege,
I. Georgiev,
S. Ghosh,
I. Hothi,
C. Höfer,
Q. Ma,
S. Munshi,
A. R. Offringa,
A. K. Shaw,
V. N. Pandey,
S. Yatawatta,
M. Choudhury
Abstract:
The power spectra of the redshifted 21-cm signal from the Epoch of Reionization (EoR) contain information about the ionization and thermal states of the intergalactic medium (IGM), and depend on the properties of the EoR sources. Recently, Mertens et al 2025 has analysed 10 nights of LOFAR high-band data and estimated upper limits on the 21-cm power spectrum at redshifts 8.3, 9.1 and 10.1. Here we…
▽ More
The power spectra of the redshifted 21-cm signal from the Epoch of Reionization (EoR) contain information about the ionization and thermal states of the intergalactic medium (IGM), and depend on the properties of the EoR sources. Recently, Mertens et al 2025 has analysed 10 nights of LOFAR high-band data and estimated upper limits on the 21-cm power spectrum at redshifts 8.3, 9.1 and 10.1. Here we use these upper limit results to constrain the properties of the IGM at those redshifts. We focus on the properties of the ionized and heated regions where the temperature is larger than that of the CMB. We model the 21-cm power spectrum with the code GRIZZLY, and use a Bayesian inference framework to explore the source parameters for uniform priors on their ranges. The framework also provides information about the IGM properties in the form of derived parameters. In a model which includes a radio background in excess of the CMB, the 95 (68) per cent credible intervals of disfavoured models at redshift 9.1 for the chosen priors correspond to IGM states with averaged ionization and heated fraction below 0.46 ($\lesssim 0.05$), an average gas temperature below 44 K (4 K), and a characteristic size of the heated region $\lesssim 14 ~h^{-1} ~\mathrm{Mpc}$ ($\lesssim 3 ~h^{-1} ~\mathrm{Mpc}$). The 68 per cent credible interval suggests an excess radio background which is more than 100 per cent of the CMB at 1.42 GHz, while the 95 per cent credible interval of the radio background efficiency parameter spans the entire prior range. The behaviour of the credible intervals is similar at all redshifts. The models disfavoured by the LOFAR upper limits are extreme ones, as they are mainly driven by rare and large ionized or heated regions.
△ Less
Submitted 1 May, 2025;
originally announced May 2025.
-
A Unified MDL-based Binning and Tensor Factorization Framework for PDF Estimation
Authors:
Mustafa Musab,
Joseph K. Chege,
Arie Yeredor,
Martin Haardt
Abstract:
Reliable density estimation is fundamental for numerous applications in statistics and machine learning. In many practical scenarios, data are best modeled as mixtures of component densities that capture complex and multimodal patterns. However, conventional density estimators based on uniform histograms often fail to capture local variations, especially when the underlying distribution is highly…
▽ More
Reliable density estimation is fundamental for numerous applications in statistics and machine learning. In many practical scenarios, data are best modeled as mixtures of component densities that capture complex and multimodal patterns. However, conventional density estimators based on uniform histograms often fail to capture local variations, especially when the underlying distribution is highly nonuniform. Furthermore, the inherent discontinuity of histograms poses challenges for tasks requiring smooth derivatives, such as gradient-based optimization, clustering, and nonparametric discriminant analysis. In this work, we present a novel non-parametric approach for multivariate probability density function (PDF) estimation that utilizes minimum description length (MDL)-based binning with quantile cuts. Our approach builds upon tensor factorization techniques, leveraging the canonical polyadic decomposition (CPD) of a joint probability tensor. We demonstrate the effectiveness of our method on synthetic data and a challenging real dry bean classification dataset.
△ Less
Submitted 28 September, 2025; v1 submitted 25 April, 2025;
originally announced April 2025.
-
First upper limits on the 21-cm signal power spectrum of neutral hydrogen at $z=9.16$ from the LOFAR 3C196 field
Authors:
E. Ceccotti,
A. R. Offringa,
F. G. Mertens,
L. V. E. Koopmans,
S. Munshi,
J. K. Chege,
A. Acharya,
S. A. Brackenhoff,
E. Chapman,
B. Ciardi,
R. Ghara,
S. Ghosh,
S. K. Giri,
C. Höfer,
I. Hothi,
G. Mellema,
M. Mevius,
V. N. Pandey,
S. Zaroubi
Abstract:
The redshifted 21-cm signal of neutral hydrogen from the Epoch of Reionization (EoR) can potentially be detected using low-frequency radio instruments such as the Low-Frequency Array (LOFAR). So far, LOFAR upper limits on the 21-cm signal power spectrum have been published using a single target field: the North Celestial Pole (NCP). In this work, we analyse and provide upper limits for the 3C196 f…
▽ More
The redshifted 21-cm signal of neutral hydrogen from the Epoch of Reionization (EoR) can potentially be detected using low-frequency radio instruments such as the Low-Frequency Array (LOFAR). So far, LOFAR upper limits on the 21-cm signal power spectrum have been published using a single target field: the North Celestial Pole (NCP). In this work, we analyse and provide upper limits for the 3C196 field, observed by LOFAR, with a strong ${\approx}80\,$Jy source in the centre. This field offers advantages such as higher sensitivity due to zenith-crossing observations and reduced geostationary radio-frequency interference, but also poses challenges due to the presence of the bright central source. After constructing a wide-field sky model, we process a single 6-hour night of 3C196 observations using direction-independent and direction-dependent calibration, followed by a residual foreground subtraction with a machine learned Gaussian process regression (ML-GPR). A bias correction is necessary to account for signal suppression in the GPR step. Still, even after this correction, the upper limits are a factor of two lower than previous single-night NCP results, with a lowest $2σ$ upper limit of $(146.61\,\text{mK})^2$ at $z = 9.16$ and $k=0.078\,h\,\text{cMpc}^{-1}$ (with $\text{d}k/k\approx 0.3$). The results also reveal an excess power, different in behaviour from that observed in the NCP field, suggesting a potential residual foreground origin. In future work, the use of multiple nights of 3C196 observations combined with improvements to sky modelling and ML-GPR to avoid the need for bias correction should provide tighter constraints per unit observing time than the NCP.
△ Less
Submitted 25 April, 2025;
originally announced April 2025.
-
Probabilistic Position-Aided Beam Selection for mmWave MIMO Systems
Authors:
Joseph K. Chege,
Arie Yeredor,
Martin Haardt
Abstract:
Millimeter-wave (mmWave) MIMO systems rely on highly directional beamforming to overcome severe path loss and ensure robust communication links. However, selecting the optimal beam pair efficiently remains a challenge due to the large search space and the overhead of conventional methods. This paper proposes a probabilistic position-aided beam selection approach that exploits the statistical depen…
▽ More
Millimeter-wave (mmWave) MIMO systems rely on highly directional beamforming to overcome severe path loss and ensure robust communication links. However, selecting the optimal beam pair efficiently remains a challenge due to the large search space and the overhead of conventional methods. This paper proposes a probabilistic position-aided beam selection approach that exploits the statistical dependence between user equipment (UE) positions and optimal beam indices. We model the underlying joint probability mass function (PMF) of the positions and the beam indices as a low-rank tensor and estimate its parameters from training data using Bayesian inference. The estimated model is then used to predict the best (or a list of the top) beam pair indices for new UE positions. The proposed method is evaluated using data generated from a state-of-the-art ray tracing simulator and compared with neural network-based and fingerprinting approaches. The results show that our approach achieves a high data rate with fewer training samples and a significantly reduced beam search space. These advantages render it a promising solution for practical mmWave MIMO deployments, reducing the beam search overhead while maintaining a reliable connectivity.
△ Less
Submitted 12 April, 2025; v1 submitted 7 April, 2025;
originally announced April 2025.
-
The impact of diffuse Galactic emission on direction-independent gain calibration in high-redshift 21 cm observations
Authors:
C. Höfer,
L. V. E. Koopmans,
S. A. Brackenhoff,
E. Ceccotti,
K. Chege,
S. Ghosh,
F. G. Mertens,
M. Mevius,
S. Munshi,
A. R. Offringa
Abstract:
This study examines the impact of diffuse Galactic emission (DGE) on sky-based direction-independent (DI) gain calibration using realistic forward simulations of Low-Frequency Array (LOFAR) observations of the high-redshift 21 cm signal of neutral hydrogen during the Epoch of Reionization (EoR). We simulated LOFAR observations between 147 and 159 MHz using a sky model that includes a point source…
▽ More
This study examines the impact of diffuse Galactic emission (DGE) on sky-based direction-independent (DI) gain calibration using realistic forward simulations of Low-Frequency Array (LOFAR) observations of the high-redshift 21 cm signal of neutral hydrogen during the Epoch of Reionization (EoR). We simulated LOFAR observations between 147 and 159 MHz using a sky model that includes a point source catalog and DGE. The simulated observations were DI-gain calibrated with the point source catalog alone, utilizing the LOFAR-EoR data analysis pipeline. A full power spectrum (PS) analysis was conducted to measure the systematic bias, relative to thermal noise, caused by DI-gain calibration using a point-source-only (PSO) sky model, when applied to simulated data that include both point sources and DGE. The results are compared to a ground truth scenario where both the simulated sky and the calibration model include only point sources. Additionally, the cross-coherence between observation pairs was computed to determine whether DI-gain calibration errors are coherent or incoherent in specific regions of PS space as a function of integration time. We find that DI-gain calibration with a PSO sky model that omits DGE introduces a systematic bias in the PS for $k_{\parallel}$ bins < 0.2 $h\,\mathrm{Mpc}^{-1}$. The PS errors in these bins are coherent in time and frequency; therefore, the resulting bias could be mitigated during the foreground removal step using Gaussian Process Regression, as demonstrated in previous studies. In contrast, errors for $k_{\parallel}$ > 0.2 $h\,\mathrm{Mpc}^{-1}$ are largely incoherent and average down as noise. We conclude that, based on our analysis prior to foreground removal, missing DGE in the sky model during DI-gain calibration is unlikely to be a dominant contributor to the excess noise observed in the current LOFAR-EoR upper limits on the 21 cm signal PS.
△ Less
Submitted 17 September, 2025; v1 submitted 4 April, 2025;
originally announced April 2025.
-
Robust direction-dependent gain-calibration of beam-modelling errors far from the target field
Authors:
S. A. Brackenhoff,
A. R. Offringa,
M. Mevius,
L. V. E. Koopmans,
J. K. Chege,
E. Ceccotti,
C. Höfer,
L. Gao,
S. Ghosh,
F. G. Mertens,
S. Munshi
Abstract:
Many astronomical questions require deep, wide-field observations at low radio frequencies. Phased arrays like LOFAR and SKA-low are designed for this, but have inherently unstable element gains, leading to time, frequency and direction-dependent gain errors. Precise direction-dependent calibration of observations is therefore key to reaching the highest possible dynamic range. Many tools for dire…
▽ More
Many astronomical questions require deep, wide-field observations at low radio frequencies. Phased arrays like LOFAR and SKA-low are designed for this, but have inherently unstable element gains, leading to time, frequency and direction-dependent gain errors. Precise direction-dependent calibration of observations is therefore key to reaching the highest possible dynamic range. Many tools for direction-dependent calibration utilise sky and beam models to infer gains. However, these calibration tools struggle with precision calibration for relatively bright (e.g. A-team) sources far from the beam centre. Therefore, the point-spread-function of these sources can potentially obscure a faint signal of interest. We show that, and why, the assumption of a smooth gain solution per station fails for realistic radio interferometers, and how this affects gain-calibration results. Subsequently, we introduce an improvement for smooth spectral gain constraints for direction-dependent gain-calibration algorithms, in which the level of regularisation is weighted by the expected station response to the sky model. We test this method using direction-dependent calibration method DDECal and physically-motivated beam modelling errors for LOFAR-HBA stations. The new method outperforms the standard method for various calibration settings near nulls in the beam, and matches the standard inverse-variance-weighted method's performance for the remainder of the data. The proposed method is especially effective for short baselines, both in visibility and image space. Improved direction-dependent gain-calibration is critical for future high-precision SKA-low observations, where higher sensitivity, increased antenna beam complexity, and mutual coupling call for better off-axis source subtraction, which may not be achieved through improved beam models alone.
△ Less
Submitted 21 July, 2025; v1 submitted 3 April, 2025;
originally announced April 2025.
-
Near-field imaging of local interference in radio interferometric data: Impact on the redshifted 21 cm power spectrum
Authors:
S. Munshi,
F. G. Mertens,
L. V. E. Koopmans,
M. Mevius,
A. R. Offringa,
B. Semelin,
C. Viou,
A. Bracco,
S. A. Brackenhoff,
E. Ceccotti,
J. K. Chege,
A. Fialkov,
L. Y. Gao,
R. Ghara,
S. Ghosh,
A. K. Shaw,
P. Zarka,
S. Zaroubi,
B. Cecconi,
S. Corbel,
J. N. Girard,
J. M. Griessmeier,
O. Konovalenko,
A. Loh,
P. Tokarsky
, et al. (2 additional authors not shown)
Abstract:
Radio-frequency interference (RFI) is a major systematic limitation in radio astronomy, particularly for science cases requiring high sensitivity, such as 21 cm cosmology. Traditionally, RFI is dealt with by identifying its signature in the dynamic spectra of visibility data and flagging strongly affected regions. However, for RFI sources that do not occupy narrow regions in the time-frequency spa…
▽ More
Radio-frequency interference (RFI) is a major systematic limitation in radio astronomy, particularly for science cases requiring high sensitivity, such as 21 cm cosmology. Traditionally, RFI is dealt with by identifying its signature in the dynamic spectra of visibility data and flagging strongly affected regions. However, for RFI sources that do not occupy narrow regions in the time-frequency space, such as persistent local RFI, modeling these sources could be essential to mitigating their impact. This paper introduces two methods for detecting and characterizing local RFI sources from radio interferometric visibilities: matched filtering and maximum a posteriori (MAP) imaging. These algorithms use the spherical wave equation to construct three-dimensional near-field image cubes of RFI intensity from the visibilities. The matched filter algorithm can generate normalized maps by cross-correlating the expected contributions from RFI sources with the observed visibilities, while the MAP method performs a regularized inversion of the visibility equation in the near field. We developed a full polarization simulation framework for RFI and demonstrated the methods on simulated observations of local RFI sources. The stability, speed, and errors introduced by these algorithms were investigated, and, as a demonstration, the algorithms were applied to a subset of NenuFAR observations to perform spatial, spectral, and temporal characterization of two local RFI sources. We used simulations to assess the impact of local RFI on images, the uv plane, and cylindrical power spectra, and to quantify the level of bias introduced by the algorithms in order to understand their implications for the estimated 21 cm power spectrum with radio interferometers. The near-field imaging and simulation codes are publicly available in the Python library nfis.
△ Less
Submitted 11 April, 2025; v1 submitted 27 March, 2025;
originally announced March 2025.
-
Square Kilometre Array Science Data Challenge 3a: foreground removal for an EoR experiment
Authors:
A. Bonaldi,
P. Hartley,
R. Braun,
S. Purser,
A. Acharya,
K. Ahn,
M. Aparicio Resco,
O. Bait,
M. Bianco,
A. Chakraborty,
E. Chapman,
S. Chatterjee,
K. Chege,
H. Chen,
X. Chen,
Z. Chen,
L. Conaboy,
M. Cruz,
L. Darriba,
M. De Santis,
P. Denzel,
K. Diao,
J. Feron,
C. Finlay,
B. Gehlot
, et al. (159 additional authors not shown)
Abstract:
We present and analyse the results of the Science data challenge 3a (SDC3a, https://sdc3.skao.int/challenges/foregrounds), an EoR foreground-removal community-wide exercise organised by the Square Kilometre Array Observatory (SKAO). The challenge ran for 8 months, from March to October 2023. Participants were provided with realistic simulations of SKA-Low data between 106 MHz and 196 MHz, includin…
▽ More
We present and analyse the results of the Science data challenge 3a (SDC3a, https://sdc3.skao.int/challenges/foregrounds), an EoR foreground-removal community-wide exercise organised by the Square Kilometre Array Observatory (SKAO). The challenge ran for 8 months, from March to October 2023. Participants were provided with realistic simulations of SKA-Low data between 106 MHz and 196 MHz, including foreground contamination from extragalactic as well as Galactic emission, instrumental and systematic effects. They were asked to deliver cylindrical power spectra of the EoR signal, cleaned from all corruptions, and the corresponding confidence levels. Here we describe the approaches taken by the 17 teams that completed the challenge, and we assess their performance using different metrics.
The challenge results provide a positive outlook on the capabilities of current foreground-mitigation approaches to recover the faint EoR signal from SKA-Low observations. The median error committed in the EoR power spectrum recovery is below the true signal for seven teams, although in some cases there are some significant outliers. The smallest residual overall is $4.2_{-4.2}^{+20} \times 10^{-4}\,\rm{K}^2h^{-3}$cMpc$^{3}$ across all considered scales and frequencies.
The estimation of confidence levels provided by the teams is overall less accurate, with the true error being typically under-estimated, sometimes very significantly. The most accurate error bars account for $60 \pm 20$\% of the true errors committed. The challenge results provide a means for all teams to understand and improve their performance. This challenge indicates that the comparison between independent pipelines could be a powerful tool to assess residual biases and improve error estimation.
△ Less
Submitted 14 March, 2025;
originally announced March 2025.
-
Deeper multi-redshift upper limits on the Epoch of Reionization 21-cm signal power spectrum from LOFAR between z=8.3 and z=10.1
Authors:
F. G. Mertens,
M. Mevius,
L. V. E. Koopmans,
A. R. Offringa,
S. Zaroubi,
A. Acharya,
S. A. Brackenhoff,
E. Ceccotti,
E. Chapman,
K. Chege,
B. Ciardi,
R. Ghara,
S. Ghosh,
S. K. Giri,
I. Hothi,
C. Höfer,
I. T. Iliev,
V. Jelić,
Q. Ma,
G. Mellema,
S. Munshi,
V. N. Pandey,
S. Yatawatta
Abstract:
We present new upper limits on the 21-cm signal power spectrum from the Epoch of Reionization (EoR), at redshifts $z \approx 10.1, 9.1, \text{ and } 8.3$, based on reprocessed observations from the Low-Frequency Array (LOFAR). The analysis incorporates significant enhancements in calibration methods, sky model subtraction, radio-frequency interference (RFI) mitigation, and an improved signal separ…
▽ More
We present new upper limits on the 21-cm signal power spectrum from the Epoch of Reionization (EoR), at redshifts $z \approx 10.1, 9.1, \text{ and } 8.3$, based on reprocessed observations from the Low-Frequency Array (LOFAR). The analysis incorporates significant enhancements in calibration methods, sky model subtraction, radio-frequency interference (RFI) mitigation, and an improved signal separation technique using machine learning to develop a physically motivated covariance model for the 21-cm signal. These advancements have markedly reduced previously observed excess power due to residual systematics, bringing the measurements closer to the theoretical thermal noise limit across the entire $k$-space. Using comparable observational data, we achieve a 2 to 4-fold improvement over our previous LOFAR limits, with best upper limits of $Δ_{21}^2 < (68.7\,\mathrm{mK})^2$ at $k = 0.076\,h\,\mathrm{cMpc}^{-1}$, $Δ_{21}^2 < (54.3\,\mathrm{mK})^2$ at $k = 0.076\,h\,\mathrm{cMpc}^{-1}$ and $Δ_{21}^2 < (65.5\,\mathrm{mK})^2$ at $k = 0.083\,h\,\mathrm{cMpc}^{-1}$ at redshifts $z \approx 10.1, 9.1$, and $8.3$, respectively. These new multi-redshift upper limits provide new constraints that can be used to refine our understanding of the astrophysical processes during the EoR. Comprehensive validation tests, including signal injection, were performed to ensure the robustness of our methods. The remaining excess power is attributed to residual foreground emissions from distant sources, beam model inaccuracies, and low-level RFI. We discuss ongoing and future improvements to the data processing pipeline aimed at further reducing these residuals, thereby enhancing the sensitivity of LOFAR observations in the quest to detect the 21-cm signal from the EoR.
△ Less
Submitted 22 April, 2025; v1 submitted 7 March, 2025;
originally announced March 2025.
-
Spectral modelling of Cygnus A between 110 and 250 MHz. Impact on the LOFAR 21-cm signal power spectrum
Authors:
E. Ceccotti,
A. R. Offringa,
L. V. E. Koopmans,
F. G. Mertens,
M. Mevius,
A. Acharya,
S. A. Brackenhoff,
B. Ciardi,
B. K. Gehlot,
R. Ghara,
J. K. Chege,
S. Ghosh,
C. Höfer,
I. Hothi,
I. T. Iliev,
J. P. McKean,
S. Munshi,
S. Zaroubi
Abstract:
Studying the redshifted 21-cm signal from the the neutral hydrogen during the Epoch of Reionization and Cosmic Dawn is fundamental for understanding the physics of the early universe. One of the challenges that 21-cm experiments face is the contamination by bright foreground sources, such as Cygnus A, for which accurate spatial and spectral models are needed to minimise the residual contamination…
▽ More
Studying the redshifted 21-cm signal from the the neutral hydrogen during the Epoch of Reionization and Cosmic Dawn is fundamental for understanding the physics of the early universe. One of the challenges that 21-cm experiments face is the contamination by bright foreground sources, such as Cygnus A, for which accurate spatial and spectral models are needed to minimise the residual contamination after their removal. In this work, we develop a new, high-resolution model of Cygnus A using Low Frequency Array (LOFAR) observations in the $110{-}250$ MHz range, improving upon previous models by incorporating physical spectral information through the forced-spectrum method during multi-frequency deconvolution. This approach addresses the limitations of earlier models by providing a more accurate representation of the complex structure and spectral behaviour of Cygnus A, including the spectral turnover in its brightest hotspots. The impact of this new model on the LOFAR 21-cm signal power spectrum is assessed by comparing it with both simulated and observed North Celestial Pole data sets. Significant improvements are observed in the cylindrical power spectrum along the Cygnus A direction, highlighting the importance of having spectrally accurate models of the brightest foreground sources. However, this improvement is washed out in the spherical power spectrum, where we measure differences of a few hundred mK at $k<0.63\,h\,\text{cMpc}^{-1}$, but not statistically significant. The results suggest that other systematic effects must be mitigated before a substantial impact on 21-cm power spectrum can be achieved.
△ Less
Submitted 25 February, 2025;
originally announced February 2025.
-
Extracting the Epoch of Reionization Signal with 3D U-Net Neural Networks Using Data-driven Systematic Effect Model
Authors:
Li-Yang Gao,
Léon V. E. Koopmans,
Florent G. Mertens,
Satyapan Munshi,
Yichao Li,
Stefanie A. Brackenhoff,
Emilio Ceccotti,
J. Kariuki Chege,
Anshuman Acharya,
Raghunath Ghara,
Sambit K. Giri,
Ilian T. Iliev,
Garrelt Mellema,
Xin Zhang
Abstract:
Neutral hydrogen (HI) serves as a crucial probe for the Cosmic Dawn and the Epoch of Reionization (EoR). Actual observations of the 21-cm signal often encounter challenges such as thermal noise and various systematic effects. To overcome these challenges, we simulate SKA-Low-depth images in South Celestial Pole (SCP) field and process them with a deep learning method. We utilized foreground residu…
▽ More
Neutral hydrogen (HI) serves as a crucial probe for the Cosmic Dawn and the Epoch of Reionization (EoR). Actual observations of the 21-cm signal often encounter challenges such as thermal noise and various systematic effects. To overcome these challenges, we simulate SKA-Low-depth images in South Celestial Pole (SCP) field and process them with a deep learning method. We utilized foreground residuals acquired by LOFAR during actual North Celestial Pole (NCP) field observations, thermal and excess variances calculated via Gaussian process regression (GPR), and 21-cm signals generated with 21cmFAST for signal extraction tests. Our approach to overcome these foreground, thermal noise, and excess variance components employs a 3D U-Net neural network architecture for image analysis. When considering thermal noise corresponding to 1752 hours of integration time, U-Net provides reliable 2D power spectrum predictions, and robustness tests ensure that we get realistic EoR signals. Adding foreground residuals, however, causes inconsistencies below the horizon delay-line. Lastly, evaluating both thermal noise and excess variances with observations up to 4380 hours and 13140 hours ensures reliable power spectrum estimations within the EoR window and across nearly all scales, respectively. The incoherence of excess variances in the frequency direction can greatly affect deep learning to extract 21-cm signals.
△ Less
Submitted 8 June, 2025; v1 submitted 21 December, 2024;
originally announced December 2024.
-
Joint Bayesian Parameter and Model Order Estimation for Low-Rank Probability Mass Tensors
Authors:
Joseph K. Chege,
Arie Yeredor,
Martin Haardt
Abstract:
Obtaining a reliable estimate of the joint probability mass function (PMF) of a set of random variables from observed data is a significant objective in statistical signal processing and machine learning. Modelling the joint PMF as a tensor that admits a low-rank canonical polyadic decomposition (CPD) has enabled the development of efficient PMF estimation algorithms. However, these algorithms req…
▽ More
Obtaining a reliable estimate of the joint probability mass function (PMF) of a set of random variables from observed data is a significant objective in statistical signal processing and machine learning. Modelling the joint PMF as a tensor that admits a low-rank canonical polyadic decomposition (CPD) has enabled the development of efficient PMF estimation algorithms. However, these algorithms require the rank (model order) of the tensor to be specified beforehand. In real-world applications, the true rank is unknown. Therefore, an appropriate rank is usually selected from a candidate set either by observing validation errors or by computing various likelihood-based information criteria, a procedure that could be costly in terms of computational time or hardware resources, or could result in mismatched models which affect the model accuracy. This paper presents a novel Bayesian framework for estimating the low-rank components of a joint PMF tensor and simultaneously inferring its rank from the observed data. We specify a Bayesian PMF estimation model and employ appropriate prior distributions for the model parameters, allowing the rank to be inferred without cross-validation.We then derive a deterministic solution based on variational inference (VI) to approximate the posterior distributions of various model parameters. Numerical experiments involving both synthetic data and real classification and item recommendation data illustrate the advantages of our VI-based method in terms of estimation accuracy, automatic rank detection, and computational efficiency.
△ Less
Submitted 9 July, 2026; v1 submitted 8 October, 2024;
originally announced October 2024.
-
Ionospheric contributions to the excess power in high-redshift 21-cm power-spectrum observations with LOFAR
Authors:
S. A. Brackenhoff,
M. Mevius,
L. V. E. Koopmans,
A. Offringa,
E. Ceccotti,
J. K. Chege,
B. K. Gehlot,
S. Ghosh,
C. Höfer,
F. G. Mertens,
S. Munshi,
S. Zaroubi
Abstract:
The turbulent ionosphere causes phase shifts to incoming radio waves on a broad range of temporal and spatial scales. When an interferometer is not sufficiently calibrated for the direction-dependent ionospheric effects, the time-varying phase shifts can cause the signal to decorrelate. The ionosphere's influence over various spatiotemporal scales introduces a baseline-dependent effect on the inte…
▽ More
The turbulent ionosphere causes phase shifts to incoming radio waves on a broad range of temporal and spatial scales. When an interferometer is not sufficiently calibrated for the direction-dependent ionospheric effects, the time-varying phase shifts can cause the signal to decorrelate. The ionosphere's influence over various spatiotemporal scales introduces a baseline-dependent effect on the interferometric array. We study the impact of baseline-dependent decorrelation on high-redshift observations with the Low Frequency Array (LOFAR). Datasets with a range of ionospheric corruptions are simulated using a thin-screen ionosphere model, and calibrated using the state-of-the-art LOFAR Epoch of Reionisation pipeline. For the first time ever, we show the ionospheric impact on various stages of the calibration process including an analysis of the transfer of gain errors from longer to shorter baselines using realistic end-to-end simulations. We find that direction-dependent calibration for source subtraction leaves excess power of up to two orders of magnitude above the thermal noise at the largest spectral scales in the cylindrically averaged auto-power spectrum under normal ionospheric conditions. However, we demonstrate that this excess power can be removed through Gaussian process regression, leaving no excess power above the ten per cent level for a $5~$km diffractive scale. We conclude that ionospheric errors, in the absence of interactions with other aggravating effects, do not constitute a dominant component in the excess power observed in LOFAR Epoch of Reionisation observations of the North Celestial Pole. Future work should therefore focus on less spectrally smooth effects, such as beam modelling errors.
△ Less
Submitted 29 July, 2024;
originally announced July 2024.
-
The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR
Authors:
J. K. Chege,
L. V. E. Koopmans,
A. R. Offringa,
B. K. Gehlot,
S. A. Brackenhoff,
E. Ceccotti,
S. Ghosh,
C. Höfer,
F. G. Mertens,
M. Mevius,
S. Munshi
Abstract:
Current radio interferometers output multi-petabyte-scale volumes of data per year making the storage, transfer, and processing of this data a sizeable challenge. This challenge is expected to grow with the next-generation telescopes such as the Square Kilometre Array. Lossy compression of interferometric data post-correlation can abate this challenge. However, since high-redshift 21-cm studies im…
▽ More
Current radio interferometers output multi-petabyte-scale volumes of data per year making the storage, transfer, and processing of this data a sizeable challenge. This challenge is expected to grow with the next-generation telescopes such as the Square Kilometre Array. Lossy compression of interferometric data post-correlation can abate this challenge. However, since high-redshift 21-cm studies impose strict precision requirements, the impact of such lossy data compression on the 21-cm signal power spectrum statistic should be understood. We apply Dysco visibility compression, a technique to normalize and quantize specifically designed for radio interferometric data. We establish the level of the compression noise in the power spectrum in comparison to the thermal noise as well as its coherency behavior. Finally, for optimal compression results, we compare the compression noise obtained from different compression settings to a nominal 21-cm signal power. From a single night of observation, we find that the noise introduced due to the compression is more than five orders of magnitude lower than the thermal noise level in the power spectrum. The noise does not affect calibration. The compression noise shows no correlation with the sky signal and has no measurable coherent component. The level of compression error in the power spectrum ultimately depends on the compression settings. Dysco visibility compression is found to be of insignificant concern for 21-cm power spectrum studies. Hence, data volumes can be safely reduced by factors of $\sim 4$ and with insignificant bias to the final power spectrum. Data from SKA-low will likely be compressible by the same factor as LOFAR, owing to the similarities of the two instruments. The same technique can be used to compress data from other telescopes, but a small adjustment of the compression parameters might be required.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Beyond the horizon: Quantifying the full sky foreground wedge in the cylindrical power spectrum
Authors:
S. Munshi,
F. G. Mertens,
L. V. E. Koopmans,
A. R. Offringa,
E. Ceccotti,
S. A. Brackenhoff,
J. K. Chege,
B. K. Gehlot,
S. Ghosh,
C. Höfer,
M. Mevius
Abstract:
One of the main obstacles preventing the detection of the redshifted 21-cm signal from neutral hydrogen in the early Universe is the astrophysical foreground emission, which is several orders of magnitude brighter than the signal. The foregrounds, due to their smooth spectra, are expected to predominantly occupy a region in the cylindrical power spectrum known as the foreground wedge. However, the…
▽ More
One of the main obstacles preventing the detection of the redshifted 21-cm signal from neutral hydrogen in the early Universe is the astrophysical foreground emission, which is several orders of magnitude brighter than the signal. The foregrounds, due to their smooth spectra, are expected to predominantly occupy a region in the cylindrical power spectrum known as the foreground wedge. However, the conventional equations describing the extent of the foreground wedge are derived under a flat-sky approximation. This assumption breaks down for tracking wide-field instruments, thus rendering these equations inapplicable in these situations. In this paper, we derive equations for the full sky foreground wedge and show that the foregrounds can potentially extend far beyond what the conventional equations suggest. We also derive the equations that describe a specific bright source in the cylindrical power spectrum space. The validity of both sets of equations is tested against numerical simulations. Many current and upcoming interferometers (e.g., LOFAR, NenuFAR, MWA, SKA) are wide-field phase-tracking instruments. These equations give us new insights into the nature of foreground contamination in the cylindrical power spectra estimated using wide-field instruments. Additionally, they allow us to accurately associate features in the power spectrum to foregrounds or instrumental effects. The equations are also important for correctly selecting the "EoR window" for foreground avoidance analyses, and for planning 21-cm observations. In future analyses, it is recommended to use these updated horizon lines to indicate the foreground wedge in the cylindrical power spectrum accurately. The new equations for generating the updated wedge lines are made available in a Python library, pslines.
△ Less
Submitted 10 December, 2024; v1 submitted 15 July, 2024;
originally announced July 2024.
-
Optimising MWA EoR data processing for improved 21 cm power spectrum measurements -- fine-tuning ionospheric corrections
Authors:
J. Kariuki Chege,
C. H. Jordan,
C. Lynch,
C. M. Trott,
J. L. B. Line,
B. Pindor,
S. Yoshiura
Abstract:
The redshifted cosmological 21 cm signal emitted by neutral hydrogen during the first billion years of the universe is much fainter relative to other galactic and extragalactic radio emissions, posing a great challenge towards detection of the signal. Therefore, precise instrumental calibration is a vital prerequisite for the success of radio interferometers such as the Murchison Widefield Array (…
▽ More
The redshifted cosmological 21 cm signal emitted by neutral hydrogen during the first billion years of the universe is much fainter relative to other galactic and extragalactic radio emissions, posing a great challenge towards detection of the signal. Therefore, precise instrumental calibration is a vital prerequisite for the success of radio interferometers such as the Murchison Widefield Array (MWA), which aim for a 21 cm detection. Over the previous years, novel calibration techniques targeting the power spectrum paradigm of EoR science have been actively researched and where possible implemented. Using recently acquired computation resources for the MWA, we test the full capabilities of the state-of-the-art calibration techniques available for the MWA EoR project, with a focus on both direction dependent and direction independent calibration. Specifically, we investigate improvements that can be made in the vital calibration stages of sky modelling, ionospheric correction, and compact source foreground subtraction as applied in the hybrid foreground mitigation approach (one that combines both foreground subtraction and avoidance). Additionally, we investigate a method of ionospheric correction using interpolated ionospheric phase screens and assess its performance in the power spectrum space. Overall, we identify a refined RTS calibration configuration that leads to an at least 2 factor reduction of the EoR window power contamination at the $0.1 \; \text{hMpc}^{-1}$ scale. The improvement marks a step further towards detecting the 21 cm signal using the MWA and the forthcoming SKA low telescope.
△ Less
Submitted 25 July, 2022;
originally announced July 2022.
-
The MWA Long Baseline Epoch of Reionisation Survey: I. Improved Source Catalogue for the EoR 0 field
Authors:
C. R. Lynch,
T. J. Galvin,
J. L. B. Line,
C. H. Jordan,
C. M. Trott,
J. K. Chege,
B. McKinley,
M. Johnston-Hollitt,
S. J. Tingay
Abstract:
One of the principal systematic constraints on the Epoch of Reionisation (EoR) experiment is the accuracy of the foreground calibration model. Recent results have shown that highly accurate models of extended foreground sources, and including models for sources in both the primary beam and its sidelobes, are necessary for reducing foreground power. To improve the accuracy of the source models for…
▽ More
One of the principal systematic constraints on the Epoch of Reionisation (EoR) experiment is the accuracy of the foreground calibration model. Recent results have shown that highly accurate models of extended foreground sources, and including models for sources in both the primary beam and its sidelobes, are necessary for reducing foreground power. To improve the accuracy of the source models for the EoR fields observed by the Murchison Widefield Array (MWA), we conducted the MWA Long Baseline Epoch of Reionisation Survey (LoBES). This survey consists of multi-frequency observations of the main MWA EoR fields and their eight neighbouring fields using the MWA Phase II extended array. We present the results of the first half of this survey centred on the MWA EoR0 observing field (centred at RA(J2000) 0 h, Dec(J2000) -27 deg). This half of the survey covers an area of 3069 degrees$^2$, with an average rms of 2.1 mJy beam$^{-1}$. The resulting catalogue contains a total of 80824 sources, with 16 separate spectral measurements between 100 and 230 MHz, and spectral modelling for 78$\%$ of these sources. Over this region we estimate that the catalogue is 90$\%$ complete at 32 mJy, and 70$\%$ complete at 10.5~mJy. The overall normalised source counts are found to be in good agreement with previous low-frequency surveys at similar sensitivities. Testing the performance of the new source models we measure lower residual rms values for peeled sources, particularly for extended sources, in a set of MWA Phase I data. The 2-dimensional power spectrum of these data residuals also show improvement on small angular scales -- consistent with the better angular resolution of the LoBES catalogue. It is clear that the LoBES sky models improve upon the current sky model used by the Australian MWA EoR group for the EoR0 field.
△ Less
Submitted 15 October, 2021;
originally announced October 2021.
-
Simulations of ionospheric refraction on radio interferometric data
Authors:
J. Kariuki Chege,
C. H. Jordan,
C. Lynch,
J. L. B. Line,
C. M. Trott
Abstract:
The Epoch of Reionisation (EoR) is the period within which the neutral universe transitioned to an ionised one. This period remains unobserved using low-frequency radio interferometers which target the 21 cm signal of neutral hydrogen emitted in this era. The Murchison Widefield Array (MWA) radio telescope was built with the detection of this signal as one of its major science goals. One of the mo…
▽ More
The Epoch of Reionisation (EoR) is the period within which the neutral universe transitioned to an ionised one. This period remains unobserved using low-frequency radio interferometers which target the 21 cm signal of neutral hydrogen emitted in this era. The Murchison Widefield Array (MWA) radio telescope was built with the detection of this signal as one of its major science goals. One of the most significant challenges towards a successful detection is that of calibration, especially in the presence of the Earth's ionosphere. By introducing refractive source shifts, distorting source shapes and scintillating flux densities, the ionosphere is a major nuisance in low-frequency radio astronomy. We introduce SIVIO, a software tool developed for simulating observations of the MWA through different ionospheric conditions estimated using thin screen approximation models and propagated into the visibilities. This enables us to directly assess the impact of the ionosphere on observed EoR data and the resulting power spectra. We show that the simulated data captures the dispersive behaviour of ionospheric effects. We show that the spatial structure of the simulated ionospheric media is accurately reconstructed either from the resultant source positional offsets or from parameters evaluated during the data calibration procedure. In turn, this will inform on the best strategies of identifying and efficiently eliminating ionospheric contamination in EoR data moving into the Square Kilometre Array era.
△ Less
Submitted 10 May, 2021;
originally announced May 2021.