-
When One Modality Is Not Enough: Multimodal Sex and Life-Stage Classification of Red Deer from Aerial RGB-Thermal Video
Authors:
Hugo Markoff,
Christoph Praschl,
Ivan Ludoški,
Sara Beery,
Michael Ørsted,
David C. Schedl
Abstract:
Aerial drone surveys increasingly support wildlife population estimation, yet a useful census is more than a count: population dynamics are defined by species composition, sex ratios and age structure, that is, by which species are present and how a herd splits into adult males, adult females and juveniles. We use red deer ($\textit{Cervus elaphus}$) as a test case, because managers act on these d…
▽ More
Aerial drone surveys increasingly support wildlife population estimation, yet a useful census is more than a count: population dynamics are defined by species composition, sex ratios and age structure, that is, by which species are present and how a herd splits into adult males, adult females and juveniles. We use red deer ($\textit{Cervus elaphus}$) as a test case, because managers act on these dynamics and because the visible cue defining adult males, the antlers, is seasonally variable. Surveys are flown nadir, high enough not to disturb the animals, so each deer occupies only a small, low-resolution patch. The two recording modalities fail in opposite conditions: in color a deer under canopy blends into the ground, while in thermal it becomes a bright blob that loses fine detail. Rather than trust either modality alone, we fuse them at every stage using self-supervised DINOv3 features. Our pipeline tracks animals in both modalities, treats an animal as confirmed only when the two cameras agree, keeps only the clear, non-occluded frames, and assigns species and sex by a vote across them; life stage is read separately from geo-referenced body size, since at survey resolution a juvenile often only differs from an adult female in size. Across four flights spanning the antler season the fused pipeline correctly classifies 25 of the 26 detected individuals (7 of 8 adult males, all 16 adult females and 2 juveniles), against 20 of 26 for either sensor alone. Multimodal species classification reaches 96.0%, while for sex classification fusing the two sensors matters most: the combined RGB+thermal model is the most robust across environments and seasons. Automating the demographic classification turns a drone flight from a count into a repeatable reading of herd structure, so the sex ratios and age structure that managers already act on can be gathered as often as a survey can be flown.
△ Less
Submitted 7 August, 2026;
originally announced August 2026.
-
Oh Deer, How Should I Handle This? Seasonal Priors for Selective Wildlife Annotation and Classification
Authors:
Hugo Markoff,
Christoph Praschl,
Anton Hjalte Jørgensen,
Christian Emil Mogensen,
Mathias Bech Skadhauge,
Sara Beery,
Michael Ørsted,
David C. Schedl
Abstract:
Fine-grained wildlife classification in aerial imagery is limited not only by model performance, but also by unreliable labels: animals occupy few pixels, key visual cues vary seasonally, and modality-specific evidence can be ambiguous. We study adult-male identification in red deer ($\textit{Cervus elaphus}$), where the antler cycle defines predictable windows of reliable evidence for both annota…
▽ More
Fine-grained wildlife classification in aerial imagery is limited not only by model performance, but also by unreliable labels: animals occupy few pixels, key visual cues vary seasonally, and modality-specific evidence can be ambiguous. We study adult-male identification in red deer ($\textit{Cervus elaphus}$), where the antler cycle defines predictable windows of reliable evidence for both annotation and prediction. Using 7,295 RGB-only, thermal-only, and matched RGB+thermal crop sets from low-altitude UAV surveys, labeled by three annotators, we show that seasonal structure links (I) annotation quality, (II) downstream classification, and (III) selective prediction. Matched RGB+thermal review resolves more samples than either single modality, recovering majority-male labels otherwise missed by RGB or thermal alone, in human-based as well as model-based classification. Months with high annotator abstention also show lower classifier confidence, and soft seasonal priors mainly benefit the season-limited thermal view. Uncertainty-band abstention further raises covered accuracy to 98.9%, though at reduced coverage and with deferral that falls disproportionately on males. Overall, a biologically grounded seasonal calendar predicts where annotation and prediction are unreliable, and can guide both annotation protocol design and modality weighting.
△ Less
Submitted 9 August, 2026; v1 submitted 3 August, 2026;
originally announced August 2026.
-
Detection and Measurement of Hailstones with Multimodal Large Language Models
Authors:
Moritz Alker,
David C. Schedl,
Andreas Stöckl
Abstract:
This study examines the use of social media and news images to detect and measure hailstones, utilizing pre-trained multimodal large language models. The dataset for this study comprises 474 crowdsourced images of hailstones from documented hail events in Austria, which occurred between January 2022 and September 2024. These hailstones have maximum diameters ranging from 2 to 11cm. We estimate the…
▽ More
This study examines the use of social media and news images to detect and measure hailstones, utilizing pre-trained multimodal large language models. The dataset for this study comprises 474 crowdsourced images of hailstones from documented hail events in Austria, which occurred between January 2022 and September 2024. These hailstones have maximum diameters ranging from 2 to 11cm. We estimate the hail diameters and compare four different models utilizing one-stage and two-stage prompting strategies. The latter utilizes additional size cues from reference objects, such as human hands, within the image. Our results show that pretrained models already have the potential to measure hailstone diameters from images with an average mean absolute error of 1.12cm for the best model. In comparison to a single-stage prompt, two-stage prompting improves the reliability of most models. Our study suggests that these off-the-shelf models, even without fine-tuning, can complement traditional hail sensors by extracting meaningful and spatially dense information from social media imagery, enabling faster and more detailed assessments of severe weather events. The automated real-time image harvesting from social media and other sources remains an open task, but it will make our approach directly applicable to future hail events.
△ Less
Submitted 7 October, 2025;
originally announced October 2025.
-
Advancing Wildlife Monitoring: Drone-Based Sampling for Roe Deer Density Estimation
Authors:
Stephanie Wohlfahrt,
Christoph Praschl,
Horst Leitner,
Wolfram Jantsch,
Julia Konic,
Silvio Schueler,
Andreas Stöckl,
David C. Schedl
Abstract:
We use unmanned aerial drones to estimate wildlife density in southeastern Austria and compare these estimates to camera trap data. Traditional methods like capture-recapture, distance sampling, or camera traps are well-established but labour-intensive or spatially constrained. Using thermal (IR) and RGB imagery, drones enable efficient, non-intrusive animal counting. Our surveys were conducted du…
▽ More
We use unmanned aerial drones to estimate wildlife density in southeastern Austria and compare these estimates to camera trap data. Traditional methods like capture-recapture, distance sampling, or camera traps are well-established but labour-intensive or spatially constrained. Using thermal (IR) and RGB imagery, drones enable efficient, non-intrusive animal counting. Our surveys were conducted during the leafless period on single days in October and November 2024 in three areas of a sub-Illyrian hill and terrace landscape. Flight transects were based on predefined launch points using a 350 m grid and an algorithm that defined the direction of systematically randomized transects. This setup allowed surveying large areas in one day using multiple drones, minimizing double counts. Flight altitude was set at 60 m to avoid disturbing roe deer (Capreolus capreolus) while ensuring detection. Animals were manually annotated in the recorded imagery and extrapolated to densities per square kilometer. We applied three extrapolation methods with increasing complexity: naive area-based extrapolation, bootstrapping, and zero-inflated negative binomial modelling. For comparison, a Random Encounter Model (REM) estimate was calculated using camera trap data from the flight period. The drone-based methods yielded similar results, generally showing higher densities than REM, except in one area in October. We hypothesize that drone-based density reflects daytime activity in open and forested areas, while REM estimates average activity over longer periods within forested zones. Although both approaches estimate density, they offer different perspectives on wildlife presence. Our results show that drones offer a promising, scalable method for wildlife density estimation.
△ Less
Submitted 5 August, 2025;
originally announced August 2025.
-
The Way Up: A Dataset for Hold Usage Detection in Sport Climbing
Authors:
Anna Maschek,
David C. Schedl
Abstract:
Detecting an athlete's position on a route and identifying hold usage are crucial in various climbing-related applications. However, no climbing dataset with detailed hold usage annotations exists to our knowledge. To address this issue, we introduce a dataset of 22 annotated climbing videos, providing ground-truth labels for hold locations, usage order, and time of use. Furthermore, we explore th…
▽ More
Detecting an athlete's position on a route and identifying hold usage are crucial in various climbing-related applications. However, no climbing dataset with detailed hold usage annotations exists to our knowledge. To address this issue, we introduce a dataset of 22 annotated climbing videos, providing ground-truth labels for hold locations, usage order, and time of use. Furthermore, we explore the application of keypoint-based 2D pose-estimation models for detecting hold usage in sport climbing. We determine usage by analyzing the key points of certain joints and the corresponding overlap with climbing holds. We evaluate multiple state-of-the-art models and analyze their accuracy on our dataset, identifying and highlighting climbing-specific challenges. Our dataset and results highlight key challenges in climbing-specific pose estimation and establish a foundation for future research toward AI-assisted systems for sports climbing.
△ Less
Submitted 19 May, 2025;
originally announced May 2025.
-
Touch Sensing on Semi-Elastic Textiles with Border-Based Sensors
Authors:
Samuel Zühlke,
Andreas Stöckl,
David C. Schedl
Abstract:
This study presents a novel approach for touch sensing using semi-elastic textile surfaces that does not require the placement of additional sensors in the sensing area, instead relying on sensors located on the border of the textile. The proposed approach is demonstrated through experiments involving an elastic Jersey fabric and a variety of machine-learning models. The performance of one particu…
▽ More
This study presents a novel approach for touch sensing using semi-elastic textile surfaces that does not require the placement of additional sensors in the sensing area, instead relying on sensors located on the border of the textile. The proposed approach is demonstrated through experiments involving an elastic Jersey fabric and a variety of machine-learning models. The performance of one particular border-based sensor design is evaluated in depth. By using visual markers, the best-performing visual sensor arrangement predicts a single touch point with a mean squared error of 1.36 mm on an area of 125mm by 125mm. We built a textile only prototype that is able to classify touch at three indent levels (0, 15, and 20 mm) with an accuracy of 82.85%. Our results suggest that this approach has potential applications in wearable technology and smart textiles, making it a promising avenue for further exploration in these fields.
△ Less
Submitted 17 May, 2023; v1 submitted 16 May, 2023;
originally announced May 2023.
-
Through-Foliage Tracking with Airborne Optical Sectioning
Authors:
Rakesh John Amala Arokia Nathan,
Indrajit Kurmi,
David C. Schedl,
Oliver Bimber
Abstract:
Detecting and tracking moving targets through foliage is difficult, and for many cases even impossible in regular aerial images and videos. We present an initial light-weight and drone-operated 1D camera array that supports parallel synthetic aperture aerial imaging. Our main finding is that color anomaly detection benefits significantly from image integration when compared to conventional raw ima…
▽ More
Detecting and tracking moving targets through foliage is difficult, and for many cases even impossible in regular aerial images and videos. We present an initial light-weight and drone-operated 1D camera array that supports parallel synthetic aperture aerial imaging. Our main finding is that color anomaly detection benefits significantly from image integration when compared to conventional raw images or video frames (on average 97% vs. 42% in precision in our field experiments). We demonstrate, that these two contributions can lead to the detection and tracking of moving people through densely occluding forest.
△ Less
Submitted 30 November, 2021; v1 submitted 12 November, 2021;
originally announced November 2021.
-
Combined Person Classification with Airborne Optical Sectioning
Authors:
Indrajit Kurmi,
David C. Schedl,
Oliver Bimber
Abstract:
Fully autonomous drones have been demonstrated to find lost or injured persons under strongly occluding forest canopy. Airborne Optical Sectioning (AOS), a novel synthetic aperture imaging technique, together with deep-learning-based classification enables high detection rates under realistic search-and-rescue conditions. We demonstrate that false detections can be significantly suppressed and tru…
▽ More
Fully autonomous drones have been demonstrated to find lost or injured persons under strongly occluding forest canopy. Airborne Optical Sectioning (AOS), a novel synthetic aperture imaging technique, together with deep-learning-based classification enables high detection rates under realistic search-and-rescue conditions. We demonstrate that false detections can be significantly suppressed and true detections boosted by combining classifications from multiple AOS rather than single integral images. This improves classification rates especially in the presence of occlusion. To make this possible, we modified the AOS imaging process to support large overlaps between subsequent integrals, enabling real-time and on-board scanning and processing of groundspeeds up to 10 m/s.
△ Less
Submitted 18 June, 2021;
originally announced June 2021.
-
An Autonomous Drone for Search and Rescue in Forests using Airborne Optical Sectioning
Authors:
D. C. Schedl,
I. Kurmi,
O. Bimber
Abstract:
Drones will play an essential role in human-machine teaming in future search and rescue (SAR) missions. We present a first prototype that finds people fully autonomously in densely occluded forests. In the course of 17 field experiments conducted over various forest types and under different flying conditions, our drone found 38 out of 42 hidden persons; average precision was 86% for predefined fl…
▽ More
Drones will play an essential role in human-machine teaming in future search and rescue (SAR) missions. We present a first prototype that finds people fully autonomously in densely occluded forests. In the course of 17 field experiments conducted over various forest types and under different flying conditions, our drone found 38 out of 42 hidden persons; average precision was 86% for predefined flight paths, while adaptive path planning (where potential findings are double-checked) increased confidence by 15%. Image processing, classification, and dynamic flight-path adaptation are computed onboard in real-time and while flying. Our finding that deep-learning-based person classification is unaffected by sparse and error-prone sampling within one-dimensional synthetic apertures allows flights to be shortened and reduces recording requirements to one-tenth of the number of images needed for sampling using two-dimensional synthetic apertures. The goal of our adaptive path planning is to find people as reliably and quickly as possible, which is essential in time-critical applications, such as SAR. Our drone enables SAR operations in remote areas without stable network coverage, as it transmits to the rescue team only classification results that indicate detections and can thus operate with intermittent minimal-bandwidth connections (e.g., by satellite). Once received, these results can be visually enhanced for interpretation on remote mobile devices.
△ Less
Submitted 10 May, 2021;
originally announced May 2021.
-
Pose Error Reduction for Focus Enhancement in Thermal Synthetic Aperture Visualization
Authors:
Indrajit Kurmi,
David C. Schedl,
Oliver Bimber
Abstract:
Airborne optical sectioning, an effective aerial synthetic aperture imaging technique for revealing artifacts occluded by forests, requires precise measurements of drone poses. In this article we present a new approach for reducing pose estimation errors beyond the possibilities of conventional Perspective-n-Point solutions by considering the underlying optimization as a focusing problem. We prese…
▽ More
Airborne optical sectioning, an effective aerial synthetic aperture imaging technique for revealing artifacts occluded by forests, requires precise measurements of drone poses. In this article we present a new approach for reducing pose estimation errors beyond the possibilities of conventional Perspective-n-Point solutions by considering the underlying optimization as a focusing problem. We present an efficient image integration technique, which also reduces the parameter search space to achieve realistic processing times, and improves the quality of resulting synthetic integral images.
△ Less
Submitted 15 December, 2020;
originally announced December 2020.
-
Search and Rescue with Airborne Optical Sectioning
Authors:
David C. Schedl,
Indrajit Kurmi,
Oliver Bimber
Abstract:
We show that automated person detection under occlusion conditions can be significantly improved by combining multi-perspective images before classification. Here, we employed image integration by Airborne Optical Sectioning (AOS)---a synthetic aperture imaging technique that uses camera drones to capture unstructured thermal light fields---to achieve this with a precision/recall of 96/93%. Findin…
▽ More
We show that automated person detection under occlusion conditions can be significantly improved by combining multi-perspective images before classification. Here, we employed image integration by Airborne Optical Sectioning (AOS)---a synthetic aperture imaging technique that uses camera drones to capture unstructured thermal light fields---to achieve this with a precision/recall of 96/93%. Finding lost or injured people in dense forests is not generally feasible with thermal recordings, but becomes practical with use of AOS integral images. Our findings lay the foundation for effective future search and rescue technologies that can be applied in combination with autonomous or manned aircraft. They can also be beneficial for other fields that currently suffer from inaccurate classification of partially occluded people, animals, or objects.
△ Less
Submitted 18 September, 2020;
originally announced September 2020.
-
Fast Automatic Visibility Optimization for Thermal Synthetic Aperture Visualization
Authors:
Indrajit Kurmi,
David C. Schedl,
Oliver Bimber
Abstract:
In this article, we describe and validate the first fully automatic parameter optimization for thermal synthetic aperture visualization. It replaces previous manual exploration of the parameter space, which is time consuming and error prone. We prove that the visibility of targets in thermal integral images is proportional to the variance of the targets' image. Since this is invariant to occlusion…
▽ More
In this article, we describe and validate the first fully automatic parameter optimization for thermal synthetic aperture visualization. It replaces previous manual exploration of the parameter space, which is time consuming and error prone. We prove that the visibility of targets in thermal integral images is proportional to the variance of the targets' image. Since this is invariant to occlusion it represents a suitable objective function for optimization. Our findings have the potential to enable fully autonomous search and recuse operations with camera drones.
△ Less
Submitted 8 May, 2020;
originally announced May 2020.
-
A Statistical View on Synthetic Aperture Imaging for Occlusion Removal
Authors:
Indrajit Kurmi,
David C. Schedl,
Oliver Bimber
Abstract:
Synthetic apertures find applications in many fields, such as radar, radio telescopes, microscopy, sonar, ultrasound, LiDAR, and optical imaging. They approximate the signal of a single hypothetical wide aperture sensor with either an array of static small aperture sensors or a single moving small aperture sensor. Common sense in synthetic aperture sampling is that a dense sampling pattern within…
▽ More
Synthetic apertures find applications in many fields, such as radar, radio telescopes, microscopy, sonar, ultrasound, LiDAR, and optical imaging. They approximate the signal of a single hypothetical wide aperture sensor with either an array of static small aperture sensors or a single moving small aperture sensor. Common sense in synthetic aperture sampling is that a dense sampling pattern within a wide aperture is required to reconstruct a clear signal. In this article we show that there exists practical limits to both, synthetic aperture size and number of samples for the application of occlusion removal. This leads to an understanding on how to design synthetic aperture sampling patterns and sensors in a most optimal and practically efficient way. We apply our findings to airborne optical sectioning which uses camera drones and synthetic aperture imaging to computationally remove occluding vegetation or trees for inspecting ground surfaces.
△ Less
Submitted 15 June, 2019;
originally announced June 2019.