Showing 1–18 of 18 results for author: Sousa, P

Earth observation embeddings are effective sub-grid descriptors for probabilistic weather downscaling

Authors: Pedro Sousa, Will Tebbutt, Sadiq Jaffer, Robin Young, Anil Madhavapeddy, Richard E. Turner

Abstract: Global weather reanalyses and forecasts resolve the evolving atmospheric state on coarse grids, but site-specific applications require predictions at arbitrary locations where near-surface conditions also depend on unresolved terrain and land-surface properties. Existing probabilistic downscalers address this gap using hand-crafted topographic descriptors. We ask instead whether Earth observation… ▽ More Global weather reanalyses and forecasts resolve the evolving atmospheric state on coarse grids, but site-specific applications require predictions at arbitrary locations where near-surface conditions also depend on unresolved terrain and land-surface properties. Existing probabilistic downscalers address this gap using hand-crafted topographic descriptors. We ask instead whether Earth observation foundation models can provide transferable sub-grid surface representations for probabilistic weather downscaling. We augment a convolutional conditional neural process that downscales coarse ERA5 reanalysis fields at ~25 km resolution with a learned local surface descriptor, obtained by compressing a patch of TESSERA embeddings at 10 m resolution. Although these embeddings summarise surface conditions over annual timescales, they improve downscaling of instantaneous 2 m temperature and 10 m wind speed by encoding persistent surface properties that capture a location's departure from the coarse-grid atmospheric state. Across five climatically diverse regions, the embedding improves point and probabilistic skill at stations held out in both space and time, overall improving CRPS skill by 11.5% for 2 m temperature and 6.2% for 10 m wind speed. We further analyse how its contribution differs by variable, finding that topography explains more of temperature's sub-grid structure, while TESSERA provides additional surface information for wind speed. These improvements persist when the coarse input is changed from ERA5 to forecasts from the Aurora AI forecasting model, and when predicting at newly deployed stations with no regional history. To our knowledge, this is the first evidence that long-timescale Earth-observation embeddings can support short-timescale weather downscaling where sub-grid departures are systematically structured by persistent surface properties. △ Less

Submitted 12 August, 2026; originally announced August 2026.

Comments: 39 pages, 12 figures, 6 tables

GLidE-SLAM: GL-Accelerated Indirect-Direct Embedded SLAM

Authors: Carlos A. Pinheiro de Sousa, Heiko Hamann, Oliver Deussen

Abstract: With the growing demand for robotics, autonomous drones, and wearable extended reality systems, the deployment of Visual SLAM on embedded devices remains challenging. Tracking must sustain high frame rates while preserving compute resources for map extension and maintenance. This paper presents GLidE-SLAM, a monocular hybrid indirect-direct framework that addresses this by architectural separation… ▽ More With the growing demand for robotics, autonomous drones, and wearable extended reality systems, the deployment of Visual SLAM on embedded devices remains challenging. Tracking must sustain high frame rates while preserving compute resources for map extension and maintenance. This paper presents GLidE-SLAM, a monocular hybrid indirect-direct framework that addresses this by architectural separation: the system performs GPU-accelerated direct tracking on intermediate frames, while reserving the full indirect pipeline for map extension and global consistency. We leverage highly parallel image-alignment operations for pose-only estimation without depth optimization or map point creation, making the workload suitable for GPU offloading and freeing CPU resources for backend tasks. We implement the direct tracker using vendor-agnostic OpenGL ES~3.1 compute shaders, enabling deployment across a broader range of commodity embedded platforms without requiring CUDA support. To our knowledge, this is the first complete direct photometric pose estimator realized via compute shaders for embedded-class devices. Experiments on target platforms demonstrate up to 9$\times$ higher frame rates than the CPU-only baseline while maintaining trajectory accuracy and improving practical deployment across commodity resource-constrained hardware. △ Less

Submitted 18 July, 2026; originally announced July 2026.

Comments: 7 pages, 6 figures, 4 tables. Accepted for presentation at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026)

TESSERA v2: Scaling Pixel-wise Earth Foundation Models

Authors: Zhengpeng Feng, Sadiq Jaffer, Ira Shokar, Jovana Knezevic, James Ball, Pedro Sousa, Mark Elvers, Madeline Lisaius, Clement Atzberger, Robin Young, Aneesh Naik, Niall Robinson, David Coomes, Anil Madhavapeddy, Srinivasan Keshav

Abstract: Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise Barlow Twins family, each evaluated on 15 diverse downstre… ▽ More Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise Barlow Twins family, each evaluated on 15 diverse downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise teachers (0.5B, 1B, and 2B) and distil the largest into compact students for embeddings-as-data deployment. In aggregate, our 44-million-parameter distilled student outperforms every open and proprietary embedding product we test, several of them an order of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. We plan to release global 10 m annual embeddings covering 2017-2025 as version 2 of the TESSERA foundation-model embeddings product. All code is available at: https://github.com/ucam-eo/tessera △ Less

Submitted 6 August, 2026; v1 submitted 4 July, 2026; originally announced July 2026.

HUydra: Full-Range Lung CT Synthesis via Multiple HU Interval Generative Modelling

Authors: António Cardoso, Pedro Sousa, Tania Pereira, Hélder P. Oliveira

Abstract: Currently, a central challenge and bottleneck in the deployment and validation of computer-aided diagnosis (CAD) models within the field of medical imaging is data scarcity. For lung cancer, one of the most prevalent types worldwide, limited datasets can delay diagnosis and have an impact on patient outcome. Generative AI offers a promising solution for this issue, but dealing with the complex dis… ▽ More Currently, a central challenge and bottleneck in the deployment and validation of computer-aided diagnosis (CAD) models within the field of medical imaging is data scarcity. For lung cancer, one of the most prevalent types worldwide, limited datasets can delay diagnosis and have an impact on patient outcome. Generative AI offers a promising solution for this issue, but dealing with the complex distribution of full Hounsfield Unit (HU) range lung CT scans is challenging and remains as a highly computationally demanding task. This paper introduces a novel decomposition strategy that synthesizes CT images one HU interval at a time, rather than modelling the entire HU domain at once. This framework focuses on training generative architectures on individual tissue-focused HU windows, then merges their output into a full-range scan via a learned reconstruction network that effectively reverses the HU-windowing process. We further propose multi-head and multi-decoder models to better capture textures while preserving anatomical consistency, with a multi-head VQVAE achieving the best performance for the generative task. Quantitative evaluation shows this approach significantly outperforms conventional 2D full-range baselines, achieving a 6.2% improvement in FID and superior MMD, Precision, and Recall across all HU intervals. The best performance is achieved by a multi-head VQVAE variant, demonstrating that it is possible to enhance visual fidelity and variability while also reducing model complexity and computational cost. This work establishes a new paradigm for structure-aware medical image synthesis, aligning generative modelling with clinical interpretation. △ Less

Submitted 24 March, 2026; originally announced March 2026.

Comments: Submitted to iEEE TPAMI (Transactions on Pattern Analysis and Machine Intelligence)

RunPBA -- Runtime attestation for microcontrollers with PACBTI

Authors: André Cirne, Patrícia R. Sousa, João S. Resende, Luís Antunes

Abstract: The widespread adoption of embedded systems has led to their deployment in critical real-world applications, making them attractive targets for malicious actors. These devices face unique challenges in mitigating vulnerabilities due to intrinsic constraints, such as low energy consumption requirements and limited computational resources. This paper presents RunPBA, a hardware-based runtime attesta… ▽ More The widespread adoption of embedded systems has led to their deployment in critical real-world applications, making them attractive targets for malicious actors. These devices face unique challenges in mitigating vulnerabilities due to intrinsic constraints, such as low energy consumption requirements and limited computational resources. This paper presents RunPBA, a hardware-based runtime attestation system designed to defend against control flow attacks while maintaining minimal performance overhead and adhering to strict power consumption constraints. RunPBA leverages PACBTI, a new processor extension tailored for the Arm Cortex M processor family, allowing robust protection without requiring hardware modifications, a limitation present in similar solutions. We implemented a proof-of-concept and evaluated it using two benchmark suites. Experimental results indicate that RunPBA imposes a geometric mean performance overhead of only 1% and 4.7% across the benchmarks, underscoring its efficiency and suitability for real-world deployment. △ Less

Submitted 14 December, 2025; originally announced December 2025.

Co-Located VR with Hybrid SLAM-based HMD Tracking and Motion Capture Synchronization

Authors: Carlos A. Pinheiro de Sousa, Niklas Gröne, Mathias Günther, Oliver Deussen

Abstract: We introduce a multi-user VR co-location framework that synchronizes users within a shared virtual environment aligned to physical space. Our approach combines a motion capture system with SLAM-based inside-out tracking to deliver smooth, high-framerate, low-latency performance. Previous methods either rely on continuous external tracking, which introduces latency and jitter, or on one-time calibr… ▽ More We introduce a multi-user VR co-location framework that synchronizes users within a shared virtual environment aligned to physical space. Our approach combines a motion capture system with SLAM-based inside-out tracking to deliver smooth, high-framerate, low-latency performance. Previous methods either rely on continuous external tracking, which introduces latency and jitter, or on one-time calibration, which cannot correct drift over time. In contrast, our approach combines the responsiveness of local HMD SLAM tracking with the flexibility to realign to an external source when needed. It also supports real-time pose sharing across devices, ensuring consistent spatial alignment and engagement between users. Our evaluation demonstrates that our framework achieves the spatial accuracy required for natural multi-user interaction while offering improved comfort, scalability, and robustness over existing co-located VR solutions. △ Less

Submitted 8 September, 2025; originally announced September 2025.

Comments: Accepted at the Gesellschaft für Informatik (GI) VR/AR Workshop 2025 (Lecture Notes in Informatics)

Bisecting K-Means in RAG for Enhancing Question-Answering Tasks Performance in Telecommunications

Authors: Pedro Sousa, Cláudio Klautau Mello, Frank B. Morte, Luis F. Solis Navarro

Abstract: Question-answering tasks in the telecom domain are still reasonably unexplored in the literature, primarily due to the field's rapid changes and evolving standards. This work presents a novel Retrieval-Augmented Generation framework explicitly designed for the telecommunication domain, focusing on datasets composed of 3GPP documents. The framework introduces the use of the Bisecting K-Means cluste… ▽ More Question-answering tasks in the telecom domain are still reasonably unexplored in the literature, primarily due to the field's rapid changes and evolving standards. This work presents a novel Retrieval-Augmented Generation framework explicitly designed for the telecommunication domain, focusing on datasets composed of 3GPP documents. The framework introduces the use of the Bisecting K-Means clustering technique to organize the embedding vectors by contents, facilitating more efficient information retrieval. By leveraging this clustering technique, the system pre-selects a subset of clusters that are most similar to the user's query, enhancing the relevance of the retrieved information. Aiming for models with lower computational cost for inference, the framework was tested using Small Language Models, demonstrating improved performance with an accuracy of 66.12% on phi-2 and 72.13% on phi-3 fine-tuned models, and reduced training time. △ Less

Submitted 27 February, 2025; originally announced February 2025.

Comments: 6 pages, 8 figures, accepted at GLOBECOM WORKSHOPS 2024

Mesh2SLAM in VR: A Fast Geometry-Based SLAM Framework for Rapid Prototyping in Virtual Reality Applications

Authors: Carlos Augusto Pinheiro de Sousa, Heiko Hamann, Oliver Deussen

Abstract: SLAM is a foundational technique with broad applications in robotics and AR/VR. SLAM simulations evaluate new concepts, but testing on resource-constrained devices, such as VR HMDs, faces challenges: high computational cost and restricted sensor data access. This work proposes a sparse framework using mesh geometry projections as features, which improves efficiency and circumvents direct sensor da… ▽ More SLAM is a foundational technique with broad applications in robotics and AR/VR. SLAM simulations evaluate new concepts, but testing on resource-constrained devices, such as VR HMDs, faces challenges: high computational cost and restricted sensor data access. This work proposes a sparse framework using mesh geometry projections as features, which improves efficiency and circumvents direct sensor data access, advancing SLAM research as we demonstrate in VR and through numerical evaluation. △ Less

Submitted 23 January, 2025; v1 submitted 16 January, 2025; originally announced January 2025.

Comments: Accepted to ENPT XR at IEEE VR 2025

Cognitive Edge Device (CED) for Real-Time Environmental Monitoring in Aquatic Ecosystems

Authors: Dennis Monari, Farhad Fassihi Tash, Jordan J. Bird, Ahmad Lotfi, Isibor Kennedy Ihianle, Salisu Wada Yahaya, Isibor Kennedy Ihianle, Md Mahmudul Hasan, Pedro Sousa, Pedro Machado

Abstract: Invasive signal crayfish have a detrimental impact on ecosystems. They spread the fungal-type crayfish plague disease (Aphanomyces astaci) that is lethal to the native white clawed crayfish, the only native crayfish species in Britain. Invasive signal crayfish extensively burrow, causing habitat destruction, erosion of river banks and adverse changes in water quality, while also competing with nat… ▽ More Invasive signal crayfish have a detrimental impact on ecosystems. They spread the fungal-type crayfish plague disease (Aphanomyces astaci) that is lethal to the native white clawed crayfish, the only native crayfish species in Britain. Invasive signal crayfish extensively burrow, causing habitat destruction, erosion of river banks and adverse changes in water quality, while also competing with native species for resources leading to declines in native populations. Moreover, pollution exacerbates the vulnerability of White-clawed crayfish, with their populations declining by over 90%. To safeguard aquatic ecosystems, it is imperative to address the challenges posed by invasive species and pollution in aquatic ecosystem's. This article introduces the Cognitive Edge Device (CED) computing platform for the detection of crayfish and plastic. It also presents two publicly available underwater datasets, annotated with sequences of crayfish and aquatic plastic debris. Four You Only Look Once (YOLO) variants were trained and evaluated for crayfish and plastic object detection. YOLOv5s achieved the highest detection accuracy, with an mAP@0.5 of 0.90, and achieved the best precision △ Less

Submitted 7 February, 2026; v1 submitted 21 December, 2023; originally announced January 2024.

Carolina: a General Corpus of Contemporary Brazilian Portuguese with Provenance, Typology and Versioning Information

Authors: Maria Clara Ramos Morales Crespo, Maria Lina de Souza Jeannine Rocha, Mariana Lourenço Sturzeneker, Felipe Ribas Serras, Guilherme Lamartine de Mello, Aline Silva Costa, Mayara Feliciano Palma, Renata Morais Mesquita, Raquel de Paula Guets, Mariana Marques da Silva, Marcelo Finger, Maria Clara Paixão de Sousa, Cristiane Namiuti, Vanessa Martins do Monte

Abstract: This paper presents the first publicly available version of the Carolina Corpus and discusses its future directions. Carolina is a large open corpus of Brazilian Portuguese texts under construction using web-as-corpus methodology enhanced with provenance, typology, versioning, and text integrality. The corpus aims at being used both as a reliable source for research in Linguistics and as an import… ▽ More This paper presents the first publicly available version of the Carolina Corpus and discusses its future directions. Carolina is a large open corpus of Brazilian Portuguese texts under construction using web-as-corpus methodology enhanced with provenance, typology, versioning, and text integrality. The corpus aims at being used both as a reliable source for research in Linguistics and as an important resource for Computer Science research on language models, contributing towards removing Portuguese from the set of low-resource languages. Here we present the construction of the corpus methodology, comparing it with other existing methodologies, as well as the corpus current state: Carolina's first public version has $653,322,577$ tokens, distributed over $7$ broad types. Each text is annotated with several different metadata categories in its header, which we developed using TEI annotation standards. We also present ongoing derivative works and invite NLP researchers to contribute with their own. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Comments: 14 pages, 3 figures, 1 appendix

MSC Class: 68T50 ACM Class: I.2.7

A framework to semantify BPMN models using DEMO business transaction pattern

Authors: Sérgio Guerreiro, Pedro Sousa

Abstract: BPMN is a specification language widely used by industry and researchers for business process modeling and execution. It defines clearly how to articulate its concepts, but do not provide mechanism to represent the semantics of the produced models. This paper addresses the problem of how to improve the expressiveness of BPMN models, proposing a definition for the semantics of a business process wi… ▽ More BPMN is a specification language widely used by industry and researchers for business process modeling and execution. It defines clearly how to articulate its concepts, but do not provide mechanism to represent the semantics of the produced models. This paper addresses the problem of how to improve the expressiveness of BPMN models, proposing a definition for the semantics of a business process within a BPMN model, and improving the completeness of the models in a systematic manner, so that models can describe far more situations with few extra managed complexity. We conceive a framework based on the business transaction patterns available in the enterprise ontology body of knowledge to prescribe the foundations of semantic BPMN models. A tool has been developed to automate the framework. Then, two industrial proof-of-concepts are used to measure its coverage, both positive and negative, and to argue about our proposal's usefulness. After that, the proposal is compared with others using a systematic literature review. A full BPMN pattern is proposed encompassing the happy flow, the declinations, the rejections and the revocations, without adding any new element to the BPMN specification. A software tool has been developed, and made publicly available, to support the automatic generation of the BPMN models from the proposed patterns. Our semantified BPMN pattern allowed the identification of a large amount of implicit, and other not implemented, situations in both proof-of-concepts. It is concluded that the usage of a semantic solution, grounded on a sound pattern, allows the systematic enrichment of the BPMN models with a bounded effort. Moreover, to simplify the BPMN executable models' implementation, its elements could be classified as implicit, explicit, or not implemented yet. Finally, related work indicates that this work is demanded, but no full solutions are available. △ Less

Submitted 17 December, 2020; originally announced December 2020.

Comments: 22 pages, 15 figures, models are available at https://github.com/SemantifyingBPMN/DEMO_complete_pattern_in_BPMN, tool available at https://github.com/SemantifyingBPMN/SemantifyingBPMN

ACM Class: D.2.10; H.1.2; H.4.2

Characterizing Human Mobility Patterns During COVID-19 using Cellular Network Data

Authors: Necati A. Ayan, Nilson L. Damasceno, Sushil Chaskar, Peron R. de Sousa, Arti Ramesh, Anand Seetharam, Antonio A. de A. Rocha

Abstract: In this paper, our goal is to analyze and compare cellular network usage data from pre-lockdown, during lockdown, and post-lockdown phases surrounding the COVID-19 pandemic to understand and model human mobility patterns during the pandemic, and evaluate the effect of lockdowns on mobility. To this end, we collaborate with one of the main cellular network providers in Brazil, and collect and analy… ▽ More In this paper, our goal is to analyze and compare cellular network usage data from pre-lockdown, during lockdown, and post-lockdown phases surrounding the COVID-19 pandemic to understand and model human mobility patterns during the pandemic, and evaluate the effect of lockdowns on mobility. To this end, we collaborate with one of the main cellular network providers in Brazil, and collect and analyze cellular network connections from 1400 antennas for all users in the city of Rio de Janeiro and its suburbs from March 1, 2020 to July 1, 2020. Our analysis reveals that the total number of cellular connections decreases to 78% during the lockdown phase and then increases to 85% of the pre-COVID era as the lockdown eases. We observe that as more people work remotely, there is a shift in the antennas incurring top 10% of the total traffic, with the number of connections made to antennas in downtown Rio reducing drastically and antennas at other locations taking their place. We also observe that while nearly 40-45% users connected to only 1 antenna each day during the lockdown phase indicating no mobility, there are around 4% users (i.e., 80K users) who connected to more than 10 antennas, indicating very high mobility. Finally, we design an interactive tool that showcases mobility patterns in different granularities that can potentially help people and government officials understand the mobility of individuals and the number of COVID cases in a particular neighborhood. Our analysis, inferences, and interactive showcasing of mobility patterns based on large-scale data can be extrapolated to other cities of the world and has the potential to help in designing more effective pandemic management measures in the future. △ Less

Submitted 27 October, 2020; originally announced October 2020.

Comments: 12 pages

Evolved Explainable Classifications for Lymph Node Metastases

Authors: Iam Palatnik de Sousa, Marley Maria Bernardes Rebuzzi Vellasco, Eduardo Costa da Silva

Abstract: A novel evolutionary approach for Explainable Artificial Intelligence is presented: the "Evolved Explanations" model (EvEx). This methodology consists in combining Local Interpretable Model Agnostic Explanations (LIME) with Multi-Objective Genetic Algorithms to allow for automated segmentation parameter tuning in image classification tasks. In this case, the dataset studied is Patch-Camelyon, comp… ▽ More A novel evolutionary approach for Explainable Artificial Intelligence is presented: the "Evolved Explanations" model (EvEx). This methodology consists in combining Local Interpretable Model Agnostic Explanations (LIME) with Multi-Objective Genetic Algorithms to allow for automated segmentation parameter tuning in image classification tasks. In this case, the dataset studied is Patch-Camelyon, comprised of patches from pathology whole slide images. A publicly available Convolutional Neural Network (CNN) was trained on this dataset to provide a binary classification for presence/absence of lymph node metastatic tissue. In turn, the classifications are explained by means of evolving segmentations, seeking to optimize three evaluation goals simultaneously. The final explanation is computed as the mean of all explanations generated by Pareto front individuals, evolved by the developed genetic algorithm. To enhance reproducibility and traceability of the explanations, each of them was generated from several different seeds, randomly chosen. The observed results show remarkable agreement between different seeds. Despite the stochastic nature of LIME explanations, regions of high explanation weights proved to have good agreement in the heat maps, as computed by pixel-wise relative standard deviations. The found heat maps coincide with expert medical segmentations, which demonstrates that this methodology can find high quality explanations (according to the evaluation metrics), with the novel advantage of automated parameter fine tuning. These results give additional insight into the inner workings of neural network black box decision making for medical data. △ Less

Submitted 14 May, 2020; originally announced May 2020.

Complex Network Tools to Understand the Behavior of Criminality in Urban Areas

Authors: Gabriel Spadon, Lucas C. Scabora, Marcus V. S. Araujo, Paulo H. Oliveira, Bruno B. Machado, Elaine P. M. Sousa, Caetano Traina-Jr, Jose F. Rodrigues-Jr

Abstract: Complex networks are nowadays employed in several applications. Modeling urban street networks is one of them, and in particular to analyze criminal aspects of a city. Several research groups have focused on such application, but until now, there is a lack of a well-defined methodology for employing complex networks in a whole crime analysis process, i.e. from data preparation to a deep analysis o… ▽ More Complex networks are nowadays employed in several applications. Modeling urban street networks is one of them, and in particular to analyze criminal aspects of a city. Several research groups have focused on such application, but until now, there is a lack of a well-defined methodology for employing complex networks in a whole crime analysis process, i.e. from data preparation to a deep analysis of criminal communities. Furthermore, the "toolset" available for those works is not complete enough, also lacking techniques to maintain up-to-date, complete crime datasets and proper assessment measures. In this sense, we propose a threefold methodology for employing complex networks in the detection of highly criminal areas within a city. Our methodology comprises three tasks: (i) Mapping of Urban Crimes; (ii) Criminal Community Identification; and (iii) Crime Analysis. Moreover, it provides a proper set of assessment measures for analyzing intrinsic criminality of communities, especially when considering different crime types. We show our methodology by applying it to a real crime dataset from the city of San Francisco - CA, USA. The results confirm its effectiveness to identify and analyze high criminality areas within a city. Hence, our contributions provide a basis for further developments on complex networks applied to crime analysis. △ Less

Submitted 24 December, 2016; v1 submitted 19 December, 2016; originally announced December 2016.

Comments: 7 pages, 2 figures, 14th International Conference on Information Technology : New Generations

MSC Class: 05C82; 91C20; 68R10 ACM Class: G.2.2; G.2.3; H.2.8; H.3.3; I.5

Journal ref: Information Technology - New Generations, Advances in Intelligent Systems and Computing, 558, 2017

Optimistic Aborts for Geo-distributed Transactions

Authors: Theo Jepsen, Leandro Pacheco de Sousa, Huynh Tu Dang, Fernando Pedone, Robert Soulé

Abstract: Network latency can have a significant impact on the performance of transactional storage systems, particularly in wide area or geo-distributed deployments. To reduce latency, systems typically rely on a cache to service read-requests closer to the client. However, caches are not effective for write-heavy workloads, which have to be processed by the storage system in order to maintain serializabil… ▽ More Network latency can have a significant impact on the performance of transactional storage systems, particularly in wide area or geo-distributed deployments. To reduce latency, systems typically rely on a cache to service read-requests closer to the client. However, caches are not effective for write-heavy workloads, which have to be processed by the storage system in order to maintain serializability. This paper presents a new technique, called optimistic abort, which reduces network latency for high-contention, write-heavy workloads by identifying transactions that will abort as early as possible, and aborting them before they reach the store. We have implemented optimistic abort in a system called Gotthard, which leverages recent advances in network data plane programmability to execute transaction processing logic directly in network devices. Gotthard examines network traffic to observe and log transaction requests. If Gotthard suspects that a transaction is likely to be aborted at the store, it aborts the transaction early by re-writing the packet header, and routing the packets back to the client. Gotthard significantly reduces the overall latency and improves the throughput for high-contention workloads. △ Less

Submitted 24 October, 2016; originally announced October 2016.

Report number: USI-INF-TR-2016-05

Combining complex networks and data mining: why and how

Authors: M. Zanin, D. Papo, P. A. Sousa, E. Menasalvas, A. Nicchi, E. Kubik, S. Boccaletti

Abstract: The increasing power of computer technology does not dispense with the need to extract meaningful in- formation out of data sets of ever growing size, and indeed typically exacerbates the complexity of this task. To tackle this general problem, two methods have emerged, at chronologically different times, that are now commonly used in the scientific community: data mining and complex network theor… ▽ More The increasing power of computer technology does not dispense with the need to extract meaningful in- formation out of data sets of ever growing size, and indeed typically exacerbates the complexity of this task. To tackle this general problem, two methods have emerged, at chronologically different times, that are now commonly used in the scientific community: data mining and complex network theory. Not only do complex network analysis and data mining share the same general goal, that of extracting information from complex systems to ultimately create a new compact quantifiable representation, but they also often address similar problems too. In the face of that, a surprisingly low number of researchers turn out to resort to both methodologies. One may then be tempted to conclude that these two fields are either largely redundant or totally antithetic. The starting point of this review is that this state of affairs should be put down to contingent rather than conceptual differences, and that these two fields can in fact advantageously be used in a synergistic manner. An overview of both fields is first provided, some fundamental concepts of which are illustrated. A variety of contexts in which complex network theory and data mining have been used in a synergistic manner are then presented. Contexts in which the appropriate integration of complex network metrics can lead to improved classification rates with respect to classical data mining algorithms and, conversely, contexts in which data mining can be used to tackle important issues in complex network theory applications are illustrated. Finally, ways to achieve a tighter integration between complex networks and data mining, and open lines of research are discussed. △ Less

Submitted 19 May, 2016; v1 submitted 29 April, 2016; originally announced April 2016.

Comments: 58 pages, 19 figures

MSC Class: 05C82; 62-07; 92C42

Stretching Multi-Ring Paxos

Authors: Samuel Benz, Leandro Pacheco de Sousa, Fernando Pedone

Abstract: Internet-scale services rely on data partitioning and replication to provide scalable performance and high availability. Moreover, to reduce user-perceived response times and tolerate disasters (i.e., the failure of a whole datacenter), services are increasingly becoming geographically distributed. Data partitioning and replication, combined with local and geographical distribution, introduce daun… ▽ More Internet-scale services rely on data partitioning and replication to provide scalable performance and high availability. Moreover, to reduce user-perceived response times and tolerate disasters (i.e., the failure of a whole datacenter), services are increasingly becoming geographically distributed. Data partitioning and replication, combined with local and geographical distribution, introduce daunting challenges, including the need to carefully order requests among replicas and partitions. One way to tackle this problem is to use group communication primitives that encapsulate order requirements. This paper presents a detailed performance evaluation of Multi-Ring Paxos, a scalable group communication primitive. We focus our analysis on "extreme conditions" with deployments including high-end 10 Gbps networks, a large number of combined rings (i.e., independent Paxos instances), a large number of replicas in a ring, and a global deployment. We also report on the performance of recovery under peak load and present two novel extensions to boost Multi-Ring Paxos's performance. △ Less

Submitted 20 April, 2015; originally announced April 2015.

Information content: assessing meso-scale structures in complex networks

Authors: Massimiliano Zanin, Pedro A. Sousa, Ernestina Menasalvas

Abstract: We propose a novel measure to assess the presence of meso-scale structures in complex networks. This measure is based on the identification of regular patterns in the adjacency matrix of the network, and on the calculation of the quantity of information lost when pairs of nodes are iteratively merged. We show how this measure is able to quantify several meso-scale structures, like the presence of… ▽ More We propose a novel measure to assess the presence of meso-scale structures in complex networks. This measure is based on the identification of regular patterns in the adjacency matrix of the network, and on the calculation of the quantity of information lost when pairs of nodes are iteratively merged. We show how this measure is able to quantify several meso-scale structures, like the presence of modularity, bipartite and core-periphery configurations, or motifs. Results corresponding to a large set of real networks are used to validate its ability to detect non-trivial topological patterns. △ Less

Submitted 17 May, 2014; v1 submitted 21 January, 2014; originally announced January 2014.

Comments: Published as: M. Zanin, P. A. Sousa and E. Menasalvas, Information content: assessing meso-scale structures in complex networks EPL 106 (3), (2014) 30001

Journal ref: EPL 106 (3), (2014) 30001