A deployable reference implementation intended to address pain points around conceptualizing data lake architectures that automatically configures the core AWS services necessary to easily tag, search, share, and govern specific subsets of data across a business or with other external businesses.
✅ End-to-end Snowflake data lake for a fictional retail bank — ingests 25,000 nested JSON customer records through a RAW → SILVER → GOLD pipeline, builds fact and dimension tables, and surfaces a Customer 360 & risk analytics dashboard via Streamlit in Snowflake.
End-to-end Real-Time Data Engineering pipeline for a music streaming platform (BeatBlast). Built with PySpark Structured Streaming to process high-velocity JSON event streams and sink into a partitioned Parquet Data Lake.
Unlocking Spotify insights with an AWS data pipeline: S3 data lake -> Glue ETL-> Athena queries -> QuickSight Dashboard
NBA Data Lake Repository contains the setup_nba_data_lake.py script, which automates the creation of a data lake for NBA analytics using AWS services. The script integrates Amazon S3, AWS Glue, and Amazon Athena, and sets up the infrastructure needed to store and query NBA-related data.
A self-hosted data platform for experimenting with various Big Data framework libraries
Built a data pipeline in Azure Synapse Analytics to load product data from Azure Data Lake into a dedicated SQL pool. Implemented data flow with inserts and upserts, handling schema drift and type 1 SCD updates, and orchestrated ingestion using Synapse Studio pipelines.
Multi-tenant data lake with per-tenant client-side encryption (HKDF plus AES-256-GCM). Ingests Postgres or MySQL, encrypts, stores Parquet in Cloudflare R2. SELECT-only DuckDB proxy with automatic tenant_id filter injection.
Backblaze B2 as an S3-compatible data lake for autonomous-vehicle multi-sensor datasets: keep nuScenes camera, LiDAR & radar blobs + annotations in B2 and stream keyframes into 3D-perception training on demand via the real nuScenes devkit. Next.js + FastAPI sample.
Apache Hudi is a data lake platform that provides incremental data processing primitives including upserts and incremental queries. It manages storage of large analytical datasets on distributed file systems with ACID transactions, timeline-based versioning, and integrations for Spark, Flink, and Hive.
Production-grade Medallion data lake (Bronze/Silver/Gold) built with PySpark, data quality framework and Docker
Data Engineer (Udacity): Project 4 Data Lakes with Spark on Amazon Web Service (AWS)
A project using AWS to combine and transform streaming data for analytics.
Dual-path Lambda Architecture on AWS Serverless — Kinesis speed layer + SQS/S3/Athena batch, Cognito JWT auth, Hive-partitioned data lake, Terraform IaC. Fail-open design. ~55ms P50.
Serverless Data Lake ETL pipeline on AWS Glue and S3 for curating IoT sensor and Customer data for Machine Learning.
Production-grade Unified Enterprise Data Lake combining Healthcare Banking and Social Media pipelines. Master PySpark analytics engine processes 140+ records from 4 MySQL databases. Built with Python Hadoop HDFS PySpark Hive Data Warehouse and Apache Airflow master DAG orchestration.
Add a description, image, and links to the data-lake topic page so that developers can more easily learn about it. 329A
To associate your repository with the data-lake topic, visit your repo's landing page and select "manage topics."