Working with SCD Type (Change Data Capture) and need a Data Vault model to test Azure Data Factory v2? - This Code with Help!
Build a data warehouse from scratch, including full load, daily incremental load, design schema, SCD Type 1 and 2.
A Data Warehousing project for retail sales using dimension modelling best practices with SCD type 2 on AWS Redshift. Utilizing AWS Lambda, Glue Workflows and Python Shell jobs to create and automate an ELT pipeline where batch data coming into S3 is loaded onto Redshift and necessary transformations are performed to meet requirements.
Design and implement a full ELT data pipeline using Snowflake and S3, featuring star schema modelling, SCD Type 1 & 2 handling, and incremental load automation
Enterprise security data warehouse with 3-layer architecture processing 10M+ records daily from 15+ security services. Snowflake | Python | Streamlit | 98% Automation | 46K Annual Savings
Production-grade ELT pipeline classifying VIX volatility regimes. Features incremental models, SCD-2 snapshots, recursive CTEs, and Slim CI/CD. Stack: dbt, DuckDB/MotherDuck, Python.
In this project we'll create real time healthcare patient data pipeline as data source and use arious services and tools like Azure Eventhubs, Azure Databricks, Delta lake and synapse analytics. also, implement medallion architecture, schema evolution and create facts and dimension tables and connect the cleaned and transformed data to PowerBI.
Production-style Slowly Changing Dimension (SCD Type 2) pipeline built with Snowflake, dbt, and AWS S3. Demonstrates secure S3 ingestion, layered bronze/silver/gold modeling, dbt snapshots for historical tracking, and analytics-ready views identifying active vs historical records.
End-to-end Azure Databricks retail data engineering project using Medallion Architecture (Bronze, Silver, Gold). Implements Auto Loader, Unity Catalog, Delta Lake, SCD Type 1 & 2 dimensions, and Fact Orders for analytics-ready star schema modeling.
A complete data integration solution that migrates data from multiple source systems into a modern data warehouse, with full documentation, quality validation, and analytical reporting.
End-to-end cloud retail intelligence platform integrating data engineering, predictive analytics, and executive BI dashboards.
End-to-End Retail Lakehouse on Microsoft Fabric | Medallion Architecture | PySpark | SCD Type-2 | DirectLake
End-to-end data engineering project using AWS S3, Snowflake, and dbt to implement Medallion Architecture with SCD Type 1 & Type 2 logic on Walmart sales data, followed by analytical visualizations using Seaborn and Plotly.
End-to-end data engineering project using AWS S3, Snowflake, and dbt to implement Medallion Architecture with SCD Type 1 & Type 2 logic on Walmart sales data, followed by analytical visualizations using Seaborn and Plotly.
End-to-end data lakehouse on Azure Databricks — Medallion Architecture, Star Schema, SCD Type 2, CI/CD, Data Quality Framework
This project simulates a real-world enterprise data migration and modernization strategy. It extracts transactional data from a simulated "On-Premise" environment (hosted on AWS EC2), performs heavy distributed processing using a Hadoop/Spark cluster, and ultimately serves the data via a Cloud-Native, serverless architecture to optimize costs .
Complete Data Vault 2.0 (DV2) data warehouse implementation with Medallion Architecture on SQL Server. 67 objects, 60 SSIS packages, watermark-based incremental ETL.
A production-grade Modern Data Stack (MDS) implementation featuring automated ELT, SCD Type 2 history tracking, and CI/CD quality guardrails using Dagster, dbt Core, DuckDB, and Soda.
E-commerce analytics pipeline using snowflake and dbt - staging, intermediate, marts, scd2 snapshots
End-to-end analytics warehouse built using dbt and DuckDB, featuring incremental pipelines, SCD Type-2 snapshots, and a star schema for e-commerce data analysis.
Add a description, image, and links to the scd-type-2 topic page so that developers can more easily learn about it.
To associate your repository with the scd-type-2 topic, visit your repo's landing page and select "manage topics."