This project provides a complete Terraform setup to automate the deployment of a real-time Change Data Capture (CDC) pipeline on Google Cloud. It streams data from a Cloud SQL for MySQL source to BigQuery, using Google Cloud Storage (GCS) as a staging area and a Dataflow streaming job for processing and enrichment.
A complete Terraform setup for creating a secure, private data replication pipeline from Cloud SQL (MySQL) to BigQuery using Datastream and Private Service Connect (PSC).
Use GCP Datastream to incrementally load PostgreSQL to BigQuery
Terraform configuration to provision a real-time Change Data Capture (CDC) pipeline on Google Cloud. This project uses Datastream to stream database changes from a private Cloud SQL for MySQL instance into Google Cloud Storage (GCS) securely via Private Service Connect (PSC).
Data sync via CDC from GCP Cloud SQL to Big Query using Datastream
Add a description, image, and links to the datastream topic page so that developers can more easily learn about it.
To associate your repository with the datastream topic, visit your repo's landing page and select "manage topics."