8000
Skip to content

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec - #31686

Merged
Gireesh Sreepathi (gisripa) merged 25 commits into
masterfrom
gireesh/30454
Oct 26, 2023
Merged

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec#31686
Gireesh Sreepathi (gisripa) merged 25 commits into
masterfrom
gireesh/30454

Conversation

@gisripa
@gisripa Gireesh Sreepathi (gisripa) commented Oct 20, 2023 •
edited
Loading
Copy link
Copy Markdown
Contributor

What

Adds ability to disable generating the final Typed+Deduped tables through a flag in ConnectorSpecification.

closes #30454

Other minor fixes

closes #11122 - Copied contents from https://github.com/airbytehq/airbyte-platform/blob/v0.43.2/airbyte-webapp/docs/HowTo-ConnectionSpecification.md
closes #31720 - Pending GSM credentials updates. local tests pass with minor fixes

BigQuery UI

bq_skip_td

Snowflake UI

snowflake_skip_td

@gisripa
Gireesh Sreepathi (gisripa) requested a review from a team as a code owner October 20, 2023 23:01
@vercel
vercel Bot commented Oct 20, 2023 •
edited
Loading
Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 26, 2023 8:33pm

@gisripa
Gireesh Sreepathi (gisripa) marked this pull request as draft October 20, 2023 23:01
@github-actions
github-actions Bot commented Oct 20, 2023 •
edited by gisripa
Loading
Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

evantahler
Evan Tahler (evantahler) reviewed Oct 20, 2023
View reviewed changes
Comment thread airbyte-integrations/connectors/destination-bigquery/src/main/resources/spec.json Outdated
@evantahler Evan Tahler (evantahler) Oct 20, 2023
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alexandre Cuoci (@Hesperide) - is this message a good enough deterrent?

@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Oct 21, 2023
evantahler
Evan Tahler (evantahler) reviewed Oct 23, 2023
View reviewed changes
Comment thread docs/connector-development/connector-specification-reference.md Outdated
Comment on lines 7 to 11
@evantahler Evan Tahler (evantahler) Oct 23, 2023
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice addition! Be sure to share this at our next team demos, or make a loom showing this off.

@gisripa Gireesh Sreepathi (gisripa) Oct 23, 2023 •
edited
Loading
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zero credit to me, just resurrected contents from a deleted file and inlined them :D

@gisripa Gireesh Sreepathi (gisripa) changed the title [WIP] Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec Oct 24, 2023
@gisripa
Gireesh Sreepathi (gisripa) marked this pull request as ready for review October 24, 2023 03:31
@gisripa
Gireesh Sreepathi (gisripa) requested a review from a team October 24, 2023 03:34
@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 24, 2023
Copy link
Copy Markdown
Collaborator

destination-bigquery test report (commit e005987ae1) - ❌

⏲️ Total pipeline duration: 11mn29s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-bigquery docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ❌
Validate metadata for destination-bigquery ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 24, 2023
Copy link
Copy Markdown
Collaborator

destination-snowflake test report (commit e005987ae1) - ❌

⏲️ Total pipeline duration: 07mn45s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-snowflake docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-snowflake ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 24, 2023
Copy link
Copy Markdown
Collaborator

destination-snowflake test report (commit 34e72b97c9) - ❌

⏲️ Total pipeline duration: 10mn25s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-snowflake docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-snowflake ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 24, 2023
Copy link
Copy Markdown
Collaborator

destination-bigquery test report (commit 34e72b97c9) - ❌

⏲️ Total pipeline duration: 07mn18s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-bigquery docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-bigquery ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@gisripa
Gireesh Sreepathi (gisripa) changed the base branch from master to edgao/skip_raw_dedup October 24, 2023 18:58
edgao
Edward Gao (edgao) reviewed Oct 24, 2023
View reviewed changes
@edgao Edward Gao (edgao) left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

had some minor comments but generally looks good! very annoyed by how github has decided to show this diff but not sure how to fix it without doing actual git stuff

Comment thread ...in/java/io/airbyte/integrations/base/destination/typing_deduping/BaseTypingDedupingTest.java Outdated
Comment thread ...in/java/io/airbyte/integrations/base/destination/typing_deduping/BaseTypingDedupingTest.java Outdated
Comment thread ...in/java/io/airbyte/integrations/base/destination/typing_deduping/BaseTypingDedupingTest.java Outdated
Comment thread ...ns/destination/bigquery/typing_deduping/BigQueryGcsRawOverrideDisableTypingDedupingTest.java Outdated
Comment thread ...irbyte/integrations/base/destination/typing_deduping/NoOpTyperDeduperWithV1V2Migrations.java Outdated
@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 25, 2023
Copy link
Copy Markdown
Collaborator

destination-bigquery test report (commit db3d1169db) - ❌

⏲️ Total pipeline duration: 09mn39s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-bigquery docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-bigquery ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 25, 2023
Copy link
Copy Markdown
Collaborator

destination-snowflake test report (commit db3d1169db) - ❌

⏲️ Total pipeline duration: 07mn30s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-snowflake docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-snowflake ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@gisripa
Gireesh Sreepathi (gisripa) changed the base branch from edgao/skip_raw_dedup to master October 25, 2023 02:31
@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 25, 2023
Copy link
Copy Markdown
Collaborator

destination-snowflake test report (commit af9623b8c4) - ❌

⏲️ Total pipeline duration: 08mn42s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-snowflake docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-snowflake ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 25, 2023
Copy link
Copy Markdown
Collaborator

destination-bigquery test report (commit af9623b8c4) - ❌

⏲️ Total pipeline duration: 08mn27s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-bigquery docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-bigquery ✅
Connector version semver check ✅
Connector version increment check ❌
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@gisripa
Gireesh Sreepathi (gisripa) removed the request for review from a team October 25, 2023 22:36
@gisripa
unused imports and unthrown exceptions
751fe17 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
wire the NoOpTypeDedupe conditionally
f9239be 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Automated Commit - Formatting Changes
abf286e
@gisripa
Fixes #11122
7a9c002 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Extend tests to verify raw table only inserts
a1b4262 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
add log stmts for standard consumer
ca640d5 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@edgao @gisripa
stuff
d8fc7d0
@edgao @gisripa
test cleanup
c917c54
@edgao @gisripa
test cleanup
0eb8d41
@gisripa
Automated Commit - Formatting Changes
0685074
@gisripa
Working integration tests with skip_raw_dedupe merge
aee046e 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Automated Commit - Formatting Changes
34997cc
@gisripa
override tests
c9538d9 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Add skipTD impl for snowflake
0851874 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Automated Commit - Formatting Changes
c1f7a68
@gisripa
Fixes 1st_1t removal failed tests
e24ee1b 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Automated Commit - Formatting Changes
38cc3b4
@gisripa
pr comments
37189e8 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
bump connector versions
958d30c 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
remove redundant assertion
299ad75 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
docs update
6698ae5 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 26, 2023
Copy link
Copy Markdown
Collaborator

destination-bigquery test report (commit 6698ae592a) - ✅

⏲️ Total pipeline duration: 10mn50s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-bigquery docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-bigquery ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@airbyte-oss-build-runner
airbyte-oss-build-runner commented Oct 26, 2023
Copy link
Copy Markdown
Collaborator

destination-snowflake test report (commit 6698ae592a) - ✅

⏲️ Total pipeline duration: 08mn59s

Step Result
Build connector tar ✅
Java Connector Unit Tests ✅
Build destination-snowflake docker image for platform(s) linux/amd64 ✅
Java Connector Integration Tests ✅
Validate metadata for destination-snowflake ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

evantahler
Evan Tahler (evantahler) reviewed Oct 26, 2023
View reviewed changes
Comment thread docs/release_notes/upgrading_to_destinations_v2.md Outdated
Gireesh Sreepathi @evantahler
Update docs/release_notes/upgrading_to_destinations_v2.md
c5dba08 
Co-authored-by: Evan Tahler <evan@airbyte.io>
@gisripa
Gireesh Sreepathi (gisripa) deleted the gireesh/30454 branch October 26, 2023 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@evantahler Evan Tahler (evantahler) evantahler left review comments
@edgao Edward Gao (edgao) edgao approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@gisripa Gireesh Sreepathi (gisripa)

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/bigquery connectors/destination/snowflake connectors/source/qualaroo

Projects

None yet

Milestone

No milestone

5 participants

@gisripa @airbyte-oss-build-runner @evantahler @edgao @octavia-squidington-iii
0