E4C4
Skip to content

Destination Bigquery / Bulk load CDK: increase message queue memory ratio - #62016

Merged
Edward Gao (edgao) merged 2 commits into
masterfrom
edgao/increase_memory_queue_ratio
Jun 23, 2025
Merged

Destination Bigquery / Bulk load CDK: increase message queue memory ratio#62016
Edward Gao (edgao) merged 2 commits into
masterfrom
edgao/increase_memory_queue_ratio

Conversation

@edgao
@edgao Edward Gao (edgao) commented Jun 23, 2025 •
edited by github-actions Bot
Loading
Copy link
Copy Markdown
Contributor

We claim to support records up to 20MiB. With default settings, this requires reserving 160MiB on the message queue. From this sync it looks like with default settings, we only have 742MiB heap, so we need to increase the message queue's ratio so that we reserve >160MiB.

Important

Auto-merge enabled.

This PR is set to merge automatically when all requirements are met.

@vercel
vercel Bot commented Jun 23, 2025 •
edited
Loading
Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 23, 2025 5:39pm

@github-actions
github-actions Bot commented Jun 23, 2025
Copy link
Copy Markdown
Contributor

👋 Greetings, Contributor!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).

📝 Edit this welcome message.

@edgao
Edward Gao (edgao) marked this pull request as ready for review June 23, 2025 16:53
@edgao
Edward Gao (edgao) requested a review from a team as a code owner June 23, 2025 16:53
@edgao Edward Gao (edgao) changed the title Destination Bigquery / Bulk load CDK: increase memory queue memory ratio Destination Bigquery / Bulk load CDK: increase message queue memory ratio Jun 23, 2025
@github-actions
github-actions Bot commented Jun 23, 2025 •
edited
Loading
Copy link
Copy Markdown
Contributor

destination-bigquery Connector Test Results

220 tests   162 ✅  30m 45s ⏱️
 11 suites   58 💤
 11 files      0 ❌

Results for commit b89b153.

♻️ This comment has been updated with latest results.

Base automatically changed from edgao/schema_parser_robustness to master June 23, 2025 17:35
@edgao
Edward Gao (edgao) enabled auto-merge (squash) June 23, 2025 17:36
@edgao
Edward Gao (edgao) merged commit b931a6d into master Jun 23, 2025
@edgao
Edward Gao (edgao) deleted the edgao/increase_memory_queue_ratio branch June 23, 2025 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@benmoriceau Benoit Moriceau (benmoriceau) benmoriceau approved these changes
@frifriSF59 Francis Genet (frifriSF59) frifriSF59 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

connectors/destination/bigquery
4DCB

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@edgao @benmoriceau @frifriSF59 @octavia-squidington-iii
0