Summary
Track consistency gaps in indexer column metadata. The archive metadata commit is transactional and idempotent, but the <dataset>_column_metadata tables do not have the same consistency guarantee.
Current inconsistency
- A compression task can succeed when its update to the indexer table fails.
- Retention-based archive GC removes archives without removing their related column metadata from the indexer table.
- Therefore,
<dataset>_column_metadata can omit schemas for existing archives and contain schemas for archives that no longer exist.
Required work
- Document the current consistency behaviour and its limits for
<dataset>_column_metadata.
- Design and implement an all-or-nothing update path. The
indexer should pass its results to the compression job commit stage so that archive metadata and indexer metadata commit together.
- Design and implement a method for archive GC to identify and remove column metadata that belongs to removed archives.
Affected areas
- Compression jobs and their commit stage.
indexer result handling and indexer-table updates.- Retention-based archive GC.
- User documentation for compression and indexer metadata consistency.
Acceptance criteria
- Documentation states that the archive metadata guarantee does not currently apply to
<dataset>_column_metadata.
- The proposed implementation defines failure and retry behaviour for combined archive and indexer metadata updates.
- Compression jobs do not report success after an indexer metadata update failure when all-or-nothing semantics are enabled.
- Archive GC keeps column metadata consistent with removed archives.
- Tests cover compression failure, indexer update failure, retry, and archive GC cases.
Context
Requested by @LinZhihao-723 during review of PR #2456.
Discussion: #2456 (comment)
Summary
Track consistency gaps in indexer column metadata. The archive metadata commit is transactional and idempotent, but the
<dataset>_column_metadatatables do not have the same consistency guarantee.Current inconsistency
<dataset>_column_metadatacan omit schemas for existing archives and contain schemas for archives that no longer exist.Required work
<dataset>_column_metadata.indexershould pass its results to the compression job commit stage so that archive metadata and indexer metadata commit together.Affected areas
indexerresult handling and indexer-table updates.Acceptance criteria
<dataset>_column_metadata.Context
Requested by @LinZhihao-723 during review of PR #2456.
Discussion: #2456 (comment)