feat(datasets): immutable DatasetVersion snapshots and Run provenance (#2701) - #2721
Open
Payal2000 wants to merge 1 commit into
Open
feat(datasets): immutable DatasetVersion snapshots and Run provenance (#2701)#2721Payal2000 wants to merge 1 commit into
Payal2000 wants to merge 1 commit into
Conversation
Adds content-addressed, immutable snapshots of a dataset's examples so an evaluation can be reproduced and two runs can be compared with confidence that they read the same test set. - `DatasetVersion` / `DatasetVersionItem` schemas. A version id hashes the dataset, the ordered contents and the source metadata, so publishing identical content is idempotent; `description` and the parent pointer are provenance only and stay out of the hash. An item id hashes the normalized example content plus the optional caller-supplied input id, so `meta` and `splits` can be re-annotated without breaking membership comparison. - `dataset_version` / `dataset_version_item` tables (alembic revision 13), with `(dataset_id, version_index)` unique so version history cannot fork. - `TruSession.create_dataset_version`, `get_dataset_version`, `list_dataset_versions` and `compare_dataset_versions`; publishing accepts a DataFrame or a `GroundTruth` sequence. - Datasets predating versioning are exposed as version zero by a compatibility loader that reads, but never rewrites, their ground truth payloads, and is materialized on the first publish so it stays loadable. `get_ground_truth(dataset_name=...)` is deliberately unchanged; pass `dataset_version_id` to read a pinned snapshot. - Optional `dataset_version_id` on `RunConfig` and `Run.SourceInfo`, persisted in run source info, read back by `run.start()`, and reported by `RunDiff.provenance()`, which now warns when two runs read different versions. Closes truera#2701
Payal2000
force-pushed
the
payal/2701-immutable-dataset-versions
branch
from
August 21, 2026 20:14
08adf0a to
3fe1a1d
Compare
This was referenced Aug 22, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2701
Problem
Datasetholds a stable identity whileGroundTruthrows point straight at it, so adding or replacing examples changes the effective test set in place. An evaluation cannot be reproduced, and two runs cannot be compared with any confidence that they saw the same examples.What this adds
Immutable, content-addressed snapshots alongside the existing mutable tables.
DatasetVersion/DatasetVersionItem(schema/dataset.py). A version id hashes the dataset, the ordered contents and the source metadata, so publishing identical content is idempotent and returns the existing version rather than duplicating or updating it.descriptionandparent_dataset_version_idare provenance annotations and stay out of the hash.dataset_versionanddataset_version_itemtables, alembic revision13.(dataset_id, version_index)is unique so version history cannot fork, andversion_indexis the deterministic ordering key for latest-version resolution.TruSession.create_dataset_version,get_dataset_version,list_dataset_versions,compare_dataset_versions. Publishing accepts a DataFrame with acolumn_spec, or aGroundTruthsequence.dataset_version_idonRunConfigandRun.SourceInfo, persisted into run source info by both the OSS and Snowflake DAOs, read back byrun.start(), and surfaced byRunDiff.provenance()— which warns when the two runs read different versions, since the deltas then mix app changes with test-set changes.contributing/database.md. API reference is generated from docstrings.Backward compatibility
Existing
DatasetandGroundTruthrows are untouched. A dataset that predates versioning is exposed as version zero, reconstructed from its ground truth rows by a compatibility loader that reads but never rewrites the original payloads. Version zero is materialized as a real row on the first publish for that dataset, so it stays loadable afterwards and becomes the parent of the first published version.The Snowflake
source_infopayload only carriesdataset_version_idwhen a version is actually pinned, so unversioned runs send an unchanged payload.Three judgment calls worth reviewing
The issue left these underdetermined. Each is documented in the code; happy to flip any of them.
Item identity excludes
metaandsplits. The issue asks for ids from "normalized example content" but also that metadata "round-trip without changing item identity unexpectedly". If metadata were part of identity, re-tagging an example would show up in a diff as removed and added. So identity isinput+expected_response+expected_contexts+input_id. To stop that from silently dropping annotations, the version hash covers the full per-item payload rather than only the ordered item ids as the issue literally specifies.get_ground_truth(dataset_name=...)is deliberately unchanged. The issue says name-only calls should resolve to the latest version, but the acceptance criteria also require that current APIs keep working. Resolving to the latest version would mean that publishing a version of a subset silently narrows what existing callers read. Legacy semantics are kept anddataset_version_idis an opt-in parameter; latest-resolution applies to the new version-aware APIs.A dataset name can map to several rows, because
dataset_idhashes name and metadata. Versions hang off the primary row (lowest id) so that reading version zero and materializing it cannot disagree.Tests
58 unit tests in
tests/unit/test_dataset_versions.pycovering version creation from DataFrames andGroundTruthsequences, content hashing, idempotency, immutability, parent provenance, split and metadata round trips, version-zero compatibility, latest and exact lookup, membership comparison, and run source provenance.A persistence-contract helper is wired into
tests/integration/test_database.pyfor sqlite, postgres and mysql, asserting that a version round-trips to the same content hash it was stored under.Verified locally: the new unit suite plus the existing run/DAO/dashboard tests pass, as does the sqlite persistence contract. Postgres and mysql need the
docker/test-database.yamlservices and will run in CI.ruff formatandruff checkare clean.Unrelated bug noticed
SQLAlchemyDB.get_datasets()raisesAttributeError: 'Dataset' object has no attribute 'name'— it readsds.name/ds.metaoff the ORM row, but those live insidedataset_json. This is broken onmaintoday and is left alone here; this PR uses its own resolution path. Worth a separate fix.