DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Source corpora. We use EHRSQL (11) and BULL (56) as source corpora. These English Text-to-SQL benchmarks cover clinical and financial analytics, respectively. Each selected source instance provides a natural-language question $q_{0}$, a relational database $\mathcal{D}_{0}$, and executable SQL $\sigma_{0}$.

Joint transformation. Both source corpora are English-only at the question–database level, whereas our task setting varies language across the complete question–workspace pair. Translating the question or database in isolation can misalign entity names, predicate values, and executable SQL. We therefore treat cross-language transformation (CLT) as a joint migration of the question, database state, and executable workload. For a source tuple $(\mathcal{D}_{0},q_{0},\sigma_{0})$, we independently choose the target languages $\ell_{D}$ and $\ell_{q}$ for the database and question:

where $\mathcal{D}_{c}$, $q_{c}$, and $\sigma_{c}$ are the transformed database, question, and SQL, respectively, and $M$ is a materialized replacement map.

Consistency-aware rewriting. To keep the same field or entity consistent wherever it appears, we link columns connected by a foreign key, a shared name, or substantial value overlap. Each linked group is translated jointly, while identifiers, codes, URLs, dates, and numbers remain unchanged. LLM-generated translations are stored as table-, column-, and cell-level mappings $M=\{M_{\mathrm{tab}},M_{\mathrm{col}},M_{\mathrm{val}}\}$. We use $M$ to deterministically rewrite the database and SQL and translate the question with the same terminology.

Validation. We retain a transformed tuple only after structural and execution checks. The rewritten SQL must reference existing translated entities and execute on $\mathcal{D}_{c}$, and its result must satisfy $\operatorname{Exec}(\mathcal{D}_{c},\sigma_{c})\cong\tau_{M}(\operatorname{Exec}(\mathcal{D}_{0},\sigma_{0}))$, where $\tau_{M}$ applies the induced value translation to the source execution result. Execution equivalence verifies database–SQL consistency but cannot establish that $q_{c}$ preserves the original intent; we therefore use an LLM judge to verify question–SQL semantic alignment. Instances that fail either check are repaired and re-executed or rejected before relational data sampling.

Task-local data diversification. Source Text-to-SQL datasets commonly associate many questions with a small number of shared database states. Directly reusing these states would produce workspaces with highly repetitive entities, values, and relational neighborhoods. We instead construct task-local relational instances by sampling rows while retaining the complete table inventory and schema. The sampled scale also makes downstream artifact rendering tractable, particularly when complete tables are transformed into long Markdown or PDF documents.

Sampling safeguards. Naively sampling each table independently can remove a condition value, break a join path, or disconnect related entities. We therefore construct a lightweight safeguard set $\mathcal{C}_{s}$ that combines primary keys, foreign keys, and known inter-table relationships with join columns, predicate and boundary values, and target entities extracted from the SQL AST.

Relationally consistent sampling. Given a seeded sampling policy $\psi$, we retain rows required by query safeguards and then propagate their key values across schema relationships. Remaining rows are selected according to the table-level sampling budget, including rows from tables not referenced by the source SQL.

Materialization and re-execution. All sampled tables are materialized as a task-local intermediate database:

The query result $\mathcal{Y}_{s}$ becomes the candidate reference. It is not required to equal the source result; sampling may change entities, aggregates, rankings, or result cardinality. A sample is accepted only if the SQL executes successfully, required relationships remain valid, and the result has not become unintentionally empty or semantically degenerate. Otherwise, the database is resampled, repaired, or rejected.

Query-independent base routing. The intermediate database fixes the relational content of a task; this stage changes how agents encounter that content. We first apply a seeded rule-based policy $\pi_{r}$, with fixed seed $z_{r}$, that assigns each sampled table to one or more compatible renderers and materializes the base workspace $\mathcal{W}_{\mathrm{base}}$ from $\mathcal{D}_{s}$. The policy considers schema properties, renderer compatibility, and batch-level modality coverage using only sampled-table metadata. Its base renderers are CSV, JSON, SQLite, Markdown, and PDF. Video is introduced separately as a task-level, query-conditioned augmentation because its construction may depend on the question, executable SQL, and candidate answer. Every sampled table receives a base representation, while the number and sizes of the resulting artifacts follow from the source table inventory, sampled contents, and applicable renderers.

Structured artifact rendering. CSV and record-oriented JSON expose individual sampled tables, whereas a task-local SQLite artifact can retain several related tables and their schema. The renderers preserve headers, cell values, nulls, and duplicate rows; the SQLite renderer additionally preserves declared column types and key relationships. Each output is parsed back into a canonical relation and compared with the rows assigned to that renderer.

Fact-grounded data documents. For a routed table $R$, let $X_{R}=\mathcal{D}_{s}[R]$ denote its complete sampled contents. The LLM-produced plan $p_{R}=\operatorname{PlanDoc}(X_{R})$ specifies a document style, record-identifying columns, semantic attribute groups, and bounded row batches. Document generation then follows

Each block repeats the identifying columns and supplies an LLM with its exact field names, types, and values. The generated sections are assembled as Markdown $d_{R}$ and converted to PDF when required. Figure 14 visualizes the alignment from source cells to generation blocks and document spans. The complete running example is provided in Appendix A.3.3.

Query-conditioned video rendering. Following DataMagic (50), we render tabular data as data-insight videos, while conditioning content selection and task integration on the query. For selected tasks, the renderer derives typed evidence atoms from the executable SQL $\sigma_{c}$ and sampled tabular result $\mathcal{Y}_{s}$, then constructs the video and integrates it into the task:

The evidence set $\mathcal{E}_{v}$ contains filter conditions and result cells. A filter condition records a column, operator, and value; a result cell records a row, field, and value. For a multi-step query, predicate abstraction assigns a stable condition to a video scene and removes it from the explicit question. For a simple query with a compact result, answer-evidence rendering distributes result cells or an intermediate value across multiple scenes. We revise the question only when evidence is moved to video. Figure 16 shows representative frames from both strategies. Appendix A.3.5 further traces the question, evidence atoms, storyboard scenes, rendered video, and resulting workspace for complete tasks. For tasks without video augmentation, $(q_{r},\mathcal{W})=(q_{c},\mathcal{W}_{\mathrm{base}})$.

Task-specific semantics. After human review, each task freezes a configuration $c_{i}=(o_{i},\{\nu_{ij}\}_{j=1}^{d_{i}})$, where $d_{i}$ is the number of reference columns, $o_{i}$ indicates whether row order is required by the question, and $\nu_{ij}$ is the canonicalization rule for reference column $j$. The rule records a semantic type—text, number, date, datetime, or Boolean—and, for numeric columns, the required integer, decimal-place, or significant-digit precision and any percentage convention. It canonicalizes both reference and predicted values: text is normalized to Unicode NFC and trimmed; numbers are parsed as decimals under the configured precision and unit; and dates, datetimes, Booleans, and nulls are converted to canonical values. Appendix A.6 gives the complete protocol, and Figure 18 shows a frozen task configuration.

Header-invariant joint column alignment. Prediction headers are not scored, and predicted columns need not follow the reference order. Instead, the evaluator searches for a one-to-one mapping from reference columns to predicted columns. Let $\Pi_{d_{i}}$ be the set of permutations of $d_{i}$ columns, and let $\pi(j)$ denote the predicted column mapped to reference column $j$. We use $\widehat{\mathcal{Y}}_{i}[:,\pi]$ to denote the prediction with its columns reordered by $\pi$, and $\operatorname{Canon}_{c_{i}}(Y)$ to normalize column $j$ of $Y$ with $\nu_{ij}$. A mapping is invalid if any predicted cell cannot be interpreted under its target rule.

Answer equivalence. Tabular results with different shapes are unequal. Otherwise, $\operatorname{Rows}_{o}(Y)$ returns the ordered row sequence when $o=1$, and the unordered row multiset when $o=0$; the latter preserves duplicate multiplicities. Define the canonical row view as $\mathcal{V}_{i}(Y)=\operatorname{Rows}_{o_{i}}(\operatorname{Canon}_{c_{i}}(Y))$. The equivalence criterion introduced in Equation 3 is then

Column alignment is evaluated over the full result, so a valid mapping must preserve the association among values within every row. Tasks requesting a ranking or another semantically ordered output are compared as row sequences; all other tasks are compared as unordered row multisets.

Aggregate metric. For $N$ tasks, the official metric is Task Accuracy,

Available versus required evidence. Figure 3(a) shows that CSV occurs in every workspace, while JSON, SQLite, Markdown, and PDF each occur in more than 93% of tasks, yet the verified paths use CSV in only 58 tasks and SQLite in 177. Long documents provide required evidence in 135 tasks; among 189 video-enabled workspaces, 97 require video.

Cross-artifact composition. The verified paths use one artifact modality for 276 tasks (67.3%), two for 115 (28.0%), and three for 19 (4.6%); thus, 134 tasks (32.7%) combine multiple modalities. At the carrier-family level in Figure 3(b), 129 tasks (31.5%) span multiple carrier families, and 202 (49.3%) require a document or video.

Operation coverage. As shown in Figure 4, projection and filtering are required by 338 (82.4%) and 321 tasks (78.3%), respectively. The benchmark also exercises ordering in 209 tasks (51.0%), aggregation in 146 (35.6%), and joins in 113 (27.6%). Cross-artifact grounding includes document and video understanding in 135 and 97 tasks, and schema and entity alignment in 97 and 90.

Compositional breadth. A task combines between two and eleven nontrivial operation types, with a median of five and a 90th percentile of seven; 248 tasks (60.5%) combine at least five types.

Execution protocol. Each run receives the task question and its complete workspace. In the backbone comparison, every task is limited to 60 model turns, 50 tool actions, and 1,800 seconds, with 4 CPUs and 16 GiB of memory. The harness comparison uses the same 1,800-second deadline but does not impose a shared action limit that would override a harness’s native control loop.

Inference and scoring. All backbone calls use the provider-default reasoning configuration through Vercel AI Gateway (46), with a maximum of 32,768 output tokens per call. The official evaluator computes Task Accuracy using the protocol in Section 4.5. Missing predictions, invalid outputs, runtime failures, and exhausted budgets count as incorrect. We additionally record token usage, API cost, tool actions, and wall-clock latency as efficiency diagnostics.

Backbone comparison. With DataSpace-Agent fixed, Grok 4.5 achieves the highest observed accuracy of 66.34%, followed by GPT-5.6 Sol at 64.63%; their totals differ by only seven correct tasks. Kimi K3 reaches 53.41%, while the remaining three backbones remain below 40%. The 37.80-point range between the strongest and weakest backbone shows substantial separation, yet even the best result solves only about two thirds of the benchmark. Across the six models, 56 tasks are solved by all models, whereas 76 are solved by none; their oracle union solves 334 tasks (81.46%). Thus, the benchmark contains both a shared hard core and model-specific successes that are hidden by aggregate ranking alone.

Harness comparison. Fixing MiMo-V2.5, Grok Build obtains 46.34% and Claude Code 44.63%, compared with 39.27% for DataSpace-Agent, 34.88% for Codex, and 30.98% for Smolagents. The resulting 15.36-point spread demonstrates that the harness substantially affects end-to-end task completion even when the backbone is held constant.

Finding 1. DataSpace remains unsaturated: the best controlled backbone reaches 66.34% Task Accuracy, while 76 tasks are missed by all six backbones. Harness design also substantially affects agent performance: with MiMo-V2.5 fixed, accuracy ranges from 30.98% to 46.34%.

Finding 2. Near-top accuracy need not require long trajectories: GPT-5.6 Sol trails Grok 4.5 by 1.71 points while using 74.2% fewer tokens, 50.3% fewer actions, and 39.2% less latency. Monetary efficiency follows a different frontier, with MiMo-V2.5 providing the lowest-cost operating point at $0.011 per task.

Language and workspace scale. Cross-language performance varies by backbone: relative to single-language tasks, MiMo-V2.5 declines by 11.8 points and Grok 4.5 by 5.1, whereas GPT-5.6 Sol and Claude Sonnet 5 improve by 6.1 and 7.2 points. The largest workspace quartile underperforms the smallest for all six models, but accuracy across the four quartiles is not monotonic. At the task level, workspace bytes have only a weak negative Spearman correlation ($-0.186$) with the number of models that solve a task.

Evidence composition. Multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points. Modality presence alone has a less uniform effect: required document evidence is nearly neutral for GPT and Kimi but substantially reduces accuracy for Grok, MiMo, Claude, and MiniMax. Required video evidence helps GPT and Kimi but hurts MiMo, Claude, and MiniMax. These contrasts locate the consistent challenge in cross-modality integration rather than in any single modality.

Relational and answer requirements. Join requirements reduce accuracy for all backbones by 9.7–19.8 points, whereas aggregation has mixed effects. Answer shape is similarly non-monotonic: multi-row answers hurt Grok and MiMo but improve Claude, while multi-column and order-sensitive groups are not less accurate overall. Answer size and order therefore show no uniform negative association with accuracy; the analytical path and task composition remain more informative.

Finding 3. Cross-modality integration and joins are the most consistent sources of degradation: multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points, and joins reduce accuracy by 9.7–19.8 points. Language configuration and answer shape instead exhibit backbone-dependent effects.

Where failures originate. Answer materialization is the largest category, accounting for 71/136 (52.2%) failures, followed by task specification and intent with 31/136 (22.8%). At the subtype level, 60 materialization failures add or omit columns after the needed internal result is available, while 17 intent failures misformulate the requested output or row grain. These two distinct routes to an incorrect answer schema comprise 77/136 (56.6%) of audited failures. By comparison, only three failures arise from selecting the wrong evidence source; extraction and semantic grounding together account for 21, showing that locating an artifact does not ensure that its values are recovered and aligned correctly.

Symptoms are not diagnoses. Figure 7 shows that the same evaluator outcome can arise at different stages. Of 74 audited column-count mismatches, 58 originate in materialization, while the remainder begin in task intent, extraction, grounding, or computation. Conversely, only 5 of the 13 no-submission outcomes are pure execution-control failures; the other eight follow an earlier persistent interpretation, extraction, grounding, or computation error. Aggregate output symptoms therefore obscure the upstream intervention required to correct a trajectory. The complete subtype taxonomy and counts appear in Appendix A.8.3.

Finding 4. Harnesses must faithfully materialize the exact requested output, rather than merely find and correctly compute the relevant values: target-result misunderstanding and faulty column projection account for 56.6% of Grok 4.5’s audited failures, while only 5 of 13 no-submission outcomes are pure termination failures.