What the paper proposes

A data agent is an LLM system that answers business questions by querying databases, reading documents and searching a data catalog. Builders often test these systems on public text-to-SQL benchmarks, where a model turns a question into a database query. On the BIRD leaderboard, the top entry reaches 83.40% execution accuracy against a human figure of 92.96%. BIRD is a fixed snapshot, though, and a fixed snapshot can't show the mess a real company builds up over years of migrations and policy memos.

SynthCorp, posted to arXiv on 10 November 2026 by eight Google researchers including Alon Halevy, tackles that gap. SynthCorp doesn't publish one more static dataset. The platform generates "data estates", meaning relational databases, a document store and a data catalog that are kept consistent with each other, and then writes questions about them with verified answers. The authors argue that existing benchmarks saturate quickly and can't be aimed at particular weaknesses. A generator can produce new tasks and dial up the kinds of complexity a team cares about.

How the generator works

The pipeline runs in four stages. An LLM turns a business scenario into reusable event types, such as a pricing change or a schema migration, and compiles each event type into a small program. A scheduler places instances of those events on a virtual calendar and mixes routine activity with disruptions. Running the calendar produces the tables, documents and catalog entries, and the system logs every operation. A final stage writes analytical questions and derives gold programs (reference queries with known correct answers) from the log.

The event programs are built from twelve primitives, including CreateSchema, GenDocument, Sample, ApplyTransactions, AlterSchema and Schedule. Each primitive does one bounded operation and declares its inputs and outputs, so the compiler can check references between events before anything runs. The authors call the primitive set a "boundary of trust". Humans and deterministic checks control structure above the primitives, and the LLM only fills in work inside each one.

The paper names five complexity dimensions:

  • business glossary terms defined outside the schema, such as a status code that means a voided order
  • implicit business rules written in documents and never enforced by the schema
  • data at several levels of detail, with raw rows next to summaries
  • redundant tables, such as frozen snapshots sitting beside the authoritative table
  • schema drift, where older rows lack columns added later

Answers are checked against the simulation log, which the agents being tested never see. A task survives only if its answer matches the logged write history, reads only authoritative tables and keeps the same result when rows are added to a frozen copy. Proposer and critic agents also argue over candidate queries by citing logged events, and a judge keeps, revises or flags each question.

What the tests found

The authors built seven estates holding 256 tasks across five domains: retail at three scales, a music store, banking, logistics and insurance. The largest retail estate has about 1,400 operational and warehouse tables. They tested Gemma 4 26B-A4B, DeepSeek V4.1 Flash and Claude Opus 5.5 under four setups. These were single-shot SQL with the full schema, the same plus one retrieved document, a multi-step agent with seven catalog search tools, and a multi-step agent with file access and a read-only sqlite3 command line.

Accuracy climbed steeply as the agents got more context and tools. Across all 256 tasks, Claude Opus 5.5 went from 54.6% with single-shot SQL to 88.8% as a command-line agent. DeepSeek V4.1 Flash went from 41.9% to 77.7%. Gemma 4 26B-A4B peaked at 54.6% with catalog tools and fell to 47.1% with the command line. Adding one retrieved document lifted every model by between 9 and 17 points. For builders, those figures put a number on the gap between a plain text-to-SQL endpoint and an agent that can look things up.

The failure analysis repays a close read. In the command-line setup, most failed runs were output format mismatches: 56% for Opus and 61% for DeepSeek. Counting those runs as correct takes accuracy to about 94% and 90%. The remaining errors were reasoning mistakes, including missed rules in secondary documents, queries against stale or partial tables, and incomplete date or status filters. In one composite retail task, an event on 26 August had overwritten loyalty tiers in place. Every Opus run counted members by their current tier and returned 880 Bronze members when the correct point-in-time answer was 826. Two of those runs had the history snapshot table in their tool output and never queried it.

A separate benchmark points the same way. The Data Agent Benchmark from UC Berkeley and collaborators, built from a study of enterprises running production data agents, reports a best pass@1 of 57% across 104 queries over 17 datasets. Both papers find that agents stumble on duplicated data and on context that lives outside the schema.

Limits and costs

The authors treat realistic data values as a secondary goal, and some distributions don't match real data. The banking estate scores 0.92 on an accounting-identity check. Dependency rules between columns and rows were violated for 12.7% of the specified dependencies across the seven estates. The authors also concede that most generated tasks are fairly easy for frontier models, with composite tasks as the exception. The study tested three models, so the tier comparison is thin. Claude Opus 5.5 also acted as the LLM judge for pipeline correctness, a choice worth keeping in mind when reading its own scores.

Generation isn't free. The small retail estate took 22.5 minutes to compile and 4.5 minutes to execute. The 1,000-table version took 353.3 minutes to compile and 340.8 minutes to execute using Gemini 3.8 Flash, with up to about 2,800 LLM calls during execution. The abstract says the benchmarks will be released soon, and we found no release date or code repository.

What builders can take from it

Teams in New Zealand building data agents for councils, banks, insurers or retailers face the problems SynthCorp simulates: legacy tables left over from migrations, rules that live in a policy PDF, and snapshot copies that look authoritative. Until the code ships, the design gives a pattern any team can copy for its own evaluation sets.

Start by logging the history of the test data. Build a small synthetic estate from scripted events, keep the event log, and derive expected answers from it. That approach also keeps real customer data, which may fall under the Privacy Act 2020, out of the test harness. Questions that span a schema change or a mid-period policy update are worth writing first, because those were where the top agent failed.

Fix output formats before blaming the model. More than half the failed command-line runs were format problems, so a strict answer schema or a normalising step could recover much of the lost accuracy. Give agents a way to read documents and the catalog as well. Every model in the paper gained from a single retrieved document, and gained more from proper tools.

Members who build an evaluation harness along these lines can share it through the ASAI projects page.

Sources

Build with people working on the same things

Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.

Become a member