Evaluation framework · pre-results
A curator + harmonizer agent pair, benchmarked against an expert on DOE ESS-DIVE soil-moisture data.
Watershed Function SFA · Berkeley Lab · authors TBD
SCAFFOLD — results pending
Press → / Space to advance · Esc for overview
Motivation
ESS-DIVE hosts diverse Watershed Function SFA datasets but has no dedicated soil-moisture reporting format — every dataset is harmonized by hand into a project schema.
If an agent pair can draft that work reliably, it unlocks scalable, FAIR-aligned reuse.
The task
Soil moisture must be reconciled across:
Target schema — 9 columns, fixed units:
qc_flag vocab: d1 (depth approx.) · g1 (coords from registry) · g2 (coords unresolved)
The system under test
From a DOI/package ID: retrieves & classifies files, resolves location, infers time-series vs discrete, detects manipulations, picks the nearest exemplar.
→ a structured bundle with an INCLUDE / EXCLUDE / FLAG decision.
From the bundle: maps variables, converts units, and emits executable Python plus a documented change-mapping JSON.
→ auditable code & provenance, not an opaque table.
Implemented as Anthropic Agent Skills — versioned instruction+tool bundles; the 19 expert harmonizations act as retrievable in-context exemplars. [Agent Skills 2025]
The intermediate artifact
Between "decide" and "executed table," the harmonizer emits a machine-readable change-mapping: one entry per dataset, with a rule for every target variable. It is the auditable record of what maps to what, and how — decoupled from the Python that implements it.
Top level (7 fields):
index, dataset_identifier, doi, archive_repositorydata_payload_files, location_metadata_files, sensor_metadata_filesharmonization_mappings — the per-variable rulesEach rule (one per target variable):
Excluded datasets collapse to "harmonization_mappings": "EXCLUDED: <reason>".
Real rules leave-one-out fold · dataset 7 · "Bradley Meadow"
Verbatim from an agent run that never saw dataset 7's gold. Source: open evaluation PRs (evaluations/fold-05-holdout-7/).
Scoring the mapping same fold · machine-readable metrics
The change-mapping is compared to the expert's, field by field:
| Field | Exact match vs gold |
|---|---|
destination_variable | 8 / 8 |
unit_conversion | 7 / 8 |
source_files | 7 / 8 |
source_pattern | 3 / 8 |
transformation (free text) | 0 / 8 |
| mapping categories covered | 8 / 8 |
The prose never matches — yet the data does.
The agent and expert describe transformations in different words (transformation 0/8),
but executing both programs on the same raw files gives:
rows recovered (P=R=1.0)
VWC & potential cells match
interval_min cells differ (ordering)
This is the whole argument for an output-equivalence endpoint: stylistic divergence in the mapping/code is irrelevant when the harmonized table is identical. The mapping JSON gives auditable semantics; the executed output gives the verdict.
Figure 1
%%{init: {"theme":"base","themeVariables":{"fontFamily":"ui-sans-serif, system-ui, Inter, sans-serif","fontSize":"15px","lineColor":"#8a8a8a","clusterBkg":"#ffffff","clusterBorder":"#e0d8c4"}}}%%
flowchart LR
classDef agent fill:#0e7c7b,stroke:#0a5a59,color:#fff,stroke-width:1.5px;
classDef artifact fill:#f3efe6,stroke:#c9bfa6,color:#2b2b2b;
classDef gold fill:#c9962f,stroke:#9c7421,color:#fff;
classDef score fill:#3b5b8c,stroke:#28406a,color:#fff;
classDef io fill:#fff,stroke:#8a8a8a,color:#2b2b2b,stroke-dasharray:3 3;
ID["dataset identifier"]:::io
EX["19 expert exemplars"]:::gold
subgraph PIPE["agent pair"]
direction LR
C["Curator
Skill 1"]:::agent --> B["bundle"]:::artifact --> H["Harmonizer
Skill 2"]:::agent --> OUT["code +
change-mapping"]:::artifact
end
RAW[("raw files")]:::io
AGT["agent table"]:::artifact
GLD["expert table
(gold)"]:::gold
CMP{{"output equivalence
cell-by-cell"}}:::score
ID --> C
EX -. exemplars .-> C
EX -. exemplars .-> H
OUT -- execute --> AGT
RAW --> AGT
RAW --> GLD
AGT --> CMP
GLD --> CMP
Modes: Skill-1 isolated · Skill-2 oracle (gold bundle) · end-to-end. Gap = oracle − end-to-end.
Evaluation design
Honest generalization, decomposable error, output-grounded scoring.
Phases
Grouped leave-one-out over 19 expert datasets. Whole source/instrument clusters held out together, so a near-duplicate sibling can't leak into the exemplar pool.
Also validates automated metrics against expert judgment.
All 19 as exemplars; agent harmonizes new ESS-DIVE datasets. Expert harmonizes the same data blind → reference.
Estimates real deployment performance.
A similarity covariate (distance to nearest exemplar) turns leakage risk into a measured generalization curve.
Modes
Curator bundle scored field-by-field vs expert labels.
Harmonizer fed the gold bundle — transform quality, independent of curator error.
Full pipeline; errors propagate.
Error-propagation gap = Mode 2 − Mode 3 isolates how much curator error degrades the final output.
Failures attributed as:
Metrics
Execute both the agent's and expert's code on the same raw data; compare tables cell-by-cell on the natural key. Two correct programs score the same regardless of style — and we avoid unreliable LLM-as-judge scoring. [Judge reliability 2026]
Harmonizer composite weighting:
Decision P/R/F1 · file-set F1 · time-series accuracy & interval error · location-source & qc_flag accuracy · manipulation detection · exemplar match · deferral calibration.
Nested data → cluster bootstrap CIs & mixed-effects models; effect sizes over p-values (N≈19); variance & pass@k across stochastic repeats.
Controls
No skill structure or tool loop — isolates the value of agentic decomposition.
Agent with the 19 references removed — isolates the value of the exemplars.
String-matching floor.
Reproducibility pinned per run: model + version, both skill versions, sampling params, seeds.
Positioning
| Work | What it does | We add |
|---|---|---|
| Harmonia Santos 2025 | LLM-agent harmonization to a schema; interactive; clinical | autonomous pair · environmental · CV + blind eval |
| CleanAgent Qi 2024 | single-agent data standardization | two-stage curate→harmonize, scored separately |
| MAST Cemri 2025 | taxonomy of multi-agent failures | operationalized cross-agent error attribution |
| KramaBench 2025 | data pipelines benchmark (best ~55% E2E) | real expert ground truth + output equivalence |
| "FMs wrangle data?" Narayan 2022 | FM beats task-specific cleaners | composed, end-to-end, executed against expert |
Lineage: Wrangler/Trifacta · HoloClean · FlashFill · schema-matching surveys (Rahm 2001). Domain anchors: ISMN, ESS-DIVE reporting formats, Maelstrom, OMOP.
Deliverables
| Result | Value |
|---|---|
| End-to-end output equivalence | [[ • ]] |
| Skill-2 (oracle) equivalence | [[ • ]] |
| Error-propagation gap | [[ • ]] |
| Curator decision F1 | [[ • ]] |
| vs single-call / no-exemplar | [[ • ]] |
| Similarity threshold | [[ • ]] |
Plus:
Every number traces to experiments/ → results/; placeholders are filled by the analysis pipeline, never by hand.
Caveats
Intended operating point: agent drafts, expert reviews, with automatic deferral above the similarity threshold and on low-confidence curator decisions.
Takeaways
First such evaluation in the environmental-data-harmonization setting. Results to follow.
Thank you
Manuscript drafts: docs/manuscript/ · Figure source: figures/figure1_workflow.mmd
Framework: config/ · src/ · skills/ · gold standard: data/gold/
Key references — Dorigo 2021 (ISMN) · Crystal-Ornelas 2022 (ESS-DIVE formats) · Santos 2025 (Harmonia) · Cemri 2025 (MAST) · Narayan 2022 (FMs wrangle data) · Wickham 2014 (tidy data) · Wilkinson 2016 (FAIR). Full list in references.md.