A curator (Skill 1) turns a dataset identifier into a structured bundle; a harmonizer (Skill 2) turns that bundle into executable transform code and a documented change-mapping. The agent's code and the expert's code are each executed on the same raw files and the resulting tables are compared cell-by-cell (primary endpoint). Three evaluation modes localize error, and the Mode 2 − Mode 3 gap quantifies how much curator error propagates downstream.
%%{init: {"theme":"base","themeVariables":{
"fontFamily":"ui-sans-serif, system-ui, Inter, Segoe UI, Roboto, sans-serif",
"fontSize":"14px","lineColor":"#8a8a8a","primaryBorderColor":"#c9bfa6",
"clusterBkg":"#ffffff","clusterBorder":"#e0d8c4"}}}%%
flowchart TB
classDef agent fill:#0e7c7b,stroke:#0a5a59,color:#ffffff,stroke-width:1.5px;
classDef artifact fill:#f3efe6,stroke:#c9bfa6,color:#2b2b2b;
classDef gold fill:#c9962f,stroke:#9c7421,color:#ffffff;
classDef score fill:#3b5b8c,stroke:#28406a,color:#ffffff;
classDef io fill:#ffffff,stroke:#8a8a8a,color:#2b2b2b,stroke-dasharray:3 3;
ID["ESS-DIVE dataset identifier
(DOI / package ID)"]:::io
EX["19 expert exemplars
(in-context references)"]:::gold
subgraph PIPE["Agent pair under evaluation"]
direction TB
C["Curator — Skill 1
retrieve · classify files · resolve location
infer time series · detect manipulation · pick exemplar"]:::agent
B["Curator bundle
decision · payload/location files · coords · interval · qc_flag"]:::artifact
H["Harmonizer — Skill 2
map variables · convert units · generate code + change-mapping"]:::agent
OUT["Python transform code
+ change-mapping JSON"]:::artifact
C --> B --> H --> OUT
end
RAW[("Raw package files")]:::io
GC["Expert transform code"]:::gold
AGT["Agent harmonized table"]:::artifact
GLD["Expert harmonized table
(gold standard)"]:::gold
CMP{{"PRIMARY ENDPOINT
output-data equivalence
cell-by-cell on the natural key"}}:::score
ID --> C
EX -. exemplars .-> C
EX -. exemplars .-> H
OUT -- execute --> AGT
RAW --> AGT
RAW --> GLD
GC --> GLD
AGT --> CMP
GLD --> CMP
subgraph MODES["Evaluation modes (error attribution)"]
direction TB
M1["Mode 1 · Skill-1 isolated — bundle vs expert labels"]:::score
M2["Mode 2 · Skill-2 oracle — gold bundle into harmonizer"]:::score
M3["Mode 3 · end-to-end — full pipeline, errors propagate"]:::score
GAP["Error-propagation gap = Mode 2 − Mode 3"]:::score
M2 --- GAP
M3 --- GAP
end
B -. scored .-> M1
OUT -. scored .-> M3