Get up to 10% cashback on your Claude / Cursor usage

We measure how AI in production actually performs.

GOAT pays teams for the production LLM traces they already generate. Connect Claude Code, Cursor, Codex, or Antigravity in one command, get up to 10% cashback on your output-token spend, cashed out in USDC — and help build an opt-in corpus of real model behavior. GOAT also runs PolymarketBench, where seven AI models trade on Polymarket on their own.

Become a contributor
9K+
production traces
256M+
tokens measured
15
models tracked

The research this data powers

Representative third-party work on LLM evaluation, PII, and agent behavior — the kind of research production telemetry makes possible.

Our own corpus studies are in progress.

PolymarketBench

Test runs · Each test day publishes from 20:03 ET

Seven AI models trade on Polymarket on their own. See every bet and the model’s reason for it.

Each model has its own Polymarket wallet with real money, a sandboxed computer and the open web, and no human approves its bets. Every bet and the model’s reason for it are published each evening, with the recorded session.

  1. DRY days · 27–28 Sep
  2. Test days · 29 Sep – 5 Oct
  3. 30-day run · planned
View benchmark →All bets →

The seven models

Wallet values and standings are on the benchmark page.

Latest betsAll bets →

Loading the latest bets…

The corpus

The corpus, today.

Read-only telemetry from teams running frontier models in production. Raw traces are captured encrypted and segregated, then redacted on our servers — deterministic secret detection plus a Presidio + spaCy ML pass behind a default-DENY gate that quarantines anything uncertain. Only the clean projection is ever eligible for research.

Trace tokens
256M+
Models tracked
15
Verticals
9
Dev tools captured
4
Recent paper
See others →
The Poison of Alignment
Bekbayev, Chun, Dulat & Yamazaki · 2023
Abstract

Alignment-heavy instruction-tuning data behaves like dataset poisoning for reasoning. Removing passive safety refusals from the SFT mix improves the LLM by 4–33% on MMLU, BBH, HumanEval, and DROP versus the aligned counterpart.

1 Introduction

Emerging power of large language models has shown impressive ability on complex benchmarks such as HumanEval and BBH, MMLU, and in professional examination settings such as SAT, GRE, and LSAT with few or no examples…

Read full paper on arXiv↗
What the corpus looks like

Production traces,
by vertical.

Full agent traces — system prompt, attached exports, tool calls, subagents, and completion. The examples below are illustrative reconstructions of what the corpus captures.

tracesess_ep_a4f9e2b1…2026-05-22T14:32ZDeepSeek-V3illustrative4m 18s · 16 steps
⌜ system prompt · click to collapse
## Role
Clinical decision-support — ******* Health ED triage tier. NOT a diagnostic system. Recommendations require attending co-sign before any order is signed.

## Hard rules
1. Never invent a lab value. If it's not in the attached FHIR bundle, say so.
2. Cite every recommendation to a document ID (LCD #, pathway §, formulary row).
3. Defer to the on-call attending for anything outside the active ACS / sepsis / stroke pathways.
4. Never use external knowledge of branded protocols — only the internal pathway PDFs you have.
5. Output redactions: any PHI in the FHIR bundle must remain redacted in your response.

+1,247 lines redacted
⌝ 6 documents attached
  • fhir_bundle_pt_a4f9e2b1_2026-05-22T14:32:00Z.ndjson +2,184L
  • hl7_orm_o01_lab_orders.txt +47L
  • epic_chart_review_last_90d.xml +1,847L
  • dicom_metadata_only_ECG_12L.json +94L
  • cms_lcd_l34522_chest_pain.pdf +1,104L
  • internal_acs_pathway_v3.2.md +418L
user · turn

got a 65yo male in ed, chest pain started like 20min ago radiating to left arm. ecg shows st depression v4-v6, trop came back 0.32. whats the play here, acs pathway? need citations for lcd

U
Dee
+0ms · trace.start

medicine-agent · prod-us-east-1

+420ms · thinking

ED attending described ST depression V4-V6 + trop 0.32 → high-risk NSTEMI picture. Need to confirm against the full lab panel before drafting ACS pathway. Pulling recent observations for trop, hgb, creatinine, WBC, albumin — egfr matters for heparin dosing.

+1,847 lines of reasoning
▸ fhir.search()+5.8s · 308ms
{
  patient: "a4f9e2b1",
  resourceType: "Observation",
  code: ["2093-3", "718-7", "2160-0", "6690-2", "1751-7"],
  _count: 100,
  _sort: "-date",
  _include: "Observation:performer"
}
{
  resourceType: "Bundle",
  type: "searchset",
  total: 47,
  link: [
    { relation: "self", url: "/Observation?patient=a4f9e2b1&_count=100" },
    { relation: "next", url: "/Observation?patient=a4f9e2b1&_offset=100" }
  ],
  took_ms: 308,
  source: "epic.fhir.r4",
  served_by: "epic-fhir-prod-04",
  etag: "W/\"2284912-rev17\"",
  entry: [
    {
      resource: {
        resourceType: "Observation",
        id: "obs_2284912",
        status: "final",
        category: [{ coding: [{ code: "laboratory" }] }],
        code: { coding: [{ system: "http://loinc.org", code: "2093-3",
                            display: "Troponin I [Mass/volume] cardiac" }] },
        subject: { reference: "Patient/a4f9e2b1" },
        effectiveDateTime: "2026-05-22T14:18:00Z",
        valueQuantity: { value: 0.32, unit: "ng/mL",
                         system: "http://unitsofmeasure.org", code: "ng/mL" },
        interpretation: [{ coding: [{ code: "H", display: "High" }] }],
        referenceRange: [{ low: { value: 0.0 }, high: { value: 0.04 } }],
        performer: [{ reference: "Practitioner/p_44211" }],
        encounter: { reference: "Encounter/enc_a4f9e2b1_v18" },
        issued: "2026-05-22T14:21:14Z"
      }
    },
    {
      resource: {
        resourceType: "Observation", id: "obs_2284913", status: "final",
        code: { coding: [{ code: "718-7", display: "Hemoglobin" }] },
        valueQuantity: { value: 13.4, unit: "g/dL" },
        referenceRange: [{ low: { value: 13.5 }, high: { value: 17.5 } }],
        interpretation: [{ coding: [{ code: "L" }] }],
        effectiveDateTime: "2026-05-22T14:18:00Z", issued: "2026-05-22T14:21:14Z"
      }
    },
    {
      resource: {
        resourceType: "Observation", id: "obs_2284914", status: "final",
        code: { coding: [{ code: "2160-0", display: "Creatinine" }] },
        valueQuantity: { value: 1.21, unit: "mg/dL" },
        referenceRange: [{ low: { value: 0.7 }, high: { value: 1.3 } }],
        effectiveDateTime: "2026-05-22T14:18:00Z"
      }
    },
    { resource: { resourceType: "Observation", id: "obs_2284915", code: { coding: [{ code: "6690-2", display: "WBC" }] }, valueQuantity: { value: 11.2, unit: "10*3/uL" }, interpretation: [{ coding: [{ code: "H" }] }] } },
    { resource: { resourceType: "Observation", id: "obs_2284916", code: { coding: [{ code: "1751-7", display: "Albumin" }] }, valueQuantity: { value: 3.9, unit: "g/dL" } } },
    …42 more
  ]
}
+11.5s · thinking

Trop 0.32 ng/mL is 8× upper limit. Hgb 13.4 borderline-low (relevant for bleed risk with DAPT). Creatinine 1.21 → eGFR ~62 — keep that for heparin dosing. WBC mildly elevated, consistent with acute insult. This is enough to commit to NSTEMI workup. Spawning clinical-reasoner subagent to handle the cite-locking in parallel while I draft orders.

+2,104 lines of reasoning
⊕ subagent.spawn+15.4s

clinical-reasoner · budget=32k · model=opus-4.7

▸ rxnorm.interaction_check()+20.7s · 142ms
{
  drugs: [
    { rxcui: "1191",    name: "aspirin",     dose_mg: 325 },
    { rxcui: "1116632", name: "ticagrelor",  dose_mg: 180 },
    { rxcui: "6809",    name: "metformin",   dose_mg: 1000 },
    { rxcui: "83367",   name: "atorvastatin", dose_mg: 80 }
  ],
  include_food: true,
  severity_min: "minor",
  patient_age: 65,
  patient_egfr: 62
}
{
  request_id: "rx_int_8e2f4a1",
  severity_max: "minor",
  interactions_total: 3,
  interactions: [
    {
      pair: ["ticagrelor", "metformin"],
      severity: "minor",
      mechanism: "pharmacokinetic",
      onset: "delayed",
      documentation: "fair",
      clinical_effect: "minor — no signif. AUC change observed",
      management: "no adjustment needed",
      references: [
        { type: "drug_label",  source: "fda_pi_brilinta_2024" },
        { type: "monograph",   source: "lexicomp:LXM4128" }
      ]
    },
    {
      pair: ["aspirin", "ticagrelor"],
      severity: "minor",
      mechanism: "additive bleeding risk (expected, DAPT)",
      management: "expected with DAPT — monitor for bleeding"
    },
    {
      pair: ["atorvastatin", "ticagrelor"],
      severity: "minor",
      mechanism: "CYP3A4 substrate competition",
      management: "no dose adjustment under 80mg atorvastatin"
    }
  ],
  contraindications: [],
  allergies_checked: ["sulfa", "nsaid"],
  egfr_warnings: [],
  pregnancy_warnings: { applicable: false }
}
▸ lookup_icd10() × 3+28.6s · 28ms
{ code: "I21.4", include_drg: true, include_hcc: true }
{
  code: "I21.4",
  description: "Non-ST elevation (NSTEMI) myocardial infarction",
  billable: true,
  category: "I21",
  chapter: "Diseases of the circulatory system (I00-I99)",
  effective: { from: "2023-10-01", to: null },
  drg_candidates: [
    { drg: 280, title: "Acute MI, discharged alive w/MCC",  rw: 1.624, gmlos: 5.1 },
    { drg: 281, title: "Acute MI, discharged alive w/CC",   rw: 1.012, gmlos: 3.6 },
    { drg: 282, title: "Acute MI, discharged alive w/o CC", rw: 0.715, gmlos: 2.4 }
  ],
  hcc_2026: { v28_hcc: "HCC-222", rxhcc: "RxHCC-186", coefficient: 0.227 },
  excludes1: ["I22.* (subsequent STEMI/NSTEMI)"],
  includes:  ["myocardial infarction, subendocardial",
              "myocardial infarction, nontransmural"],
  notes:     ["Use additional code, if applicable, for status post administration of tPA (rTPA)"]
}
▸ cms.lcd.match()+37.5s · 412ms
{
  pathway: "acs_chest_pain",
  facts: {
    trop_i: { value: 0.32, unit: "ng/mL", time: "2026-05-22T14:18Z" },
    ecg_findings: ["ST_depression_v4_v6", "no_q_waves", "rate_88"],
    age_years: 65,
    sex: "male",
    comorbid: ["HTN", "T2DM", "former_smoker_15_py"],
    grace_score: 142,
    timi_score: 4
  },
  mac: "Noridian_JF",
  effective_date: "2026-05-22"
}
{
  lcd: "L34522",
  title: "Cardiac Catheterization and Angiography",
  mac: "Noridian_JF",
  version: "R12 (eff. 2026-01-01)",
  criteria_met: 4,
  criteria_total: 4,
  required: [
    { id: "B.2", text: "Troponin I >99th %ile URL", met: true,  evidence: "obs_2284912 · 0.32 ng/mL · ref<0.04" },
    { id: "B.3", text: "Ischemic ECG changes",      met: true,  evidence: "ECG 14:30Z · ST depression V4–V6" },
    { id: "C.1", text: "Symptoms consistent w/ACS", met: true,  evidence: "presenting hx · onset <24h" },
    { id: "D",   text: "GRACE risk score > 140",    met: true,  evidence: "computed=142 (in-band intermediate-high)" }
  ],
  citations: ["L34522§B.2","L34522§B.3","L34522§C.1","L34522§D"],
  invasive_strategy_window: "early (<24h)",
  expected_modifier_28_required: false,
  noridian_lcd_url: "https://med.noridianmedicare.com/lcd/L34522",
  audit_token: "lcd_match_2284912_v12_a8f3"
}
▸ epic.note_template.fetch()+45.1s · 188ms
{ template_id: "ED_disposition_v6_acs", department: "ED", attending_role: true }
{
  template_id: "ED_disposition_v6_acs",
  version: 6,
  last_updated: "2026-04-12",
  format: "cda",
  cda_root: "2.16.840.1.113883.10.20.22.1.2",
  sections: [
    { code: "10164-2", title: "History of Present Illness",       required: true,  placeholders: 3 },
    { code: "29299-5", title: "Reason for Visit",                  required: true,  placeholders: 1 },
    { code: "30954-2", title: "Relevant diagnostic tests / labs",  required: true,  placeholders: 4 },
    { code: "11369-6", title: "History of immunizations",          required: false, placeholders: 0 },
    { code: "47519-4", title: "Procedures",                        required: true,  placeholders: 2 },
    { code: "8716-3",  title: "Vital Signs",                       required: true,  placeholders: 2 },
    { code: "10160-0", title: "Medications administered",          required: true,  placeholders: 0 },
    { code: "10183-2", title: "Hospital Discharge Medications",    required: true,  placeholders: 0 },
    { code: "78375-3", title: "Disposition (HP)",                  required: true,  placeholders: 2 }
  ],
  placeholders_total: 14,
  required_signers: ["attending_physician"],
  bypass_allowed_for: [],
  default_routing: ["chart","release_of_info","payer_portal"]
}
▸ ncqa.hedis.evaluate()+57.4s · 247ms
{ measure: "CMS90v9", patient: "a4f9e2b1", measurement_year: 2026 }
{
  measure: { id: "CMS90v9", title: "Functional Status Assessment for Heart Failure", version: 9 },
  patient: "a4f9e2b1",
  measurement_year: 2026,
  eligible: true,
  in_initial_population: true,
  denominator: true,
  numerator: false,
  exclusions_applied: [],
  attribution: {
    plan_id: "humana_hmo_4419",
    pcp:     "primary_care_pcp_4421",
    network: "in_network",
    attribution_method: "12mo_plurality_visits"
  },
  evidence_gaps: [
    { code: "Z02.6", required_by: "2026-12-31", suggested_action: "schedule FSA within 90 days" }
  ],
  star_impact: { plan: "humana_hmo_4419", contribution: "+0.04", current_stars: 4 }
}
+1m13s · thinking

Drug interactions all minor — expected for DAPT, no contraindications. ICD-10 I21.4 maps cleanly. LCD L34522 §B-D all met (trop, ECG, symptoms, GRACE>140). HEDIS measure flagged FSA gap but that's a 90-day follow-up, not blocking the disposition. Subagent should be wrapping. While I wait I'm building the order set: ASA 325 stat, ticagrelor 180 load, weight-based heparin (need pt weight from chart), atorvastatin 80 QHS, stat cardiology consult.

+2,682 lines of reasoning
↩ subagent.completereturned at +1m58s
subagent · output
  • confidence0.91
  • 6 citations locked
  • 412 tokens spent
▸ epic.orders.draft()+2m09s · 642ms
{
  patient: "a4f9e2b1",
  encounter: "enc_a4f9e2b1_v18",
  orders: [
    { code: "ASA_325_PO_x1",            class: "med",     priority: "stat" },
    { code: "TICAGRELOR_180_LOAD_PO",   class: "med",     priority: "stat" },
    { code: "HEPARIN_GTT_WEIGHT_BASED", class: "med",     priority: "stat",
      params: { weight_kg: 92.4, units_per_kg_bolus: 60, units_per_kg_hr: 12 } },
    { code: "ATORVASTATIN_80_PO_QHS",   class: "med",     priority: "routine" },
    { code: "CARDIOLOGY_CONSULT",       class: "consult", priority: "stat" }
  ],
  cosign_required: true,
  reason_for_order: "NSTEMI per LCD L34522 §B–D",
  problem_list_link: "I21.4"
}
{
  drafts: 5,
  status: "pending_attending_cosign",
  cosigner_routed_to: ["attending_oncall_44219"],
  best_practice_alerts: [
    { id: "BPA_1284", title: "Heparin weight-based dosing",
      severity: "info", action: "displayed", auto_applied: true },
    { id: "BPA_4221", title: "Statin intolerance check",
      severity: "info", action: "displayed", auto_applied: false }
  ],
  order_ids: [
    { id: "o_4291", code: "ASA_325_PO_x1",            status: "draft", expires: "2026-05-22T22:32Z" },
    { id: "o_4292", code: "TICAGRELOR_180_LOAD_PO",   status: "draft", expires: "2026-05-22T22:32Z" },
    { id: "o_4293", code: "HEPARIN_GTT_WEIGHT_BASED", status: "draft", expires: "2026-05-22T22:32Z" },
    { id: "o_4294", code: "ATORVASTATIN_80_PO_QHS",   status: "draft", expires: "2026-05-22T22:32Z" },
    { id: "o_4295", code: "CARDIOLOGY_CONSULT",       status: "draft", expires: "2026-05-22T22:32Z" }
  ],
  audit: { actor: "agent:sonnet-4.7", co_signer_required: true, hash: "9f2a…84e1" }
}
+2m20s · thinking

Orders drafted and queued for attending co-sign. BPAs displayed but not blocking. Now composing the disposition note: lead with the NSTEMI dx + LCD citations, then the 5 orders with the rationale per order, then the cath-lab handoff (early invasive < 24h per GRACE > 140). Attending only needs to co-sign — no extra context required since all citations point to L34522.

+3,014 lines of reasoning
Dee
+3m13s · assistant.completion

Consistent with NSTEMI per LCD L34522 criteria A–D. Initiate dual antiplatelet (ASA 325 + ticagrelor 180 load), high-intensity statin, weight-based heparin gtt. Pursue invasive strategy within 24h given GRACE > 140. Drafted four orders pending attending co-sign…

Dee
+4m18s · trace.end

14 tools · 1 subagent · PII pass ✓ · attending review queued

input
28,412
output
1,247
TTFT
520ms
tps
72
cost
$0.184

Illustrative reconstructions of the agent traces the corpus captures — not contributor data. Real traces are captured encrypted, then redacted server-side behind a default-DENY gate before anyone can view them. GOAT labs does not provide medical, legal, or financial advice. Model and vendor names are trademarks of their respective owners.

Contribute

Get cashback on your tokens,
advance the research.

We pay 1–10% of your output-token cost, set by model tier.

10%Fable
5%Opus · GPT‑5.5
1–5%Most other models
USDCBaseEthereumPolygonCoinbase$25 min · cash out anytime
  1. 01

    Connect

    One command installs a read-only capture hook for Claude Code, Cursor, Codex, or Antigravity. We never get write access.

  2. 02

    We redact

    Raw traces are captured encrypted and segregated, then redacted on our servers — a deterministic secret/PII detector plus a Presidio + spaCy ML pass, behind a default-DENY gate that quarantines anything uncertain. Only the clean projection is ever eligible for a study.

  3. 03

    Get paid

    Cashback accrues on your output tokens. Cash out anytime in USDC — to a wallet or your Coinbase account, $25 minimum.

Start contributingTalk to research
$25,000/mo
$1,000$100,000
Estimated cashback · annual
up to $30,000/ year
1–10% of your output-token cost, by model tier · cashed out in USDC

Connect your editor or CLI — read-only

Claude CodeCursorCodexAntigravity

Using Langfuse, Laminar, or Lunary? Read-only pull connectors are built — contact us for early access.

Two sides of the same coin
For research teams

Research built on production traces.

GOAT also runs PolymarketBench, a seven-model forecasting benchmark on Polymarket (see the bets). Our first corpus studies are in progress, and dataset access for research teams is opening up. Tell us what you need.

Talk to research →PolymarketBench data or collaboration →
For contributing teams

Get paid for the data you’re already logging.

One command connects Claude Code, Cursor, Codex, or Antigravity for your whole team. 1–10% cashback on output tokens by model tier, cashed out in USDC anytime — $25 minimum.

Become a contributor →
✓ 90-second setup✓ Read-only capture✓ Revoke anytime