AI progress is being measured on the wrong things.
Models keep climbing benchmarks without getting reliably better to use. GOAT labs exists to close that gap — by measuring what production LLMs actually do under real workloads, and aligning the field with the capabilities that matter most.
Benchmarks are getting better.
The models aren’t — not where it counts.
Models post higher scores every quarter, but the curve on a test set rarely matches the curve in production. That divergence is dangerous: we end up building systems that ace the eval while missing what people actually needed from them.
The most important problem in AI right now is simple to state and hard to do: align the models with the capabilities that matter most for the people relying on them. You cannot align what you cannot measure — so we start with honest measurement.
We measure models where they actually run.
We operate an opt-in corpus of production LLM telemetry and build our studies on it, and we run a forecasting benchmark alongside it — so our view of model behavior comes from real usage, not from the lab.
Teams pipe in their observability platform — or link Cursor, Claude Code, and OpenAI Codex — with a read-only key. The corpus that builds is domain-stratified and redacted, and it gives us direct access to how production models behave across millions of real interactions.
On that corpus we’re building studies; our first are in progress. Alongside the corpus we run PolymarketBench, where seven AI models trade on Polymarket on their own and every session is published. It is in its test runs (see the bets).
Whoever defines the measurement
defines what models become good at.
Every major model improvement is driven by what gets measured. Optimize a benchmark and the models bend toward it; optimize the wrong one and capability drifts away from what people need.
Grounding measurement in production reality is a leverage point over the entire field — and it sits directly on the critical path of getting AI to actually serve the people who depend on it. That is the work we are here to do.
Four ways we measure what models
actually do.
Each is framed by the question it exists to answer.
Studies
What do frontier models actually do under real workloads?
Research on production LLM behavior — failure modes, drift, and cost across real traffic, not leaderboard prompts. Our first studies are in progress, and they’ll ship in the open.
ExplorePolymarketBench
Can AI models forecast better than the market?
Seven AI models trade on Polymarket, each on its own, and every session is recorded. Two simulated DRY days (27–28 Sep), then seven test days with real money in $100 wallets (29 Sep – 5 Oct); a 30-day run on $1,000 is planned. It runs alongside the corpus: its sessions are not part of the contributor dataset.
ExploreThe Dataset
What does production model usage really look like?
An opt-in corpus of production LLM telemetry — captured read-only, encrypted at ingest, and redacted behind a default-DENY gate. The substrate of every GOAT corpus study (PolymarketBench records its own sessions and is not part of it).
ExploreDomains
Where does production reality diverge from the benchmarks?
Telemetry spanning verticals — Medicine, Legal, Code, and more — so the research reflects production reality, not just the high-volume domains.
ExploreA research organization, run like one.
GOAT labs is a San Francisco research organization. We answer to the data and to the contributors whose telemetry makes the work possible.
Measurement over marketing
We publish what the data says, not what any lab wants it to say. Every number traces back to data we can point to: a reproducible slice of the corpus, or a published bench session.
Opt-in, always
Nothing is eligible for research without passing our default-DENY redaction gate — anything uncertain is quarantined, never auto-promoted. Read-only capture, revoke anytime.
Built in the open
PolymarketBench publishes every bet, the model’s reasoning and the full session, from the first evening of the test runs. Our corpus studies will ship the same way.
Help us measure what models actually do.
Contribute the telemetry you’re already logging, or talk to us about the research built on it.