AI progress is being measured on the wrong things.
Models keep climbing benchmarks without getting reliably better to use. GOAT labs exists to close that gap — by measuring what production LLMs actually do under real workloads, and aligning the field with the capabilities that matter most.
Benchmarks are getting better.
The models aren't — not where it counts.
Models post higher scores every quarter, but the curve on a test set rarely matches the curve in production. That divergence is dangerous: we end up building systems that ace the eval while missing what people actually needed from them.
The most important problem in AI right now is simple to state and hard to do: align the models with the capabilities that matter most for the people relying on them. You cannot align what you cannot measure — so we start with honest measurement.
We measure models where they actually run.
We operate an opt-in corpus of production LLM telemetry, and we build research products on top of it — so our view of model behavior comes from real usage, not from the lab.
Teams pipe in their observability platform — or link Cursor, Claude Code, and OpenAI Codex — with a read-only key. The corpus that builds is domain-stratified and redacted, and it gives us direct access to how production models behave across millions of real interactions.
On that substrate we're building studies and running live benchmarks like Polymarket Bench, where frontier models bet real money on real-world outcomes. Our first studies are in progress — and our live benchmark, PolymarketBench, already publishes every bet, transcript, and score in the open.
Whoever defines the measurement
defines what models become good at.
Every major model improvement is driven by what gets measured. Optimize a benchmark and the models bend toward it; optimize the wrong one and capability drifts away from what people need.
Grounding measurement in production reality is a leverage point over the entire field — and it sits directly on the critical path of getting AI to actually serve the people who depend on it. That is the work we are here to do.
Four ways we turn production traffic
into honest signal.
Each is framed by the question it exists to answer — and each feeds the same corpus of real model behavior.
Studies
What do frontier models actually do under real workloads?
Research on production LLM behavior — failure modes, drift, and cost across real traffic, not leaderboard prompts. Our first studies are in progress, and they'll ship in the open.
ExplorePolymarket Bench
Can AI models actually predict the future?
Eight frontier models bet real money on Polymarket every day. A live, adversarial benchmark for forecasting and judgment under uncertainty — settled by reality, not by a rubric.
ExploreThe Dataset
What does production model usage really look like?
An opt-in corpus of production LLM telemetry — captured read-only, encrypted at ingest, and redacted behind a default-DENY gate. The substrate every GOAT study is built on.
ExploreDomains
Where does production reality diverge from the benchmarks?
Telemetry spanning verticals — Medicine, Legal, Code, and more — so the research reflects production reality, not just the high-volume domains.
ExploreA research organization, run like one.
GOAT labs is a San Francisco research organization. We answer to the data and to the contributors whose telemetry makes the work possible.
Measurement over marketing
We publish what the data says, not what any lab wants it to say. Every number traces back to a reproducible slice of the corpus.
Opt-in, always
Nothing is eligible for research without passing our default-DENY redaction gate — anything uncertain is quarantined, never auto-promoted. Read-only capture, revoke anytime.
Built in the open
PolymarketBench publishes every bet, transcript, and score live. Our corpus studies will ship the same way — in the open, for everyone.
Help us measure what models actually do.
Contribute the telemetry you're already logging, or talk to us about the research built on it.