AI progress is being measured on the wrong things.
Models keep climbing benchmarks without getting reliably better to use. GOAT labs exists to close that gap — by measuring what production LLMs actually do under real workloads, and aligning the field with the capabilities that matter most.
Benchmarks are getting better.
The models aren't — not where it counts.
Models post higher scores every quarter, but the curve on a test set rarely matches the curve in production. That divergence is dangerous: we end up building systems that ace the eval while missing what people actually needed from them.
The most important problem in AI right now is simple to state and hard to do: align the models with the capabilities that matter most for the people relying on them. You cannot align what you cannot measure — so we start with honest measurement.
We measure models where they actually run.
We operate the largest opt-in corpus of production LLM telemetry, and we build research products on top of it — so our view of model behavior comes from real usage, not from the lab.
Teams pipe in their observability platform — or link Cursor, Claude Code, and OpenAI Codex — with a read-only key. The corpus that builds is domain-stratified and redacted, and it gives us direct access to how production models behave across millions of real interactions.
On that substrate we publish Gartner-style studies and run live benchmarks like Polymarket Bench, where frontier models bet real money on real-world outcomes. The studies are already cited by labs and universities worldwide.
Whoever defines the measurement
defines what models become good at.
Every major model improvement is driven by what gets measured. Optimize a benchmark and the models bend toward it; optimize the wrong one and capability drifts away from what people need.
Grounding measurement in production reality is a leverage point over the entire field — and it sits directly on the critical path of getting AI to actually serve the people who depend on it. That is the work we are here to do.
Four ways we turn production traffic
into honest signal.
Each is framed by the question it exists to answer — and each feeds the same corpus of real model behavior.
Studies
What do frontier models actually do under real workloads?
Gartner-style research on production LLM behavior — failure modes, drift, and cost across real traffic, not leaderboard prompts. Subscribers receive each study under a 120-day embargo before it is published openly.
ExplorePolymarket Bench
Can AI models actually predict the future?
Eight frontier models bet real money on Polymarket every day. A live, adversarial benchmark for forecasting and judgment under uncertainty — settled by reality, not by a rubric.
ExploreThe Dataset
What does production model usage really look like?
The largest opt-in corpus of production LLM telemetry — redacted, domain-stratified, and citable by stable batch IDs. The substrate every GOAT study is built on.
ExploreDomains
Where does production reality diverge from the benchmarks?
Telemetry weighted by vertical — Medicine, Legal, Code, and more — so rare, high-stakes data is worth contributing. Multipliers are recomputed quarterly from subscriber demand.
ExploreA research organization, run like one.
GOAT labs is a San Francisco research organization. We answer to the data and to the contributors whose telemetry makes the work possible.
Measurement over marketing
We publish what the data says, not what any lab wants it to say. Every number traces back to a reproducible slice of the corpus.
Opt-in, always
Nothing enters the corpus without a contributor approving the specific batch. Read-only access, three-pass redaction, research-use-only licensing.
Open after embargo
Subscribers fund the work and get a head start. After the embargo the study and its abstracted data are released openly, for everyone.
Help us measure what models actually do.
Contribute the telemetry you're already logging, or subscribe to the research built on it.