Studies

Research on the
live corpus.

One forecasting benchmark GOAT runs alongside the corpus, one published paper, and the first corpus studies in progress.

BENCHMARK30-DAY RUN · PAPER MONEY4 market categories
GOAT labs

PolymarketBench

Seven AI models research and paper-trade on Polymarket on their own; every bet, reason and session is published. After the test runs (simulated DRY days, then seven days with real money on $100 wallets), the 30-day run trades paper money: $1,000 per model, orders filled in simulation against Polymarket’s live order book.

Wallets
Paper money in the 30-day run (USDC · Polygon in the test runs)
Cap
$5 in tests · $20 in the 30-day run, per market per day
Sessions
Captured in full
Scoring
Profit, checked against a no-skill trader
models7
stage30-day run, paper money
test wallets$100, real money
30-day run wallets$1,000, paper money
View benchmark →
PAPERScience
Bekbayev, Chun, Dulat & Yamazaki · arXiv 2023

The Poison of Alignment

Alignment-heavy instruction-tuning data behaves like dataset poisoning for reasoning. Removing passive safety refusals from the SFT mix improves the LLM by 4–33% on MMLU, BBH, HumanEval, and DROP versus the aligned counterpart — while fine-tuning on aligned data alone often fails to beat the base model.

SFT pairs
3M+
Base model
LLaMA 2 7B
Benchmarks
MMLU · BBH · HE · DROP
Alignment removed
~33%
MMLU Δ+8.1%
BBH Δ+4.1%
HumanEval Δ+33%
DROP Δ+24%
Read the paper on arXiv →
In progress

Our first corpus studies — tool-use reliability, model comparisons on real workloads — are being built on the live corpus.

Talk to research →PolymarketBench data or collaboration →