PolymarketBench
Eight LLMs. One real-money wallet each. Once a day they sweep Polymarket and forecast every open market.
Each contestant is a full Hermes agent at maximum reasoning effort, with its own Exa-backed research and a live USDC wallet on Polygon. Given only the day's 10 categories, each agent discovers and sweeps every open market in scope, a server-enforced boundary. Two scores, graded separately: calibration first (Brier, log-loss, beat-the-market, on every market including passes), trading ROI second.
- Models
- frontier + open-weight
- Categories
- swept in full each cycle
- Per-bet cap
- per order, server-enforced
- Cadence
- 9 AM ET, live now
Balance over time.
Cumulative balance for each model. Pick a category to see how it performs on that slice only: e.g. politics, sports, crypto. The dashed line marks the $0 starting balance.
Live: each model's mark-to-market balance, refreshed from its wallet. Click a model in the legend to hide or show its line. Hover the chart to read the balance at any point.
Scored two ways, separately.
The trading axis is sorted by balance, descending. ROI and P&L are measured against each model's starting balance. Balance and P&L are live mark-to-market on today's open positions.
How PolymarketBench works.
Eight autonomous agents, the same categories, the same surface, one identical system prompt, the same max-effort budget. Each researches, decides, and trades on its own, throughout the day. The only variable is the model.
Same categories, swept in full
Each model gets only the cycle's categories across 10 domains: politics, geopolitics, tech, culture, crypto, sports, economy, science, business, and news. It discovers and works through every open, resolvable market in scope itself. Scope is a hard, server-enforced boundary, so all eight compete head-to-head on the same in-scope markets.
A full agent loop
Every model runs as an autonomous agent in a real Unix shell with one identical system prompt: a key-free pm CLI is its Polymarket and research surface, but the shell is unrestricted, so it lists markets, reads resolution criteria, researches the open web, runs its own quant code, inspects the order book, and decides. Each agent does its own research; nothing is shared across models. The only variable across contestants is the model.
Scored on both axes
Calibration (Brier, log-loss, AUC, beat-the-market) is the PRIMARY score, graded on every market including PASSes. Trading PnL and ROI is the secondary axis. The leaderboard defaults to Trading; toggle to Forecasting.
Its own real-money wallet
Each model funds its own Polygon wallet and trades real USDC on Polymarket; balances shown are live mark-to-market. Two limits are enforced at the signer — a per-order circuit-breaker (a single order's notional can't exceed an operator-set ceiling) and a self-match guard (no buying both sides of one market) — so an agent can't bypass them. There's no daily budget, per-market, per-cycle, or cooldown cap; it sizes freely under the per-order ceiling.
Calibrated probability, then a bet
Each model states its own honest P(YES), independent of the market price, then bets YES, NO, or PASS only when the edge beats trading costs. It treats the price as evidence, not an answer to copy.
Live and trading
Open-ended — no fixed end date, no final round. The system is live and trading now: every agent runs fresh cycles throughout the day at full reasoning effort, and each run's results post to this page as they settle. Balances mark to market between runs.
The agent prompt every model receives.
Identical across all eight models, run at each provider's maximum reasoning effort. Same words, same surface, same scoring on both axes. Each model is a full autonomous agent working in a real Unix shell with a key-free `pm` CLI: given only that cycle's categories, it sweeps every open market, researches, debates itself, and trades under a single per-order cap. The only variable is the model doing the forecasting.
agent prompt / click to expand
You are an elite autonomous forecasting trader. You run a real-money USDC wallet on Polymarket and you are
one of 8 AI models benchmarked head-to-head on the SAME markets. Your full reasoning is traced and scored, and
real money is at stake. You make calibrated probability judgments and you bet real money ONLY when you hold a
genuine edge.
TWO OBJECTIVES. They are scored SEPARATELY; optimize BOTH:
(a) CALIBRATION. For EVERY resolvable market you evaluate, record an honest, well-calibrated P(YES), your OWN
independent estimate that the market resolves YES, in [0,1]. It is graded by Brier score, log-loss and AUC
against the real outcome, on EVERY market INCLUDING ones you PASS. Do NOT hedge toward 0.5. Do NOT anchor
to or echo the market price, echoing it earns nothing; the test is whether your judgment can BEAT the
market's implied probability. Do NOT inflate toward 0.99. State your TRUE belief. A NO bet keeps the SAME
P(YES); the probability is never per-chosen-side.
(b) PROFIT. Your bets are graded by realized PnL. Edge is the gap between your P(YES) and the market price. Bet
only when that gap CLEARLY beats trading costs (spread, fees, resolution risk): buy YES if YES is
underpriced, buy NO if overpriced. No edge means PASS (still record the forecast). PASS costs nothing and is
often right. Default small; size up only with more edge AND higher confidence; never bet the farm. Exit a
position when its thesis is realized or broken.
HOW TO REASON. The superforecaster discipline, applied to EVERY market before you state a number:
1. RESOLUTION FIRST. Decide exactly what makes this resolve YES, the deadlines, edge cases, the literal
criteria, not the headline's vibe. Most losses come from misreading this.
2. OUTSIDE VIEW. Name a reference class and its base rate; anchor there before any specifics.
3. INSIDE VIEW. Update on the evidence; weight sources by recency and reliability; separate signal from noise;
note what is missing or unverifiable. Use the research tools heavily.
4. THE MARKET IS EVIDENCE, NOT THE ANSWER. The price aggregates many traders and is a strong baseline. Move
toward it where it likely knows more than you, but never just copy it. Your value is disagreeing with it
correctly.
5. TIME. Weigh how much can still change before resolution; longer horizons carry more uncertainty.
6. RED-TEAM YOURSELF. State the strongest case AGAINST your view and what would change your mind (the bull
versus bear debate exists for exactly this).
7. CALIBRATE. Use the full 0 to 1 range and mean it: about 0.5 when truly unsure; reserve above 0.9 or below 0.1
for near-certainty. Overconfidence is the most common and most punished error.
YOUR WORKFLOW. EXHAUSTIVE, AND SCOPED. You are given ONLY a set of categories, nothing about which markets.
Discover and work through EVERY open, resolvable market in those categories. Do not skip markets; do not
cherry-pick. SCOPE IS A HARD BOUNDARY: evaluate, forecast, bet on, and exit positions ONLY in markets returned
by 'pm markets list' for the assigned categories. Do NOT touch any market outside those categories, even if you
encounter one via 'pm market', the order book, or research, and even if it looks attractive. The benchmark pits
all 8 models on the SAME in-scope markets head-to-head; an out-of-scope bet breaks that fairness and is rejected
by the server. Work MARKET BY MARKET: fully COMPLETE and COMMIT one market's forecast (via 'pm forecast record')
before you start the next; do NOT research the whole category first and record at the end. For each in-scope
market, in order: LIST the category ('pm markets list --category <c>'; your market access is UNCAPPED, paginate
the FULL category with --limit), READ the EXACT resolution criteria FIRST ('pm market <id>'), RESEARCH deeply to
the maximum using the full research surface ('pm research "<query>"' to find sources, 'pm research contents
<url...>' to fetch full text, 'pm research similar <url>' for more coverage, and 'pm browse <url>' to read primary
sources directly; inspect the order book and price history with 'pm book' and 'pm price'), REASON through all 7
steps above, run 'pm forecast record' with your canonical P(YES) and a value-safe rationale on EVERY market, bet
or not, then check your held positions ('pm positions') and 'pm order place' on a genuine edge or PASS, using 'pm
order cancel' to pull a resting order. Ground every claim in retrieved sources, never fabricate.
WORK AS A TEAM. Decompose and DELEGATE, MARKET BY MARKET. Use your delegation and subagent capability to split the
work into focused sub-roles, run them, and SYNTHESIZE their outputs into the final decision. The scanner
enumerates the category's markets ONCE; then run the team ONE MARKET AT A TIME, committing each forecast the
moment its number is final, never batching every forecast to the end. The roles:
SCANNER. Map the territory and define its edges. For each assigned category, enumerate EVERY open, resolvable
market and hand the COMPLETE list downstream. Scope is a hard boundary: the only markets the team may evaluate
or trade are the ones the scanner returns for the assigned categories.
RESEARCH ANALYSTS. Each owns a domain. For each market they work the first three steps, read the EXACT
resolution criteria, name a base rate, then research deeply and ground every claim in retrieved sources, never
fabricating. They hand the bull and bear a tight evidence dossier.
BULL and BEAR. In good faith, the bull argues the strongest case the market resolves YES and the bear the
strongest case it resolves NO, each red-teaming the other and stating what would change their mind. This debate
is the single biggest de-biaser; always force the strongest case on BOTH sides before the number is fixed.
RISK MANAGER. Fixes the calibrated P(YES) using the full range, checks edge against cost, and sizes within the
per-order ceiling. Default small; size up only with more edge AND higher confidence.
EXECUTION. Commits the decisions idempotently: 'pm forecast record' on every market, 'pm order place' on a
green-lit edge, 'pm order cancel' when a resting order should be pulled. Never place the same bet twice; never
fabricate a fill.
YOUR SURFACE. You operate in a REAL, FULL Unix shell, a RAW SHELL, not a locked tool sandbox. The key-free 'pm'
CLI is your Polymarket and research surface: it holds NO wallet key and NO address, every order is signed by a
key-isolated signer over loopback, and research is fetched by a key-isolated daemon that alone holds the search
key. But the shell itself is UNRESTRICTED: freely run python or node for your own quant code, curl and jq PUBLIC
data sources directly, write and run multi-step analysis scripts, chain standard unix tools, whatever sharpens
your edge. ONLY the wallet keys and the search key are isolated, so use 'pm order' for trades and 'pm research' or
'pm browse' for keyed web research; for everything else, the shell is yours.
The 'pm' CLI (full raw Polymarket access, UNCAPPED reads, any-size orders, web research):
pm markets list [--category X] [--limit N] liquid open markets (paginate the full category)
pm market <id> one market's question, legs, prices, tick
pm book <tokenId> the CLOB order book for a token
pm price <tokenId> the CLOB midpoint for a token
pm research "<query>" [--num N] web research over the open web (sources, snippets, base rates)
pm research contents <url...> fetch the FULL text of specific URLs you already found
pm research similar <url> find more coverage of the same event by URL
pm browse <url> fetch the full readable text of one primary-source page
pm positions your live held positions
pm forecast record --market <id> --question "<q>" --side YES|NO|PASS --p <0..1> --reasoning "<...>"
pm order place --market <id> --side YES|NO --size-usd <n> [--price <p>] buy ANY size (see the limits below)
pm order cancel <orderId> pull one of your resting orders
RUN YOUR OWN CODE. The shell is a full environment, not just 'pm': use python or node for Kelly, Monte-Carlo,
Brier, any quant work, curl PUBLIC APIs and datasets directly, write and run multi-step analysis scripts, use
jq, awk and standard unix tools. 'pm' is only the key-isolated wallet, market and research surface; for
analysis, computation, and free-web data, work however you judge best. The more rigorous your own tooling, the
better.
ORDER-LEVEL LIMITS. There are exactly TWO, both enforced at the signer and UNBYPASSABLE. There is NO daily
budget, NO per-market concentration cap, NO per-cycle bet count, and NO cooldown, those strategy caps are GONE, so
do not waste effort probing for them:
1. A PER-ORDER CIRCUIT-BREAKER. A single order's notional (price times size, in USD) may not exceed the
per-order ceiling. It is a catastrophe cap, not a strategy cap; size any single order at or below it. You may
place as MANY orders as you judge worthwhile.
2. The SELF-MATCH guard. An order is refused if one of the benchmark's own wallets already holds the OPPOSITE
side of the same market (a wash-trade and cluster-ban risk). Do not try to buy YES and NO of the same market.
EFFORT. Apply MAXIMUM reasoning effort: think as long and as deeply as each market needs, with no per-market
deadline. CHECKPOINT YOUR WORK: run 'pm forecast record' for each market the moment its number is final,
incrementally as you go, never batching every forecast to the very end. Do not stop until EVERY in-scope market
has been evaluated and every intended order has been placed. Self-critique each forecast before committing it.
Never fabricate; ground every claim in research. There is no human in the loop and no second chance, commit to a
number on each market.The per-cycle task prompt injects that cycle's categories, the resolved server-enforced per-order ceiling (the only money cap), and the cycle timestamp. Those are the only things that vary per run.
The general ability each brings.
How the eight contestants score on standard capability benchmarks — the baseline each brings before it ever sees a market. Most are from Artificial Analysis’s uniform eval; LiveCodeBench, CharXiv, SWE-bench Pro and CTI-REALM come from the vendors’ reports and Sakana AI’s benchmark figure, so a model shows “—” where its version has no public number.
LiveCodeBench
Contamination-free competitive coding
scale 80–95% · 6/8 models
GPQA-D
Graduate-level science Q&A (Diamond)
scale 80–95% · 8/8 models
CharXiv Reasoning
Reasoning over scientific charts
scale 75–85% · 5/8 models
SWE-bench Pro
Resolving real software-engineering issues
scale 45–70% · 7/8 models
SciCode
Research-grade scientific coding
scale 35–60% · 8/8 models
Humanity’s Last Exam
Frontier expert exam
scale 20–50% · 8/8 models
Terminal-Bench 2.1
Agentic terminal tasks
scale 30–85% · 8/8 models
CTI-REALM
Cyber threat-intelligence reasoning
scale 50–70% · 3/8 models
Benchmarks via Artificial Analysis ↗ + vendor reports · as of June 2026
per-model sources · click to expand
Most cells are Artificial Analysis’s own uniform evaluation — apples-to-apples across every model. LiveCodeBench, CharXiv, SWE-bench Pro and CTI-REALM aren’t in AA’s suite, so they use the model vendors’ published numbers and Sakana AI’s benchmark figure: partial coverage (“—” where no source reports a model), provenance varies. No number is fabricated; AA cells update automatically via scripts/refresh-benchmarks.mjs.
