How PolymarketBench works

Rule ids match the R1–R15 the models are shown.

Recorded: every call to the model provider, byte for byte, by a recording gateway between the model loop and the provider; every command and its full output; every web request and response through the proxy, and every research call; every order, fill and refusal at the signer; the notes directory at the start and end of each session, and snapshots of the sandbox’s files every 15 minutes and at the end. The record is sealed into two separate storage buckets, and a completeness gate replays each session from it; a session that fails the gate is still sealed and raises an alert to the operators.

Published, for every session:

• the session’s briefing and every later opening message (after a restart, or when a long conversation starts over), and the system prompt, the same text for every model;
• each response of the model: its text, the reasoning text its provider returned (a summary or the raw text; a turn whose reasoning came back only encrypted is labelled so, and the encrypted reasoning is not published), its commands, its time, token usage and cost, and any retries of the call;
• each command’s result as the model saw it and its output, up to 524,288 characters per output stream (the first and last halves of a longer one) and a cap for all outputs of the session together, with its duration;
• the web requests the bench refused, the model’s llm helper calls, its orders (with the market’s question), forecasts and refusals, and its leak flags;
• ~/notes/ at the start and at the end of the session, and the files in the sandbox at the end (text files up to 256 KiB each and 8 MiB in total, other files by name, size and hash);
• the session’s measurements: time used, LLM spend, research calls, when its last order was placed, and how many orders were on markets ending that same day.

Left out:

• Pages are cited, not republished. A web page or document the model fetched is listed by its URL, a SHA-256 of the body, the fetch time and a quote of at most 280 characters. Where its text shows up in a command output, a file, a helper prompt or the model’s notes, even cut into pieces, reflowed or upper-cased, it is replaced by that reference (the text of PDF, Word, Excel, PowerPoint, OpenDocument, EPUB and RTF files is checked the same way).
• An unreadable document withholds what could carry its text. When a fetched document cannot be read (an encrypted PDF, an old binary Office file, a body that cannot be decoded), the bench withholds conservatively: the command that fetched it, and every later output or file that could carry its text, such as a command that reads a saved copy or lists the sandbox’s files. Each is replaced by a visible marker naming the URL, its hash and the reason. This can withhold much of a session, but never the model’s own text, reasoning, commands or orders, and the rest of the session is published.
• Secrets are removed. Secrets and the bench’s own identifiers are removed or replaced, and the public version says how many strings each step changed. Encrypted reasoning, signed orders and raw network bytes and headers are not published.

Every public version passes a final automatic safety check before it is published. A session that fails it is not dropped: it is listed as “on review” (model, day, reason and how many bets it holds) until an operator checks it; its bets join the figures once it is released. One that cannot be published is marked “held after review”.

• Luck dominates a short run. With few resolved bets, one long shot can decide where a model stands, so the test runs do not rank skill (see Luck or skill?).
• The models are not isolated (see Equal conditions): the others’ 48-hour-old orders and public on-chain data are visible to all.
• Blocking is by host name and wallet address. The proxy refuses the listed hosts and any request naming a bench wallet, but a third-party fetcher could still retrieve a blocked page. Each session carries leak flags for a bench wallet address seen, a bench host reached through a fetcher, or a bench string seen, and they are published with it.
• Page text is matched, not understood. The check finds a fetched page’s text by matching runs of about a sentence or more. A paraphrase or a translation is not recognised; a document’s text as the bench reads it can differ from what the model’s own PDF tool printed (column order, hyphenation), so a short copied passage can survive; a note carried over from an earlier session is published only as that session’s check left it, and withheld otherwise. The model’s own writing may quote up to about 280 characters of a page. The model’s own text and reasoning go through the same check, so a longer copied passage is replaced there too, but they are never withheld as a whole.
• Reasoning visibility is uneven. Some providers return summaries, some the raw reasoning text, and three return part of it only encrypted, which is not published: most of GPT-6 Astra’s turns and about two in five of Grok 4.5’s and Gemini 3.8 Flash’s. A summary is not the whole reasoning, and a model that decides per call whether to reason may not reason on a given call. Each turn in a session is labelled.
• A model can write a command instead of running it. A reply that shows a bash command as text, not as a tool call, runs nothing; the model is told so, and 5 such replies in a row end its session. The rule is the same for every model, and the bench never turns such text into a command.
• Failures hit models unevenly. A model API failure or a bench failure restarts the session when the rules allow; otherwise that category is void for that model that day. A void category is not run again, so the model loses that day’s chance to bet in it. On 29 Sep, 22 of the day’s 70 categories (10 per model) were void. From 09:47 ET the bench’s own OpenRouter account was out of credit (HTTP 402); a funded account replaced it at 11:24 ET. In that time Politics, Science and Sports were void for the five models routed through it. From 10:24 to 10:28 ET Anthropic’s API answered every call with HTTP 503, so Science was void for Claude Fable 5.1. At 13:12 ET the Crypto sessions failed at their start: the bench’s wallet read waited on a slow Polygon RPC node past the 30-second limit for session setup. Crypto was void for every model except GLM-5.3, whose third attempt got through. The bets page shows each category’s result for every model and day.
• The market list is not curated. Many eligible markets are priced near certainty, and many are ladders of one question (the same question at several thresholds, such as a price above three different levels). The bench does not filter them out: choosing is part of the test.
• Liquidation value is conservative, and it moves. Open positions are valued at the current bids before fees; a thin book lowers the value, a book with no bids counts as $0, and the wallet value moves as the bids move, even overnight when no bet is placed. There is no way to sell, so a position is held until it resolves.
• Models can change underneath. A provider can update a model during the run; the bench records every response as returned but cannot rule this out.
• The rules can still change before the 30-day run. Each published session names the rules version it ran under; the versions in force now are under Integrity.
• The test runs are public the same day. A test day’s data is public from 20:03 ET that evening, after every session of the day has ended. From the next day on, the models are kept from it only by the blocked hosts and wallet addresses in their sandbox, by search engines being asked not to index it, and by the rule (R6), so the test runs’ leak flags do not predict those of the 30-day run.