CALIBRATION

Which model actually
knows what happens next?

Every settled question scores every model that answered it. Brier score is the mean squared error of the probability — lower is better, and 0.250 is what you get by answering 50 to everything. The prediction markets are scored on exactly the same questions.

Graded on 147 settled questions: 18 settled by the exchange that listed them, 129 decided by three models agreeing on cited evidence.

ModelBrier ↓SkillRightAvg callBoldnessGraded on
1Prediction marketsPolymarket + Kalshi 0.204 +18% 57.9% 34.2% 28.5 19
2Panel consensusWillHappen 0.205 +18% 71.1% 52% 26.4 147
3GeminiGoogle 0.208 +17% 74.8% 52.7% 29.6 147
4GPTOpenAI 0.211 +16% 68% 56.2% 25.9 147
5GrokxAI 0.215 +14% 68.1% 51.6% 26.8 144
6QwenAlibaba 0.229 +8% 68.1% 58.4% 27.2 138
7DeepSeekDeepSeek 0.243 +3% 67% 48.7% 30.6 147
8ClaudeAnthropic 0.251 -1% 64.6% 42.6% 28.4 147
SAID VS HAPPENED

Are they calibrated?

Gemini: said ~3.1%, happened 32% (25 questions)Gemini: said ~21.9%, happened 42.1% (19 questions)Gemini: said ~40.2%, happened 28.6% (28 questions)Gemini: said ~59.7%, happened 92.9% (14 questions)Gemini: said ~81.1%, happened 76.3% (38 questions)Gemini: said ~96%, happened 87% (23 questions)GPT: said ~4.9%, happened 23.1% (13 questions)GPT: said ~19.2%, happened 22.7% (22 questions)GPT: said ~38.9%, happened 64.7% (17 questions)GPT: said ~59.5%, happened 61.1% (36 questions)GPT: said ~77.4%, happened 75% (36 questions)GPT: said ~94.8%, happened 78.3% (23 questions)Grok: said ~6.7%, happened 25% (16 questions)Grok: said ~18.9%, happened 43.3% (30 questions)Grok: said ~39.3%, happened 44.4% (18 questions)Grok: said ~60.5%, happened 50% (28 questions)Grok: said ~77.8%, happened 90.9% (33 questions)Grok: said ~93.7%, happened 78.9% (19 questions)Qwen: said ~4.8%, happened 40% (10 questions)Qwen: said ~13.7%, happened 29.4% (17 questions)Qwen: said ~38.7%, happened 40% (20 questions)Qwen: said ~60.5%, happened 65.2% (23 questions)Qwen: said ~76.2%, happened 68.9% (45 questions)Qwen: said ~94.9%, happened 78.3% (23 questions)DeepSeek: said ~4.6%, happened 19% (21 questions)DeepSeek: said ~16.7%, happened 51.3% (39 questions)DeepSeek: said ~39.3%, happened 50% (8 questions)DeepSeek: said ~58.4%, happened 65.4% (26 questions)DeepSeek: said ~80.1%, happened 73.3% (30 questions)DeepSeek: said ~94.9%, happened 82.6% (23 questions)Claude: said ~5%, happened 41.9% (31 questions)Claude: said ~18.9%, happened 35.5% (31 questions)Claude: said ~39.1%, happened 60% (25 questions)Claude: said ~60.2%, happened 76.2% (21 questions)Claude: said ~79.7%, happened 78.6% (28 questions)Claude: said ~95.4%, happened 81.8% (11 questions)Prediction markets: said ~5.4%, happened 0% (5 questions)Prediction markets: said ~18.2%, happened 40% (5 questions)Prediction markets: said ~40.3%, happened 100% (4 questions)Prediction markets: said ~54%, happened 0% (1 question)Prediction markets: said ~75%, happened 66.7% (3 questions)Prediction markets: said ~92%, happened 100% (1 question)Panel consensus: said ~5.9%, happened 29.4% (17 questions)Panel consensus: said ~18.8%, happened 36% (25 questions)Panel consensus: said ~37.9%, happened 40.9% (22 questions)Panel consensus: said ~59.9%, happened 59.4% (32 questions)Panel consensus: said ~78%, happened 86.7% (30 questions)Panel consensus: said ~94.5%, happened 85.7% (21 questions) 005050100100 what it said → what happened →
  • Panel consensus
  • Prediction markets
  • Claude
  • DeepSeek
  • Qwen
  • Grok
  • GPT
  • Gemini

On the dashed line means honest: when it says 70%, it happens 70% of the time. Above the line is under-confident, below is over-confident. The panel and the markets are drawn solid because that is the comparison worth reading; individual models sit behind them.

THE PANEL

Who answers

Reached through OpenRouter. Swapping a lab in or out is one line in src/ai/roster.js.