About

I built a calibration study of Kalshi's daily NYC high-temperature prediction markets, comparing market-implied probabilities against actual observed weather to test whether the market meaningfully predicts weather or just prices in near-settled outcomes.
The workflow pulls settled contracts from Kalshi's public API, cross-references daily high temperatures from Open-Meteo's historical archive, and samples each market's price from 12–18 hours before close (via Kalshi's candlesticks endpoint) to capture a real forecast rather than the settlement snapshot. It then derives YES/NO outcomes from actual temperature versus contract strike and computes both an overall Brier score and a binned reliability curve.
Result: across ~50 markets that survived filtering, the market scores a Brier of 0.215 against a naive-baseline 0.25 — 14% skill over a coin flip, so it's genuinely informative. Calibration is strong at the extremes (near 0% and near 100% predictions), where most trading volume sits. The mid-range (30–70%) shows noise and slight overconfidence, likely driven by thin volume on genuinely uncertain days — though the small sample size per mid-range bin means those individual points shouldn't be overinterpreted.