Products we run
AI Rating API
Which model should this workload use today? Three public sources — a catalog, a quality index, and a human-preference arena — reconciled every morning into ranked, price-tiered lists, served as a versioned JSON API and a dashboard.
- 3 independent sources, each covering a different axis
- daily rebuilt every morning at 06:00 UTC
- 4 × 10 price tiers, top ten models each
The itch
Every project that calls a model embeds a stale opinion about which model is best.
We kept hardcoding model choices into our own systems, and the choices kept going stale — the model landscape turns over monthly, opinions about it turn over slower. So we made the opinion data: rebuilt daily, reconciled from sources that measure different things, and served behind a stable API our own projects can ask instead of hardcode. Model identifiers in the responses are OpenRouter-compatible, so the answer can be routed with directly.
What we built
One question in, one defensible answer out.
A single endpoint answers the real question: the best text or vision models in my
price range, today. Ask for a category and a tier and you get the top ten, each
with its capability score, prices, context window, and which sources vouched for it. A
health endpoint states exactly how fresh the data is and which sources succeeded; the
contract is versioned under /v1 and published as an OpenAPI spec. The
dashboard is the same data made readable — server-rendered, no framework, fast.
How it works
Three sources, three axes: what exists and costs what, how capable it is, and what humans prefer.
Every morning one run fetches all three sources in parallel — with timeouts, retries, and backoff — then matches, scores, and tiers the field. The result is written as a complete versioned snapshot before a single pointer flips to publish it: readers see the old rankings or the new ones, never a half-written run. If a required source is down, the pointer simply doesn’t flip — the API serves yesterday’s data and says so in its health endpoint.
The hard part: three names for every model
Each source names models its own way, and one of them splits models into effort variants.
Reconciling catalogs is where ranking projects quietly go wrong: match too loosely and a small model inherits a big model’s score; match too strictly and half the field disappears. Our matcher allows a match only when one name is a prefix of the other and the leftover is provably metadata — a date, a version tag, “preview” — or a known effort marker like “thinking”. When a source lists several effort variants of one model, the strongest is kept and labelled as such.
Decisions worth noting
- Price is a tier, not a score. The tiers already segment by cost; scoring price too would double-count cheapness and let a weak-but-cheap model outrank a strong one. The score measures capability alone.
- A fixed scale, so scores mean something over time. Scores are scaled against a fixed reference band rather than each day’s field — a 71 this month is comparable to a 71 last month.
- Arena ratings break ties; they never rank. Every frontier model saturates the top of the preference scale, so it cannot separate leaders — it only orders near-equals.
- The expensive tier is gated by recency. Models that are costly merely because they are old do not belong beside current flagships, so the top tier requires a recent release date.
- No provider allowlist. A new provider appears in the rankings the first morning it has cross-source data — nothing to maintain, nothing to forget.
Built with
- Cloudflare Workers
- KV
- Hono