JetBrains Built Its Own Benchmark for Claude Fable 5 — and Opus 4.8 Lost by 16 Points

On JetBrains’ private Python evaluation set, Claude Fable 5 posted a 44.3% pass rate against 28.2% for Anthropic’s own Opus 4.8 — a roughly 16-point gap measured on a benchmark JetBrains built itself, not a public leaderboard. Anthropic published the finding August 13 in an interview with JetBrains CTO Vladislav Tankov, whose developer tools serve 12.5 million active users.

Tankov’s team doesn’t rely on public benchmark scores. It tests models against large private-repository evaluation sets, including JetBrains’ own monorepo, and maintains standing leaderboards tracking quality, cost-per-task, and speed. In head-to-head runs, Claude Fable 5 solved 18 Python tasks that Opus 4.8 missed while losing only 2, and needed roughly 22% fewer steps to reach a solution. Tankov’s operating split: Opus as the reliable workhorse, Fable 5 for complex reasoning and long-running agentic work — implementing sophisticated components, rewriting applications across frameworks. On safety, JetBrains told Anthropic it prioritizes infrastructure safeguards over tuning the model itself, and accepts limited data retention specifically to investigate serious classification failures — a deliberate tradeoff the company frames as reasonable for running frontier AI in production developer tools.

JetBrains isn’t alone in distrusting public scores. In a separate case Anthropic published a month earlier, Cursor built its own benchmark after engineer Nate Schmidt noticed public benchmark results had stopped correlating with how developers actually rated a model’s output. Claude Fable 5 scored 72.9% at Max effort on that benchmark — a new high for it — and the practical effect for Schmidt was that he stopped needing to re-supply context or audit every result, handing off gnarlier, more ambiguous refactors instead.

Two separate engineering-tools companies, evaluating independently, arrived at the same conclusion by building their own tests rather than citing Anthropic’s marketing numbers. For consulting engagements advising on coding-AI vendor selection, that’s the more durable signal than any single benchmark score: the companies with the most at stake aren’t trusting the vendor’s leaderboard, and neither should a client comparing models for production use.