TechSafety
Grok 4.5 first-day benchmarks put it fourth on Artificial Analysis with a doubled hallucination rate
Snorkel AI · 2026-07-10 · Source ↗
Grok 4.5 first-day benchmarks put it fourth on Artificial Analysis with a doubled hallucination rate
Snorkel AI · 2026-07-10 · Source ↗
Two days after SpaceXAI released Grok 4.5, Snorkel AI published a testing writeup that puts the model's first-day production behavior in the sharpest relief the benchmark community has produced this cycle. Snorkel ran the model against a 2,000-task sample drawn from GDPval+ Edition 1, an expert-authored dataset of workplace reasoning tasks across 44 knowledge occupations in 9 economic sectors that Snorkel built as a private extension of OpenAI's public GDPval work. The results are the largest independent professional-work test the model has faced.
"Grok 4.5 outperforms other frontier models on real-world professional tasks. On the GDPval+ benchmark of expert-created workplace reasoning tasks across the economy: Grok 4.5 achieves a 29 percent mean pass rate."— Snorkel AI, GDPval+ testing writeup
The 29 percent lead
Grok 4.5 pulled 29 percent mean pass on the 2,000 sampled tasks. GPT-5.5 hit 22 percent. Claude Opus 4.8 hit 21 percent. The lead concentrated in domains that demand professional judgment rather than raw code or math. Legal work: 40 percent for Grok, 27 to 28 percent for the field. Education: 58 percent for Grok, 35 to 42 percent for the field. Healthcare: 35 percent for Grok, 23 to 25 percent for the field. QA analysis: 37 percent for Grok, 19 to 27 percent for the field. Those numbers matter because GDPval+ is not a synthetic reasoning suite. The tasks are things like drafting a legal memo, marking up a lesson plan, or writing a nursing hand-off note, evaluated by domain experts who authored the original prompts and produced the reference deliverables.
The hallucination doubling
The same test surface produced the story's darker number. Grok 4.5's hallucination rate on the GDPval+ tasks was 54 percent, up from 25 percent on Grok 4.3. The doubling is not a random-error phenomenon. Snorkel's writeup attributes it to increased model capacity: the model knows more, so when it is wrong it is more confidently wrong, and it fabricates supporting detail that reads plausible to a non-expert reviewer.
"The model knows more, but it is also more confident when it is wrong."— Snorkel AI, GDPval+ testing writeup on Grok 4.5
Artificial Analysis and the SWE Marathon claim
Artificial Analysis ranked Grok 4.5 fourth on its Intelligence Index at a score of 54. Claude Fable 5 held first. GPT-5.5 took second. Claude Opus 4.8 took third. Elon Musk claimed on X that Grok 4.5 had taken the top position on the SWE Marathon long-horizon coding benchmark. Third-party runs published Friday morning confirmed the claim on the specific SWE Marathon variant xAI cited but not on the widely used SWE-bench Pro leaderboard, where the model sits at fourth. The gap between the two benchmarks became a mini-controversy on Hacker News through Friday afternoon.
"Grok 4.5 has landed at number 1 on multiple benchmarks, performing better than expected."— Elon Musk, CEO, SpaceXAI
Token efficiency and the CursorBench contamination flag
xAI's own token-efficiency disclosure put Grok 4.5 at 15,954 output tokens on the SWE-bench Pro tasks against 67,020 for Opus 4.8, a 4.2x efficiency gap that translates to roughly 17x lower cost per task at published API rates. The comparison held up in independent runs. The complicating factor: Awesome Agents flagged an unresolved training-data overlap between Grok 4.5's Cursor IDE session training corpus and the tasks in CursorBench, one of the two benchmarks xAI cited to advertise the model's coding performance. Until that overlap is characterized, the CursorBench numbers can't be treated as a generalization signal.
The composite picture that emerged from Friday: Grok 4.5 leads on real professional work by a real margin, hallucinates more than any current frontier model on the same tasks, requires output validation for any accuracy-dependent workflow, and prices at $2 input and $6 output per million tokens against Opus 4.8's $15 and $75. For volume agentic pipelines where the operator can afford to validate outputs, the price-performance math points at Grok. For accuracy-critical single-shot work with no downstream validator, it does not.