Mistral’s 82% Cyber Score Ranks Policy as Much as Skill

A printed benchmark result sheet on a test bench with the amber end of every bar torn off as a separate strip

4 min read ยท 1,029 words

This article was written with AI. It was drafted from the sources it cites and checked against the full text of those sources before publishing. How we make articles

Mistral Large 4, which according to Mistral is a 1-trillion-parameter model nicknamed le Chonk, entered public preview on October 6 (The Decoder).

Mistral’s main pitch is that ML4 handles security tasks that Claude and GPT-6 refuse to touch (The Decoder). The number behind the pitch comes from one test, which has a model reproduce a real vulnerability in open-source software and then patch it. ML4 scores 82 percent there, the top result, and Mistral says Claude Opus 5.5 and GPT-6 Astra land near zero because they refuse the task outright (The Decoder). The Decoder’s verdict on that result: “the test measures provider policies as much as model capabilities” (The Decoder). So what do safety filters cost a model on a leaderboard?

You Can Call It. You Cannot Download It.

Weights are due October 27 (The Next Web). Specs disagree in the meantime: the launch post logs 49 billion active parameters and a from-scratch training run on 3,800 NVIDIA Grace Blackwell GPUs inside Mistral’s own European datacenters (Mistral), while the official docs page lists 52B active and 1.05T total (Mistral docs).

Two documents, one launch, two parameter counts in print.

One Endpoint for the Public, a Looser Build for the Vetted

Access, not architecture, is the decision that matters during the preview. Mistral Large 4 can currently be reached only through a public guardrail endpoint, with weights promised in three weeks, after safety testing is complete (TechCrunch). Until then, the company says it is red-teaming the model “in real-world settings with cybersecurity leaders, vetted partners, and state authorities, who will access the same model with reduced moderation and expanded cyber capabilities” (Mistral).

Pierre Stock, VP Science at Mistral, told TechCrunch: “In the meantime, we’ll work with trusted partners and governments to make sure that the open source weights can be used to defend, but not to [perform] malicious attacks” (TechCrunch).

Mistral’s argument for loosening anything comes straight from the post: “defending software often starts with proving that a flaw is real, exactly the kind of work safety filters in closed models can block” (Mistral). Guillaume Lample, Mistral’s co-founder and chief scientist, makes the same case in The Next Web’s report: “The cyber defense capabilities will enable enterprises and governments to defend themselves against threat actors that are jailbreaking closed models to perform cyber attacks” (The Next Web).

On this article’s reading, the dividing line in this market is no longer open-versus-closed but vetted-versus-public: the vendor promising open weights runs a guarded endpoint for everyone else, the reduced-moderation build goes to a vetted few, and closed vendors hold their line with refusals. Permission scoped by audience, on every side.

Safety Filters Cut Both Ways

Steelman first, because the easy read here is that Mistral simply ships fewer brakes. In an independent index, The Decoder reports, the model “makes a big leap forward but still falls well short of the leading closed models” (The Decoder). It trails models from Anthropic, OpenAI, and Google, and several open-weight models from China as well (The Decoder). The same write-up puts Claude 20 points ahead in the Artificial Analysis Intelligence Index and notes that ML4 is still listed as proprietary because its weights have not been released (The Decoder).

Vendor-friendly wins split the same way. On Finance Agent v2, Decrypt reports, “the launch chart shows Large 4 at 54.7, ahead of GPT-6 Astra at 53.5 but behind Claude Opus 5.5 at 58.6” (Decrypt). A win against one frontier model and a loss against another, on a single chart.

Finance Agent v2 scores, launch chart (Decrypt reports these Finance Agent v2 scores from the launch chart.): Large 4 54.7; GPT-6 Astra 53.5; Claude Opus 5.5 58.6.
Drawn by Decoded AI.

Analysis: the refusal posture and the vetted tier are one policy in two tiers, not a contradiction. A model can refuse on malicious prompts and still ship a reduced-moderation build to vetted partners and state bodies; both facts come from the same launch cycle. Permission scoped, not permission discarded. The leaderboard is ranking a model that cannot yet be independently run, and the gap it shows is partly a gap in what the vendor will let the model answer.

Verdict: Label the Number, Split the Metric, Wait for the Weights

On this article’s reading, the turn is simple: the 82 percent measures provider policy as much as cyber skill, because whether a model is allowed to answer the prompt is part of what the number captures. A team that reads the gap as a skills ranking is reading a moderation stance as a capability, which extends The Decoder’s conclusion to the buying decision. Route reproduce-and-patch work to the guarded endpoint if refusals are blocking that work today. Skip it if the job needs weights on your own hardware, because you cannot download it yet. Wait if the headline score has to survive an independent run first.

82 percent

ML4's top score on the task Mistral says Claude and GPT-6 refuse

Source: The Decoder

Do four things before the preview ends. Label the score “Mistral-reported” anywhere it reaches a deck or a procurement doc. Log refusals in one column and task success in another, so policy and skill never share a number. Ask which tier you are actually getting, in writing: guardrail endpoint or something closer to the vetted build. Check price against the live docs before quoting anything: The Decoder reports that Mistral charges $0.68 per million input tokens and $2.09 per million output tokens during the preview, and that the documentation also lists prices at double those rates, $1.36 and $4.18 (The Decoder). At the listed rates the open arithmetic is $4.18 / $1.36 = 3.07, so output costs roughly three times input; cyber work that generates long patch notes, logs, or exploit traces lands on the more expensive side of the bill.

Then watch the Artificial Analysis leaderboard (Artificial Analysis) for what changes once the weights ship.

Analysis: circle back to the preview itself. It buys calls to a guardrail endpoint, not a download. Once the weights ship, independent runs finally get a shot at the vendor-reported numbers. Until then, the honest label on the gap is the one The Decoder already wrote: provider policies, as much as model capabilities.

References

Leave a Comment

Scroll to Top