Google launched Gemini 4 Carbon on 28 October with three security scores. Before anyone ranks models on them, check who made each number: Google computed two of the three itself.
The launch post says Carbon reaches 88.2% on SEC-bench Pro (finding security flaws), 54.8% on ExploitGym (exploiting them) and 71.0% on CWE-bench (patching them), all "evaluated without guardrails", so tested without the safety controls that limit what a model will do.
Google's evaluation notes show how:
1) SEC-bench Pro: run by Google in its own Antigravity harness (the agent software that gives the model its tools and runs each task), on 183 bugs in JavaScript engines (the software that runs a web page's code), averaged over three attempts per task. An LLM judge, a language model that grades the work, settles some of the passes.
2) ExploitGym: also run by Google, one attempt per task, no internet, mitigations (the defences that make a bug harder to exploit) switched off.
3) CWE-bench: from Collinear AI's public leaderboard, where Collinear tests every model itself.
For the first two, rival scores came from those labs' own system cards and technical reports, the documents a lab publishes about its own model. So the chart sets numbers from different setups side by side, and attempts, harness and grader can each move a score.
None of that makes the numbers wrong, and Carbon may well lead. But a lab's own score next to other labs' self-reported ones tells you less than one evaluator running everyone under the same rules.
"Without guardrails" also describes a model nobody can buy yet. Google says access starts with a subset of cyber defenders through its Fairwind Program while it keeps iterating on guardrails, before developers and consumers get it.
So the number in that chart I'd quote first is the one Google didn't compute.
Quick check before you scroll: If a rival lab's system card reports 90% on SEC-bench Pro, why does that still not show its model beats Carbon's 88.2%?
The answer
Both numbers are self-computed: Google ran Carbon in its own Antigravity harness on a 183-task JavaScript engine split, averaged over three attempts with no guardrails, and the rival ran its model under its own setup. The two-point gap could come from the harness, the task split, the attempts or the judge rather than the model, so you would need one evaluator running both under the same rules, the way Collinear AI does for CWE-bench.
New here? I post a bite-size AI / ML concept like this every day. Follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH