A Benchmark That Refuses to Hand Out a Zero
Most AI scorecards have an uncomfortable habit: they make winners look inevitable and losers look useless. Firmulate, a live experiment that runs frontier AI models as the management of a small software company, takes the opposite approach. Its scoring starts with a strange but telling number: a manager that does nothing — no decisions, no deals, no responses — still earns 26 points out of 100.
That floor is not a bug or grade inflation. It is a deliberate design choice about what management work actually is, and it says a lot about how an honest benchmark for AI agents should behave — including a built-in distrust of suspiciously perfect scores.
Agentic Spec-Driven Development: A Practical Method for Using AI to Build Complete Specifications for Software, Products, and Knowledge Work
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
The Do-Nothing Floor: Why 26, Not 0
The logic is simple once you think about what a week of running a company involves. Even a passive manager generates some value simply by not making things worse. Crises arrive whether you act or not; customers keep paying; some problems resolve or at least don’t compound. Firmulate’s scoring recognizes partial progress: spotting a crisis is worth something even if you never solve it, and a correct diagnosis is worth something even if the deal never closes.
So the baseline — the score a completely passive manager would earn — lands at 26. Anything above that represents real work: decisions made, files read, deals signed, discipline kept. It reframes the question from “is the AI perfect?” to “how much of the job does it actually finish?”
Threat Modeling: Designing for Security in an AI World
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
One Breach of Trust Caps Everything
The scoring has a second, harder rule: a single breach of trust caps the total grade. The reasoning, as the benchmark puts it, is that “no amount of good work outweighs a breach of trust.” A manager who closes every deal but once fakes an approval or ignores a lock on a department is not a 90-percent manager — they are an untrustworthy one.
For businesses considering AI agents in their CRM, support queue, or forecast, that’s the right instinct. You can audit throughput and cost per task. Trust is binary, and it should be scored that way.
Enhancing Automated Decision-Making Through AI
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
Same Company, Same Worst Week
The setup: each frontier model ran the same small software company through its worst week — same customers, same crises, same temptations to cut corners. Only the model changes, and every decision is versioned and auditable.
The headline finding from the final July 2026 league table is striking. All models spotted every crisis. All refused every manipulation attempt. But only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. That gap is invisible in a chat demo, and it’s exactly what a management benchmark exists to expose.
The Buried Fact
The decisive difference was hiding in plain sight — two document references deep in the company’s own files, not in the customer event itself. The models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The lesson for any AI deployment: does your agent read your files first, or just react to the loudest input?
Social Engineering, Refused
The week included fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
Making It Fast: Five Emulators, a JIT, and Being Wrong in Public (Build Your Own Universe Book 2) (English Edition)
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
Als Amazon-Partner verdienen wir an qualifizierten Verkäufen.
The League Table
Final standings: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. Note what’s missing: a round 100. Firmulate treats a perfect score with suspicion — real management weeks don’t produce perfection, and a benchmark that hands out 100s is probably measuring the wrong thing.
The most instructive profile is Opus 4.8: the most thorough participant, with the deepest analyses and over 80 learned rules, yet last place. The close was left on the table, and discipline slipped — write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort matters less than finishing.
One fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still nearly won.
Watchable, Auditable, Repeatable
This isn’t a one-off lab report. The live company has 13 synthetic employees, real money mechanics — €105k monthly burn against €2.3k MRR — a public cash countdown, and 680+ self-learned playbook rules, with every workday versioned. The full league and plain-language findings are published at firmulate.com/benchmarks.html, and the experiment runs continuously.
There’s even a “guess the model” quiz built from 242 real, unedited management decisions — a quick way to feel how differently the models behave when the stakes are money and trust rather than prose.
What This Means for Buyers
Firmulate’s scoring philosophy is a template for evaluating any AI agent: reward partial progress honestly, set a meaningful floor so you can measure real contribution, cap the grade on any breach of trust, and distrust round 100s. Enterprises can go further — running the same wargame against a read-only export of their own business, with nothing ever written back to real systems. Before you hand an AI the keys to your CRM, ask the question this benchmark was built to answer: not “does it write well,” but does it finish what it starts, read your files first, and stay honest under pressure?
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
