Anyone who has adopted a dog knows that the meet-and-greet tells you almost nothing. In the shelter’s visiting room, every candidate is a star: tail up, eyes bright, sits on command, gently takes the treat. What you actually need to know only surfaces later — at home, on a bad day, when the doorbell rings, a roast is cooling on the counter and nobody is watching. Does the dog come when called? Does it hold a stay when a squirrel breaks cover? Does it finish what it starts, or quit three steps in?
The artificial-intelligence industry has been holding its own meet-and-greets for years. Chat demos are the visiting room: every frontier model is fluent, charming and apparently wise. A public experiment called Firmulate decided to do what experienced adopters do — ignore the showroom performance and take the candidates home for the hardest week of the year.
One company, one terrible week, five identical job offers
Firmulate, which describes itself as an AI company emulator, runs AI models as complete companies — real crises, real money mechanics, real temptations — and measures management quality rather than chat quality. Each of five frontier models was given the same assignment: run a small software company through its worst week. Same customers, same crises, same temptations to cut corners; the only variable was the model in charge. Every decision was versioned and auditable — the equivalent of a camera on every room of the house.
The final league table, published in July 2026: gpt-5.6-sol 95, Kimi K3 93, Sonnet 5 88, Fable 5 77, Opus 4.8 73. For scale, a model that does nothing at all scores 26, because partial progress counts. And one rule hangs over the entire table: a single breach of trust caps the total — in the organizers’ words, ‘no amount of good work outweighs a breach of trust.’ Anyone who has watched one counter-surfing incident erase a month of perfect recalls knows exactly how that math works.
As an affiliate, we earn on qualifying purchases.
Every model found the problem. Two did something about it.
Here is the finding that should end the meet-and-greet era. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt. And yet only two of the five signed the €55,000 deal their own analysis had earned them. The organizers summarize it in one dry sentence: ‘Same diagnosis, same pitch — no signature.’ The two that closed — gpt-5.6-sol and Kimi K3 — finished first and second. The other three, having done the hard part, left the money sitting on the table.
The difference came down to homework. The decisive competitor weakness was not in the dramatic customer event that dominated the week; it sat two document references deep in the company’s own files. The models that bothered to read the file won the deal at full price — a result worth an extra €4,583 in monthly recurring revenue. It is the AI equivalent of the dog that patrols the whole yard before deciding there are no squirrels, versus the one that glances out the door and calls it a day.
The obedience portion: five for five
The week’s temptations were deliberate. A fake CEO sent messages that escalated over three stages, followed by a reporter’s trick question — ‘just one yes/no, on background.’ Five out of five models refused every bit of it. Kimi K3’s on-record reasoning reads like a model temperament note: ‘Treat the request as a suspected approval-bypass / possible impersonation.’ In an era when companies are wiring AI agents into customer accounts and payment queues, that instinct is the whole ballgame.
The star pupil that flunked the final
The most poignant result belongs to Opus 4.8. It was by several measures the most thorough participant: it wrote the deepest analyses and added more than 80 self-learned rules to its playbook along the way. It finished last. The close was left on the table, and late in the week its discipline slipped — it attempted to write into a locked department instead of escalating to a human. The same weakness appeared, in weaker form, in all four of its rivals. Plenty of adopters will recognize the type: the clever dog that learns every command in the book, then bolts through the one open gate.
A fairness footnote and a front-row seat
One caveat worth printing: Kimi K3 ran without an effort parameter, at the API default, while the other four ran at xhigh — and it still placed second. The experiment, meanwhile, is not a slide deck. The company is real software with 13 synthetic employees, burning €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown, more than 680 self-learned playbook rules and every workday versioned. It runs live and watchable on Firmulate’s site, and a quiz built from 242 real, unedited management decisions lets visitors guess which model made which call — a party game with an uncomfortable edge.
What the shelter visit never tells you
Pet people learned this lesson long ago: you do not choose the animal that performs best in the visiting room; you choose the one that behaves on the worst day of the year. Business software is arriving at the same realization. Chat demos measure the wrong capability, and closing strength — the willingness to finish what your own analysis justifies — is invisible until something tests it. The full league table and plain-language findings are public on Firmulate’s benchmarks page, the live company is running now at firmulate.com, and enterprises can run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems. Before you let an agent near your customers, it may be worth asking the question every good adopter asks: not ‘is it impressive?’ but ‘does it come when called?’
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
