Real-SWE tests AI models on private enterprise codebases
A new benchmark draws its tasks from licensed private production codebases and scores model-and-harness combinations rather than models alone.
Specific has released Real-SWE, a benchmark that evaluates frontier AI models on private, real-world enterprise codebases. Every task comes from a private production codebase licensed from an actual company, drawn from problems that company's engineers work on.
The team argues that expert-written or synthetic tasks may be well designed but are not the verbatim work engineers do inside real companies. Real-SWE tasks differ along two axes: the underlying coding artifact and how specific the instruction is.
How the tasks are built
Prompts describe the change that is needed and leave agents to discover implementation details in the codebase and surrounding tooling. Any behaviour the verifier requires must be stated or reasonably discoverable. Specific says this makes its prompts slightly underspecified, roughly on par with DeepSWE and Terminal Bench.
One sample task involves fixing invoice billing so each business is charged the correct tax. Some businesses maintain their own rate, some price each invoice against the buyer's destination through a tax authority provider, some collect nothing, and a customer with an exemption on file is never taxed. A single change of this kind spans several parts of the application.
Setup and early numbers
Each agent runs in an isolated sandbox. Tasks are in Harbor format, and verifiers are injected at grading time, either inspired by or lifted directly from the existing test suites in the codebase. Specific uses native harnesses, evaluating model-and-harness combinations rather than models in isolation.
One reported figure: 71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts. Failures are grouped by observed submission behaviour using a shared taxonomy following DeepSWE, with percentages calculated out of each model's failed runs rather than all runs.
Access to the sample task set is available on request.
