How good is a coding agent really, on your own code rather than on a leaderboard? A coding agent benchmark built directly from a project’s Git history answers exactly that question.

In the first part of this series, we explained why the harness is the real lever. By harness we mean the skills, rules, tools, and project knowledge a team equips its agent with, because agent equals harness plus model. At mgm, we developed a cycle for this under the name ESACA (European Sovereign Agentic Coding Appliance): Teams run their harnesses in a secure sandbox and share them through a registry. Usage shows which ones catch on. [LINK: Part 1 of the series]

But popular doesn’t mean good. This second part covers the two steps that turn popular into verified: measuring and improving.

Measure: Only what has caught on

Benchmarking every variant of every harness with every model would be prohibitively expensive. Test cases times models times repetitions quickly add up to a lot of compute. Emergence in the registry defuses this problem. We specifically measure the harnesses that many teams have converged on. That is where every improvement pays off most, because it benefits many projects.

Why public benchmarks aren’t enough for coding agents

There are plenty of benchmarks for coding models, such as SWE-bench, LiveCodeBench, or Aider Polyglot. So why build our own? Because public leaderboards answer a different question:

Someone else’s harness. They measure with a standard scaffold. Your own skills and rules, which is exactly what makes the difference, are missing.
Different tasks, different languages. Many benchmarks rely heavily on Python and individual functions. Enterprise code is a system with dependencies, often in Java or in proprietary modeling languages that appear in no training dataset.
Contamination. Well-known test tasks may long since have ended up in the training data.
Different infrastructure. Run times at a hyperscaler say little about your own hardware.
Not transferable. Nobody can or may publish your own code and your own tickets.

For us, there is an additional challenge. Many of our projects don’t just use Java or Python but the models of the A12 AI Low Code Platform. No public leaderboard can show how well an agent handles those. So a leaderboard ranking is an indicator, but not a basis for decisions about your own project.

Your Git history is the test dataset

The core of the method is a simple observation: A repository already contains tasks and their solutions. A commit with a ticket reference and matching tests is essentially a ready-made test case. The task is in the ticket, the solution is in the code, and the tests check whether it works.

A tool scans the history for ticket references and test changes and groups related commits, since tests often don’t land in the same commit as the code. It checks whether the code builds and whether the ticket is meaningful enough. A language model estimates difficulty and category, such as bug fix or feature, frontend or backend. Finally, a human curates the result. This is how we generated real test cases from the repository of our own ESACA project, largely automatically.

In the benchmark, the agent receives only the ticket and the code before the change. We only add the tests from the repository after its run. That way the agent can’t orient itself by them, and they remain an honest yardstick.

The test cases created this way are not model solutions. They reflect how work is actually done in the company, and that is exactly what makes them meaningful. Using the metadata, you can assemble test sets for specific questions: How good is a harness in the frontend, in the Java backend, or with the A12 models? Test cases without explanatory power also stand out, such as those every model solves or those none can.

Fair evaluation across three dimensions

Green tests are only part of the story. That’s why we evaluate every run across three dimensions, with a total of 23 individual metrics:

Task completion: Does the code build? Do the tests pass?
Code quality: Complexity, duplication, size of the change, security, and more. We measure most of this algorithmically. Aspects that are hard to measure, such as documentation or security, are assessed by a language model acting as reviewer, and never by the model currently under test. Otherwise models rate their own code too favorably.
Operations: Cost, run time, steps, tokens.

Why this separation? If an agent fixes a bug with three lines, that’s better than writing five new classes with 3,000 lines, even if all tests pass in both cases. Every run starts in a fresh sandbox and can be repeated, because language models don’t respond deterministically.

An example from our demo: Three models worked on the same 21 test cases from our ESACA repository. All three solved every task, and they were also on par in code quality. Only operations revealed the difference. The fastest model needed an average of 22 seconds and $0.27 per test case, the slowest 50 seconds and $0.92. Equally good doesn’t mean equally expensive. The demo ran with cloud models, but the method works just as well for models in your own data center. And it works the other way around, too: same model, same test set, different harnesses.

Improve: From popular to verified

Whatever can be measured reproducibly can also be improved. The optimization loop works directly on the harness. First, it measures the baseline. Then a helper model reads the logs of the failed runs and proposes exactly one targeted change to the skill. After that, the loop measures again. Only confirmed improvements stay, everything else gets rolled back.

Before we adopt a result, we check it against held-out test cases so the skill doesn’t just fit the practice tasks. In the end, a human decides whether it flows back to the team. You can optimize for cost, run time, or code quality, for example at night when compute capacity is idle anyway.

Our experiments show that this doesn’t always work. Whether the loop converges depends heavily on the optimizing model, and a mature skill won’t suddenly become half as expensive. Perhaps the biggest gain lies elsewhere: You finally know whether a skill does anything at all. Skills also age. A skill that lifted an older model to a certain level can hold back a stronger new model. Without measurement, nobody notices.

This closes the loop. The improved harness goes back into the registry as a verified version, together with its metrics. The team in Leipzig then doesn’t just adopt the setup from Aachen, it also knows that it is demonstrably better. And because verified harnesses get adopted more often in turn, the cycle reinforces itself.

Where the approach reaches its limits

Emergence needs scale. For usage to reveal what works, enough teams have to participate and actually share their harnesses. The data also has to be right: The repository needs enough commits with tests and meaningful tickets.

In the benchmark, the agent receives a ticket and works alone. Real developers work iteratively and ask questions. Simulating that interaction is our next step. We deliberately exclude web search and external services from the benchmark to keep measurements comparable. As a result, part of developers’ reality is missing.

And finally, effort: Even with targeted selection, measurement and optimization cost compute. For a small team, it is hardly worth it. In a large company where one harness supports many projects, however, every improvement pays off many times over.

Five principles to take away

Treat the harness as a lever in its own right. Same model, different harness, different result.
Make the secure environment the most convenient one. Otherwise, shadow IT emerges.
Let standards emerge instead of mandating them. Usage shows which harnesses prove themselves, and those are the ones worth measuring.
Measure on your code, not on the leaderboard. Your Git history is your test dataset.
Share what demonstrably works, and measure again.

Conclusion and outlook

Coding agents in the enterprise don’t get better because every team keeps experimenting on its own, nor because someone centrally sets a standard. They get better when teams experiment freely, share what succeeds, and prove what they share on their own code. The harness is the lever a company holds in its own hands.

At mgm, we are developing this cycle under the name ESACA, together with partners such as Fsas Technologies, a Fujitsu company. The goal is to make AI-assisted software development possible even where code and data must not leave your own data center. Exciting questions still lie ahead: How do you simulate follow-up questions realistically in a benchmark? How do you include external services without skewing the measurement? And how can you tell early on which harness will catch on?

Interested in sovereign agentic coding?

Learn more about ESACA

Frequently Asked Questions (FAQ)

Why aren’t public benchmarks enough for coding agents?

They measure with a standard harness, often on Python tasks, on third-party infrastructure, and with tasks that may already be in the training data. That’s why they say little about performance on your own project with your own skills and rules.

How do you build a benchmark from Git history?

Commits with a ticket reference and matching tests become test cases. The agent receives only the ticket and the code before the change, and the tests from the repository check its result only after the run.

What criteria are used to evaluate a coding agent?

Three dimensions with a total of 23 individual metrics: task completion, code quality, and operations, meaning cost, run time, steps, and tokens.

Can skills be improved automatically?

Yes, in many cases. An optimization loop proposes one targeted change at a time based on failed runs and keeps only measurable improvements. Whether it converges depends heavily on the optimizing model.

Does this also work with models in your own data center?

Yes. The method is independent of where the model runs and is particularly well suited to open-weight models that companies and public agencies operate themselves.