---
title: You Do Not Need a Server for Evals
description: 'Most evals do not need LangSmith, Braintrust or MLflow. Datasets, the runner, and the history all fit in your git repo - and your coding agent reviews the results for free.'
date: 2026-07-24
tags:
  - ai
  - evals
  - agents
  - rust
  - developer-experience
---

I run evals for a few projects - [yolop](https://github.com/everruns/yolop) (a coding agent), [bashkit](https://github.com/everruns/bashkit) (a sandboxed bash), [Everruns](https://everruns.com/), and a couple of private ones. Every time this comes up, someone asks which eval platform I use. LangSmith? Braintrust? MLflow Evals?

None. My evals live in the git repo. Datasets, the runner, the whole run history - all committed, right next to the code they measure.

And honestly, for most evals, that is all you need.

## What an Eval Actually Is

Strip away the dashboards and an eval is three things:

- a **dataset** - a list of cases with expected outcomes
- a **runner** - something that feeds each case to a model and scores the result
- **results** - what happened, kept around so you can compare later

A dataset is a file. A runner is a script. Results are files. Git already stores files, versions them, and shares them with your team. So why rent a server for it?

![The eval loop living inside your repo: dataset and results are committed files, the runner is a script, and your agent reviews the results folder](/blog/evals-in-the-repo.svg)

## The Feature List, Point by Point

Eval platforms sell a feature list. Let me go through it with a repo in mind.

1. **Available to the whole team** → same with a repo. Everyone already has it cloned.
2. **Datasets versioned and maintained** → same with a repo. That is literally what git is for. A dataset change is a diff you can review.
3. **Runs the evals** → same with a repo. The runner is a script. It runs on your laptop and it runs in CI.
4. **Compare and review results** → here is the fun part. The results are plain JSON in a folder. Point your coding agent at it and ask. More on this below.
5. **Runs large, long-running evals** → okay, this one is real. Servers win here. I will be honest about it at the end.

Four out of five, the repo just does. The fifth is a genuine trade-off, not a reason to move everything to a server.

## What This Looks Like

Here is one of these studies - bashkit's eval crate. Datasets in `data/`, the study in `src/`, run history in `results/`. And the history is not a figure of speech: that is 78 result files checked into git.

![Terminal: tree of the bashkit-eval crate - data, src, results - then 78 committed result files and the git log line for the results folder](eval-structure.png)

A dataset row is boring on purpose - one JSON object per line:

```json
{
  "id": "smoke_echo",
  "prompt": "Print 'hello world' to stdout.",
  "expectations": [
    { "check": "stdout_contains:hello world" },
    { "check": "exit_code:0" }
  ]
}
```

And a saved run stamps the git commit it ran against, so a number from three weeks ago is still interpretable today:

```json
"environment": {
  "git": { "commit": "1e187f3…", "branch": "…", "dirty": false },
  "mira_version": "0.2.0"
},
"summary": { "scored": 174, "passed": 159, "failed": 15, "total_tokens": 1260246 }
```

No dashboard. Just files you can `git log`.

## The Fun Part: The Agent Reviews Results

This is the point everyone underestimates. Once results are plain JSON in a folder, comparing and reviewing them stops being a product feature. It becomes a prompt.

Even plain `jq` pulls a run's headline numbers straight out of a committed file - pass rate, tokens, cost, tool calls:

![Terminal: jq .summary on a committed swebench meta.json - 13 of 13 passed, total tokens, cost, and tool calls](run-summary.png)

The agent goes further. I open it in the repo and ask things like:

- "Compare the last two runs in `evals/swebench_verified/results` - which tasks regressed?"
- "Chart pass rate by category from the newest `report.json`."
- "The django cases fail more than the rest. Read the transcripts and tell me why."

It reads the folder, does the diff, and gives me the answer - often with a chart. This works _as a charm_ in agents, because the whole substrate is files and JSON, which is exactly what they are good at. No SDK, no query language, no export step.

The dashboards on eval platforms exist to answer these questions. The agent answers them from the raw files, and it answers the weird one-off questions too - the ones no dashboard has a widget for.

## Mira: Evals as Runnable Scripts

You could hand-roll all of this. It is genuinely easy - a loop, a scorer, write JSON to a folder. I did exactly that at first, and the flat `eval-…-2026-02-07.md` files in bashkit's history are the leftovers.

But then you re-invent the matrix, the retries, the resume, the reporting. So I moved everything onto [Mira](https://github.com/everruns/mira) (from the Ukrainian word _міра_, meaning "measure") - a small, code-first eval runner where an eval _is_ a runnable script.

Here is the part I actually care about: **I never write these studies by hand.** I ask the agent. Mira ships an [agent skill](https://github.com/everruns/mira/tree/main/skills/mira) - `npx skills add everruns/mira --global` - so my coding agent already knows the API, the CLI, and the conventions. I just describe what I want:

- _"evaluate yolop on SWE-bench Verified with https://github.com/everruns/mira"_
- _"eval the yolop system prompt - two variants, same 20 tasks"_
- _"generate a Mira dataset from our yolop Braintrust traces"_

and it writes the study, runs it, and hands me the report. So the eval is agent-driven on both ends - the agent authors it, and [the agent reviews the results](#the-fun-part-the-agent-reviews-results).

The study it writes stays small. Here is a complete one - a single file, no `Cargo.toml`, dependencies right in the cargo-script header:

```rust
#!/usr/bin/env -S cargo +nightly -Zscript
---
[package]
edition = "2024"

[dependencies]
mira-eval = "0.4"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
---
use mira::scorer::{contains, succeeded};
use mira::subject::subject_fn;
use mira::{eval, Eval, Transcript};

#[eval]
fn greet() -> Eval {
    Eval::new("greet")
        .sample("hi", "Say hi and tell me the answer to life.")
        .sample("formal", "Greet me formally and tell me the answer to life.")
        .sample("casual", "Greet me casually and tell me the answer to life.")
        .subject(subject_fn(|_sample, _cx| async move {
            Transcript::response("Hi! The answer is 42.")
        }))
        .scorer(succeeded())
        .scorer(contains("42"))
        .build()
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    mira::Study::registered().serve().await
}
```

The study serves itself to a host CLI, discovered like `cargo test`, and you run it:

```bash
mira run --study greet.rs
mira run --study greet.rs --tag smoke        # just a slice
mira run --study greet.rs --format html --out report.html
```

![Running mira run --study greet.rs - the greet study passes on the offline sim model and the run is saved under ./results](/blog/mira-run.gif)

Every run saves to `./results/<run_id>/` - `report.json`, a self-contained `report.html`, one `result.json` per case. That is the folder you commit.

And it is not a Rust thing. A study is just a program that speaks the protocol, so it can be Rust, Python, TypeScript, or any binary - pick whatever the thing you are testing already lives in. yolop's SWE-bench study is [one Python file](https://github.com/everruns/yolop/blob/main/evals/swebench_verified/swebench_verified.py); bashkit's is [Rust](https://github.com/everruns/bashkit/tree/main/crates/bashkit-eval); Mira ships [Python](https://github.com/everruns/mira/tree/main/examples/greet-python) and [TypeScript](https://github.com/everruns/mira/tree/main/examples/greet-typescript) starters.

The one idea worth holding onto: a **target is whatever is under test** - not just a model. It can be a raw model, or a whole agent. Evaluating yolop means the target _is_ yolop:

```rust
.targets([
    Target::cli("yolop"),           // the agent under test
    Target::cli("claude-code"),     // compared against another agent
    Target::openai("gpt-5"),        // or a raw model, same run
])
```

Mira takes the cross-product of samples × targets × any axes you add, runs it with bounded concurrency, and returns a non-zero exit when a case fails - so `mira run …` drops straight into CI. No server to stand up, no keys to hand a third party.

## Where a Server Actually Wins

I said I would be honest, so: point 5 is real.

If you are running SWE-bench Verified end to end - 500 instances, each spinning a Docker container, for hours - a repo and your laptop are the wrong tool. That wants a fleet of workers, a queue, and shared storage. A platform earns its keep there. Same if non-engineers need to browse results without cloning anything, or you have compliance-grade retention requirements.

But that is the heavy tail. The daily eval - a couple hundred cases, run on a change, compared against last week - has none of those problems. For that, the server is overhead you carry to solve a problem you do not have.

Keep datasets in the repo. Keep results in the repo. Let the agent read them. Reach for a server the day you actually outgrow git - not before.