Benchmarks

Memory that knows when it's wrong, measured.

Every tool run by us, on the same machine, repos and tasks, scored against answer keys none of the tools produced.

fewer outdated facts served as current: Up to5x; mex 20%, other memory tools 90 to 100% in their no-LLM modes.
of outdated facts flagged when returned: 78.6%; Every other memory tool tested flagged 0%.
keyword search at finding the code a fix needs: 2.2x; Hit@5 on 656 SWE-Explore issues: 65.9% vs 30.2%.
updates identical to a clean rebuild: 24 / 24; Other graph tools drifted by up to 124.6%.
more of what it returns is relevant: 4.4x; Python issues: 44% of returned lines relevant vs 10% for keyword search.
fewer tokens with a real agent: 72.5%; 12-task pilot, Claude Sonnet, vs file search. 9 of 12 correct vs 6.

Read the full report Full method and raw data

01At a glance

Only one column fills top to bottom.

mex against nine tools and a plain AGENTS.md file, on four capabilities that decide whether an agent can trust its memory. Every cell comes from a test we ran, not from a feature list.

Table 1 · What each tool does, as measured

Capability scoreboard across the tools tested
Capability	mex	Hindsight	cognee	mem0	agentmemory	Plain AGENTS.md	code-review-graph	codebase-memory-mcp	graphify	codegraph	GitNexus
	mex	Memory tools					Code graph tools
Flags a fact when the code it describes changes	Yes78.6%	No0%	No0%	No0%	No0%	No0%	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable
Project knowledge linked to specific code	Yes	No	No	No	No	No	No	No	Nodocs only	No	No
Graph identical to a clean rebuild after every update	Yes24/24	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	No21/24	No12/14	No0/23 hook	No2/24	No12/22
Memory you can review in a git pull request	Yes	No	No	No	No	Yes	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable

Table 1. Filled: yes. Open: no. Dash: not applicable, because the tool has no project-knowledge layer or no code graph. Notes are measured results from the sections below.

In practice: mex is the only tool tested that is git-native, linked to specific code, and checked against every code change with no LLM.

02Stale memory

Other memory tools serve outdated facts as current. mex flags them.

We replayed 367 commits of the Hono web framework, then asked each tool for 132 project facts. 30 of them had been made false by the code changes. Tools with no link to code had no way to tell.

Figure 1 · Outdated facts served as current (lower is better)

Figure 1. Hono, checkpoint C5: 132 facts, 30 outdated. Competitors in their native no-LLM modes. mex sync runs your coding agent and uses an LLM to re-check flagged facts; plain mex check uses no LLM.

In practice: Your agent gets told when a fact it's about to rely on may be out of date, instead of acting on it.

78.6% vs 0%

Outdated facts flagged when returned. No other memory tool flagged a single one.

82.9% vs 53.0%

Balanced accuracy, counting true and false facts equally: mex check + mex sync against the best other tool. Plain mex check: 68.4%.

One repo, 132 facts written with AI, answer key audited by AI; a maintainer spot check is pending. Warnings have a cost: plain mex check also flags some still-true facts, which the report covers in full.

Full knowledge results

03Finding code

Finds the code a fix needs, 2.2x as often as keyword search.

SWE-Explore gives a tool a real GitHub issue and asks for the code that matters. We scored the 656 issues in languages mex supports with the benchmark's own evaluator.

Figure 2 · Issue Hit@k on SWE-Explore

Figure 2. Issue Hit@k: the share of issues where the top k results contain code the fix needed. Compared with keyword search (BM25); no other graph tool has a published SWE-Explore result.

In practice: The first five things your agent reads are far more likely to include the code it actually has to change.

Full SWE-Explore results

04Staying exact

Zero drift after every update. Every other graph tool drifted.

We replayed commits one at a time, plus one 80-commit jump, and compared each tool's incrementally updated graph with its own clean rebuild.

Figure 3 · Incremental update vs clean rebuild, one square per check

Figure 3. One square per check, consistent with the tool's own rebuild. This measures whether updates stay in step with a full rebuild, not whether a graph is complete.

In practice: After you commit, the map of your code your agent works from is the same one a fresh rebuild would give.

05Search

The right symbol, ranked first.

On code-review-graph's public task set, mex had the highest search MRR and put the right symbol first in two of three searches.

Figure 4 · Search MRR on code-review-graph's task set

Figure 4. 15 searches; a small set, so read differences as directional. On the 9 multi-hop tasks, code-review-graph with embeddings scored highest (0.889 vs 0.556 for mex).

In practice: Your agent spends fewer steps deciding which of several matches is the one it asked for.

06Real agent

Less to read, less to pay, more right answers.

The same coding agent, Claude Sonnet, on 12 tasks across two repositories: once with plain file search, once with mex. Same tasks, same model, same prompts.

Figure 5 · Real agent, files only vs with mex (files only = 100)

Figure 5. 12-task pilot from August 2026, one run each. Correct answers: 9 of 12 with mex, 6 of 12 with files only.

In practice: The agent reached more correct answers while reading and spending far less.

Full pilot results

07Footprint

Nothing to host. Memory you can review in a PR.

mex is a CLI. Memory is Markdown in your repo plus a local SQLite index. Here is what the other memory tools needed to run in our tests.

Table 2 · Memory tools: what has to run

Memory tools: what has to run
Tool	What has to run	Reviewable in a git PR?
mex	The CLI. Markdown plus local SQLite, no background service	Yes
mem0	878 MB Python env + 88 MB model; local vector DB	No
cognee	1.7 GB env + about 750 MB models	No
agentmemory	About 1 GB + background daemon on 4 ports	No
Hindsight	HTTP server + embedded Postgres	No
Plain AGENTS.md	Nothing	Yes

In practice: No daemon to keep alive and no new service to secure; memory changes go through code review like everything else.

We tested only what runs free and locally. Tools that need an LLM provider or a hosted service are listed in the report.

Read the reports. Run it yourself.

Report · Oct 2026When memory goes stale: benchmarking mex against nine tools Pilot · Aug 202690% less noise, up to 70% fewer tokens: the numbers behind mex Method · GitHubBENCHMARKS.md: full method, harness and raw results

Try the free CLI