Every Code Change Has a History

Git blame tells you who touched a line. It does not tell you why the line is that way, or what broke the last time someone changed it.

That answer usually lives in a review comment from months ago, or in a bug report filed weeks after a merge. Nobody searches for it before approving a PR, and neither do AI coding tools.

For the Graph Hacks hackathon (WeMakeDevs x FalkorDB) I built Provenance: it turns a GitHub repo's history into a FalkorDB knowledge graph, then answers one question before a change merges: what could this break, and what is the evidence?

Repo: https://github.com/CodewithEvilxd/provenance

A real example: PR #22832

In vitejs/vite, PR #22832 changed resolveConfig so the project root always went through realpath. Later, issue #23197 was filed with the regression label, naming #22832 as the likely cause. The fix, PR #23198, says the same in its description.

Nothing in git links those three things automatically. In Provenance they are one path in the graph:

Function(resolveConfig) -> Commit / PR #22832 -> Incident(regression #23197)

The next person touching resolveConfig gets that path as a warning, with links to the real GitHub sources.

Why not just embed the code?

The usual approach is to chunk files into a vector index and retrieve the top matches. That works for "where is X defined". It is a weak fit here because the questions are about relationships:

• Which functions call this one, up to a few hops away?
• Which earlier decisions constrain it, and have they been superseded?
• Which incidents were traced back to changes in it?
• What did we know as of the day this PR merged?

These are traversals, so I modeled them as a graph.

Architecture

1. Ingestion: Octokit pulls commits, PRs, issues and review comments (paginated, cached, idempotent MERGE loaders). Tree-sitter parses TypeScript/JavaScript into Function nodes and a call graph.
2. FalkorDB: one graph per repo, plus separate graphs for per-developer memory and shared agent state.
3. Agents: a reviewer that scores change risk, and a decision extractor that turns review threads into structured Decision nodes.
4. MCP server: 10 tools over stdio, so Claude Code, Cursor or any MCP client can query the graph before editing.
5. Next.js UI: PR review, ask-the-graph, bus factor, time slider.

The data model

Nodes: Person, Commit, PR, Issue, Comment, File, Function, Incident, Decision.

Key edges:

(Person)-[:AUTHORED]->(Commit|PR)
(Comment)-[:ON]->(PR)
(Decision)-[:DERIVED_FROM]->(Comment)
(Decision)-[:CONSTRAINS]->(File|Function)
(Decision)-[:SUPERSEDES]->(Decision)
(Commit)-[:MODIFIED_FUNCTIONS]->(Function)
(Function)-[:CALLS]->(Function)
(Incident)-[:CAUSED_BY]->(PR)
(Incident)-[:RESOLVED_BY]->(Commit)

Every Decision has a DERIVED_FROM edge to a real review comment. A decision without a source comment and at least one real File or Function target is rejected before it is written.

Turning review threads into Decisions

I filtered review comments (non-bot, enough length, rationale cues), grouped them into threads, and asked an LLM to extract constraints. Everything the model returns is validated:

• the JSON must pass a Zod schema,
• every cited comment ID must exist in the thread and in FalkorDB,
• every target must resolve to an existing node.

Result: 38 source-verified Decisions from vercel/next.js, each with deterministic IDs and embeddings. A vector query for "caching" returned 10 hits, 7 of them real cache-related decisions.

Honest note: my first extraction run on Vite produced decisions like "avoid using this pronoun". They were real model outputs, but documentation nits, not engineering constraints. I tightened the prompt and bumped the cache version instead of counting them.

Querying the graph

Blast radius is a variable-length traversal over CALLS:

MATCH path = (caller:Function)-[:CALLS*1..3]->(target:Function)
WHERE target.id IN $functionIds
RETURN caller.id AS callerId, target.id AS targetId, length(path) AS depth

Active decisions are the ones not replaced by a newer one:

MATCH (d:Decision)-[c:CONSTRAINS]->(target)
WHERE target.id IN $targetIds
  AND NOT (d)<-[:SUPERSEDES]-(:Decision)
OPTIONAL MATCH (d)-[:DERIVED_FROM]->(src:Comment)
RETURN d.id, d.summary, c.strength, src.url

(Simplified. The production query also applies the as-of cutoff described below.)

Risk scoring

The score is a weighted sum of five signals, each normalized to 0..1:

Bands: low below 30, medium 30-59, high 60 and above. A hard decision violation, or a direct incident in the last 12 months, floors the score at medium.

The graph supplies every input except one: whether a diff violates a decision is an LLM judgment, run only on diff hunks that overlap the constrained function. If a check fails or times out, it is recorded as unavailable, never as "no violation", and the review reports insufficient_evidence.

The demo: PR #23333

I ran the reviewer on Vite PR #23333 (dev server integration). It overlaps hard DevTools decisions and the regression-affected resolveConfig.

Result: risk 89 (high), evidence status complete, 18 decision verdicts, and graph paths ending in both a Decision and an Incident. Clicking a path animates it in the UI and opens the source GitHub discussion.

Time slider: no future leakage

To evaluate fairly, the graph has to answer "what was known on merge day". Edges carry timestamps and every query takes an as-of cutoff.

Tests inject post-cutoff incidents and decisions into the graph and check that scores and reasons do not change.

Bus factor

For each module I compute commit concentration (bots filtered) and the smallest number of authors covering 50% of commits.

On Vite (1,915 commits, 98 modules) the average is 1.26 and 73 modules are single-maintainer concentrated. This measures commit concentration, not who actually knows the code. A maintainer who reviews a lot but commits little is invisible here.

MCP: put it where the agent works

The same graph is exposed as 10 MCP tools (explain_code_constraint, check_change_safety, trace_decision_evidence, assess_change_risk, my_context, and more). An agent can ask "is it safe to change this function?" and get graph-grounded answers with source URLs. Per-developer memory lives in its own graph (mem:<login>), and a shared state graph lets one agent hand a task to another.

Evaluation, and its limits

I labeled 21 real Vite PRs (10 later linked to regressions, 11 clean) and ran the reviewer as of each merge time.

The first run caught nothing. Two causes: my dataset had placeholder timestamps that filtered out all history, and missing evidence was being scored as zero risk. I fixed both and set the threshold to 30 to match the documented medium band.

Read the table with care:

• 21 PRs is a small sample.
• The threshold was tuned after seeing these results, so 70% recall and 87.5% precision are optimistic.
• Labels come from issue and PR links, so regressions nobody reported count as "clean".

I would trust the direction more than the numbers.

What is not proven

• The GitHub webhook comment bot passes local simulation tests (signature checks, idempotent task claims in FalkorDB), but I never ran a live delivery.
• Everything here is JS/TS only. The call graph is approximate, and function-to-commit links are line-range based.

Try it

git clone https://github.com/CodewithEvilxd/provenance.git
cd provenance
pnpm install
docker compose up -d
cp .env.example .env.local
pnpm --filter @provenance/web dev

Then open the Vite review, scrub the time slider, or ask the graph why a function is constrained.

What I learned

• Make the graph hold evidence, and let the model do only the narrow judgment calls.
• Fail closed: "unknown" must never turn into "safe".
• Fix your evaluation before you trust your model. My 0% baseline was a data bug, not a modeling result.

If you review code, how do you keep decisions from review threads findable? I'd like to hear in the comments.