DEV Community: TECHDEV

Build headlines said 92x faster. I measured my own crate first.

TECHDEV — Mon, 02 Nov 2026 15:18:29 +0000

A few weeks ago the Rust corner of my feed lit up: a new toolchain promising dramatically faster builds, one benchmark, one headline number. I have seen this movie before: a number from somebody else's crate, somebody else's machine, and then a week of arguing about whether it applies to mine.

I did not want to argue about toolchains. I wanted a ruler: something that tells me, on my crate and my machine, where the seconds actually go (per crate, per build script, per link step), then tells me which fix is cheapest, and lets me verify the delta after I apply it.

So I built cargoscope. It measures. It recommends. It verifies. It never touches your toolchain.

The three commands

cargo cargoscope measure [--json out.json] shells out to cargo build --timings twice. The first build is timed; the second is a no-change rebuild. That second run is the trick: cargo omits fresh-skipped units from the timing report, so a build-script-run unit that still shows real duration in the no-change rebuild was genuinely executed again with no source changes. Its freshness is broken: typically a script that rewrites its own rerun-if-changed inputs. That is measured, not guessed from the script source.

The report shows wall time, top crates by compile time, build scripts that re-run on every build, and the codegen+link share of the final binary. On stable, the report is honest about what it cannot separate: "link step not separable on stable — the bin unit's duration covers codegen+link together."

cargo cargoscope recommend ranks fixes cheapest-first with exact config snippets: cargo check for the edit loop, fixing re-running build scripts, mold/lld linker, sccache, cranelift (gated behind --nightly-ok). It changes nothing. The snippets are yours to review and apply.

cargo cargoscope compare before.json after.json prints the measured delta and a verdict line. Only measured numbers; nothing is promised.

A real demo, honest numbers

The animated demo below is a real terminal recording on the repo's small fixture crate (verbatim transcript also in the repo). Watch the loop: measure flags a build script with RE-RUNS ON EVERY BUILD, recommend ranks the rerun-if-changed fix, I apply it, measure again, compare:

wall:   0.7s -> 0.7s  (-6.6%)
units:  5 -> 4
verdict: -6.6% wall time faster (0.7s -> 0.7s)

Small crate, small numbers, no hype. The re-run flag is gone; the per-crate deltas are printed, wins and losses. That is the whole point: the tool's job is to make the second measurement possible, not to impress you with the first.

The bug the nightly CI caught

To separate the link step, cargoscope parses nightly -Ztime stderr. While dogfooding on a real nightly toolchain I found that rustc 1.101-nightly removed -Ztime entirely (unknown unstable option). The old code set RUSTFLAGS=-Ztime whenever it detected nightly, and cargo forwards RUSTFLAGS into its own internal rustc probe, so measure broke completely on new nightlies (cargo exit 101).

The fix: probe first (rustc -Ztime --print=sysroot; --version short-circuits before -Z validation, so it cannot be the probe), set the flag only when the probe passes, and fall back gracefully otherwise. A nightly CI job now runs measure on a real nightly toolchain every night and asserts the graceful fallback. Finding your tool broken on a toolchain you claimed to support is the best possible use of a nightly job.

Where the rivals beat us

Honesty section, as always with my tools:

cargo-blaze wins if in-place recompilation fits your workflow: it actually makes rebuilds faster instead of measuring them. It is a young, invasive bet (replaces the compiler driver). cargoscope complements it: it can measure blaze's own delta.
cargo build --timings wins on zero install. The raw HTML timeline has everything cargoscope parses. cargoscope adds ranking, re-run detection, and before/after verification on top.
cargo-bloat answers a different question (binary size, not build time).

Honest limits

It does not speed up your builds. It measures. If you want the build itself to go faster without measuring first, this is the wrong tool.
Timing numbers are your machine's, your crate's, your profile's. A −6.6% here is not a claim about any other setup.
Link-step timing needs a nightly toolchain whose rustc still accepts -Ztime (removed by 1.101-nightly; support is probed, never assumed).
cargo --timings output format is not a stable API. The parser is defensive, records the tested cargo version, and fails loudly instead of guessing.

Try it

The npm package is not published yet, so install from source for now:

git clone https://github.com/techdevfun/cargoscope.git
cd cargoscope && npm install && npm run build
# then, inside any Rust crate:
npx cargo-cargoscope measure

Requires Node 20+ and a Rust toolchain (stable is fine). Zero runtime dependencies, MIT licensed.

Repo: github.com/techdevfun/cargoscope. Issues and PRs welcome; please include your measure output (redacted paths are fine) when reporting wrong numbers. Also listed in the TECHDEV catalog.

My agent burned $31 in a week. I asked which tool ate it - so I built the receipt.

TECHDEV — Mon, 02 Nov 2026 12:11:09 +0000

I can tell you my AI bill to the cent. Ask me which tool call caused it and I used to shrug.

Last week I stared at my provider dashboard: a clean per-key total, zero line items. Which MCP tool ate the budget? Which framework surface did the agent re-read forty times? Was that "heavy" framework actually heavy for my workload? The dashboard had no answers. Hacker News had opinions - 215 points on ChatGPT/Codex limit merges, 291 on the AI cost of JavaScript frameworks. One dev.to author cut their MCP server from 15 tools to 4 by hand-counting tokens with tiktoken. All vibes. No receipts.

So I built the receipt: aibill, a small CLI that reads your real agent session transcripts and attributes tokens and estimated USD to every tool call.

What it does

Four commands, one idea: measure first, cut second, verify third.

aibill scan ~/.claude/projects/my-project/

TOOL                   CALLS   MEAN TOK   P95 TOK   TOTAL TOK       USD*
------------------------------------------------------------------------
mcp__doc_lookup           30     23,265    26,947     697,938     $16.55
mcp__codebase_search      24     15,427    18,889     370,241      $7.81
...

aibill scan parses Claude Code session JSONL (or a generic MCP tool-call JSONL you feed it) and prints per-tool rows - call counts, mean and p95 tokens, totals, USD - plus per-model and per-framework-surface breakdowns. aibill cut --top 3 then suggests cut candidates: duplicated calls with identical inputs, tools whose huge outputs were never referenced again, costly one-shot tools. Candidates, not autopilot - it changes nothing on its own.

Then the part I actually wanted: aibill diff --before old.jsonl --after new.jsonl measures the before/after. On the demo run: 137 calls, 1,425,208 tokens, $31.68 estimated, one doc-lookup tool eating 52% of the bill. After trimming the obvious waste: -49.0% tokens, -52.3% USD - measured on the two transcripts, not promised about your future. aibill budget --max-usd 20 exits non-zero when projected spend exceeds the cap, so the same check runs in CI.

(The GIF runs on synthetic seeded fixtures - run scan on your own transcripts for real numbers.)

The honest fine print

USD is an estimate from public rate cards (verified 2026-11-02, sources in src/pricing.ts), not your invoice. Discounts, caching, tiers and regional multipliers change the real number. Unknown models price at $0 and get flagged, never guessed. Bring contracted rates with --pricing-file.
Claude Code logs token usage per message, so when one message carries several tool calls the usage is split evenly and flagged as shared attribution in the report.
aibill measures. It does not reduce tokens, rewrite prompts, or compress anything. Any "% saved" you see from diff was measured on the two transcripts you gave it.
It reads local transcripts only - never calls billing APIs, never touches credentials, never sends anything anywhere. Transcripts contain your proprietary code; the tool is 100% offline by design. Zero runtime dependencies, Node 20+.

Where rivals beat us

The AWS "MCP Token Cost" piece measures something aibill does not: the upfront tool-schema/context tax - 55k tokens just to describe GitHub MCP's 93 tools before a single call happens. aibill attributes per-call tokens from transcripts; it is blind to that schema tax. The two are complementary: use their number for "should this MCP server be connected at all", use aibill for "now that it is, which 3 calls are eating the bill." Provider dashboards win on authoritative totals; hand-rolled tiktoken audits win on nothing except proving the demand is real.

Try it

Install from source for now (aibill-cli isn't on npm yet - publish pending):

git clone https://github.com/techdevfun/aibill.git
cd aibill && npm install -g .
aibill scan <your-transcript.jsonl>

The point was never the $31. It's that every cost conversation about coding agents is currently vibes - headline benchmarks measured on someone else's workload, gut-feel tool cuts, surprise limit changes. A receipt won't make the bill smaller by itself. But at least now you can argue with it.

Repo: https://github.com/techdevfun/aibill - MIT, issues and redacted-transcript bug reports welcome.

Before you rent the GPUs, check the map: I built a CLI that picks your LLM serving engine

TECHDEV — Sun, 01 Nov 2026 12:07:21 +0000

DeepInfer collected ~19k stars in a single week (GitHub Trending scan 2026-10-28). The headlines say "production inference engine". My weekend said: it serves exactly one model (DeepSeek-V4.1-Flash), needs SM100+ or Ascend950 silicon, and runs offline-only with no KV cache system, no PD disaggregation, no load balancing. Checked against the upstream repo 2026-11-01.

That is not a knock. That is the gap a tool should close: before you rent GPUs or rewire your stack, you need the map — what you actually have, what the model actually costs in memory, and which engines are actually viable on that combination.

So I built servefit: a zero-dependency Python CLI that inventories your hardware, does the memory math, checks a dated engine-support matrix, and recommends an engine with reasons. Two minutes, no account, and the offline path needs no network at all.

What it does

servefit inventory — parses nvidia-smi (GPUs, VRAM, compute capability with name fallback), AMD via rocm-smi, CPU/RAM. Apple Silicon is reported as detected-with-limits, never guessed.
servefit fit --model Qwen/Qwen3-8B — weights + KV cache + overhead math with every assumption printed, then viable engines from a curated 6-engine matrix. Every row carries a check date (all 2026-11-01); CI fails if any row goes 90 days stale, so staleness is loud, not silent. One recommendation, one starter config. Missing production components (load balancing, disaggregation) are listed per engine, never hidden.
servefit probe --endpoint <url> — OpenAI-compatible SSE probe measuring TTFT and approximate tokens/sec on your hardware.
Every command also writes --json manifests for automation.

A real run

fit on a Qwen3-8B lookup from the Hugging Face Hub: pulls the real arch (36 layers, 8 kv-heads, 128 head-dim) and the 8.19B param count from safetensors metadata, computes 14.9 GiB weights + 4.5 GiB KV cache at 32k context = 20.4 GiB total, and recommends SGLang; DeepInfer is ruled out with the chip and model reasons printed. The demo transcript labels what is synthetic (the inventory runs against a synthetic RTX 4090 profile) and what is scripted (the probe runs against a local fake OpenAI-compatible server); the timing it measures is this machine's, honestly labeled.

Method, stated honestly

Memory math: weights = params × bytes-per-param; KV cache = 2 × layers × kv_heads × head_dim × context × bytes-per-param; plus a stated overhead line. Every assumption is printed, so you can argue with it. Hand-computed fixtures in the test suite (8B bf16: 14.9 GiB weights, 4.5 GiB KV @ 32k, 20.4 GiB total) all match; 37 tests, all green, CI across Python 3.10–3.13.

The engine matrix is the opinionated part: a preference order (SGLang > vLLM > TensorRT-LLM > DeepInfer > llama.cpp > Ollama) gated by chip, model-family, and memory rules. Verdicts are never silent — every "viable" or "non-viable" comes with its reasons.

What it is NOT

It does not replace profiling. Memory math gets you to the right engine; real throughput comes from measuring your workload.
Apple Silicon: GPU VRAM is reported unknown by design; recommendations lean on llama.cpp/Ollama with a manual RAM check note.
Probe token counts are a whitespace-split approximation, labeled in the output — not the model's tokenizer.
The matrix covers 6 engines checked 2026-11-01. New engines and hardware appear as rows with fresh check dates, or they fail CI.

Where rivals beat us

Honest comparison, checked 2026-11-01:

Hugging Face's model pages and hub calculators win on data freshness and model coverage: every new model's arch is there the day it drops. servefit does its own lookup but only maintains a curated engine matrix.
Per-engine docs (vLLM, SGLang, TensorRT-LLM) win on depth: the full config surface, performance tuning, and multi-node guides. servefit gives one starter config, not a manual.
DeepInfer wins on the exact thing it was built for: serving DeepSeek-V4.1-Flash fast on SM100/Ascend hardware. servefit's matrix correctly rules it out for everything else — that is the point of the map.

Install

The servefit name on PyPI is still free; until a release is published there, install from source:

pip install git+https://github.com/techdevfun/servefit
servefit inventory
servefit fit --model Qwen/Qwen3-8B

Roadmap: more engines, multi-GPU/NUMA topologies, and a servefit watch mode that re-checks the matrix on a schedule. Repo: github.com/techdevfun/servefit. Apache-2.0 licensed.

My AI writes 400-line diffs in 30 seconds. I built a 2-second gate that reviews them.

TECHDEV — Sat, 31 Oct 2026 12:07:46 +0000

My AI pair programmer writes a 400-line diff in 30 seconds. I used to spend 20 minutes reviewing it: scroll, squint, guess whether that new function is "a bit complex" or "merge-blocking complex", wonder if the helper it added already exists somewhere else in the file. Tests pass. The diff looks plausible. Ship it.

A week later we find the duplicated retry logic in three places and a 14-branch function nobody remembers writing.

The bottleneck in AI-assisted coding is not writing code. It is reviewing diffs at the speed the model produces them. Eyeball review does not scale to model throughput, and "tests pass" says nothing about the shape of the code that was just added.

Gate the diff, not the repo

I built diffgate, a zero-config CLI that checks changed code only for complexity, duplication, and PR growth. It runs in CI in under a second and returns a verdict: pass or fail, with reasons.

Four gates, all scoped to the diff (v1):

Gate	Default	What it checks
`complexity`	max 10	a touched function above the max fails
`delta`	max +15	a function gaining more than this at once fails
`dupe`	50 tokens	a normalized token block repeated in the diff fails
`growth`	+500 lines	a single file growing past this fails ("split this PR")

The point of scoping to the diff: pre-existing tech debt is context, never a failure. You do not get punished for the legacy function you did not touch. You get stopped when this PR makes things worse.

A real run

A sample PR with three violations: a checkout function at complexity 13, a 106-token duplicated block in a Go file, and one JSON file growing by 622 lines. diffgate fails it with the exact locations. Fix the branch, re-run: PASS, all functions at complexity 5 or below. In CI it emits ::error annotations that land on the PR's Files tab:

diffgate check --base origin/main --format github

- run: npx -y diffgate-cli check --base ${{ github.event.pull_request.base.sha }} --format github

There is also a measure mode that prints numbers with no verdict, for the "measure first, gate later" crowd. A 500-file diff checks in about 0.7 seconds on a laptop. Zero runtime dependencies, Node 20+.

Method, stated honestly

Complexity is a token-based cyclomatic approximation: count branch keywords and operators (if, for, while, &&, ||, ??, ternaries, elif, except...) per function. Same class as lizard: fast and deterministic, not compiler-grade.

Duplication detection normalizes identifiers to id and literals to str/num, then finds repeated token windows with a rolling hash, so copy-paste with renamed variables still matches. Strings are blanked to a single str marker rather than nothing, which avoids mass false hits on fixtures full of string literals. I hand-computed complexity fixtures in JS, Python, and Go (all 5) and the tool agreed on all of them; the suite is 38 tests, all green.

What it is NOT

It does not prove correctness. Structural gates do not prove semantics.
It does not detect AI-generated code and makes no authorship claims. There is no snake-oil "AI detector" inside.
Complexity numbers are heuristic-class, not universal truths. Thresholds are team-tunable via flags or a diffgate.json in the repo root.

Where rivals beat us

Honest comparison, checked 2026-10-31:

unclebob/ub-mode wins on process depth: it guides the whole agent workflow (planning, review rounds, verification habits). diffgate is a 2-second CI check, not a workflow. Complementary, not competing.
SonarQube / SonarCloud wins on history: trend dashboards, leak-period tracking, and 30+ languages. diffgate sees one diff, three languages, no server.
lizard wins on language breadth (20+ languages) and editor integrations. It reports numbers; diffgate adds verdicts, PR gating, and annotations.
Semgrep wins on custom rules: you can encode your team's exact patterns. diffgate ships fixed gates with sane defaults, no rule authoring, no rule maintenance.

Install

git clone https://github.com/techdevfun/diffgate.git
cd diffgate
npm install && npm run build
npm link   # or run ./bin/diffgate directly

(The diffgate-cli npm package is being published; until the registry listing is live, install from source. The README will be updated the moment npm install -g diffgate-cli works.)

Then drop diffgate check into CI and let it argue with your pull requests. It is deterministic, local, and has no model, no account, and no network to phone home to.

Roadmap: new-lines-without-coverage when an lcov/cobertura file is supplied, more languages, and per-file growth budgets. Repo: github.com/techdevfun/diffgate. MIT licensed.

What would an infostealer find on YOUR machine? I built a CLI that answers exactly that

TECHDEV — Fri, 30 Oct 2026 12:07:30 +0000

In October 2026, Check Point Research reported the Aokigahara campaign: infostealers sweeping developer machines for authentication files, session data, and configs of AI coding assistants -- Claude, ChatGPT Desktop, Codex, Cursor, Windsurf, GitHub Copilot, Gemini CLI, and others. Reading that report made me ask an uncomfortable question: if one of those landed on MY machine, what exactly would it find?

I had no good answer. So I built sessguard: a local, read-only, zero-network CLI that inventories AI-assistant session and credential material on your machine -- and hands you a per-app revocation checklist. It never prints, logs, or exports token values: findings carry paths, file metadata, and pattern-match counts only.

The gap it fills

Secret scanners like TruffleHog own the repository surface. They are excellent at that, and to be fair, that is where they beat sessguard: if your secrets already leaked into git history, you want TruffleHog, not this tool. OSQuery can answer endpoint questions, but only with hand-written queries per app that you maintain yourself. Nobody maps "AI session material on THIS machine" to "here is exactly how to revoke each one". That one-to-one mapping is the gap: curated, dated path knowledge for 10 AI assistants, strict no-value reporting, and an actionable revocation checklist.

What it does

Two commands:

inventory -- scan well-known session/config paths, report paths, file metadata, and token-pattern match counts. Never values.
checklist -- per-app revocation checklist: what each store typically holds, where to revoke it, how.

A real run against the synthetic demo profile (all fixtures fake, values never leave the machine):

$ SESSGUARD_ROOT=examples/demo-profile node dist/cli.js inventory
AI SESSION STORES ON THIS MACHINE
Token values are never shown -- paths, file metadata, and match counts only.

Apps with stores found: 6/10

- Claude Code / Claude CLI:
    stores: .claude.json, .claude
    files: 2, total size: 634 bytes
    newest: today, oldest: today
    files matching token patterns: 1 (6 matches -- values hidden)
- Codex CLI:
    stores: .codex/auth.json, .codex
    files: 2, total size: 378 bytes
    newest: today, oldest: today
    files matching token patterns: 1 (6 matches -- values hidden)
- Cursor:
    stores: .cursor
    files: 1, total size: 231 bytes
    newest: today, oldest: today
    files matching token patterns: 1 (2 matches -- values hidden)
...

And the checklist tells you exactly where to click:

$ SESSGUARD_ROOT=examples/demo-profile node dist/cli.js checklist
REVOCATION CHECKLIST

## Claude Code / Claude CLI
Stores: .claude.json, .claude
Typically holds: API keys, OAuth session tokens, project session transcripts
Revoke: https://console.anthropic.com/settings/keys
How: Revoke API keys at the console. For OAuth sessions, sign out inside the app as well.
Also rotate: Any API keys you used with Claude Code

Every path entry carries the date it was last verified (2026-10-30 for all ten apps in this release), because apps move their storage locations and stale path lists are worse than none. Terminal recording of a real run:

What it is NOT

It does not prevent infostealer infection.
It does not detect exfiltration.
Absence of findings is not evidence that a machine is clean.

It is a hygiene and incident-response checklist tool. It answers "what AI session material lives on this machine, and how do I revoke each one" -- nothing more.

How it is built

TypeScript, zero runtime dependencies, Node.js 20+. Read-only: the scanner never writes, follows no symlinks, and reads at most 2 MB per file. Zero network: CI runs the full suite with networking disabled, and a static test asserts no networking API exists in the source. The redaction self-test plants fake secrets in the demo profile and fails the build if any of them reaches the JSON output. The path list is public knowledge (documented by vendors and threat research), so the tool cannot be used as an exfiltration aid -- and it makes no network calls anyway.

Try it

git clone https://github.com/techdevfun/sessguard.git
cd sessguard
npm ci && npm run build
node dist/cli.js inventory
node dist/cli.js checklist

MIT licensed. Roadmap: an npm package (so npx sessguard inventory just works), more assistants as their storage paths get verified, and scheduled re-verification of revocation URLs.

Repo: https://github.com/techdevfun/sessguard

Node 26 is LTS today. Should you upgrade? Measure your own app instead of trusting a headline benchmark.

TECHDEV — Wed, 28 Oct 2026 12:44:40 +0000

Node 26 entered Active LTS today, 2026-10-28. If you run Node 20 or older,
your runtime gets no more security fixes. If you run 22 or 24, you need a
plan for the move. And if you are choosing between Node and Bun, what you
have today is headline benchmarks: one author's app, on one machine, that
is not yours.

Nobody's tool answers, for your app on your machine: does it load on the
target runtime, how fast is it there, and should you upgrade — with
reasons, not a headline. So I built one: runverdict.

What it does

runverdict is a local, zero-dependency TypeScript CLI. No account, no
network beyond localhost, no downloads, no data leaves the machine. It
never downloads a runtime: it measures only runtimes already installed
and reports the rest as not-installed.

For each runtime it reports:

Version and EOL status from a built-in schedule table (Node only).
A load check: the entry is spawned, and a failure records the stderr tail verbatim — which is what names a broken native addon or a missing module.
Startup timing: median of 5 spawns, spawn to first stdout line.
An HTTP workload with the built-in http client only: requests/second, p50/p95 latency, and peak RSS, using the same workload, in the same order, on the same machine for every runtime.

Then it returns a deterministic verdict:

Verdict	Rule
`BLOCKED`	The target runtime fails the load check, or no runtime produced a measurement. Reasons name the failing runtime and its stderr tail. Exit code 1.
`UPGRADE-READY`	The current runtime is EOL or Maintenance, the target runtime loads, and its req/s is not worse than the current runtime's by more than 10%. Exit code 0.
`STAY`	Otherwise, with reasons and numbers. A measured regression beyond 10% is a reason to stay, stated with both numbers. Exit code 0.

The current runtime is the first runtime in --runtimes order with a
measurement; the target is the best-performing other measured runtime,
if any. With a single measured runtime there is no target to compare
against, so the verdict is STAY.

Built-in Node EOL schedule

Node major	Status	EOL
20	EOL	2026-04-30
22	Maintenance	2027-04-30
24	Maintenance	2028-04-30
26	Active LTS	2029-04-30

An unknown Node major is reported as UNKNOWN, never guessed. Bun has
no LTS/EOL schedule: runverdict reports the Bun version only and never
invents an EOL date for Bun. Deno is explicitly out of scope in v1: it
is not detected, and passing --runtimes deno is an error, not a silent
omission.

How to run it

Requires Node >= 20.

git clone https://github.com/techdevfun/runverdict
cd runverdict
npm install
npm link

The repository ships a zero-dependency sample app with /health and
/json endpoints (examples/sample-app.js):

runverdict runtimes
runverdict measure --entry examples/sample-app.js --url /json --runtimes node,bun --duration 5 --connections 20 --json out.json
runverdict --version

--duration defaults to 10 seconds and --connections defaults to 20.
The JSON manifest carries runtime versions, EOL statuses, every measured
number, run counts, the verdict with reasons, and the machine note:
"single-machine measurement, not a general claim".

Exit codes: 0 when the verdict is UPGRADE-READY or STAY; 1 when the
verdict is BLOCKED or when no usable runtime measurement exists; 2 for
a usage error.

Demo

A real demo run on the build machine (node v24.22.1, bun 1.4.4) measured
the sample app: node startup median 114 ms, 12,675.8 req/s, p50 1.0 ms,
p95 4.0 ms, 81.3 MB peak RSS; bun startup median 33 ms, 12,614.8 req/s,
p50 1.0 ms, p95 4.0 ms, 60.8 MB peak RSS — verdict UPGRADE-READY,
because node is Maintenance and bun loaded within the 10% regression
allowance. These numbers are that machine's only; re-running produces
new numbers, because the numbers are measurements.

Limits, stated plainly

A load check proves the entry loads, not full compatibility. A passing load check is not a guarantee the app behaves identically on another runtime.
Numbers are this machine's. A faster runtime here is not a claim about any other machine, app, or workload.
Node 26 is EOL-table-only in v1 until it is measured live on a machine that has it installed. The tool reports not-installed runtimes as skipped, never downloads them.
It never downloads runtimes, never touches the network beyond localhost, and never reads files outside the app directory except the runtime binaries.

Where rivals beat us

clinic.js beats us on profiling depth: flame graphs and event-loop diagnosis. It profiles one runtime's bottleneck; it has no cross-runtime comparison, no EOL schedule, no upgrade verdict.
hyperfine beats us on statistical rigor: outlier detection and warmup modeling. It is a generic timer; it knows nothing about JS runtimes, EOL lines, native addons, or upgrade risk.
autocannon beats us on HTTP load options. It gives HTTP numbers only; no runtime awareness, no verdict, no EOL, no addon/load check.

v1 does none of those deeply. It does the upgrade decision, which none
of them do.

The code is on GitHub under MIT, zero runtime dependencies:
https://github.com/techdevfun/runverdict

Measured on real public CI logs: what two small tools actually did, and where they fell short

TECHDEV — Tue, 27 Oct 2026 09:09:57 +0000

Most tool READMEs show numbers from their own fixtures. We wanted numbers from logs nobody on our side wrote, so we ran two public repos, context-diet 0.5.0 and ci-failure-detective 0.1.0, cloned from GitHub and built and run exactly as their READMEs describe, on two real public CI logs from other projects. Everything below is what the runs actually printed, including the misses. Lines and bytes are exact. Token counts are the tools' own chars/4 estimates, labeled as estimates throughout, never as measured tokens. We did not run a real tokenizer, so we cannot say whether chars/4 overcounts or undercounts these logs.

The two logs

Log A: prettier/prettier, workflow "Dev", run 40391802031 (PR #20363, "Update babel to v8.0.8", 2026-10-26), job "[Full Test] Node.js 26 on Linux". Downloaded log: 2,589 lines, 251,942 bytes, 62,967 estimated tokens (chars/4). Ground truth from the log itself: 1 failed suite (tests/format/js/errors/export-default-from/format.test.js), 4 failed snapshot tests, 1,577 passed suites, 103,099 passed tests, 5 skipped, 103,108 total. The failures are snapshot mismatches: babel v8 changed parser error wording ("Unexpected token" versus "Unexpected token, expected from"), so stored snapshots no longer match. That is a real test failure caused by an upstream message change, not a bug in prettier's own formatting code.

Log B: jestjs/jest, workflow "Node CI", run 40415519415 (branch renovate/eslint-plugin-unicorn-79.x, 2026-10-26), job "Windows with shard 1_4, Node v25.x". Ground truth: 2 failed tests in e2e/tests/customReporters.test.ts ("Custom Reporters Integration" - "prints reporter errors" and "supports reporter written in ESM"), 2 failed / 11 skipped / 1,394 passed / 1,407 total in the failing invocation. The failure cause visible in the log is a Windows temp-directory cleanup error (EPERM, permission denied, on fs.rmSync). A later invocation in the same job log passed everything (1,396 passed, 11 skipped, 1,407 total). In other words: a Windows-only environment flake, not a code regression.

Method note: read this before the numbers

GitHub's downloadable logs prefix every line with a timestamp. ci-failure-detective's test parsers match markers at line starts, so on raw downloaded logs they extract nothing; we verified this on both logs (zero tests extracted from each raw log, framework reported as unknown). To reproduce the input the tool was built for (piped test output, no timestamps), we stripped only the timestamp prefixes and ANSI color codes from Log B before running the parser, and verified the strip changed nothing else: same line count, every line identical after the strip. Log A was run both raw and stripped; the parser gap there is different (see below) and survives the strip. context-diet was run on Log A exactly as downloaded.

context-diet on Log A

We replicated the procedure from the tool's own README example: pack with a 400-token budget and the query "test failure". We also ran dedupe alone and extract alone, because a single pack number hides which stage did the work.

Run	Lines	Bytes	Est. tokens
Input (Log A)	2,589	251,942	62,967
dedupe alone, raw log	2,589	251,942	62,967
pack, budget 400, query "test failure"	13	1,580	395
pack, budget 4000, same query	163	15,771	3,941
extract alone, same query	1,694	149,870	(not reported)

Three findings, in order of importance.

First, dedupe removed nothing from the raw log: 0 duplicate lines, 0 duplicate blocks. Every line carries a unique timestamp, so no two lines are identical. The tool does not normalize timestamps today, so on raw GitHub logs its dedupe stage is idle. As a secondary check only, we ran dedupe on the same log with timestamps and color codes stripped: it then removed 266 lines (115 duplicate lines plus 26 blocks covering 151 lines), taking 2,589 lines to 2,323 (10.3% of lines; bytes 174,192 to 158,340 on the stripped input). Near-duplicate collapsing found nothing here, which is expected: this log contains no repeated traces differing only in memory addresses. The timestamp effect is therefore measurable on this log: 0% versus 10.3% of lines.

Second, the headline pack saving is mostly budget truncation, not duplicate removal. At budget 400, the manifest reads: strip-ansi removed 2,698 bytes and 0 lines; dedupe removed 0 lines; extract kept 10 chunks (2,589 to 1,694 lines); summarize dropped 1,663 lines as over-budget plus 18 noise lines. So the 99.5% line saving (2,589 to 13) is the budget doing the cutting, and every output line is still verbatim from the input (verbatim: true). That is a property worth keeping, but it is not evidence that the log was 99.5% duplication.

Third, at budget 400 the actual failure did not survive. The 13-line output contains two node warnings and eleven PASS lines, but no FAIL line and no mention of the failing suite. The mechanism is in the summarize code: it picks lines by priority (error-matching lines first) then document order until the budget is met, and its error pattern also matches PASS lines under errors/ directories. In the extracted text, 50 such lines precede the FAIL line, and the 400-token budget was spent before reaching it. At budget 4000 the failure does survive (163 lines, FAIL present), at a 93.7% line saving that is still mostly budget truncation (summarize dropped 1,531 lines). The honest summary: on this log, pack is a lossy budget summary, and the budget you pick decides whether the failure you care about is still in the output. Protect the lines that must survive, or check the output before trusting it.

ci-failure-detective on Log A and Log B

On Log A, the tool found the first error correctly: line 449 of the downloaded log, FAIL tests/format/js/errors/export-default-from/format.test.js, with the snapshot diff as context. That matches ground truth. But it extracted zero failing tests and reported the framework as unknown, raw and stripped alike. The reason is visible in the parser code: the jest parser needs bullet failure blocks or checkmark/cross result lines, and prettier's custom format-test runner emits neither. Its failing checks appear as "snippet: #0 > format" blocks, which no parser pattern matches. Verdict: unknown, with no history supplied. The tool pointed at the right suite and stopped there; it could not name the four failed checks.

On Log B (after the timestamp and color strip described above), the tool found the first error correctly: FAIL e2e/tests/customReporters.test.ts (9.769 s), and reported the framework as jest. It named both failing tests correctly ("Custom Reporters Integration" - "prints reporter errors" and "supports reporter written in ESM"), matching ground truth. But it attributed both to the wrong file (packages/jest-environment-node/src/tests/globals_cleanup_3.test.ts instead of customReporters.test.ts). The reason is structural: the bullet failure blocks sit in the summary section at the end of the log, after all file lines, so the parser's "current file" is whatever file line came last, not the failing suite's file. Verdicts for both tests: unknown, confidence 0, reason "No history supplied". That is the honest answer here, but it is not the flaky verdict the ground truth would support.

Why verdicts cannot go further on logs like these: the tool's record command stores failing tests plus per-test pass records, but pass records only exist when the parser sees individual passes, which needs verbose test output. Jest's own CI log contains zero checkmark/cross lines, so no pass records exist at all. Reading the scoring code, without pass records the same-commit verdict (0.95), the flip-rate verdict (0.70), and the regression verdict (0.80) cannot fire; the reachable verdicts are unknown or "real" at 0.55 for a test that failed before and never passed in recorded history. The in-log retry heuristic also needs retry markers, and this log contains none. Separately, the file misattribution above means history keys (file plus test name) would not match pass records even if verbose logs existed. We did not manufacture history from the same run to force a verdict; that would be circular.

Where rivals beat us

Stated plainly, because these are the gaps the measurements exposed:

GitHub's own failure annotations beat this tool on position: they show the failing suite directly on the run page, with no parsing step and no format caveats.
A plain text search for FAIL beats it on Log A: the first error was one grep away, while per-test verdicts needed markers the log does not have.
Real tokenizers beat chars/4 on measurement: our token numbers are estimates by construction, and we did not verify them against a real tokenizer.
The projects' own READMEs already list the honest comparisons: ci-failure-detective names hosted flaky platforms (dashboards, alerting, auto-quarantine) that it does not aim to replace, and context-diet keeps every output line verbatim instead of rewriting, which is a smaller claim than tools that summarize freely. We keep both framings: local, deterministic, no log leaves the machine, and narrower than the platforms above.

What we did not measure

Two logs are not a benchmark. We did not test other frameworks' logs, other budgets beyond 400 and 4000, or history-based verdicts, because the logs we used cannot support them (see above). We did not measure tokens with a real tokenizer. Nothing here claims these tools beat any rival on detection quality; the claim is narrower: on two real public logs, first errors were found correctly twice, per-test verdicts were absent twice for stated, checkable reasons, and pack savings track the budget you choose more than the duplication in the log.

Sources

Log A run page: https://github.com/prettier/prettier/actions/runs/40391802031 (job "[Full Test] Node.js 26 on Linux", job id 121814403282, 2026-10-26; log downloaded 2026-10-27 via the repo's public run logs)
Log B run page: https://github.com/jestjs/jest/actions/runs/40415519415 (job "Windows with shard 1_4, Node v25.x", 2026-10-26; log downloaded 2026-10-27 via the repo's public run logs)
context-diet 0.5.0: https://github.com/techdevfun/context-diet
ci-failure-detective 0.1.0: https://github.com/techdevfun/ci-failure-detective
Method note: GitHub's downloadable logs add per-line timestamps; the timestamp and color strip applied to Log B removed only those prefixes and codes (line count unchanged, all other bytes identical). context-diet ran on Log A exactly as downloaded.

A stolen token turns your own CI into a secret dump. I built a local detector that watches for the act

TECHDEV — Tue, 27 Oct 2026 07:10:04 +0000

Your private repository's CI can betray you without a single strange domain ever appearing. That is the shape of Gheist, a campaign Wiz Research has tracked since September 15, 2026: more than 13,500 planted branches recovered from more than 850 compromised accounts, from personal projects to enterprise private repositories.

The mechanism is brutally simple. A stolen token — Wiz points mainly to infostealer logs and credentials exposed in earlier supply-chain incidents — creates a branch named ci-sync- plus six random characters, then commits a workflow at .github/workflows/sync-check.yml with the message ci: sync config. The push triggers the workflow, which prints repository secrets and environment variables after base32-encoding them. GitHub masks secret values in logs only when they appear verbatim, so an encoded secret prints in full. Seconds after the job finishes, the attacker's tool downloads the workflow log through the GitHub API and deletes the run. No external command-and-control is needed, because the secrets leave through GitHub's own API. A second variant, seen October 19, 2026, targets self-hosted runners with a job named sync-host, hunting the host filesystem, process environments, and mounted containers for 31 credential formats.

Deleting the branch or the run does not undo the exposure. Every secret the repository could read must be rotated, and the token that pushed the branch must be revoked.

Why the existing answers miss it

GitHub's control from July 28, 2026 holds suspicious workflow runs in public repositories pending collaborator approval, and it stopped every public run Wiz checked. It does not cover private repositories or GitHub Enterprise Server — exactly where Gheist succeeds.

The organization audit log is ground truth: it records the created, completed, and deleted workflow-run events, plus the tool's user agent. But the audit-log API is available to organization owners only, and nothing ships a packaged Gheist rule — the operator must already know the pattern, write the query, and interpret the result.

Static secret scanners scan content. A dump workflow contains no secret text, so it is invisible to them by construction — and deleting the offending line does nothing to the credential anyway. Rotation is the only step that neutralizes exposure.

SIEM forwarding works, but it needs an enterprise plan, streaming setup, and hand-written queries. Manual audit runbooks are exact and free, but they are one-off: no scheduling, no report, no false-positive discipline, no checklist.

What I built: gheist-guard

gheist-guard is a local, deterministic, read-only behavioral detector. It watches for the act, from the evidence that survives it: the branch name, the audit-log sequence, the tool fingerprint, and the rotate checklist.

One command, one verdict per repository: CLEAR, SUSPECT, or CONFIRMED.
Three commands: scan (REST path over branches, workflows, and runs), audit --org (audit-log path, or --events with recorded payloads), and checklist (re-print the rotate checklist from a saved report).
Seven rules: the ci-sync- branch pattern, the sync-check workflow identity, base32/base64 dump content, completed-then-deleted within 60 seconds, the gheist user agent, a generic unreviewed-workflow rule that also catches the adjacent GhostAction shape, and the self-hosted sync-host variant.
Read-only by default. The checklist lists secret NAMES only — it never prints a secret value, never deletes anything, never rotates anything itself. A partial signal is SUSPECT with its next check, never silently CLEAR and never CONFIRMED. The name "Gheist" appears in a verdict only when the full documented pattern matches.

Install from source (the tool is not on npm):

git clone https://github.com/techdevfun/gheist-guard
cd gheist-guard
npm ci
npm run build

Or use it as a GitHub Action: techdevfun/gheist-guard@v1.

Measured numbers (build 2026-10-27)

Test suite: 52/52 green, TypeScript strict clean.
False-positive probe: 24 legitimate branch names + 5 legitimate workflows → 0 CONFIRMED, 1 SUSPECT (ci-sync-config matches the six-character pattern by design; documented boundary).
Rule-evaluation latency (in-process, 500 iterations on the planted fixture): median 0.0045 ms, p95 0.0148 ms per repository.
Live REST end-to-end (real GitHub API, read-only): 3 real repositories scanned, all CLEAR (vetgate, context-diet, breakcheck); plus a private test repository with a branch-only plant: SUSPECT before a real adapter fix (it only read the default branch) and CONFIRMED after (R1+R2+R3).
Demo (real CLI runs on fixtures, not staged): a 9.4-second GIF plus the full transcript in the repo's examples.

Where rivals beat us

Honestly, and this stays in the README too:

GitHub's native controls beat us on authority and position. The audit log is ground truth; the approval hold stops public execution before any dump. We only read what the platform records.
Socket, StepSecurity, and GitGuardian beat us on threat-intelligence scale. They see campaigns across the ecosystem; we see only your repositories. We will never be the first to name a new campaign.
Gitleaks and TruffleHog beat us on secret-pattern breadth in code and history. We do not scan secrets in code at all — run one of them alongside this tool if you need that.
SIEM tooling beats us on central correlation across sources. We are a single-purpose local tool.
Manual runbooks beat us on zero-install immediacy. Copy-paste commands work today with no install. We have to earn the install with scheduling, reporting, false-positive discipline, and the checklist.
The July 28 approval hold beats us on prevention for public repositories. Our value concentrates where the hold does not reach: private repositories, GitHub Enterprise Server, and the response after a run has completed and been deleted.

Limits, stated plainly

The audit-log path is fixture-tested end to end. No organization-owner account was available for a live audit-log test.
GitHub Enterprise Server logs are documented but untested.
Pull-request association is an approximation: a workflow commit with an associated pull request is treated as reviewed.
The tool cannot see a run that was deleted before the first scan unless the branch or the audit log still carries it.
"Gheist" is Wiz's name for the campaign. This tool credits Wiz as the source of the name and pattern; there is no link to or endorsement by Wiz.

The code is MIT, on GitHub: https://github.com/techdevfun/gheist-guard

Your agent installs packages before anyone vets them. I built a gate that runs first.

TECHDEV — Mon, 26 Oct 2026 07:22:43 +0000

Your agent installs packages before anyone vets them. I built a gate that runs first.

My coding agent resolves a task, picks a package, and runs the install. The postinstall hook fires before I read any output. By the time a post-install audit, a PR check, or a dashboard tells me something was wrong, the payload has already run.

That is the moment I wanted to gate: after the agent has chosen the artifact, before the install or run command executes.

What I built

vetgate is a local, deterministic pre-execution gate. It is an MCP server and a CLI mirror. Call it instead of running the install: one call, one verdict (ALLOW, DENY, or REVIEW), every finding carrying the exact evidence string it matched, and a manifest that makes the verdict re-checkable without re-running anything.

It vets three kinds of artifact:

vet_package / npm:name@version — downloads the published tarball, inspects it, never executes it.
vet_action / action:owner/repo@ref — a local form check: SHA-pinned, mutable tag, or unpinned.
vet_script / url:https://... — fetches the script text, scans it, never executes it.

There is no model, no account, no API key, and no verdict score. The default path sends nothing anywhere except downloading the artifact itself (vet_package) and fetching the script text (vet_script, labeled in the verdict manifest). All analysis is local.

The rules are stated, not learned. A finding without evidence is a bug. Any high finding means DENY; otherwise any finding means REVIEW with a safeAction — for example, install with --ignore-scripts, pin to a digest, or ask the user. A single weak signal alone (an install script that only echoes, a missing license field, a mutable tag) yields REVIEW, never a silent ALLOW and never a DENY.

A real run

From the CLI, on hand-built fixtures (no real malware, nothing fetched in fixture mode):

$ vetgate check local:tests/fixtures/exfil
vetgate: DENY — 2 finding(s)   # exit 1

$ vetgate check local:tests/fixtures/benign
vetgate: ALLOW — 0 finding(s)  # exit 0

$ vetgate check action:actions/checkout@v4
vetgate: REVIEW — 1 finding(s) # exit 1, pin to a full commit SHA

The full transcript is in the repo at docs/demo/transcript.txt, and the README shows a 30-second demo rendered from that transcript.

As an MCP server, it exposes the same three tools over stdio:

{
  "mcpServers": {
    "vetgate": { "command": "npx", "args": ["-y", "vetgate", "serve"] }
  }
}

I verified it with a real MCP session: the server lists all three tools, vet_action on actions/checkout@v4 returns REVIEW, and vet_package on the real published lodash@4.17.21 returns ALLOW with the tarball SHA-256 6a087ac9e5702a0c9d60fbcd48696012646ec8df1491dea472b150e79fcaf804.

Measured numbers, not claims

Measured locally at build time on the fixture set (analysis only; network download time excluded). The measurement script is scripts/measure-latency.mjs in the repo, so you can re-run it:

Fixture suite: 15 hand-built fixtures — ALLOW 4, DENY 6, REVIEW 5. Every malicious-pattern fixture DENYs, every benign fixture ALLOWs, every weak-signal fixture REVIEWs.
False-positive probe: 10 benign fixture contents — 0 DENY.
Determinism: the same package contents vetted twice produce identical verdicts after stripping manifest.checkedAt.
Latency over the fixture set (3,000 timed checks): median 0.006 ms, p95 0.075 ms, mean 0.026 ms per check.
Test suite: 26 tests, all passing (npm test).

Where rivals beat us

Honesty first, because a security gate that overclaims is worse than useless:

SafeDep vet beats us on breadth and on malicious detection. More ecosystems, Docker and SBOM support, policy expressions, and cloud active scanning for malicious packages in the wild. Our v1 is npm-only with a fixed local rule set and cannot match that breadth or that cloud detection power.
Socket beats us on behavioral detection depth. Its malware database, obfuscation analysis, and telemetry detection come from operating a cloud scanner at scale. Our local rules cover only the documented install-script and pattern surface.
Snyk and DeepSource beat us on monitoring and fixing. Continuous re-evaluation and fix PRs are a different, genuinely valuable job; we do neither. We vet one artifact once, before execution.
OpenSSF Scorecard beats us on maintainer-risk signals. Branch protection, review discipline, and fuzzing adoption are real risk signals our artifact inspection does not measure.
OSV and the advisory databases beat us on known-vulnerability breadth. Any local-only v1 misses known CVEs that a database lookup would catch. There is no enrichment mode in v1.
A registry firewall beats us on position. Blocking at the registry edge stops a bad package before download; we inspect after download and before execution, by design, because we are the gate for agents that install from anywhere, not just one registry.

The wedge is narrow and deliberate: a gate that sits inside the agent loop at the moment of execution, returns a deterministic verdict with quotable evidence, and runs fully locally with no account, no key, and no data leaving the machine. If you need breadth, cloud detection, monitoring, or fix PRs, use the tools above — vetgate does not replace them.

What is NOT in v1

Other ecosystems (PyPI, Maven, Go, Ruby, Rust, PHP). npm only.
Agent skill / MCP-server-config vetting — a real pain, a different detection domain, and the first follow-up candidate.
Container image layer inspection, monitoring, re-scanning, fix PRs, dashboards, accounts, cloud APIs, or policy languages.
Any claim about malware families beyond the stated rule list. The tool reports what its rules matched, nothing more.

The roadmap is in the README: skill/config vetting first, then ecosystems one at a time, then optional labeled network enrichment, off by default.

Try it

npm ci
npm run build
npm test

Requires Node 20 or later. The repo is MIT licensed, and the code, rules, fixtures, and measurement script are all in the open:

Repo: https://github.com/techdevfun/vetgate
README: https://github.com/techdevfun/vetgate#readme

If you run it on a package your agent was about to install, I would like to hear what it caught — or what it missed. The rule list is explicit, so a miss is a concrete bug report, not a vague score to argue about. What would you want the gate to check next: another ecosystem, or skill/config vetting?

Your agent isn't running out of context because of one big file

TECHDEV — Sun, 25 Oct 2026 12:07:37 +0000

Your agent isn't running out of context because of one big file. It is
running out because of accumulation: tool outputs, test logs, build logs,
repeated turns, and the same content pasted again in a slightly different
form. I kept watching sessions die on the tenth retry of the same failing
test — not because any single output was huge, but because nobody ever
threw anything away.

So I built Context Diet: an MCP server (plus a CLI) that shrinks the
content an agent actually sends, deterministically, with receipts. Every
output line is copied verbatim from the input. Every dropped line is
counted and categorized in a manifest. No model, no download, no data
leaves the machine.

What it does

Five tools, one promise — nothing invented:

diet_count — lines, bytes, and estimated tokens of a text.
diet_dedupe — removes exact duplicate lines and repeated blocks.
diet_extract — relevance-ranked excerpts for a query, with context around each hit.
diet_summarize — an extractive summary to a token budget. It keeps headings, error and failure lines, and the first and last lines of each section. It never rewrites a word.
diet_pack — the full pipeline: dedupe, then extract if you give it a query, then summarize to a budget, plus a manifest for every line.

The manifest is the point. A summary that silently rewrites your log is
a liability; a summary that says "kept 23 lines, dropped 54 as
duplicates, 4 in extraction, 88 in summarization" is something you can
diff and trust.

Measured, not claimed

On a real session log (verbose test runs, a README excerpt, a build
output — examples/session.log in the repo), packed to a 400-token
budget with the query test failure:

	Lines	Bytes	Est. tokens
Before	169	7,917	1,942
After	23	1,587	392

Token counts are estimates (a documented chars/4 rule); lines and
bytes are exact. The manifest adds up: 169 = 23 kept + 54 deduped + 4
extracted away + 88 summarized away. Verbatim: true.

How to run it

From source (the npm package arrives with the first npm publish):

git clone https://github.com/techdevfun/context-diet
cd context-diet
npm install && npm run build
node dist/cli.js pack --input examples/session.log --budget 400 --query "test failure"

As an MCP server over stdio, point your client at
node /path/to/context-diet/dist/server.js. The repo has a demo GIF
recorded from a real terminal session, CI on Node 20 and 22, and a test
suite that runs the server through a real MCP client.

How it compares (checked 2026-10-25)

This is a crowded corner, and the honest version is that each rival is
better at something:

mcp-token-optimizer minifies code with AST awareness and restores formatting with Prettier. It works on code files; it does not touch logs, outputs, or conversation turns, which is where session context actually accumulates.
LLMLingua / LLMLingua-2 compress harder (up to 20x on benchmarks) — but they need a model download and a Python/PyTorch stack, and their drops are unauditable at token granularity.
Entroly claims 78% fewer tokens per request across whole codebases, with learned weights and a dashboard. That figure is self-reported, and the weighting is a black box: you cannot list exactly why a line was dropped.
Vendor tool search / code mode (Claude Code deferral, Cloudflare Code Mode) report the biggest percentages, up to 99.9% — but they shrink tool definitions, not the content the agent accumulates. Different layer, complementary, not competing.

Where rivals beat us

Plainly: LLMLingua compresses harder than our typical 2–5x on logs.
mcp-token-optimizer understands code structure and restores formatting;
we treat code as text. Entroly covers whole codebases with a dashboard;
we are one focused pipeline. Vendor tool search fixes tool-schema bloat
at percentages we will never match, because that is not our layer. We
do not have those things in v1, and we do not claim them.

What we have instead is the one thing none of them offer: a
deterministic, auditable diet for session content, where every cut is
accounted for and nothing is invented.

Roadmap (not in v1)

LLM summarization (opt-in), model-based compression, AST-aware code
minification, cross-session cache, dashboard, auto-apply to agent
configs, JUnit/XML parsing, SARIF output, secret redaction.

Repo: https://github.com/techdevfun/context-diet — MIT licensed,
no account, no network, no model calls. If your sessions also die by
accumulation, try it on your own ugliest log and check the manifest.

Your CI failed. Was it a flake or a real break?

TECHDEV — Sat, 24 Oct 2026 12:08:16 +0000

Your CI failed. Was it a flake or a real break?

Last week I watched a team re-run a red CI job three times, merge on the third green run, and ship the bug anyway. The test was not flaky. The retry just hid it for one run.

The opposite happens just as often: a genuinely flaky test blocks a good PR for an afternoon while someone scrolls a 40,000-line log looking for the first real error.

I built a small step that answers three questions before anyone re-runs anything:

Where did it actually break first?
Is each failing test flaky, or genuinely broken?
Has this exact failure happened before?

It is called CI Failure Detective, and it runs entirely inside your runner. No log ever leaves your repo. No signup, no server, no bill.

How it works

Add one step after your test command:

- name: Run tests
  run: npm test 2>&1 | tee ci.log

- name: Diagnose failures
  if: failure()
  uses: techdevfun/ci-failure-detective@v1
  with:
    log-file: ci.log

It posts a diagnosis to the job summary: the first error line with context, a verdict per failing test with a confidence score and a reason, a stable 12-character signature for the failure, and suggested next steps. Suggestions only — it never quarantines, retries, or edits anything by itself. Automatically silencing failures hides real bugs, so that part stays human.

As a CLI it works the same way, with zero dependencies (Node.js 18+):

npx ci-failure-detective analyze ci.log
cifd analyze ci.log --format md >> $GITHUB_STEP_SUMMARY

Flaky detection needs history, so you commit a small JSONL file and append each run:

cifd record --history .cifd-history.jsonl --commit $GITHUB_SHA < ci.log
cifd history   # show failure clusters

The rules are deliberately boring and deterministic:

A test that fails and passes on the same commit is flaky, confidence 0.95.
A test that flips frequently across recent runs is flaky, 0.70.
A test that fails consistently on a commit where it used to pass is a real regression, 0.80.
No history at all? The verdict is unknown, with a reason. It never guesses.

Same-commit verdicts need per-test pass records, so run tests verbose (jest --verbose, pytest -v). Plain logs only record failures, so same-commit evidence is unavailable from them — flip-rate and regression verdicts still work.

What I verified before publishing (2026-10-24)

Test suite: 46/46 passing locally; CI green on Node 20.x and 22.x.
Six fixture logs classify correctly: Jest, pytest, go test, Mocha, npm install failure, and tsc failure.
A real failing node:test run gets the right category (test failure) and the right first error line.
Same-commit flaky detection works end to end through the real CLI: record a passing run, then analyze a failing run on the same commit, and it reports flaky at 0.95.
Live GitHub Action run on a real pull request: I ran the composite action on a private test repo's PR with a deliberately failing test. Both action invocations succeeded — one on the real failing log, one on a Jest fixture log — and a verify step asserted both produced a non-empty verdict and a 12-hex-character signature (real-log signature 41968f39ccc7, fixture signature 4f709b012d30). The run itself stayed red by design, because the test step failed. That is the point of a failure-diagnosis tool.
Parsers today: Jest, pytest, go test, Mocha. Failure classification and first-error extraction work on any raw log, even without a test framework.

How this compares (checked 2026-10-24)

Tool	What it does	What it costs you
BuildPulse ($99/month Startup) / Trunk / Datadog / Mergify	Flaky detection, dashboards, trend, auto-quarantine, alerts	You upload all test results and logs to their cloud
gh-failmap (OSS)	Clusters failed runs by shared error line	CLI only, needs Python + `gh` login, GitHub only, no flaky verdicts
flakestat (OSS, zero-dependency)	Local-first flaky hunting	Only hunts flakes via re-runs or JUnit XML; no log reading, no clustering, no Action
CI Failure Detective	First error + flaky-vs-real + clustering, in the runner, zero dependencies	No dashboard, no alerting, no auto-apply

Where rivals beat us (honestly)

BuildPulse and friends have dashboards, long-term trend analytics, auto-quarantine, multi-channel alerting, and AI grouping — we have none of that in v1. gh-failmap's clustering is a focused, proven approach; we have not run a head-to-head cluster comparison against it yet, so we claim no advantage there. flakestat's JUnit XML input is something we do not have yet.

CI Failure Detective is not a replacement for those platforms. It is a specialized triage layer for teams that cannot or will not send logs out, or that want a deterministic answer inside the runner before deciding whether a retry is even sensible.

Honest limits

No native node:test parser yet: a real node:test log gets the right category and first error, but its tests are not listed individually.
No JUnit XML input yet, no dashboard, no alerting, no auto-quarantine apply, no SARIF output.
History is a JSONL file you own and rotate yourself; there is no automatic rotation yet.
If your log contains a secret, that secret is in the log you already have; the summary shows to the same audience that can see the log. Redaction is on the roadmap.

Roadmap (not in v1)

JUnit XML as an additional input, Slack webhook output, more frameworks (PHPUnit, RSpec, .NET), SARIF output. Quarantine stays suggest-only by design.

The repo is MIT, the Action is techdevfun/ci-failure-detective@v1, and the README has the full input/output reference. If your team re-runs red jobs on hope, try the step that tells you whether hope was justified.

Your agent asked "can I run this?" You said yes. It ran it 40 minutes later.

TECHDEV — Fri, 23 Oct 2026 12:00:55 +0000

I run coding agents the way most of us do now: it proposes, I approve, it executes.
Last week I caught something uncomfortable in my own workflow. I told an agent
"yes, run the deploy" — then got pulled into a meeting. Forty minutes later it
ran the deploy. Different directory. Code that had changed twice since I'd
looked at it.

I had approved the command at 10:00. I had not approved the command at 10:40.
But nobody wrote that down, so my yes quietly survived past its own relevance.

The pattern has a name: stale approvals

DZone recently ran a piece on authorization leases for agentic workflows, and it
clicked: the problem isn't that agents are disobedient. It's that our approvals
have no expiry date, no scope, and no receipt. A human approval is a moment in
time — but we hand it to the agent as if it were a permanent permission.

So I built the lease

leaseguard turns "yes" into a
signed, time-boxed, directory-scoped lease:

leaseguard grant "npm run deploy" --for 10m --note "hotfix for #42"
leaseguard run --lease $TOKEN -- npm run deploy

The gate checks: standing deny policy (deny always wins, even over a valid
lease) → lease signature → expiry → directory scope → command pattern. Then it
runs the command, passes through the exit code, and appends every decision to
a hash-chained ledger. leaseguard audit shows what ran, what was denied,
which approvals died quietly unused, and whether anyone tampered with the log.

One file, zero dependencies, Node 18+. The part I'm proudest of is what it
doesn't claim: it won't stop a determined adversary with your file
permissions, and it says so in the README. It stops stale approvals, accidents,
and casual misuse — which is the actual problem I had.

Try the demo: grant a 3-second lease, use it, watch it expire, then audit.
The stale approval dies quietly, and nothing runs on your old yes.