SecretsChecker

Every developer has pasted an API key into a config 'just to test something', pushed, and moved on — and sometimes a bot finds it in eleven minutes and your AWS bill gains a zero. The interesting part isn't the leak, it's why it keeps happening to careful people. Secret scanners fail on a human level before a technical one: regex-only tools scream at every Base64 string and EXAMPLE_KEY in a README, and after the third false positive developers stop reading the bot and start clicking 'merge anyway'. A scanner nobody believes is worse than none — it creates the feeling of safety without the substance.

Regex is fast, deterministic and dumb — it has no idea whether token = 'test_1234...' is a real secret or a unit-test fixture. Large models understand context but are slow and expensive, and running one over every line of every diff burns a GPU budget for nothing. So detection is a funnel: a cheap, merciless regex pass for API-key-shaped assignments, AWS keys, PEM headers and high-entropy blobs, then progressively more expensive judgment only on what survives. Regex provides recall; the model provides judgment; neither is allowed to override the other completely.

Between regex and the model sits a Shannon-entropy filter whose only job is to stop the 'high entropy string' pattern from flagging every hex color and UUID in the codebase. Random-looking data has high entropy; predictable data like 'aaaaaaaa' or a repeated hex pattern doesn't. Only strings that survive both the regex match and the entropy bar get promoted to the expensive stage — that's the whole trick to keeping recall high without setting the ML layer on fire with noise.

The model layer never sees raw secrets — every candidate is masked first (sk_live_51H8s... becomes sk_l****), because a scanner that forwards the thing it's meant to protect into a third-party context window is a special kind of embarrassing bug. Masked candidates are batched into one structured prompt and asked for exactly one thing: a plain YES/NO plus a confidence number per finding, nothing conversational. Then that verdict is fused with the regex signal in a small weighted formula — language models are confident liars often enough that you never let one be the sole vote.

The system needs to remember the last commit it scanned between runs — but dragging in Postgres or Redis for what is fundamentally one string didn't fit the project's lean spirit. So a pinned GitHub Issue becomes an append-only ledger: every successful scan appends a status comment rather than editing old state, so history is naturally auditable and nothing can be silently overwritten. First run triggers a full scan; every run after is a targeted git diff against the last known-good commit, with a [full-scan] tag or manual dispatch forcing the exhaustive path. Reach for the platform you're already standing on.

GitHub Actions minutes are not the place to load a multi-gigabyte model on every push, so the Action's only job is to figure out what changed, package it as JSON and ship it to an external FastAPI service — deployable on a plain EC2 box or a serverless Modal container that scales to zero when idle. The service exposes a single /scan endpoint and does all the real thinking, so detection logic can be re-tuned or swapped to a bigger model without touching workflow YAML. A CLI runs the same pipeline in miniature for local sanity-checks before anything reaches CI.

The judge model is a config value, not a hardcoded import, because the right choice is a permanent tension, not a bug to resolve once. The README's default is distilbert — chosen for CI stability and a tiny footprint rather than accuracy, cheap to boot — while the classifier has since moved to a causal instruction-following model prompted for a YES/NO/confidence verdict: a more capable judge at the cost of being heavier. Free CI tier or warm GPU container decides which one you want, which is exactly why MODEL_NAME lives in config.

Nobody wants a security tool — people want to not think about security and have it quietly be true anyway. Success was never 'catches every possible secret'; a regex that aggressive is unusable within a week. It was: filter hard with cheap deterministic rules, spend expensive reasoning only on what survives, mask everything before it leaves the process boundary, and store history where nobody can quietly edit it. Get that right and the tool earns the one thing that matters for adoption — the next PR author trusting the red X enough to stop and look instead of clicking past it.