GitHub's secret-scanning service has been quietly running on regular-expression patterns for years. It looks at every commit pushed to public repositories, matches strings against patterns for known credential formats — AWS keys, Stripe secrets, GitHub tokens, database connection strings — and raises an alert when something matches.
The system works, in the sense that it catches real secrets. It also produces a steady stream of false positives. Developers learn to ignore the alerts. The worst possible outcome for a security tool is to be ignored.
This week, GitHub published a post describing how it added large-language-model verification to the scanner and significantly reduced false positives without losing detection accuracy. The architecture is worth studying, because the same pattern shows up everywhere ML meets infrastructure scale.
Why pure pattern matching was failing
The fundamental problem with regular expressions is that they have no context. A regex for an AWS access key matches anything that looks like AKIA followed by sixteen alphanumeric characters. That pattern fires in three very different situations:
- A real production AWS key, accidentally committed.
- A fake example value in documentation or a test fixture.
- A randomly generated string that happens to match the pattern.
The regex cannot tell them apart. To the pattern matcher, all three are the same.
At GitHub's scale — millions of commits per day across hundreds of millions of repositories — the noise compounds fast. If even one in a hundred matches is a false positive, the absolute number of bad alerts overwhelms the good ones. Security teams stop responding. Developers stop reporting. The signal-to-noise ratio collapses.
The obvious fix is to write smarter regexes. The less obvious fix is to admit that regex alone cannot solve a problem that requires understanding context.
Why pure LLM scanning was not the answer either
The seductive alternative is to throw an LLM at every diff. Feed each code change into a model that has been trained to recognise secrets, and let it decide.
This does not work at GitHub's scale. The reasons are practical:
- Cost. Even at favourable per-token pricing, scanning billions of commits per year through a large model becomes a budget line nobody is willing to sign.
- Latency. Secret scanning needs to flag credentials before they propagate. An LLM inference that takes seconds is too slow for a system that needs to run on every push, in real time.
- Hallucinations. LLMs occasionally invent secrets that are not there, or fail to recognise patterns they should. For a security-critical system, neither failure mode is acceptable.
So pure LLM scanning is too expensive, too slow, and too unreliable on its own. Pure regex is too noisy. Neither approach by itself is viable.
The hybrid architecture GitHub actually built
GitHub's solution is a two-stage filter. Each stage does what the other cannot.
The first stage is the existing regex matcher. It runs on every commit, against every file, looking for candidate secrets. It is fast, cheap, and slightly paranoid — designed to over-match rather than miss anything real. This is the candidate generator.
The second stage is the LLM verifier. It only fires on the small subset of candidates the regex produced. For each candidate, it examines the surrounding context: the variable name the secret is assigned to, the comments above and below, the file path (is this a test file? a documentation file?), neighbouring function names, and whether the string appears in a context that implies a real credential or a placeholder.
The model is asked one question: given this context, is this candidate a real secret? Its answer determines whether an alert fires.
| Stage | Approach | Strengths | Weaknesses | Volume |
|---|---|---|---|---|
| Stage 1 | Regex / pattern | Fast, cheap, exhaustive | No context, high noise | Every commit |
| Stage 2 | LLM verifier | Context-aware, accurate | Slow, expensive per call | Only stage-1 matches |
The economics work because the second stage operates on orders of magnitude fewer inputs than the first. The bulk of GitHub's commits never reach the LLM. Only the candidates the regex already flagged are paid for.
Why this pattern generalises
This is the same shape that already shows up in mature ML pipelines across the industry. Spam filtering uses cheap heuristics to flag candidates, then expensive ML to verify. Fraud detection uses rule-based flags to narrow the search, then deep models on the narrow set. Content moderation runs cheap classifiers first, then human or expensive-model review on the survivors.
Think of it like a smoke detector that, before sounding the alarm, calls a person to look at the kitchen. The smoke detector is cheap, fast, and slightly paranoid — exactly what you want at the entry point. The person is slow and expensive but has context. Each does the job the other cannot. Neither alone would work as a household alarm.
The cheap stage exists to make the expensive stage affordable. The expensive stage exists to make the cheap stage trustworthy.
The reason this pattern keeps reappearing is that no single technique is both cheap and accurate. Cheap techniques are noisy. Accurate techniques are expensive. Layering them inverts the cost equation: you pay accuracy prices only on inputs that have already passed a cheap filter.
What this means for builders
If you are building ML-augmented infrastructure of any kind, four implications follow.
Default to hybrid architectures, not pure ones. "We rebuilt our system to use AI" is rarely the right answer. "We added an AI verification stage to the noisy candidate generator we already had" almost always is. The hybrid version is cheaper, faster, and more reliable.
Engineer the cheap stage to over-match. The cheap stage should be slightly paranoid, optimised for recall over precision. Its job is to feed candidates to the expensive stage, not to be right. If the cheap stage is too strict, real signals get filtered out before the expensive stage can recover them.
Budget the expensive stage by volume, not per call. The cost question is not "how much does one LLM call cost." It is "how many calls does the cheap stage send to the expensive stage, in expectation." Tune the cheap stage's threshold to control downstream cost.
Treat the verification stage as a model boundary, not a black box. GitHub can swap its LLM in stage two without changing stage one. That modularity matters as model quality and pricing change. If your verification stage is fused into your candidate generator, you cannot upgrade either independently.
Conclusion
GitHub's secret-scanning improvement is a small story on the surface — a security tool got more accurate. Underneath, it is a clean example of a pattern that is becoming the default for infrastructure-scale ML: cheap candidate generation feeding expensive verification, with neither stage trying to do the other's job.
The hybrid pattern is not a compromise. It is the only architecture that survives both the cost constraints and the accuracy constraints at scale. The teams that internalise this early will design more durable ML systems than the teams that keep looking for one model to do everything.