A blog about learning, agile product development and software testing.

LLMs have made bug discovery nearly a commodity capability. What was groundbreaking a year ago is now standard, and security teams are finding more true positive vulnerabilities than they can realisti...

Welcome to the Open Security Conference (osco), the people-centred international gathering for everyone interested in cybersecurity. Join us 2-5 October 2025 in Rückersbach, Germany.

Choosing between AppSec and ProdSec? Explore the nuances of application security, from secure coding to automating DevSecOps pipelines. Learn how to protect software and prevent malicious executables.

At 4:11 AM UTC on November 24th, a number of our SDKs and other packages were compromised, with a malicious self-replicating worm - Shai-Hulud 2.…

In our last post, we shared how we began to scale our security code review process with SAGE. We discussed how we gathered historical Product Security (ProdSec) review records to create a 1Password-specific ruleset, the three-stage Finder/Critic/Judge pipeline, and the limitations of our v1 implementation. Above all, human ProdSec reviewers still had to bring full context to the findings: where the trust boundaries lie, which directories are sensitive, and whether mitigations exist elsewhere in the codebase. Our goal for v2 was to help SAGE understand our entire codebase. Many of our GitHub repositories are _huge_ , including our client and server monorepos. That means we have way too much information to fit within any LLM’s context window. We had to find a way to let SAGE perform deeper reasoning about the PR diffs it reviews without the codebase itself. There was another hurdle. As we built v2, we ran into a fundamental LLM trait: they can’t reliably produce the same output twice. We knew we had to do our best to manage this nondeterminism so we could trust SAGE to be a relatively consistent security reviewer. We had two things to figure out: how to fit a lot of data into a context window, and how to get consistent output from inherently inconsistent tools. If we could solve those riddles, SAGE wouldn’t just know 1Password, it would finally understand it. And it would earn the name SuperSAGE. ## Compressing context with scaffolding As it turns out, our Security Research team had already developed a Python proof of concept designed to compress our code context. It was a set of LLM prompts that generated one SCAFFOLDING.md file per source directory. Those scaffolding files carried compressed structural context like sensitivity ratings, attack surfaces, trust boundaries, and file summaries. It was a great foundation; we just had to productionize it as a Go rewrite on top of SAGE v1’s model-agnostic llm.Client harness. To start, the PoC took inventory of our code structure. Any well-organized codebase is shaped like a tree: a root that branches into directories and subdirectories, all the way down to individual files. The PoC walked that tree once from the bottom up, so by the time it reached any given directory, everything beneath it had already been analyzed. This is a classic map-reduce pattern: each directory was summarized on its own (the map), and those summaries were folded upward, child into parent, all the way to the root (the reduce). This worked well for a point-in-time snapshot of our codebase, but 1Password has hundreds of hard-working engineers, so there are sections of our code that change _every day._ On the other hand, other sections, like our cryptography layer, are trusted, well-vetted, and rarely touched. Re-running that full bottom-up walk every night, across a codebase the size of ours, would be slow and expensive. So we chose to make it incremental: only regenerate scaffolding for directories in which the code had _actually_ changed. But skipping unchanged directories is only half the picture. When a directory's code has changed and its scaffolding needs to be regenerated, the model hands back **new** prose every __time, even if nothing more than a line of whitespace was added. An LLM rarely, if ever, describes the same code the same way twice. Had we kept folding each directory's model-written summary into the one above it, those harmless rewordings would have piled up: a rephrased child summary would make its parent look changed, and that parent its own parent, all the way to the root, leaving us regenerating the whole tree every night to chase code changes that weren’t semantically different. It left another problem sitting in front of us, even for a single directory. We'd been letting the model's wording alone decide what counted as a change. We needed a way to answer one question without depending on the prose at all: Did this directory _truly_ change? There it was: the nondeterminism problem. ## Working with LLM nondeterminism The easiest fix for inconsistent LLM output is to force temperature=0, which helps reduce randomness from the generation. But two of the three models in our SAGE pipeline don’t support that setting at all, so we had to solve our nondeterminism issue architecturally. At first we explored an alternative to plain-text comparison. Instead of checking if the new summary’s _text_ was identical to the original, we considered checking if the new summary’s _meaning_ was close enough to the original. It sounded good on paper, but ultimately we rejected the idea because a genuinely important security change (like adding a new endpoint that’s reachable from outside the trust boundary) might only shift the similarity score a small amount, and would be indistinguishable from ordinary LLM-rephrasing noise. For a security tool, silently missing that kind of change is worse than being too cautious. Instead, we narrowed what SAGE v2 considers a change. Rather than comparing everything within a directory's SCAFFOLDING.md file, we only track the SHA256 hashes of a small, fixed set of structural facts — things like the files in the directory, their sensitivity ratings, and trust-boundary designations. If any of _those_ hashes shift, we treat it as a real change worth propagating throughout the scaffolding. The written analysis generated for humans is intentionally left out of the comparison because it's the piece most likely to come back worded differently from one scan to the next without any meaningful changes. We made one other deliberate choice: The LLM never gets to decide what counts as a change, either. The hashes themselves are computed by our Go harness, which is completely deterministic. The LLM no longer needs to infer what has changed, so we’re not asking the model to grade its own homework. Long story short, we let the structure decide, not the prose. A new file in a directory is a real signal; a reworded summary of the same code is just noise. And by eliminating that noise, SAGE keeps its understanding of our code fresh. ## Teaching SAGE what matters: The next chapter Implementing context compression and solving for model nondeterminism are what let SAGE go from reviewing diffs in isolation to reasoning about the code around them: where a change lives, why __it matters, and whether a risk is real. And it can keep that knowledge current, day after day, without unnecessary costs incurred by LLM drift. The results speak for themselves. SAGE now distills a directory’s raw source into a security summary a fraction of its size — often 30 times smaller — and refreshes that map every night for a few dollars, re-deriving only the small slice of code that actually moved while skipping the rest for free. It has _earned_ the name SuperSAGE. We’ve come a long way. But we still have work to do. To this point, SAGE has only scanned select PRs: those voluntarily tagged with the sage-review label and those assigned to the ProdSec team for review. The next step is to roll it out to every PR across all repositories. But scaling a tool is its own test. A reviewer that can't tell a real finding from a false positive is manageable on a handful of PRs, and a liability on _all_ of them. Before we have SAGE scan every PR throughout the organization, we need to teach our AI-assisted reviewer which of its findings actually matter. Which means SAGE needs to _learn_. And that’s exactly where we’re headed next. Stay tuned for Part 3. ### Want to work with 1Password? Our Security team is hiring. If this work sounds exciting to you, we [encourage you to apply](https://jobs.ashbyhq.com/1password).

What Al-generated apps get wrong: a vulnerability study across models, vendors, languages, and a real-world app