A blog powered by Hashnode

Deploying with an AI agent? Pick the region near your users first. A checklist with a host-by-region table for Europe, Latin America and Asia.

https://losthills.com.co/

Bridging the gap between development and leadership roles. Read the latest articles by Dev Leader on Hashnode.

Hashnode는 개발자가 마크다운으로 글을 쓰고 커스텀 도메인을 연결해 기술 블로그를 운영할 수 있도록 돕는 블로깅 플랫폼입니다. 이 글은 Hashnode의 특징과 장단점을 먼저 정리한 뒤, Medium, Dev.to, Ghost, WordPress, Substack을 대안

If Claude Code weakens a test in the same commit that breaks the code, my test gate passes that commit. Codex is told to look for weakened and deleted tests when it reviews, and the only hard check in the scripts is for a run where zero tests ran. Kartik pointed out this weak spot under my Hashnode post on how Claude Code and Codex review each other's work. I gave him the short answer in the comments. This post is the full answer, with the commands I ran to check it. ## Why the tests in the same commit are the weak spot two-engines-gate is the free, MIT licensed version of how I make Claude Code and Codex check each other. `gate.sh` runs your tests and writes a pass record named after the commit, `review.sh` gets the other engine to review it, and `merge-check.sh` refuses the merge unless the commit has both. I wrote about the review loop and the commit hash records before. `gate.sh` runs whatever tests are in the commit. When a commit changes the code and the tests together, the same commit also decides what a pass means. I checked this on 23 October with the public scripts and a small Python project. It had an `add` function and two `assertEqual` tests. First I broke `add` so it returned `a + b + 1`, and the gate failed as it should: FAILED (failures=2) gate: FAIL (exit 1) on 8390173080fb; no pass record; full log .ai/tests/8390173080fb.log Then, in the next commit, I changed the two tests to `assertTrue(add(2, 3))` and `assertIsNotNone(add(-1, 1))`. Both hold for the broken function, so the gate wrote a pass record: gate: running 'python3 -m unittest -q' on 266c4d979b71 gate: PASS on 266c4d979b71, record .ai/tests/266c4d979b7102820377e64ab3f20b069e41bf01.pass The test command exited 0 and two tests still ran, so `gate.sh` had nothing to refuse. Catching this falls to the reviewer. ## What the reviewer is told The free `review-prompt.md` asks for a static review. Codex reads the commits, the diff and any file in the checkout, and runs no tests or builds. This is the line that covers tests: > Check in particular: behaviour the commits promise but the diff does not deliver, error paths and edge cases, tests that would still pass on the old code, and anything that weakens an existing check. The same prompt says a blocker or should-fix needs "a concrete, reproducible failure". For a weakened assertion that failure is easy to write down. Call `add(2, 3)` on the new code, get 6, and watch the new test pass anyway. A reviewer that reads the test diff next to the code diff has everything it needs to flag it. In my own projects the prompt Codex gets is stricter. Every test change has to fail on the old behaviour, nothing can be weakened without a stated reason, and the reviewer has to check the diff for any weakened or deleted test. A loosened assertion with no reason comes back as CHANGES REQUIRED. The README tells you to edit `review-prompt.md` to add project rules, so you can put the same three rules in your copy. That is still the reviewer's judgement. The README says so directly: "A review can miss things, and tests only check what they test." If Codex misses the weakened test and answers ACCEPT, `merge-check.sh` finds a pass record and an ACCEPT for the commit and lets it through. ## The hard check: a run with zero tests One case is checked by the script itself. `gate.sh` refuses a pass when the test command prints nothing, or when the test summaries it knows add up to zero tests that ran. It reads the summary lines of pytest, unittest, `node --test`, mocha, Jest and Vitest. That covers deleting every test, a filter or file list that selects nothing, and, for every runner except unittest, a run where every test was skipped. In my Python project, deleting the test file made unittest exit 5 with "NO TESTS RAN", so the gate already failed on the exit code. To see the zero-test check on its own, I ran the same command with `|| true` so it always exited 0: gate: the test summaries in the output show that no test ran; set GATE_COUNT_RE or GATE_MIN_TESTS (README) gate: FAIL (exit 1) on f93de2d8e8a6; no pass record; full log .ai/tests/f93de2d8e8a6.log Deleting one of the two tests is a different case. One test still ran, so the default check passed the commit. If you want the gate to notice a drop in the count, set `GATE_COUNT_RE` to a pattern for the count line your runner prints, and `GATE_MIN_TESTS` to the number of tests you have now. With `GATE_COUNT_RE='^Ran ([0-9]+) tests? in '` and `GATE_MIN_TESTS=2`, the same commit failed: gate: the output reports 1 tests, fewer than GATE_MIN_TESTS=2; set GATE_COUNT_RE or GATE_MIN_TESTS (README) gate: FAIL (exit 1) on 1ea182ad01af; no pass record; full log .ai/tests/1ea182ad01af.log The minimum is set where you run the gate, outside the commit, so you raise it by hand when you add tests. go test and cargo test need `GATE_COUNT_RE` before any count check works for them, and the README has an example for each. `GATE_MIN_TESTS=0` turns the check off. ## A habit that covers new tests New tests get one more step, which I described in the post on how I make Codex review Claude Code's commits. The engine that writes the code often writes the tests too, and a test that already passes on the old code proves nothing about the change. So the writer runs each new test once against the base commit and pastes the failing output into the commit body. That does nothing for an old test that got weaker, which is why the prompt rule matters. ## What isn't built A script that reads the diff and flags removed or loosened assertions does not exist. I told Kartik it would be a good addition and that I'd look at it. Today neither the free gate nor the kit has one, so a single weakened assertion gets caught only if the reviewer notices it. ## What to set up in your project 1. Add the test rules to your `review-prompt.md`: new tests must fail on the old behaviour, and no test gets weakened or deleted without a stated reason. 2. Leave the zero-test check on, and set `GATE_COUNT_RE` if you use go test or cargo test. 3. Set `GATE_MIN_TESTS` to your current count if a deleted test should fail the gate. 4. Have the writing engine run each new test against the base commit and paste the output into the commit body. ## The same rule in the kit The paid Two Engines Kit (currently v1.0.2) ships a review prompt template whose project rules include "tests must fail on the old behaviour; nothing weakened without reason", and its list of risks to check includes "any weakened test". The Two Engines Kit is built on information gathered since the first Codex and Claude CLIs. ## Try it Clone two-engines-gate and run `bash demo.sh` to watch one full loop with a stand-in reviewer, so no model or account is used. `tests/run.sh` has the zero-test cases for all six test runners, plus the `GATE_COUNT_RE` and `GATE_MIN_TESTS` cases.

Gamify your goals and conquer your peaks. A React Native mobile app for long-term goal tracking and weekly time-blocking, featuring Local Calendar sync and visual progression. - ClimbStack/climbsta...

MisarBlog vs Hashnode in 2026: which is the best free custom-domain blog for developers? Compare SEO, monetization, AI tools, and developer features.

Dev.to vs Hashnode in 2026: which is the best place for developers to blog? Compare audience, SEO, custom domains, monetization, and community features.