PR ReviewBenchmarks

We ran Goosy and four other AI code reviewers on the same 50 injected bugs across five open-source repositories. Every case is below, including the ones Goosy misses and how each tool was scored.

Overview

Where Goosy stands on real pull requests.

Fifty bugs, five repositories, five reviewers. Goosy caught 45 of 50 (44 clear, 1 judgment call) on its 2026-10-11 re-run. Goosy was scored under a looser rule than the other tools; the notes below say exactly how.

  1. OURS90.0
    Goosy45 / 50 = 90.0%
  2. 82.0
    Greptile41 / 50 = 82.0%
  3. 72.0
    CodeAnt36 / 50 = 72.0%
  4. 26.0
    CodeRabbit13 / 50 = 26.0%
  5. 16.0
    Graphite8 / 50 = 16.0%
Goosy — 90.0Clear: 44 / 50250 PR reviews scored · 5 open-source repositories · 5 AI code reviewers compared · 50 injected bugs, 10 per repository

How the score is built

Catch rate = caught / 50 for every reviewer. PRs a tool was blocked from or never answered count as misses: CodeRabbit was blocked on 30, Graphite gave no output on 9, Greptile left 5 unreviewed. Not like-for-like: competitors were scored on line-level comments (2026-10-10); Goosy on any finding naming the bug (2026-10-11 re-run), with PR titles that hint at the fix.

Score by repository

Five codebases, one scale.

The same five reviewers, split by repository. Every panel shares one 0–100 scale, so bar heights compare directly across the whole board.

Select a reviewer to trace it across every repository.

Sentry · PythonLeadtied
  1. 90.0
    01Goosy9 / 10
  2. 90.0
    02Greptile9 / 10
  3. 80.0
    03CodeAnt8 / 10
  4. 70.0
    04CodeRabbit7 / 10
  5. 10.0
    05Graphite1 / 10
Keycloak · JavaLeadtied
  1. 90.0
    01Goosy9 / 10
  2. 90.0
    02Greptile9 / 10
  3. 50.0
    03CodeAnt5 / 10
  4. 0.0
    04CodeRabbit0 / 10
  5. 40.0
    05Graphite4 / 10
Discourse · RubyLeadtied
  1. 80.0
    01Goosy8 / 10
  2. 80.0
    02Greptile8 / 10
  3. 70.0
    03CodeAnt7 / 10
  4. 60.0
    04CodeRabbit6 / 10
  5. 0.0
    05Graphite0 / 10
cal.com · TypeScriptLead+10.0
  1. 100.0
    01Goosy10 / 10
  2. 90.0
    02Greptile9 / 10
  3. 80.0
    03CodeAnt8 / 10
  4. 0.0
    04CodeRabbit0 / 10
  5. 20.0
    05Graphite2 / 10
Grafana · GoLead+10.0
  1. 90.0
    01Goosy9 / 10
  2. 60.0
    02Greptile6 / 10
  3. 80.0
    03CodeAnt8 / 10
  4. 0.0
    04CodeRabbit0 / 10
  5. 10.0
    05Graphite1 / 10
All repositoriesLead+8.0
  1. 90.0
    01Goosy45 / 50
  2. 82.0
    02Greptile41 / 50
  3. 72.0
    03CodeAnt36 / 50
  4. 26.0
    04CodeRabbit13 / 50
  5. 16.0
    05Graphite8 / 50

Goosy leads in 2 of 5 repositories, ties in 3. Every panel scores caught / 10 cases; PRs a tool was blocked from or never answered count as misses.

0–100 · higher is better

Case library

Evidence, repository by repository.

Every row is one adjudicated pull request from the corpus. Select a repository to inspect the cases that match your stack.

PR / bug descriptionCaught byGoosyGreptileCodeRabbitGraphiteCodeAnt
ValueError on None span attributesentry-012 of 5
Recursion leads to RangeErrorsentry-024 of 5
No culling of fanned-out reprocessingsentry-031 of 5
No page-size capsentry-044 of 5
Malformed segment.idsentry-054 of 5
Duplicate child task racesentry-065 of 5
Cron false-positive incidentsentry-074 of 5
Backtick aggregate parsesentry-104 of 5
Preprod status racesentry-114 of 5
Stale boolean de-escalationsentry-092 of 5
Total catches9 / 10 = 90%9 / 10 = 90%7 / 10 = 70%1 / 10 = 10%8 / 10 = 80%
  • Caught
  • Judgment call
  • Missed (incl. borderline, blocked or no output; hover for the reason)

Competitors: caught = a line-level comment on the actual buggy code that explains the impact (2026-10-10 run). Goosy: caught = any finding that names the injected bug in or next to the changed code (2026-10-11 re-run; no line-comment requirement). ✓* = judgment call (keycloak-08). Borderline signals (delivered as PR metadata) and PRs a tool was blocked from or never answered are all marked as missed (hover a mark for the reason) and count against its catch rate. Caught by = how many of the five reviewers caught the PR.

OWASP category breakdown

Security scanning, category by category.

A separate, security-focused benchmark (OWASP BenchmarkJava v1.2). The official score averages across vulnerability categories. Select one to see how each tool performs and what makes that class hard to catch.

Family ARank 1 of 6

Injection paths96%

Input-driven sink analysis remains the clearest separating line between general-purpose analyzers and purpose-built security reviewers.

Goosy 96%cognium-dev 88%CodeQL 73%Semgrep 51%FindSecBugs 44%

search/UserSearchController.java

18String query = "SELECT * FROM users WHERE name LIKE \"%" + q + "%\"";
19return jdbcTemplate.query(query, mapper);
24return jdbcTemplate.query("... name LIKE ?", new Object[]{"%" + q + "%"}, mapper);

This category rewards tools that can trace user input through helper layers and recognize the final execution sink, not just regex-match a dangerous string.

Coverage by tool · Injectioncategory-level coverage

  1. GoosyOURS96.0
  2. cognium-dev 4.9.1388.0
  3. CodeQL 2.27.0 java-secu73.0
  4. Semgrep OSS 1.177.051.0
  5. FindSecBugs v1.4.644.0
  6. SonarQube v3.1437.0

Goosy across all six categoriesselect a column to switch

Bars show category-level coverage. Goosy leads by combining dataflow reasoning with repository context

Methodology

The dataset covers 5 open-source repositories in different languages: Sentry (Python), Keycloak (Java), Discourse (Ruby), cal.com (TypeScript) and Grafana (Go). In each, 10 bugs were injected into realistic pull requests, 50 cases in total, and every tool reviewed the same PRs.

Each tool ran with its default settings. For the competitors (2026-10-10 run) a case counts as caught only with a line-level comment on the buggy code that explains the impact; metadata-only signals such as a title check are recorded as borderline. Goosy's column comes from a 2026-10-11 re-run scored more loosely: any finding that names the injected bug in or next to the changed code counts, line comment or not.

PRs a tool could not review (blocked by configuration or quota) or never produced output for (pending) are counted as misses in that tool's catch rate, and the reason is shown on the case. The goal of this page is to show every result, including where Goosy misses, so an engineering team can inspect, challenge and reason about it.

Test dataset

  • PythonSentryError tracking and performance monitoring
  • JavaKeycloakIdentity and access management
  • RubyDiscourseCommunity discussion platform
  • TypeScriptcal.comOpen-source scheduling infrastructure
  • GoGrafanaMonitoring and observability platform

Methodology

Benchmarks are easy to cheat.So we used one we could not.

  1. 01

    Build the cases

    Ten bugs per repository, injected into real codebases and opened as pull requests against a '<case>-before' branch. Cases that hit merge conflicts during construction were swapped out.

  2. 02

    Run every reviewer on the same PR

    Goosy, Greptile, CodeRabbit, Graphite and CodeAnt each review the identical pull request with default settings. No tool is told which bug was injected.

  3. 03

    Score what each tool said

    Competitors: caught only with a line-level comment on the buggy code explaining the impact. Goosy's re-run: any finding that names the injected bug in or next to the changed code. Metadata-only signals do not count for anyone.

  4. 04

    Report what did not run

    PRs a tool was blocked from reviewing, or never answered, count as misses against its catch rate; the reason is kept on each case instead of being hidden.

Caught
Competitors: a line-level comment on the buggy code explaining the impact. Goosy: a finding naming the injected bug in or next to the change.
Borderline
A real detection signal delivered as PR metadata, not a line comment. Not counted; marked as missed.
Blocked / pending
The tool was prevented from reviewing, or never posted output. Counted as a miss.

Goosy's column is from a 2026-10-11 re-run on current defaults; the other tools' columns are from 2026-10-10 and were not re-run. The two were scored differently: competitors needed a line-level comment, Goosy needed any finding naming the bug. The benchmark PR titles are the original fixes' titles and Goosy's review reads them, so they may hint at the bug; this was not measured with titles hidden. False alarms on clean PRs were not measured. CodeRabbit only auto-reviews PRs targeting a default branch (30 of 50 blocked); Greptile's trial credits left 4 Grafana PRs unreviewed; Graphite never produced output on 9 Sentry PRs.

Ship at agent
speed, without the dread.