We put Google's search on 484,464 German court decisions and measured it

Google's two off-the-shelf options for searching a legal corpus — Programmable Search (CSE) and Vertex AI Search — measured against our own retrieval stack on real user queries, on 2,941 held-out case-law questions, and on 116 landmark research questions: we win two of three, and the loss taught us the most.

• On real user queries, our engine answers 91% in the top 5; Vertex AI Search over the same corpus answers 77%; Google's web index manages 32%.
• On 2,941 held-out case-law questions, our production API finds the correct decision 2.4 times as often as Vertex AI Search: recall@10 0.659 vs 0.275, both measured live against the serving systems.
• On 116 landmark research questions Vertex edges us out at rank 10 (0.552 vs 0.517), and the two engines find different decisions: combined they would reach 72%. That fusion is on our roadmap.

Why we ran this

Our search over recht.nulegal.eu — every German federal statute plus, at benchmark time, 484,464 court decisions — is built in-house: hybrid lexical + dense retrieval with a fine-tuned German-legal query encoder, citation-graph priors, and an LLM rerank path for natural-language questions. The obvious question any team should ask itself: would off-the-shelf Google do just as well?

We tested both ways Google can sit on top of a corpus like ours:

1. Google Programmable Search (CSE) restricted to our public site: Google's web index of the corpus.
2. Vertex AI Search with our full corpus — 584,465 documents: 100,001 statute sections with body text (§/Art./Anlage provisions rather than heading-only rows) plus the 484,464 decisions — pushed into a datastore: Google's retrieval engine, with no index-coverage excuse.

Setup

Three evaluation sets, none synthetic:

• Organic probe: 119 real user queries from our search logs, stratified by the query styles people actually type (citations, keywords, lay questions, docket numbers, typos). Judged by Gemini 2.5 Pro at temperature 0, with a second pass of the same model at temperature 0.7 as a stability cross-check (agreement on the headline metric: 0.90–0.92). The judge is Google's own frontier model, a deliberate choice: if the judging leans anywhere, it leans toward the Google baselines, not us. Metric: answered@5 — would the searcher's information need be satisfied by the top-5 results?
• case_eval: 2,941 held-out questions, each with one known-correct decision from 2026. Judge-free: the gold decision is either in the top-k or it isn't.
• landmark-v2: 116 verified research questions, each targeting one landmark decision (our permanent holdout set).

Results

¹ CSE cannot be scored on case_eval: the gold decisions were largely absent from Google's index of our young site, so a recall number would measure crawl coverage, not retrieval.

² The Vertex case_eval run had 40 of its 2,941 queries fail to fetch (retries exhausted). The 40 are counted as misses, so 0.275 is a floor: had every failed query been a hit, the ceiling would be about 0.289. The failures are logged as fetch_errors in the published run artifact, results/run_vertex_case_eval_20260727.json in the research repository.

³ Offline measurement against the production dense retriever. The live full-scale run of the production API on the same questions measured 0.659 — see the method notes; the summary above quotes that live-vs-live pair.

⁴ CSE returns at most 10 results per query, so recall@20 cannot exceed recall@10.

The CSE number is an index-coverage story, not a retrieval story. At measurement time (July 2026) the site was weeks old and Google's crawl covered a small fraction of it: 74 of the 116 landmark questions returned zero results. As of that measurement, "just put a Google search box on it" answered one query in three.

Vertex AI Search is a serious baseline, and it still loses on what users actually type. With the full corpus indexed, it answers 77% of real queries against our 91%, and it trails badly on precise case-law lookup: 0.275 vs 0.610 in the table's offline measurement, 0.275 vs 0.659 live against live. Domain-tuned retrieval — the citation graph, the fine-tuned encoder, German legal query handling — earns its keep exactly where precision matters.

The interesting result is the one we lost. On long, doctrine-heavy research questions, Vertex's semantic retrieval finds landmark decisions our stack misses at rank 10. The headline gap (0.552 vs 0.517) is four questions out of 116, well within sampling noise at this size; what makes the result real is the diff, not the delta. The diff is also what makes it actionable: of 116 questions, both engines find 40, only ours finds 20, only Vertex finds 24, neither finds 32. The engines are complementary — a fused candidate pool would lift recall@10 from 52% to a 72% ceiling. That fusion experiment is on our roadmap.

Method notes

• The case_eval pair is same questions, same gold decisions, same metric; the measurement paths differ. Vertex was run against its live API. The 0.610 in the table was measured offline against the production dense retriever — the same encoder and index the live system serves — without the serving layer (reranking, blending) on top. A subsequent full-scale live run of our production API on the same 2,941 questions (27 July 2026) measured recall@10 = 0.659: live against live, our engine finds the correct decision 2.4 times as often as Vertex AI Search, and that pair is the honest headline comparison. The landmark-v2 rows are both live-API runs.
• Judged runs follow our standing rules: a strong judge, structured output, no cherry-picking. The cross-check on judged metrics is two passes of the same judge model at different temperatures — Gemini 2.5 Pro at 0 and at 0.7. That is a stability check on one judge, not two independent judges.
• The Vertex datastore held a same-day snapshot of exactly the documents our engine searches, so corpus coverage is identical by construction.
• Two judge calls on the organic probe were arguably generous to Google (accepted answers on shaky rationales); we count them anyway. The gap doesn't depend on them.
• Costs, for anyone repeating this: the entire exercise (datastore, 10.6 GiB import, ~6,200 queries across all runs) cost under 30 euros in API fees.

@techreport{nulegal_google_search_vs_our_retrieval,
  title  = {We put Google's search on 484,464 German court decisions and measured it},
  author = {Bork Morfaw},
  year   = {2026},
  institution = {nu:legal research},
  url    = {https://recht.nulegal.eu/forschung/google-search-vs-our-retrieval}
}