Codebase Copilot

Context and problem

Joining an unfamiliar project means finding where the business rules live, how a flow crosses the layers and which modules do what. Asking a language model without controlled context produces answers that sound convincing but cannot be checked.

Codebase Copilot indexes the repository on the user's machine, retrieves the relevant excerpts for each question and only accepts an answer whose citations point to excerpts the model actually received. Every claim can be checked in the code with one click. When there is not enough evidence, the tool says it does not know.

Why RAG

A language model answers with what it learned during training. It has never seen your repository, so any answer about it is a well-written guess.

RAG (Retrieval-Augmented Generation) changes the question the model answers. Instead of "what do you know about this code?", it becomes "based on these excerpts, what can be stated?". For every question, the system:

1. retrieves the most relevant excerpts from the repository;
2. hands those excerpts to the model, numbered as evidence;
3. requires every claim to cite the evidence it came from.

It is the difference between a closed-book exam and an open-book exam in which every answer must point to the page in the book. The model still writes the text, but the source of truth becomes the code itself. In this project, every step (indexing, retrieval, context assembly and citation validation) is implemented explicitly, tested and measured.

Architecture

The first row runs once per version of the repository. The second runs for every question.

• Import: the application only reads folders inside the authorized roots, or extracts a ZIP with limits against zip bombs and path traversal.
• Pre-analysis: before indexing, it shows what is included, the languages, the size and what was excluded (dependencies, generated artifacts, .env, keys and binaries).
• Indexing: files become chunks with path and lines, identifier terms and vectors, stored in an immutable snapshot. Reindexing without changes reprocesses nothing.
• Hybrid search: three channels. The lexical one uses an identifier tokenizer (getUserById → get user by id). The vector one uses pgvector. The path channel finds files mentioned by name.
• Fusion and context: rankings are combined with Reciprocal Rank Fusion (k = 60) and the selected excerpts fit into a token budget, labeled E1…En.
• Citation validation: the model answers in a strict structured format, and every citation is checked against the evidence and lines it received before it reaches the screen.

The backend is a modular monolith in Java 25 and Spring Boot 4.1, with workspace, import, indexing, retrieval, answering, provider and history modules. The frontend is React 19 with Vite and TypeScript, and the API types are generated from the OpenAPI contract.

Technical decisions and trade-offs

Modular monolith, not microservices

Less operational complexity for a product that runs on the user's machine. Modules and internal ports make it possible to extract a worker later, if measurements show the need. Indexing uses a table-backed queue with SKIP LOCKED, heartbeats and a retry limit: an interrupted job resumes and reaches the same result as a clean run.

Swappable providers

Embeddings and generation sit behind internal ports (EmbeddingProvider, GenerationProvider). The official openai-java SDK only exists inside the providers module, and an architecture test (ArchUnit) ensures its types do not leak. Spring AI was used at first and then removed: its chat model confused missing tokens with zero and could log the prompt.

Local-first

With providers turned off (none, the default), nothing leaves the computer. With a remote provider, only the excerpts selected as evidence and the question text are sent, never excluded files.

Token budget

Context is assembled in fusion order: up to 8 pieces of evidence by default, up to 3 excerpts per content, without overlapping lines already chosen and within 6,000 estimated tokens in the final prompt. An excerpt that does not fit is skipped and the next ones are still tried. There is no reranking: it only comes in if evaluation shows it is worth the cost and latency.

Explicit SQL in retrieval

Spring Data JDBC for small aggregates and JdbcClient with explicit SQL for indexing and search, without JPA. Hybrid ranking has to be observable and testable, and generic abstractions would hide that behavior.

Security and privacy

The imported repository is treated as untrusted content: it may contain secrets, malicious code, prompt injection instructions, huge files or malformed paths.

• No imported code is ever executed, at any step. The importer reads text and never runs scripts, build tools or hooks.
• Restricted reads: paths are canonicalized and validated inside the root, symbolic links are not followed, and there are limits on count, depth and size. ZIPs are checked against path traversal, absolute paths and excessive expansion, counting the bytes actually written.
• Secrets: sensitive files are excluded by name, under rules that cannot be turned off, and by content (private key blocks and keys in published formats), before any chunk, embedding or upload.
• Prompt injection: each piece of evidence goes into a block delimited by a random 128-bit nonce per request, so a file cannot close its own block. Generation has no tools, the output is strict JSON and the backend decides the final status.
• API bound to the local machine: the backend listens on 127.0.0.1, rejects other hosts and origins and sends a strict Content Security Policy.
• Logs without content: API keys, code and prompts never appear in logs or error messages.

How I measured quality

Evaluation exists before any prompt optimization or model change. The corpus has 80 questions about four repositories (a synthetic fixture, Codebase Copilot itself, pallets/click and honojs/hono), half in Portuguese and half in English. Fourteen require abstention, and another fourteen test instruction injection or secret leakage.

Custom harnesses measure:

• Retrieval: Recall@k, MRR@10, Precision@10 and file coverage for questions that cross modules.
• Answers: citation validity, answers supported by at least one valid citation, correct and unwarranted abstention, and leakage of forbidden text.
• Operations: tokens, estimated cost and mean and p95 latency.

The first published baseline is the lexical channel, over 66 questions with an expected range. It became a no-regression floor: no change is promoted if results fall below it.

The performance targets, measured with simulated providers, were met:

The comparison of the vector and hybrid channels and the evaluation of answers with the real model have not run yet. They come out of the same harness, with no new code. Until then, two guarantees hold by construction and by test: no invalid citation is silently accepted, and no excluded content is sent to the provider.

Screens

• 

• 

• 

• 

Next steps

• Run the evaluation of the vector and hybrid channels and of the answers with the real model, followed by a human audit, and set targets from that measurement.
• Structural analysis (AST) adapters for priority languages.
• Snapshot comparison.
• Architecture map export.
• Local model provider.

Links

• Repository on GitHub ↗ (opens in a new tab)
• Technical documentation ↗ (opens in a new tab): PRD, architecture, RAG pipeline, security, evaluation and the decision log.