Post cover image

August 1, 2026

okf-rs: A New Rust Tool for Turning Codebases into AI-Readable Knowledge Bases

Building a fast, open-source Rust toolkit for generating, validating, and serving Open Knowledge Format (OKF) knowledge bases from source…

By Jeremy JEANNE

8 min read

Building a fast, open-source Rust toolkit for generating, validating, and serving Open Knowledge Format (OKF) knowledge bases from source code.

The problem: AI agents keep re-reading the same code

If you've worked with an AI coding agent — Claude Code, GitHub Copilot, or any other MCP-aware tool — you've probably watched it do this dance: it needs to know who calls a function, so it greps the codebase, opens a handful of matching files, and reads through hundreds of lines just to confirm which hits are actually call sites. Every file it opens costs its full size in context tokens. Do that a few dozen times in a session, and you've burned an enormous amount of context on something that is, fundamentally, a lookup.

This is the problem I set out to solve with okf-rs, a new open-source Rust CLI available https://github.com/jyjeanne/okf-rs.

What okf-rs actually does

At its core, okf-rs turns a codebase into a portable Open Knowledge Format (OKF) knowledge base: plain Markdown files with YAML frontmatter, cross-linked into a real call graph, and readable by both humans and AI agents.

$ okf-rs generate .
Generated 146 concepts into knowledge
  Module       16
  Struct       18
  Enum         6
  Function     87
  Method       19
$ okf-rs validate
knowledge — no issues found

Run generate against a repository, and you get a knowledge/ directory full of ordinary .md files — one per module, struct, enum, function, or method — each with a small YAML header and a body describing signatures and relationships. A single generated concept looks like this:

 — -
type: Rust Method
title: verify_token
resource: src/main.rs#L4-L6
generated:
 by: okf-rs/0.1.0
 — -

Signature

`fn verify_token(&self, token: &str) -> bool`’

Calls

- [decode_jwt](../../../functions/src/Auth/decode_jwt.md)

That's it. No proprietary database, no vector store, no SDK required to read it back. It's git-diffable, greppable, and renders natively on GitHub.

Why not just another graph database or context blob?

Most codebase-analysis tools land in one of two camps: they build a proprietary graph database you need their runtime to query, or they generate an AI-specific context blob that's opaque to everything except the model that consumed it. Neither is inspectable with tools you already own.

**okf-rs** takes a different approach, built on a few explicit principles:

  • Open — the output is the artifact. No runtime or SDK is required to read or write it.
  • Fast — a native Rust core using tree-sitter for parsing, rather than spinning up a full compiler frontend.
  • Deterministic— identical source always produces byte-identical output. No timestamps, no unordered-map noise leaking into the result.
  • AI-ready without requiring AI — the knowledge base is structured well enough that an LLM can consume it directly, but no LLM is involved in producing it.

Under the hood

okf-rs is a Cargo workspace of small, single-purpose crates, with okf-cli acting as a thin wrapper so the underlying logic can be embedded by other Rust tools. A few of the pieces I'm most pleased with:

  • Multi-language semantic extraction. The extractor understands packages, modules, types, functions, and methods across eleven languages — Rust, Python, TypeScript, JavaScript, Go, Java, C#, PHP, Kotlin, C/C++, and Swift — including public/private API-surface detection tuned to each language's actual visibility rules (explicit opt-in for Rust and Java, opt-out-by-default for PHP and Kotlin, section-based for C++, capitalization-based for Go).
  • A resolved call graph, covering bare calls, method/self/this calls, static and scoped calls, and qualified module calls, consistently across all supported languages.
  • **Optional LSP-backed disambiguation.**Tree-sitter alone resolves unambiguous calls by name. When a name is ambiguous project-wide, okf-rs generate — lsp can ask the project's real language server (rust-analyzer, pyright) to resolve it via textDocument/definition — verified end-to-end against real servers, with timeout handling and correct handling of paths containing spaces or non-ASCII characters.
  • Incremental indexing and watch mode. generate caches each file's extraction by content hash, so re-runs only reparse what changed. okf-rs watch keeps a bundle current as you edit, debouncing bursts of file changes.
  • Validation built for CI. Schema checks, dangling-link detection, orphan detection, and duplicate-identity checks, with a  — ci flag that treats orphaned concepts as hard failures once other tooling starts depending on the bundle being correct.

The part that actually matters for agents: the MCP server

The knowledge base is useful sitting on disk, but the real payoff comes from okf-mcp, a Model Context Protocol server that exposes a bundle's search and graph queries — search, graph_callers, graph_callees, graph_api, graph_cycles, graph_modules, graph_path — directly to any MCP-aware coding agent.

Registering it with Claude Code is one line:


claude mcp add okf-rs — /path/to/okf-mcp /path/to/project

And that's really the point: because okf-mcp speaks plain MCP over stdio, it isn't tied to a single vendor or agent. The exact same binary works with Claude Code, opencode, or any other MCP client — you just point that client's stdio transport at the okf-mcp binary and a project root. No per-agent integration, no proprietary plugin format, no re-implementing the graph queries for each tool. Build the bundle once with okf-rs generate, and every MCP-compatible agent in your toolchain — whatever you use today, whatever you switch to tomorrow — gets the same fast, structured access to it.

Where the token savings come from

Once registered, a question like "who calls verify_token?" no longer means grepping, opening files, and reading enough surrounding code to confirm real call sites. It means one graph_callers call that returns the answer directly — no source file enters the agent's context at all.

To put a number on it: on the okf-rs codebase itself, answering "who calls cmd_generate?" by hand means opening a 672-line, ~24 KB file and reading enough of it to find the caller — roughly 6,000 tokens by the usual rule of thumb. The equivalent graph_callers call returns one line, at around 15 tokens — a ~400x reduction for that single question.

That gap isn't a one-off, and it compounds in two ways over a real agent session:

  • Per query.Every call-graph or API-surface question — "what calls this?", "what does this module expose?", "is there a cycle here?" — pays the same 6,000-vs-15 token ratio, because the expensive part (parsing and resolving the call graph) already happened once, at okf-rs generate time, instead of being re-paid on every query.
  • Per session.Without a structured index, an agent re-opens the same large files repeatedly as its context window fills up and gets compacted — each reopen costs the file's full size again. With okf-mcp, the agent asks a targeted question and gets a targeted answer every time, so context usage stays roughly flat instead of growing with session length.

In practice, this means longer agent sessions on large codebases before hitting context limits, lower per-task token cost (and therefore lower API cost if you're paying by usage), and — because the agent isn't skimming irrelevant code to answer structural questions — fewer wrong assumptions creeping into its reasoning. And since it's just MCP, that benefit isn't locked to one coding assistant; it travels with you across whichever agent you're running.

Getting started

A prebuilt binary is available from the GitHub Releases page, or install directly via Cargo:


cargo install — git https://github.com/jyjeanne/okf-rs okf-cli

Then, from an existing project:


cd /path/to/your-existing-project
okf-rs init .
okf-rs generate
okf-rs validate

init writes an okf.toml recording your bundle's default location, and idempotently updates CLAUDE.md, AGENTS.md, and .github/copilot-instructions.md with a marked section pointing agents at the bundle — existing content in those files is left untouched. From there, okf-rs watch keeps things current as you work, and wiring okf-rs generate — no-cache && okf-rs validate — ci into CI ensures a stale bundle never ships silently.

What's new: from a call-graph tool to a knowledge platform

Since the first version, okf-rs has shipped an entire additional phase of work — Phase 2 (deep language coverage, LSP disambiguation, incremental indexing, MCP server) and Phase 3 (search, interop, and intelligence) are both complete, along with a full round of competitive gap-closing against adjacent codebase-knowledge-graph tools. The core idea — a deterministic, git-diffable Markdown bundle — hasn't changed. What's changed is how much you can do with that bundle once it exists.

  • Ranked and semantic search. Alongside the original exact/substring search, okf-rs search — ranked now does relevance-scored full-text search (via Tantivy) across titles, descriptions, signatures, and tags, with camelCase/snake_case boundary matching so a query for verifyToken still finds verify_token. Going further, okf-rs search — semantic layers cosine-ranked embedding search on top, against any OpenAI-compatible /embeddings endpoint.
  • Optional AI enrichment — genuinely optional.okf-rs generate — enrich fills in missing descriptions for functions, methods, modules, and packages by calling any OpenAI-compatible chat/completions endpoint — Ollama, LM Studio, LocalAI, or a cloud provider, never a hard dependency on one vendor. It never re-queries or overwrites a description that already exists, human-written or previously generated. On top of that, okf-rs suggest-links uses the same enrichment layer to propose plausible missing relationships between semantically close concepts — advisory only, nothing gets silently written into the bundle.
  • Deterministic architecture extraction — no AI required. A new okf-arch crate derives real structural insight straight from the call graph: okf-rs graph layers computes each package's depth in the dependency graph, graph domains finds which packages actually collaborate, and graph communities goes further with proper modularity-based clustering (Clauset–Newman–Moore), which — verified by dogfooding on the project's own 18-crate workspace — actually splits a codebase that plain connected-components collapses into one blob. graph patterns flags structural signals for Builder, Singleton, Factory, and Visitor; graph features flags REST endpoints, database models, and event-flow participants by naming convention.
  • Change-impact analysis and PR review automation.okf-rs impact <ref-a> <ref-b> scores every concept added, removed, or changed between two git refs by transitive-caller count ("blast radius"), public-API membership, and cycle participation. okf-rs review <ref-a> <ref-b> renders that as a sticky-comment-ready Markdown report, with  — fail-on-risk for CI gating — and ships with a ready-to-use GitHub Action (pr-review.yml) to wire it straight into pull request review.
  • More export formats. Beyond the original HTML and Markdown, okf-rs docs now also generates paginated PDF (with a bookmark per concept), GraphML (for Gephi, yEd, or any graph-visualization tool), and Obsidian vaults ([[wikilinked]] notes). A new okf-dita crate adds a genuine two-way DITA bridge: exporting a bundle to DITA topics, and importing an existing DITA corpus back in as first-class Document concepts that participate in the same search index and graph as extracted code — verified by round-tripping the project's own 715-concept DITA export back through generate — dita with zero data loss.
  • A composite explore query. Instead of chaining several search/graph_* calls, okf-rs explore <concept> (and the matching explore MCP tool) returns a concept's signature, description, callers, callees, blast radius, public-API membership, and cycle membership in a single call — one more step toward keeping an agent's token budget for reasoning rather than tool-call overhead.

Sturdier validation. The validator now checks relationship targets (not just markdown links), detects isolated concepts with zero call-graph edges, flags redundant links, and catches Calls/CalledBy asymmetry across the whole bundle — plus okf-rs coverage and okf-rs graph stats for at-a-glance bundle health metrics.

All of this reached users the same way the original release did: dogfooded against okf-rs's own, now roughly 850-concept codebase, with each feature verified end-to-end rather than assumed — including one case where dogfooding caught a real bug (a DTD-handling mismatch between the DITA exporter and importer) that a hand-written test fixture alone would never have surfaced.

What's next

Phase 1 through Phase 3 of the roadmap, plus the full competitive-gap-closing pass, are complete. Phase 4 — Ecosystem — is next: okf-server (a REST + GraphQL API over the knowledge graph for multi-repository, organization-wide serving), okf-rs as an LSP server (hover, go-to-definition, and find-references, reachable from any LSP-capable editor), an interactive graph visualizer, and continuous indexing at organization scale. The project is MIT/Apache-2.0 dual-licensed, and issues and pull requests are welcome.

If you're building or maintaining AI coding agents and finding that context budget is your real bottleneck rather than model capability, I'd genuinely like to hear whether this approach helps — or where it breaks down on your codebase.

Repository: github.com/jyjeanne/okf-rs