A self-hostable RAG toolkit — FAISS, sentence-transformers, and pluggable LLMs behind one CLI.

Context

Every RAG tutorial ends the same way: sign up for a vector DB SaaS, pick one LLM vendor, wire your docs into their pipeline, done. That's fine for a demo. It's not fine when your corpus is internal documentation you can't send off-box, when your budget doesn't survive a per-token pricing model, or when the “pick one LLM” decision is exactly the thing you want to defer.

I wanted a RAG stack that runs on a laptop, in a container, or on a workstation with a local model — with the same commands, the same index format, and no vendor in the middle. The output was markdown-tree-parser: a Python toolkit that ingests a markdown corpus, builds a persistent FAISS index, and answers questions through whichever LLM you happen to have available today.

Architecture

Ingest and query are separate lifecycles. The index is a directory on disk (vector_store/), swappable between machines. The LLM adapter is the only piece that changes when you switch providers — everything else keeps its state.

Constraint

• No SaaS in the loop. Vector store, embeddings, and (optionally) inference all have to run local.
• Three providers, one interface. OpenAI, Ollama, and a local HuggingFace pipeline must be swappable through config alone — no code changes, no different CLI.
• Persistence over ceremony. Ingest once, query forever. The index has to survive a process restart without a rebuild.
• CLI-first. A human should be able to ingest, query, stats, and drop into an interactive REPL without writing Python. A programmatic API is a bonus, not the main event.
• Structure-aware chunking. Naive fixed-window chunking destroys markdown. Chapter → section → subsection boundaries matter for retrieval quality.

Move

• Hierarchical parser first, chunker second. Markdown is parsed into a tree (level, title, path, content), then chunked with configurable size and overlap. Each chunk carries its full path back to the source heading — so citations read “Chapter 3 › Neural Networks › Training”, not “chunk 187”.
• FAISS IndexFlatIP on normalized embeddings. Exact cosine similarity, no approximate tricks — correct default for corpora up to hundreds of thousands of chunks. IndexIVFFlat is documented as the upgrade path when a user outgrows it.
• LLM adapter with three concrete implementations. OpenAI (Chat Completions), Ollama (HTTP against 127.0.0.1:11434), and a local HuggingFace text-generation pipeline. All three expose the same generate(prompt, context) → answer surface.
• Config-driven, not code-driven. config.json controls provider, model, chunk size, overlap, similarity threshold, temperature, and token budget. .env holds secrets. Switching providers is a one-line change.
• Persistent store on disk. vector_store/faiss.index + chunks.pkl + metadata.json. Load-on-init, save-on-ingest. No hidden global state.
• CLI as the first-class surface. ingest, query, stats, interactive — each with its own flags (--k, --json, --no-sources, --force). The Python API wraps the same primitives.
• Similarity threshold as a floor, not a filter. Chunks below the configured similarity score are dropped from the LLM context entirely — the model never sees weak retrievals and hallucinates less as a result.

Result

• Ingest → query → answer in three commands. ingest sample_document.md, then query "what are neural networks?" — grounded answer with cited paths and per-chunk similarity scores.
• Three interchangeable LLM backends (OpenAI, Ollama, local HF), driven entirely by config.json.
• Persistent FAISS store that survives restarts and moves between machines by copying a folder.
• Interactive REPL with runtime config overrides (set similarity_threshold 0.4, set llm_provider ollama) — useful for tuning retrieval without re-ingesting.
• Docker recipe included. python:3.9-slim → install requirements → production_app.py interactive. Deploys as a container without further changes.
• JSON output mode for machine consumers — the same query surface feeds either a human REPL or another script.

What I'd change

• An evals harness. Recall@k on a labeled question set, per provider, would replace “this answer looks good” with a number. Right now tuning similarity_threshold and chunk size is a spot-check exercise.
• Embedding cache. Re-ingesting an unchanged corpus re-embeds unchanged chunks. A content-hash keyed cache would cut wall-clock time and (for hosted embedding models) cost.
• Streaming responses. OpenAI and Ollama both support token streaming; the CLI still waits for the full completion. First-token latency dominates perceived quality — this is the highest-value polish item.
• IndexIVFFlat auto-upgrade. The README calls out IndexIVFFlat for large datasets; the code should switch to it above a chunk-count threshold rather than making the user re-read the docs.
• Structured multi-doc support. Ingest today is one-markdown-at-a-time. A ingest --dir docs/ mode with per-file provenance is the obvious next feature.
• Guardrails on the LLM adapter. Rate limiting, retry-with-backoff, and a cost meter (tokens in / out) would earn their keep the moment this runs in anything other than a single-user session.