Retrieval-Augmented Generation · Financial NLP

Ask questions about SEC 13F filings — and trust the answer.

A RAG pipeline that answers questions about institutional-investor holdings grounded in the actual filings, not the model's memory. Every claim is traceable to a specific accession number, share count, and filing date.

View on GitHub Design Notes & Evaluation See it in action
100 institutional funds 4 quarters of 13F-HR filings +2.70 faithfulness gain (of 5) 6-stage pipeline
The problem

A “Private Data Gap” a vanilla LLM can't cross.

Recent 13F filings land after a model's training cutoff, so off-the-shelf LLMs either refuse or hallucinate plausible-looking share counts. For financial research, a fabricated CUSIP is worse than no answer at all. 13F-INSIGHT indexes real quarterly filings, retrieves the relevant rows, and forces the generator to answer only from them — then audits every response against the source corpus. The result says “I don't have that” when it should, and cites exact figures when it can.

See it in action

Same question. One model. Retrieval changes everything.

Query: “Analyze the evolution of Nvidia (NVDA) holdings across the four reported quarters of 2025.”
Baseline vanilla LLM · no retrieval

“The 2025 calendar year has not yet concluded, and therefore four quarters of 2025 data do not exist. […] As of today, the most recent completed filing period is Q3 2024.”

Confidently wrong. It reasons from a stale world model — the filings exist, they're just past its training cutoff — so it invents “2024 trends” to fill the gap.
Grounding: indeterminate (refusal, no CUSIPs) · Judge score 1.5/5
13F-INSIGHT retrieval-augmented

“BNY Mellon Investment Management reported NVIDIA CORP (CUSIP 67066G104): period ending 2025-03-31 (Accession 0001390777-25-000071) … 75,774,092 shares; period ending 2025-06-30 (Accession 0001390777-25-000119) … a marked decrease. No data was available for Q4 2025.”

Real share counts and real accession numbers from the actual filings — and where data is genuinely missing (Q4 2025), it says so instead of inventing it.
Grounding: PASSED — CUSIP 67066G104 verified · Judge score 3.5/5

Abridged from demo_log.txt, which carries the full side-by-side with dual-judge scores and per-chunk retrieval provenance.

How it works

Four stages from filing to audited answer.

01 Ingest

  • SEC EDGAR download (rate-limit safe)
  • XML <informationTable> parsing
  • Table-aware chunking w/ metadata headers

02 Retrieve

  • ChromaDB (MiniLM) + BM25
  • Hybrid fusion via RRF
  • FlashRank cross-encoder rerank
  • Entity / temporal / diversity filters

03 Generate

  • Routed Stuff / Map-Reduce chains
  • Verbatim-preservation prompting
  • Gemini 3.1 Flash-Lite

04 Evaluate

  • Dual-judge: GPT-4o-mini + Gemini 2.5 Pro
  • 5 MB CUSIP grounding check
  • Per-query audit trail w/ accessions
Results

Retrieval more than doubled faithfulness.

Ten diverse queries — fact lookup, numerical extraction, comparative synthesis, CUSIP lookup, hallucination traps, out-of-scope periods — scored 1–5 by two independent LLM judges from different vendors.

1.75
Baseline avg faithfulness / 5
4.45
Enhanced avg faithfulness / 5
+2.70
Improvement (7/10 grounded)
Baseline Enhanced (13F-INSIGHT)
Q01
Q02
Q03
Q04
Q05
Q06
Q07
Q08
Q09
Q10

Q06 & Q07 are the traps — a non-existent entity and an out-of-scope period. There a good system should refuse, and the enhanced pipeline does, scoring full marks for it. The flat Q09 and the Q04 grounding false-positive are dissected honestly in the design notes.

Pipeline

Six reproducible notebook stages.

StageNotebookPurpose
1stage1_acquisitionDownload up to 4 quarterly 13F-HR filings per fund for 100 CIKs from SEC EDGAR, rate-limit compliant.
2stage2_parsingExtract <informationTable> XML → DataFrames → ≤300-word Markdown table chunks with metadata headers.
3stage3_vectordbBuild ChromaDB (all-MiniLM-L6-v2) + BM25 index; wire the hybrid retriever with Reciprocal Rank Fusion.
4stage4_rag_analysisFull RAG pipeline: entity/temporal/diversity filters, reranking, Stuff/Map-Reduce chains, audit-trail logging.
5stage5_evaluation10 diverse queries, baseline vs enhanced, dual-judge scoring and grounding checks.
6stage6_demo3 thematic demo queries with side-by-side output; generates the demo log.
Techniques

What's under the hood.

Hybrid search + RRFBM25 keyword precision fused with ChromaDB semantics via Reciprocal Rank Fusion.
Cross-encoder rerankingFlashRank, dual-width — narrow (top-8) for entity queries, wide (top-20) for synthesis.
Routed Map-ReduceVerbatim Stuff chain for lookups; abstractive Map-Reduce for cross-fund synthesis.
Precision filtersEntity-scoped, temporal (period-of-report), and fund-diversity post-retrieval filters.
Cross-vendor dual judgeGPT-4o-mini (OpenAI) + Gemini 2.5 Pro (Google) — two vendors, no self-grading.
CUSIP grounding checkScans up to 5 MB of source filings to verify every cited identifier; refusals → indeterminate.