IR / RAG Evaluation Lab
A professional information retrieval and RAG evaluation workbench covering BM25, Dense, Hybrid, Rerank, RAG citation, LLM judge, and text-mining analytics.
IR / RAG Evaluation Lab is a benchmark and evaluation lab, not a chatbot UI. It supports custom JSONL, BEIR/MS MARCO/OpenAlex ingestion, BM25 baseline, dense fallback, hybrid search, reranker interface, retrieval metrics, bad case analysis, RAG citation checking, local llama.cpp judging, query rewrite experiments, text-mining network analysis, and a bilingual React dashboard.
Verified Demo Recording 1
Copied by the portfolio quality pass from existing project demo media.
Media overview
Browse the project screenshots and recorded walkthroughs.
Project links and demo readiness
live
Live Demo
Open the verified live demo when available, or use the internal run guide for manual demo preparation.
github
View GitHub
Open the public repository when available, or review the source-access status captured by this case study.
video
Demo Recording
Open the demo recording when available, or use this section as the recording checklist for the project.
documentation
README
Open the project README when available, or use this case study as the documentation baseline.
Project overview
IR / RAG Evaluation Lab is designed to prove retrieval and RAG evaluation quality, not merely produce answers. Users can upload `documents.jsonl` and `queries.jsonl` or use the included sample datasets. The system persists data in a DuckDB dataset registry, checks data quality, runs BM25/dense/hybrid/rerank evaluations, and reports Precision@K, Recall@K, MRR, MAP, nDCG@K, latency, and zero-result rate. The frontend is a bilingual evaluation dashboard with Query Evaluator, Retrieval Comparison, Evaluation Analytics, RAG Citation Checker, Bad Case Viewer, LLM Evaluation, Text Mining, and Experiment Runs. The auto-starting guide walks an interviewer across the actual feature areas and explains what the metrics and charts mean, instead of showing unexplained visuals. local llama.cpp is used inside the evaluation workflow for claim support, bad case root-cause suggestions, query rewrite experiments, faithfulness judgments, and analyst notes. The UI labels LLM judge output as an assistive signal, not ground truth, and tracks latency, confidence, invalid JSON rate, and run history. The full experience is verified by a Playwright walkthrough covering the guided assistant, Pipeline Journey, query evaluation, analytics heatmaps, RAG citation, LLM dashboard, text-mining network, and zh/en switching. The screenshots and recording linked on this project card come directly from that capture pipeline.
My role
Independent Developer / IR, RAG, and Evaluation Workflow Architect
Problem
Many RAG demos show an answer after putting embeddings into a vector database, but they do not prove whether retrieval found the right evidence or explain failed queries, citation coverage, bad cases, or LLM judge reliability. This project was built as a reproducible, explainable, demo-ready IR/RAG evaluation lab.
Solution
I built a FastAPI + DuckDB backend for dataset registry, BM25/dense/hybrid/rerank retrievers, experiment persistence, retrieval metrics, bad case root-cause workflow, RAG claim-to-evidence evaluation, llama.cpp local judge history, and text-mining analytics. The React + TypeScript + Vite frontend provides an auto-starting guided assistant, Pipeline Journey, interactive heatmaps, scatter plots, networks, Sankey-style flows, and chart dashboards, all verified with Playwright screenshots and a walkthrough recording.
Current outcome
An interview-ready evaluation platform: users can upload JSONL or use sample datasets, follow an assistant through the full data-to-evaluation journey, compare retrievers, inspect failures, review RAG citations, verify local llama.cpp judging, and export reproducible evidence.
Highlights
- Comparable BM25, Dense, Hybrid, and Rerank workflows with every search/evaluation run persisted to DuckDB.
- Evaluation Analytics includes metric matrices, failure heatmaps, rank movement, retriever battles, query diagnostics, and deterministic insight summaries.
- RAG Citation Checker splits answers into claims, maps evidence/citations, and uses local llama.cpp to assist with supported, partially supported, unsupported, or contradictory judgments.
- Text Mining combines co-occurrence networks, collocations, association rules, Sankey flows, community detection, and a Gephi-like network workbench.
- An app-like guided assistant auto-starts on every visit, jumps to feature targets, and explains results through spotlighted UI and the Pipeline Journey.
Engineering challenges
- Keeping the product focused on benchmark/evaluation workflows instead of drifting into a generic ChatGPT interface.
- Designing fallback and real llama.cpp modes so offline demos still run while real LLM judge runs remain auditable.
- Turning raw JSON payloads into structured cards, badges, confidence bars, history dashboards, and reviewer workflows.
- Using Playwright to verify the guided tour, every major page, long-page screenshots, recording, and bilingual switching.
Target users
- Portfolio reviewers and interviewers
- Technical readers who need a quick view of purpose, stack, and maturity
Technical highlights
- Detected technical signals: Python, FastAPI, DuckDB, React, Vite, TypeScript, BM25, RAG, llama.cpp, ECharts, Recharts, Playwright
- README evidence exists and can support a fuller reviewed case study
- A public GitHub repository is linked for source traceability
Architecture
This case study is generated from the portfolio catalog pipeline using README, Git metadata, package/build configuration, and media signals. The final architecture narrative still needs source-level review. Current detected technology signals include: Python, FastAPI, DuckDB, React, Vite, TypeScript, BM25, RAG, llama.cpp, ECharts, Recharts, Playwright.
Data flow
A public data-flow narrative is not fully reviewed yet. If the project includes data processing, AI pipelines, or backend APIs, the next pass should document input, processing, storage, and UI/output flow end to end.
Project structure
ir-rag-evaluation-lab/ README.md # project documentation, when available source files # implementation reviewed by local audit package/build config # detected capability signals
Setup / Run guide
This project does not expose a verified runnable web command yet. Review the README/source tree and add exact install, run, test, and build commands before interview use. No verified build command was detected. Treat the current portfolio page as a case-study placeholder until build steps are reviewed.
Future improvements
- Complete the production-quality README, screenshots, and demo recording
- Add architecture diagrams, data-flow notes, and key technical decisions
- Verify build/test status and update the portfolio release report
Interview notes
- State the current maturity and demonstrable scope first
- Focus on verified stack, source structure, and completed behavior
- Do not claim unverified deployment, video, or test coverage as finished
Next steps
- Connect larger BEIR/MS MARCO/OpenAlex benchmark corpora with deeper qrels coverage.
- Add a production cross-encoder/reranker adapter and Prompt Studio for evaluation prompts.
- Grow the report into an interactive benchmark artifact with more stakeholder-friendly conclusions.