Controllable Complex RAG
Plans, retrieves, and verifies answers to complex multi-step questions over your own documents.
The README maps the main flow through PDFs, summaries, vector stores, retrieval, verification, and external LLM APIs, and attributes ideas to Self-RAG, Plan-and-Solve, babyagi, and an adviser. Requiring service keys through a .env file shows limited privilege awareness. Deductions apply because there is no permission model, pre-action confirmation, key-protection or log-privacy guidance, telemetry/data-retention boundary, provider disclosure detail, revocation, or rollback mechanism. Many dependencies are pinned, but several central packages are not, and no vulnerability scanning, update policy, or supply-chain controls are evidenced.
The README, architecture narrative, and dependency list mostly describe a coherent RAG workflow. However, calling the graph deterministic sits uneasily with continuous LLM-driven planning, while absolute claims such as hallucination prevention, solely grounded answers, and hallucination-free verification lack supplied evaluation results. The very large dependency set includes unpinned LangChain packages, pyppeteer, and urllib3, and the installation sections contain duplicated headings and numbering. No evidence covers user-facing errors, recovery behavior, degraded operation, or failure diagnostics.
The material identifies a useful audience and scenario: Python/RAG users analyzing complex questions over their own long-form data. It supplies a Harry Potter example, Docker and local paths, OpenAI/Groq options, and a graph-oriented workflow. Deductions reflect unclear boundaries around document scale, supported formats, model compatibility, termination conditions, and unsuitable cases. The retrieve-versus-answer routing is described conceptually but no precise, statically reviewable trigger rules are supplied.
The README has recognizable sections for features, architecture, use case, installation, usage, technology, contribution, and attribution, plus notebook and Streamlit entry points. The complete Apache-2.0 text fully supports the license criterion. Deductions cover confused Docker/non-Docker organization and numbering, no real FAQ or explicit limitations, no releases or changelog evidence, and unstable unpinned names among central dependencies. PR, issue, social, and community routes are mentioned, but no maintainer duties, support commitment, or update policy are defined; unknown publisher identity is not treated as a quality or safety failure.
Task decomposition, multiple retrieval representations, replanning, and Ragas-oriented evaluation offer plausible marginal value beyond simple similarity search. A tutorial and visualization entry point make the proposed answer workflow reasonably usable. Scores are limited because no actual answer artifact, quantitative evaluation table, baseline comparison, or failure example is supplied. Repeated LLM summarization, planning, distillation, and verification may impose substantial latency and API cost, yet token, runtime, infrastructure, and benefit tradeoffs are not documented.
Architecture steps, the dependency manifest, the license, and research references provide partial traceability. The central quality claims, however, are not tied to supplied metrics, tests, experiment configurations, or result artifacts. Cross-source support is largely limited to consistency among README, requirements, and LICENSE and does not independently corroborate reasoning or anti-hallucination performance. Promotional absolutes, design intentions, and established results are not clearly separated, so fact-versus-inference handling remains thin.
- Do not treat hallucination prevention or solely data-grounded answers as verified guarantees; the supplied material contains no evaluation results or test artifacts supporting those absolutes.
- Before use, review what document content is sent to OpenAI, Groq, or other providers, including telemetry and retention policies, and separately protect API keys, logs, and sensitive PDFs.
- Pin the unresolved core dependencies and perform a current vulnerability and compatibility review before deployment; the large mixed-version dependency set can drift.
- Measure answer quality, API cost, and latency on small non-sensitive corpora first, and design failure recovery and rollback procedures independently.
What does this agent do, and when should you use it?
This repository implements an advanced RAG workflow for questions that basic semantic-similarity retrieval cannot readily answer. It processes PDFs and builds vector stores from text chunks, LLM-generated chapter summaries, and selected book quotations. A deterministic graph orchestrates question anonymization, planning, task decomposition, retrieval or answering, grounding checks, and iterative re-planning. The final answer is produced from accumulated context, with verification intended to reduce unsupported generation. The project is self-hosted through a Python environment or Docker and exposes a Jupyter tutorial plus a Streamlit visualization rather than a documented production API.
The pipeline loads PDFs, separates them into chapters, cleans their text, and calls an LLM to generate extensive chapter summaries. It encodes book chunks, summaries, and quotations in FAISS Vector Store indexes. For each question, it replaces named entities with variables, creates a high-level plan for the anonymized form, restores the entities, and breaks the plan into retrievable or answerable tasks. Each task either retrieves and distills relevant vector-store content or generates an answer from the available context; the workflow then checks grounding against the original context and revises the remaining plan. It combines the accumulated material into a final answer. sophisticated_rag_agent_harry_potter.ipynb presents the workflow step by step, simulate_agent.py supplies real-time Streamlit visualization, and Ragas measures answer correctness, faithfulness, relevancy, context recall, and answer similarity.
- A researcher with a long book or PDF collection who must answer questions requiring facts and causal links from several chapters.
- An ML engineer prototyping internal knowledge QA who wants explicit control over planning, retrieval, answering, verification, and re-planning.
- A team evaluating a RAG pipeline that wants Ragas measurements covering correctness, faithfulness, relevancy, context recall, and similarity.
- An instructor or learner studying advanced RAG who wants a notebook demonstration of anonymization, decomposition, vector retrieval, and adaptive planning.
- A developer presenting an agent workflow who needs a Streamlit interface that visualizes execution in real time.
What are this agent's strengths and limitations?
- A deterministic graph explicitly controls planning, task execution, verification, and re-planning instead of relying on a single similarity-search pass.
- It indexes text chunks, chapter summaries, and quotations, giving the retrieval process several representations of the source material.
- Question anonymization, retrieval distillation, and context-grounding checks directly address pretrained-knowledge bias and unsupported generation.
- The included Ragas evaluation covers five distinct quality dimensions rather than relying only on demonstrations.
- A notebook tutorial, Streamlit visualization, and Docker path make the workflow accessible for experimentation and demonstrations.
- The documented workflow centers on book PDFs and one Harry Potter example; there is no evidence of connectors for websites, databases, or continuously changing sources.
- Generating chapter summaries and maintaining multiple vector stores adds preprocessing time, provider calls, and storage overhead.
- Execution requires network access and credentials for an external LLM provider; the example environment names only OpenAI and Groq.
- Authentication, concurrency, monitoring, a persistent service API, and production scaling are not documented, so adopters must supply that engineering.
- Although flexible LLM integration is claimed, the source provides no provider-by-provider compatibility matrix or description of feature tradeoffs.
How do you install or deploy this agent?
Prerequisites are Python 3.8+ and an API key for the selected LLM provider.
For a local installation:
git clone https://github.com/NirDiamant/Controllable-RAG-Agent.git
cd Controllable-RAG-AgentCreate .env in the repository root, using .env.example as a reference, and fill the applicable provider key:
OPENAI_API_KEY=
GROQ_API_KEY=Install the packages:
pip install -r requirements.txtAlternatively, after configuring the environment variables, build and run with Docker:
docker-compose up --buildHow do you use this agent?
Open and run sophisticated_rag_agent_harry_potter.ipynb for the step-by-step tutorial. To launch the real-time visualization locally, run:
streamlit run simulate_agent.pyFor the Docker deployment, visit http://localhost:8501/ after the container starts. The documented example uses the first Harry Potter book so users can inspect whether answers rely on retrieved evidence rather than pretrained knowledge; applying it to another corpus requires preparing and processing the relevant PDFs. No separate HTTP API, command-line question interface, or production service deployment is documented.
How does this agent compare with similar options?
Compared with basic RAG that retrieves passages only by semantic similarity, this project adds question anonymization, high-level planning, task decomposition, retrieval distillation, grounding verification, and adaptive re-planning for non-trivial multi-step questions. That additional control also makes the pipeline more complex and introduces more LLM calls, index preparation, and evaluation work. The README presents RAG_Techniques as a complementary collection of other RAG methods, not as a direct replacement.
FAQ
Is OpenAI required?
OPENAI_API_KEY and GROQ_API_KEY. Configuration instructions for additional providers are not supplied.Does it guarantee hallucination-free answers?
Is there a ready-made chat API?
streamlit run simulate_agent.py, and the Docker-hosted Streamlit page.