Kubeflow Docs Agent

What does this agent do, and when should you use it?

This is the official LLM implementation of the Kubeflow Documentation Assistant, a retrieval-augmented generation system that lets users search across Kubeflow documentation and receive contextual answers. Retrieval runs on Milvus: production uses RAG v4 hybrid search combining 768-dimensional MPNet dense vectors with native BM25 sparse vectors and a typed release_date field. Generation is served by KServe with a vLLM backend running RedHatAI/Llama-3.1-8B-Instruct, with auto tool choice and the llama3_json tool-call parser enabled. Ingestion is implemented as Kubeflow Pipelines that fetch the kubeflow/website content/en directory, parse and chunk it canonically, call the in-cluster TEI embeddings service, and write to Milvus in batches. Two APIs are provided: a WebSocket server in server/app.py and a FastAPI HTTPS server in server-https/app.py, both supporting streaming and non-streaming responses, tool calling and de-duplicated citation collection. The repository also ships an MCP server, kagent controller manifests, a Terraform platform stack and a gateway-guardrails Helm chart, targeting deployment on Kubernetes such as OKE.

The ingest side fetches .md and .html docs from GitHub (by default kubeflow/website under content/en), parses and re-chunks them in canonical_rag_ingest.py, extracts release_date, obtains 768-dimensional embeddings from the in-cluster TEI service, and batch-inserts dense plus native BM25 vectors into Milvus via pymilvus. At query time, docs-agent-mcp/mcp-server/server.py applies a deterministic intent router (SEARCH_MODE=auto) over Milvus v4 hybrid search, then passes the retrieved passages to the KServe llama3.1-8B endpoint, which produces answers plus structured citations — the README states the LLM never renders URLs. Clients reach it either through the WebSocket handler (handle_websocket, stream_llm_response) or the HTTPS endpoint POST /chat, which emits SSE events of type content, tool_result, citations and done for streaming, or JSON with response and citations otherwise; GET /health backs Kubernetes probes. Infrastructure is declared in docs-agent-mcp/terraform across namespaces.tf, knative.tf, kubeflow_pipelines.tf, milvus.tf, kagent.tf and gateway_guardrails.tf, while the gateway-guardrails chart supplies the Istio gateway with TLS, CORS lockdown, a 60 req/min token bucket and zero-trust AuthorizationPolicies. CI runs through oke-cicd.yaml and tests.yml.

1. A Kubeflow platform operator answering a user's question about creating a pipeline, and needing the reply to cite official documentation.
2. A docs maintainer re-running the KFP ingestion pipeline after a website release to refresh the Milvus collection without rebuilding the stack.
3. A platform team deploying an internal-only documentation Q&A service on its own Kubernetes or OKE cluster and reproducing it through Terraform.
4. A frontend engineer embedding the assistant into the Kubeflow website by calling POST /chat and consuming the SSE events plus the citations field.
5. An agent developer wanting to reuse the same retrieval and citation logic through an MCP client instead of reimplementing chunking and vector search.
6. A security reviewer needing the public chat endpoint to ship with TLS, a CORS allowlist and rate limits straight from the gateway-guardrails defaults.

How do you install or deploy this agent?

Prerequisites: Kubernetes 1.20+, Helm 3.x, Kubeflow Pipelines, GPU nodes for LLM inference, and an SSL certificate from a trusted CA (browsers block WebSocket and HTTPS connections without one).

Install Milvus into the docs-agent namespace first:

helm repo add milvus https://milvus-io.github.io/milvus-helm/
helm repo update

helm upgrade --install my-release zilliztech/milvus -n docs-agent \
  --set cluster.enabled=false \
  --set standalone.enabled=true \
  --set etcd.replicaCount=1 \
  --set etcd.persistence.enabled=false \
  --set minio.mode=standalone \
  --set minio.replicas=1 \
  --set pulsar.enabled=false \
  --set pulsarv3.enabled=false \
  --set standalone.podAnnotations."sidecar\.istio\.io/inject"="false"

Verify the connection:

from pymilvus import connections
connections.connect("default", host="my-release-milvus.docs-agent.svc.cluster.local", port="19530")
print("Connected to Milvus successfully!")

Then apply the KServe ServingRuntime and InferenceService (model RedHatAI/Llama-3.1-8B-Instruct, vLLM backend, HF_TOKEN from huggingface-secret):

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: llm-runtime
  namespace: docs-agent
spec:
  supportedModelFormats:
    - name: huggingface
      version: "1"
      autoSelect: true
  containers:
    - name: kserve-container
      image: kserve/huggingfaceserver:latest-gpu
      command: ["python", "-m", "huggingfaceserver"]
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
          nvidia.com/gpu: "1"
        limits:
          cpu: "6"
          memory: "24Gi"
          nvidia.com/gpu: "1"

If Kubeflow Pipelines must reach Milvus, add the RBAC grants:

kubectl create role milvus-access \
  --namespace docs-agent \
  --verb=get,list,watch \
  --resource=services,endpoints

kubectl create rolebinding kfp-to-milvus-editor \
  --namespace docs-agent \
  --role=milvus-access \
  --serviceaccount=kubeflow:default-editor

The wider platform (Knative, Istio, KServe, KFP, Milvus, kagent, gateway guardrails) is provisioned declaratively from docs-agent-mcp/terraform; exact variable names and credential wiring are not enumerated in the README and must be filled in from variables.tf and kagent.tf.

How do you use this agent?

Run the ingestion pipeline to load documentation into Milvus:

python docs-agent-mcp/pipelines/kubeflow-pipeline.py

Start one of the two API servers:

# WebSocket API
python server/app.py

# HTTPS API
python server-https/app.py

Streaming request (Server-Sent Events):

curl -X POST "https://your-domain.com/chat" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{"message": "What is KServe?", "stream": true}'

Non-streaming request:

curl -X POST "https://your-domain.com/chat" \
  -H "Content-Type: application/json" \
  -d '{"message": "What is KServe?", "stream": false}'

import requests
import json

response = requests.post(
    'https://your-domain.com/chat',
    json={
        'message': 'What is KServe?',
        'stream': False
    }
)

data = response.json()
print(f"Response: {data['response']}")
if data.get('citations'):
    print(f"Sources: {data['citations']}")

Key environment variables: KSERVE_URL, MODEL (default llama3.1-8B), PORT (default 8000), MILVUS_HOST, MILVUS_PORT (default 19530), MILVUS_COLLECTION (default kubeflow_docs), EMBEDDING_MODEL (default sentence-transformers/all-mpnet-base-v2). Pipeline parameters include repo_owner, repo_name, directory_path (default content/en), chunk_size (1000), chunk_overlap (100), base_url and milvus_host. For debugging:

kubectl get pods -n docs-agent | grep milvus
kubectl get inferenceservice -n docs-agent
kubectl logs -f deployment/docs-assistant-api

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ