WeldCraft: Hybrid‑Search RAG Assistant for Technical Documentation

Challenge

WeldCraft's field engineers and support team needed instant, trustworthy answers from thousands of pages of technical PDFs, spec sheets, and web content. Keyword search missed contextually relevant results, users wanted conversational answers instead of sifting through manuals, and the support queue was dominated by questions whose answers already lived in the docs.

Solution

We built a hybrid‑retrieval RAG assistant with citations, evaluations, and observability baked in from day one.

• Hybrid retrieval with rerank. BM25 over OpenSearch fused with dense embeddings in a vector store using reciprocal rank fusion, followed by a cross‑encoder reranker (bge‑reranker‑v2) that trims to the top chunks per query. A structure‑aware splitter keeps tables, part numbers, and procedure steps intact across page breaks.
• Conversational RAG. A LangGraph workflow handles query rewriting, retrieval, answer drafting, and a citation‑audit step that fails closed when a claim lacks a resolvable source span. Conversation state is persisted so multi‑turn troubleshooting flows survive page reloads and handoffs.
• Evaluation harness. A Ragas suite (faithfulness, answer relevancy, context precision/recall) plus a curated golden set of real support questions runs in CI on every prompt or index change. Promotion gates block regressions before they reach users.
• Observability. Langfuse captures every trace in production with PII scrubbing; weekly reports surface unanswered queries, low‑confidence answers, and retrieval drift so the content team knows what to write next.

Results

• Users find relevant information 3× faster than with the previous keyword search.
• 85% reduction in documentation‑related support tickets within the first quarter.
• Multi‑turn conversation memory replaces manual cross‑referencing across multiple documents.
• Every answer ships with inline citations back to the source PDF or page.

Technical Highlights

• Hybrid BM25 + dense retrieval with reciprocal rank fusion and cross‑encoder reranking.
• Structure‑aware chunking tuned for technical PDFs and tabular content.
• LangGraph orchestration with a citation‑auditor node that fails closed on unsourced claims.
• Ragas + golden‑set evaluations wired into CI; Langfuse traces and drift reports in production.