The challenge
A support team was answering the same questions repeatedly from documentation scattered across a help centre, PDFs, and internal notes. An off-the-shelf chatbot had been tried and abandoned because it confidently invented answers that weren't true — which eroded trust faster than it saved time.
Discovery
Discovery focused on two things: which questions actually recurred and were safe to automate, and what 'correct' meant for each. Rather than aim for a general-purpose bot, the scope was narrowed to grounded answers with sources, and an explicit 'I don't know' for anything outside the documented material.
The solution
A retrieval-augmented assistant indexed the company's real content and answered only from it, citing the source passages. A human-reviewed evaluation set of real questions measured accuracy before launch, and low-confidence answers were routed to a person instead of guessed. Cost per answer and response latency were monitored from day one.
Architecture
A Next.js application with a PostgreSQL database using pgvector for semantic search, embeddings and generation via a hosted LLM provider, and an evaluation harness that scored answers against a curated question set. Retrieval, prompts, and guardrails were versioned so quality changes were visible.
What we learned
The value wasn't the model — it was the retrieval, the evaluation, and the willingness to say 'I don't know.' Grounding and measurement are what turned a discredited chatbot into something people actually used.
Features delivered
- Retrieval-augmented answers grounded in real content
- Source citations on every response
- Evaluation set measuring accuracy before launch
- Human handoff for low-confidence questions
- Cost and latency monitoring
Outcomes
- Answers the team could trust because they were grounded and cited
- Repetitive questions deflected without inventing answers
- A measurable baseline to improve against over time