The Vibe Check That Shipped a Hallucinating Chatbot
An Indian SaaS company builds a RAG-powered support chatbot. The team tests it with ten questions. The answers look reasonable. The demo impresses the leadership team. The chatbot goes live. Three weeks later, a customer receives a confidently worded answer about a refund policy that does not exist. The support team spends a day cleaning up the confusion. The root cause is not the model. It is the evaluation: ten prompts and a gut feeling are not a test.
This is the most common failure in enterprise AI deployment. The system is built, a few examples are tried, and the output “looks right.” The team moves to production. The failure surfaces later, in front of customers, when the cost of correction is highest.
AI evaluation is the discipline that prevents this. It is not a single test. It is a continuous process of measuring whether an AI system produces accurate, grounded, relevant, and safe outputs—before it reaches users, and continuously after it does. For Indian businesses deploying LLMs, RAG pipelines, and AI agents, evaluation is the difference between a demo and a production system.
The stakes are high because LLMs are probabilistic. They do not throw errors when they are wrong. They produce fluent, confident, incorrect output. A deterministic unit test that checks “input X produces output Y” fails to capture this. What is needed is a statistical evaluation against a curated dataset, using metrics that measure quality dimensions, combined with online monitoring that catches degradation in production .
India adds specific considerations. DPDPA applies to evaluation datasets that contain personal data, requiring anonymisation and purpose limitation. MeitY AI governance expects transparency and accountability for AI systems, which evaluation documentation supports. Indian language evaluation is a distinct challenge—global benchmarks are English-centric, and models may perform differently on Hindi, Tamil, or Hinglish inputs . And cost management matters because evaluation itself consumes LLM tokens, and a poorly designed evaluation can be expensive without being informative.
This guide is written for Indian CTOs, AI engineers, product managers, and quality leads evaluating AI systems. We will cover the offline and online evaluation layers, the metrics that matter, the golden dataset, real costs in rupees and dollars, DPDPA compliance, and a phased implementation framework with clear go/no-go gates.
Offline vs Online: The Two Evaluation Layers
Evaluation is not a one-time gate. It has two distinct layers, each answering a different question.
Offline Evaluation: The Laboratory
Offline evaluation happens before deployment. It answers the question: does this system meet the quality bar on a known set of cases?
It requires a golden dataset: a curated collection of inputs, expected outputs (or reference answers), and source documents. The system is run against this dataset, and metrics are computed. When a prompt changes, a model is swapped, or a chunking strategy is adjusted, the dataset is run again to see whether the system improved or regressed .
The purpose is optimisation and regression testing. Without a golden dataset, every change is a guess. With one, every change is measured.
Online Evaluation: The Production Floor
Online evaluation happens after deployment. It answers the question: is the system behaving well on real user inputs?
In production, there is no ground truth. You cannot know what a customer will ask, and you cannot manually verify every output. Online evaluation runs lightweight scoring models alongside the main pipeline to grade live interactions. If a response scores poorly on factual alignment, it is flagged before the user sees a hallucination .
The purpose is safety and anomaly detection. It catches drift, edge cases, and failures that the golden dataset did not anticipate.
The Continuous Loop
The two layers must feed each other. Online telemetry identifies unusual interactions or borderline failures. Those cases are cleaned, scrubbed of personal data, and added to the golden dataset. The evaluation laboratory gets smarter from real-world usage, and the production system becomes more resilient .
Offline vs Online Table
| Dimension | Offline Evaluation | Online Evaluation |
|---|---|---|
| When | Before deployment | After deployment |
| Data | Golden dataset | Live traffic |
| Ground truth | Available | Not available |
| Purpose | Optimisation, regression testing | Safety, anomaly detection |
| Frequency | Every change | Continuous |
| Metrics | Reference-based, LLM-judge | LLM-judge, heuristics |
For teams building the RAG pipelines and AI applications that evaluation protects, our AI Development and Generative AI Services practices cover the full stack.
The Metrics That Matter for RAG
For RAG systems, evaluating only the final answer is insufficient. If the answer is wrong, the cause could be retrieval (the wrong document was found) or generation (the model ignored the document). The pipeline must be evaluated at each stage.
The Three Core Metrics
Context Precision. Did the retrieval system find the right information? If the retriever pulls irrelevant documents, the LLM has no chance of generating a correct answer. This measures the quality of data ingestion and retrieval .
Faithfulness (Groundedness). Is the generated answer completely grounded in the retrieved documents? If the model introduces external knowledge or fabricates facts that were not in the source text, it is hallucinating. This measures the reliability of the model .
Answer Relevance. Does the response actually address the user’s query? A model can retrieve perfect information and write a factually accurate response, but fail completely if it ignores the core question asked .
Additional Metrics for Agentic Systems
For AI agents that call tools and take actions, additional metrics apply:
Tool Use Quality: Did the agent select the right tool, with the correct arguments, in the correct schema?
Trajectory Quality: Did the agent follow a sensible sequence of steps toward the goal?
Task Success: Did the agent achieve the goal, as observed from outcomes and confirmations?
Metrics Table
| Metric | What It Measures | Pipeline Stage |
|---|---|---|
| Context Precision | Retrieval quality | Retrieval |
| Faithfulness | Grounding in sources | Generation |
| Answer Relevance | Query alignment | Generation |
| Tool Use Quality | Tool selection and arguments | Agent |
| Trajectory Quality | Step sequence | Agent |
| Task Success | Goal achievement | Agent |
Building the Golden Dataset
The golden dataset is the foundation of offline evaluation. Without it, evaluation is subjective. With it, evaluation is repeatable.
What the Dataset Needs
Representative queries. The dataset should cover the range of questions the system will face: common queries, edge cases, ambiguous queries, and adversarial inputs. For an Indian support chatbot, this includes Hinglish queries and queries about India-specific policies.
Reference answers. For each query, a trusted answer. This may be written by a domain expert or verified from source documents.
Source documents. The documents the system should retrieve to answer the query. This enables context precision measurement.
Diversity. The dataset should not be ten variations of the same question. It should cover different intents, different document sources, and different difficulty levels .
How Many Examples
There is no fixed number. A dataset of 100–300 diverse cases is sufficient for meaningful offline evaluation. Fewer than 50 is unlikely to capture enough variation. More than 500 becomes expensive to maintain without proportional benefit.
Synthetic Data
When real data is scarce, synthetic data can supplement. An LLM can generate realistic queries based on the domain. However, synthetic data must be reviewed for quality—it can introduce unrealistic patterns or miss edge cases that real users produce .
Golden Dataset Table
| Element | Purpose | Source |
|---|---|---|
| Queries | Represent real inputs | Production logs, expert writing, synthetic |
| Reference answers | Ground truth | Domain experts, source documents |
| Source documents | Retrieval evaluation | Document store |
| Adversarial cases | Safety testing | Red teaming, known failure modes |
Tools and Frameworks for Evaluation
The evaluation tooling landscape has matured. Indian teams have options ranging from open-source frameworks to cloud platform services.
Open-Source Frameworks
DeepEval is a widely used open-source LLM evaluation framework. It provides metrics like G-Eval (custom criteria evaluated by an LLM), test case management, and integration with pytest. It supports RAG evaluation, agent evaluation, and tracing of nested components .
RAIL (Responsible AI Labs) provides a scoring API that evaluates AI outputs across eight dimensions including safety, fairness, reliability, transparency, privacy, accountability, inclusivity, and user impact. It includes a compliance engine that checks against 63 requirements across six regulatory frameworks including India’s DPDP Act and India AI Governance Guidelines. It also offers an Indian Responsible AI Benchmark with 212 adversarial prompts across 22 India-specific safety categories including caste, region, and Hinglish code-switching .
Cloud Platform Services
Google Cloud’s Gemini Enterprise Agent Platform offers agent and model evaluations with over 20 pre-built metrics spanning quality, safety, grounding, agent tool use, and trajectory. It supports adaptive rubrics (case-specific pass/fail criteria generated by an LLM judge), online monitors for continuous production evaluation, and issue clustering to group failures into actionable categories .
Microsoft Azure provides evaluation guidance and SDKs for testing AI workloads, with a focus on separating model evaluation (development) from system testing (whole system validation). It emphasises groundedness, fairness, and agent workflow validation .
India-Specific Benchmark
AI4Bharat’s Indic LLM Arena is a crowd-sourced platform for evaluating LLMs on Indian languages. It tests models across three pillars—language, context, and safety—using human votes to rank performance on Hindi, Tamil, Hinglish, and other Indian language inputs. For enterprises deploying multilingual AI, it provides a benchmark that global leaderboards do not .
Tools Table
| Tool | Type | Best For |
|---|---|---|
| DeepEval | Open-source | RAG, agents, pytest integration |
| RAIL | API / SDK | India-specific safety, DPDPA compliance |
| Google Agent Platform | Cloud | Adaptive rubrics, online monitors |
| Azure AI Evaluation SDK | Cloud | Groundedness, agent validation |
| Indic LLM Arena | Benchmark | Indian language evaluation |
Real Costs in India (2026)
AI evaluation costs have two components: the evaluation infrastructure/tooling and the human effort to build datasets and review results.
Implementation Costs
| Scope | Cost (₹) | Cost ($ approx.) | Timeline |
|---|---|---|---|
| Basic evaluation setup (golden dataset, 2–3 metrics) | ₹1–3 lakh | $1,200–$3,600 | 3–6 weeks |
| RAG evaluation (context precision, faithfulness, relevance) | ₹3–8 lakh | $3,600–$9,600 | 6–12 weeks |
| Agent evaluation (tool use, trajectory, task success) | ₹5–15 lakh | $6,000–$18,000 | 10–16 weeks |
| Continuous evaluation (offline + online, monitoring) | ₹10–25 lakh | $12,000–$30,000 | 14–24 weeks |
Running Costs
| Component | Monthly Cost (₹) | Monthly Cost ($ approx.) | Notes |
|---|---|---|---|
| LLM judge API | ₹10,000–₹60,000 | $120–$720 | Evaluation consumes tokens |
| Platform fees | ₹5,000–₹40,000 | $60–$480 | Cloud evaluation services |
| Human review | ₹15,000–₹75,000 | $180–$900 | Dataset curation, result review |
| Maintenance | ₹10,000–₹40,000 | $120–$480 | Dataset updates, metric tuning |
The Evaluation Cost Trade-off
Evaluation costs money. Running a golden dataset of 200 cases through an LLM judge costs approximately $0.50–$2 per run depending on the judge model and case complexity. If the evaluation is run on every commit, the cost is manageable. If it is run once and forgotten, the cost is wasted.
The real cost is not the token bill. It is the failure that evaluation prevents. A hallucinated refund policy that reaches 100 customers, or an agent that sends incorrect invoices, costs far more than the evaluation that would have caught it.
India-Specific Compliance: DPDPA and MeitY
DPDPA and Evaluation Datasets
Evaluation datasets often contain real user queries and documents. Under the DPDP Act:
Anonymisation: Personal data in the golden dataset should be anonymised or synthetic.
Purpose limitation: Data collected for support should not be repurposed for evaluation without a lawful basis.
Retention: Define retention periods for evaluation data.
Access control: Restrict access to evaluation datasets containing sensitive data.
MeitY AI Governance
MeitY’s guidelines expect transparency and accountability for AI systems. Evaluation documentation—what metrics were used, what the results were, what thresholds were set—supports this. For significant data fiduciaries, Rule 13 of the DPDP Rules requires due diligence that algorithmic software does not pose a risk to data principals’ rights. Evaluation evidence is part of that due diligence .
Indian Language Evaluation
Global benchmarks are English-centric. Models may perform differently on Hindi, Tamil, or Hinglish inputs. Evaluation datasets should include Indian language queries if the system serves Indian users. The Indic LLM Arena provides a benchmark for comparing models on Indian languages .
Compliance Checklist
| Requirement | Implementation | Priority |
|---|---|---|
| Dataset anonymisation | Remove or mask PII | High |
| Purpose limitation | Document evaluation use | High |
| Retention policy | Automated purge | Medium |
| Access control | Role-based dataset access | High |
| Evaluation documentation | Metrics, results, thresholds | High |
Implementation Framework: A Phased Approach With Cost Gates
Phase 1: Define Quality Criteria (Week 1–2)
Objective: Decide what "good" means for the specific use case.
Activities:
Define the quality dimensions that matter (accuracy, groundedness, safety).
Identify the failure modes that are unacceptable (hallucinated policies, PII leakage).
Set thresholds for pass/fail.
Choose initial metrics.
Cost: ₹50,000–₹1.5 lakh ($600–$1,800).
Go/No-Go: Quality criteria documented. Thresholds set.
Phase 2: Build Golden Dataset (Week 3–6)
Objective: Create a representative evaluation dataset.
Activities:
Collect real queries from logs or experts.
Write or verify reference answers.
Map queries to source documents.
Include edge cases and adversarial inputs.
Anonymise PII.
Cost: ₹1–3 lakh ($1,200–$3,600).
Go/No-Go: Dataset covers diverse cases. Reference answers verified.
Phase 3: Offline Evaluation (Week 7–12)
Objective: Run the system against the dataset and measure quality.
Activities:
Implement evaluation pipeline with chosen metrics.
Run baseline evaluation.
Identify failure patterns.
Fix issues (prompts, retrieval, chunking).
Re-run to measure improvement.
Cost: ₹2–6 lakh ($2,400–$7,200).
Go/No-Go: Metrics meet thresholds. Failure patterns understood.
Phase 4: Online Monitoring (Week 13–20)
Objective: Add production monitoring.
Activities:
Implement online scoring of live interactions.
Set up alerts for quality degradation.
Build dashboards for quality metrics.
Establish feedback loop to golden dataset.
Cost: ₹5–15 lakh ($6,000–$18,000).
Go/No-Go: Online monitoring works. Alerts fire correctly.
Phase 5: Continuous Improvement (Ongoing)
Objective: Maintain and improve quality.
Activities:
Add production failures to golden dataset.
Re-run evaluation on changes.
Tune metrics and thresholds.
Expand to new use cases.
Cost: ₹10,000–₹40,000/month ($120–$480).
Phase Summary Table
| Phase | Duration | Key Deliverable | Cost (₹) | Go/No-Go Criteria |
|---|---|---|---|---|
| 1. Define Criteria | 2 weeks | Quality dimensions, thresholds | 50,000–1.5 lakh | Criteria documented |
| 2. Golden Dataset | 4 weeks | Evaluation dataset | 1–3 lakh | Diverse, verified |
| 3. Offline Evaluation | 6 weeks | Metrics, failure analysis | 2–6 lakh | Thresholds met |
| 4. Online Monitoring | 8 weeks | Production scoring, alerts | 5–15 lakh | Monitoring works |
| 5. Continuous | Ongoing | Feedback loop | 10,000–40,000/month | Quality maintained |
Risks, Pitfalls, and Governance
The Most Common Evaluation Failures
The vibe check. Testing with ten prompts and declaring the system ready. Mitigation: Golden dataset, statistical metrics.
Evaluating only the final answer. Not measuring retrieval quality, so the root cause of failures is invisible. Mitigation: Component-level metrics (context precision, faithfulness).
No production monitoring. Quality degrades silently after launch. Mitigation: Online evaluation, drift alerts.
Stale golden dataset. The dataset does not reflect current user behaviour. Mitigation: Feed production failures into the dataset.
Over-reliance on LLM judges. The judge model may have its own biases. Mitigation: Combine LLM judges with code-based metrics and human review.
Ignoring Indian languages. The system is evaluated only in English but deployed to Hindi-speaking users. Mitigation: Include Indian language cases.
Risk Summary Table
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Vibe check | High | High (production failures) | Golden dataset |
| Final-answer-only | High | Medium (misdiagnosis) | Component metrics |
| No monitoring | High | High (silent drift) | Online evaluation |
| Stale dataset | High | Medium | Production feedback |
| LLM judge bias | Medium | Medium | Human review, code metrics |
| English-only | High | High (Indian users) | Indian language cases |
Benchmark Summary and Decision Framework
Decision Matrix: Evaluation Approach by System Type
| System Type | Primary Metrics | Evaluation Approach |
|---|---|---|
| RAG chatbot | Context precision, faithfulness, relevance | Offline + online |
| AI agent | Tool use, trajectory, task success | Offline + online + simulation |
| Summarisation | ROUGE, factual consistency | Offline (reference-based) |
| Classification | Accuracy, precision, recall | Offline (labeled data) |
| Multilingual | Per-language accuracy, cultural safety | Offline + Indic benchmark |
Evaluation Priority Table
| Priority | Capability | Why |
|---|---|---|
| 1 | Golden dataset | Foundation of all evaluation |
| 2 | Faithfulness/Groundedness | Prevents hallucination |
| 3 | Context precision | Diagnoses retrieval failures |
| 4 | Answer relevance | Ensures query alignment |
| 5 | Online monitoring | Catches production drift |
| 6 | Indian language cases | Serves Indian users |
The One-Paragraph Decision Rule
If you are deploying an LLM or RAG system, do not ship on a demo. Build a golden dataset of 100–300 representative cases, measure faithfulness and context precision, set thresholds, and block deployment until they are met. Add online monitoring before launch, not after. Feed production failures back into the dataset. Include Indian language and India-specific adversarial cases. And document the evaluation—it is both a quality tool and a governance asset.
Frequently Asked Questions
1. What is AI evaluation and why does it matter?
AI evaluation is the process of measuring whether an AI system produces accurate, grounded, relevant, and safe outputs. It matters because LLMs are probabilistic—they produce confident wrong answers without throwing errors. Evaluation catches failures before users see them.
2. What is a golden dataset?
A golden dataset is a curated collection of queries, reference answers, and source documents used for offline evaluation. It is the foundation of repeatable, objective testing .
3. What is the difference between offline and online evaluation?
Offline evaluation runs before deployment against a known dataset. Online evaluation runs in production against live traffic, using scoring models to catch anomalies and drift .
4. What metrics matter for RAG?
Context precision (did retrieval find the right documents), faithfulness (is the answer grounded in those documents), and answer relevance (does the answer address the query) .
5. What tools are available for LLM evaluation?
DeepEval (open-source), RAIL (India-specific safety and compliance), Google Agent Platform evaluations, Azure AI Evaluation SDK, and Indic LLM Arena for Indian language benchmarking .
6. How many examples do I need in a golden dataset?
100–300 diverse cases is sufficient for meaningful evaluation. Fewer than 50 is unlikely to capture enough variation.
7. How does DPDPA affect AI evaluation?
Evaluation datasets may contain personal data. Anonymisation, purpose limitation, retention policies, and access control apply.
8. How do I evaluate Indian language performance?
Include Hindi, Tamil, Hinglish, and other Indian language cases in the golden dataset. Use Indic LLM Arena for benchmarking model performance on Indian languages .
9. How much does AI evaluation cost in India?
Basic setup costs ₹1–3 lakh ($1,200–$3,600). RAG evaluation costs ₹3–8 lakh ($3,600–$9,600). Agent evaluation costs ₹5–15 lakh ($6,000–$18,000). Running costs add ₹10,000–₹60,000/month for LLM judge APIs.
10. How often should I re-run evaluation?
Run offline evaluation on every change (prompt, model, retrieval). Run online evaluation continuously. Feed production failures into the golden dataset.
11. What is the vibe check and why is it dangerous?
The vibe check is testing with a few prompts and judging outputs by feel. It is dangerous because it misses edge cases, does not measure statistical quality, and gives false confidence .
12. How can Innovative AI Solutions help?
Innovative AI Solutions builds AI evaluation capabilities for Indian businesses—golden datasets, RAG metrics, agent evaluation, online monitoring, and DPDPA-compliant data handling. Our Delhi-based team handles assessment, implementation, and ongoing evaluation. Visit https://innovativeais.com to discuss your requirements.
Contact Innovative AI Solutions
Ready to know whether your AI is actually ready for production?
[One-line supporting sentence.]
Contact Information
Innovative AI Solutions
📍 Netaji Subhash Place, Pitampura, Delhi – 110034
🌐 Website: https://innovativeaislabs.com 📧 Email: info@innovativeais.com
📞 Phone: +91 7464 099 059 / +91 96899 67356
Business Services
AI Automation
AI Development
AI Consulting
Machine Learning Solutions
Deep Learning Solutions
Generative AI Services
NLP Solutions
AI Agents
AI Chatbots
Voice AI
CRM Development
Custom Software Development
Website Development
Mobile App Development
About the Author
Abhishek Kumar
Founder & CEO, Innovative AI Solutions
5+ years building production AI systems for Indian businesses. Based in Delhi, serving clients across India.
Ready to build AI solutions for your business?
Innovative AI Solutions — Delhi's leading AI development company. Free consultation available.
Get Free Consultation →
A practical 2026 guide for Indian teams shipping LLMs—golden datasets, RAG metrics, offline and online evaluation, real rupee costs, and DPDPA and MeitY-compliant governance.
#AIEvaluation #LLMTesting #LLMAccuracy #GoldenDataset #RAGEvaluation #Faithfulness #AnswerRelevance #LLMAsJudge #AIQualityAssurance #HallucinationTesting #DPDPA #MeitY #AIRedTeaming #LLMRegressionTesting #DelhiNCR #InnovativeAISolutions
Copyright ©️ 2015–2026 Innovative AI Solutions. All Rights Reserved. | Privacy Policy | Terms & Conditions
