Where Does Retrieval Fail? Evaluating RAG Architectures for Agricultural Advisory
Abstract
Retrieval quality in RAG systems is commonly reported as a single aggregate score, which can hide large differences across query types and language conditions. We study this problem in Bengali agricultural advisory, where farmer queries are often colloquial while official advisory documents use formal scientific terminology. We construct a test collection of 1,000 queries and 2,882 knowledge nodes extracted from 284 official Bangladeshi agricultural publications (with 19,768 entities and 17,501 factual triples), and evaluate five retrieval architectures and six embedding models across three controlled language conditions (BN→BN, EN→BN, EN→EN). Results show that Hybrid RRF achieves the highest overall R@10 (0.539), while dense retrieval exhibits a sharp register divide: R@10 collapses to 0.093 on colloquial farmer queries but reaches 0.970 on formal safety queries due to a severe symptom-to-entity lexical gap (96.4% Jaccard < 0.10). Furthermore, cross-lingual querying causes BM25 to collapse by 99% (R@10: 0.506 → 0.004) while dense retrieval falls only 8% (0.464 → 0.425).
Key Empirical Receipts & Quantitative Highlights
- Hybrid RRF (Gemini + BM25) achieves the highest overall retrieval score in Bengali (R@10=0.539, MRR=0.551).
- Dense retrieval exhibits a bimodal register failure: R@10 is 0.093 on colloquial farmer queries vs. 0.970 on formal safety queries.
- Symptom-to-entity register gap: 96.4% of queries have Jaccard < 0.10 with gold documents; only 3.5% of gold entity names appear verbatim in farmer queries.
- Cross-lingual script collapse: BM25 drops by 99% under EN→BN querying (0.506 → 0.004), while dense retrieval preserves 92% performance.
- Configuration audit: task-type mismatch reduces Gemini Dense R@10 by 7× (0.464 → 0.063); 128-token truncation drops ColBERT by 23%.
Problem Context & Real-World Motivation
In high-stakes domain-specific RAG pipelines, aggregate evaluation metrics like Mean Reciprocal Rank (MRR) obscure fatal localized retrieval breakdowns. When smallholder farmers ask for advice, they express problems in regional colloquial terms ("leaves curling with yellow spots"), whereas extension manuals index knowledge under formal botanical pathogen entities ("Xanthomonas oryzae"). This investigation dissects where, why, and how different retriever architectures fail across language boundaries, query registers, and chunking configurations.
Principal Research Contributions
Engineered an empirical test collection for low-resource Bengali retrieval comprising 284 official PDFs, 2,882 atomic knowledge nodes, 19,768 extracted entities, 17,501 triples, and 1,000 annotated queries.
Developed a canonical node representation unifying natural language content, structured facts, and deterministic source metadata with 100% human-verified provenance.
Conducted systematic cross-architectural diagnosis comparing BM25, Gemini Dense, BGE-M3 Dense, ColBERT multi-vector, and Hybrid Reciprocal Rank Fusion (RRF).
Uncovered the "symptom-to-entity register gap" and demonstrated the massive configuration vulnerability of embedding API task-type assignments.
Dataset Construction, Taxonomy & Verification
2,882 structured knowledge nodes, 1,000 queries, 19,768 entities, 17,501 factual triples
900 answerable queries (300 farmer-anchored, 400 KG-grounded, 200 safety-critical) + 100 low-agreement queries withheld
Bengali Agricultural RAG, Dialectal Information Retrieval
284 authoritative publications from BRRI, IRRI, DAE, SRDI, and MoA spanning 1999–2024
3 independent agricultural expert annotators (Fleiss κ = 0.81); bounded open IE schema
Task Breakdown & Evaluation Tracks
Farmer-Anchored Queries
Colloquial farmer phrasing collected from field interviews; high dialectal variance, zero explicit entity keywords.
KG-Grounded Relational Queries
Constructed from verified knowledge-graph triples requiring multi-hop reasoning across varieties, symptoms, and pesticides.
Safety-Critical Advisory Queries
Formal, technical queries demanding exact pesticide dosage, banned chemicals, and toxicity precautions.
Controlled Language Conditions
Bengali to Bengali (BN→BN), Cross-lingual English to Bengali (EN→BN), and English to English (EN→EN).
Experimental Setup & Benchmarked Models
Asymmetric query/document embeddings; cosine similarity with FAISS IndexFlatIP; 10,000 BCa bootstrap resamples; Holm-Bonferroni correction.
Dual Nvidia RTX 4090 (24GB) and Google Cloud Vertex AI Embedding API harness.
Recall@1, Recall@5, Recall@10, MRR, nDCG@10, and Semantic Separation Score.
Quantitative Evaluation Results & Baselines
Table 1: Bengali Architecture Comparison (900 queries, BN→BN Native Condition)
| Architecture | Recall@1 | Recall@5 | Recall@10 | MRR | nDCG@10 |
|---|---|---|---|---|---|
| Hybrid RRF (Gemini + BM25) | 0.291 | 0.466 | 0.539 | 0.551 | 0.461 |
| BM25 (Sparse Keyword) | 0.205 | 0.405 | 0.506 | 0.481 | 0.407 |
| Dense (Gemini-embedding-001) | 0.320 | 0.431 | 0.464 | 0.514 | 0.431 |
| ColBERT (BGE-M3 Multi-Vector) | 0.255 | 0.414 | 0.487 | 0.324 | 0.416 |
| Dense (BGE-M3 Native) | 0.281 | 0.369 | 0.408 | 0.432 | 0.370 |
Table 2: Query-Register Divergence (Recall@10 by Query Type)
| Architecture | Farmer (Colloquial) | Safety (Formal) | KG-Grounded (Relational) |
|---|---|---|---|
| Dense (Gemini-embedding-001) | 0.093 [0.06, 0.13] | 0.970 [0.94, 0.99] | 0.489 [0.44, 0.54] |
| BM25 (Sparse) | 0.523 [0.47, 0.58] | 0.539 [0.47, 0.60] | 0.478 [0.43, 0.53] |
| ColBERT (BGE-M3) | 0.210 [0.17, 0.26] | 0.675 [0.60, 0.74] | 0.600 [0.55, 0.65] |
Table 3: Language Boundary Effect (Script Invariance Audit)
| Setting | Dense (Gemini-001) | BM25 (Sparse) | Performance Margin |
|---|---|---|---|
| Native Bengali (BN→BN) | 0.464 | 0.506 | BM25 leads (+9.0%) |
| Cross-Lingual (EN→BN) | 0.425 | 0.004 | Dense leads (100x advantage) |
| Translated English (EN→EN) | 0.442 | 0.384 | Dense leads (+15.1%) |
Diagnostic Error Analysis & Failure Modes
Dense retrieval in low-resource settings suffers from severe bimodal polarization: it achieves near-perfect recall (0.970) on formal safety queries but catastrophic collapse (0.093) on colloquial farmer queries. This stems from the symptom-to-entity lexical gap, where 96.4% of queries have a Jaccard overlap under 0.10 with the target document.
Symptom-to-Entity Lexical Disconnect
Only 3.5% of official botanical entity names appeared in raw farmer queries. Farmers describe visual observations ("yellow leaves, drying tips") while documents index under disease taxonomy ("Rice Tungro Spherical Virus").
Asymmetric Task-Type Sensitivity
Failing to declare RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT in the Gemini Embedding API reduced Recall@10 from 0.464 to 0.063—a 7.3× performance penalty caused purely by bad configuration.
Context Truncation in Bengali Multi-Vector Encoders
At standard 128-token limits, Bengali knowledge nodes (average 1,180 chars) are truncated by 95%, causing ColBERT to fail. Expanding context to 512 tokens restored ColBERT to 0.487.
Practical Significance & South Asian AI Implications
Demonstrates that hybrid retrieval (BM25 + Dense RRF) is non-negotiable in low-resource production systems. Dense models cannot handle colloquial register gaps alone, and keyword search fails completely under cross-lingual querying.
Open-Access Research Artifacts
AgriTrust-RAG Dataset (Hugging Face)
2,882 nodes + 1,000 queries with triple annotations
RRF Evaluation Harness (GitHub)
FAISS + BM25 evaluation pipeline
arXiv Preprint
Full 16-page paper with appendix
BibTeX Citation (Preprint)
@article{reza2026where,
title={Where Does Retrieval Fail? Evaluating RAG Architectures for Agricultural Advisory},
author={Reza, Khan Raiyan Ibne and Maria, Sanjana Aktar and Nimi, Sumaiya Tabassum},
journal={arXiv preprint arXiv:2608.14886},
year={2026}
}