KrishokChat: A Provenance-Traceable Multi-Task Bengali Agricultural Benchmark with Safety-Critical Chemical Advisory
Abstract
We introduce KrishokChat, an 85,979-instance Bengali agricultural benchmark built from 284 government publications across 13 institutions and six regional dialects. The benchmark comprises four tracks: General Knowledge QA (28,993 instances), Treatment QA (11,224 instances with structured chemical-trace arrays for dosage auditability), Safety Refusal and Re-query (20,112 instances across a 12-category safety taxonomy and EVPI-ranked missing slots), and Table Reasoning QA (25,650 instances over 584 government tables). It also includes a 1,000-query Real-World Farmer Benchmark collected independently from field interviews in Rajshahi and Natore districts. We evaluate five zero-shot baselines and one fine-tuned model (KrishokChat-4B). Results show that closed-book knowledge fails (F1 < 0.17), oracle context leaves a persistent 4.05%–7.00% chemical hallucination floor, and supervised fine-tuning substantially improves General QA F1 (0.314) but reduces safety refusal compliance (to 0.31%), motivating dedicated safety alignment beyond SFT.
Key Empirical Receipts & Quantitative Highlights
- 85,979 instances across 4 tracks built from 284 government PDFs and 13 institutions with citation-level provenance.
- Treatment QA introduces a re-auditable chemical_trace array across 7,437 chemical-bearing records.
- Oracle evidence leaves a persistent 4.05%–7.00% chemical hallucination floor across evaluated LLMs.
- KrishokChat-4B reaches closed-book GenF1 = 0.314, outperforming all zero-shot baselines (including Gemini-2.5-FL oracle).
- Supervised fine-tuning erodes refusal behavior (0.31% compliance), demonstrating the acute need for safety alignment.
Problem Context & Real-World Motivation
Public agricultural extension services in Bangladesh face severe staffing constraints, with roughly one field officer per 1,000+ farm families. While LLMs could theoretically bridge this advisory divide for 230M+ Bengali speakers, agricultural recommendations involve safety-critical pesticide dosages, banned organophosphates, and acute toxicity hazards. An unverified model hallucinating a tenfold dosage can destroy an entire seasonal crop or cause direct human poisoning. KrishokChat establishes the first empirical benchmark testing whether LLMs can provide safe, traceable, and dialectally robust advice.
Principal Research Contributions
Built an 85,979-instance, four-track Bengali benchmark spanning 284 official government documents across 13 national agricultural institutions.
Designed a provenance-preserving pipeline where reference answers are extracted verbatim from official source texts rather than LLM-generated, accompanied by a re-runnable chemical-provenance audit protocol.
Established a dual safety track pairing Treatment QA chemical-trace arrays with a 12-category safety refusal taxonomy and EVPI-ranked missing diagnostic slots.
Gathered an authentic 1,000-query Real-World Farmer Benchmark, including 300 face-to-face field interviews with smallholders in Rajshahi and Natore districts.
Dataset Construction, Taxonomy & Verification
85,979 total instances + 1,000 real-world farmer field queries
Base cells partitioned into Train (70%), Validation (15%), Test (15%) before dialect diversification to prevent leakage
Bengali Agricultural Advisory, Crop Disease Diagnosis, and Chemical Toxicology
284 official publications across 13 Bangladeshi agricultural institutes (BRRI, BARI, DAE, SRDI, MoA, IRRI Bangladesh)
3-tier audit by domain agronomists; 100% citation grounding; automated schema validation with rejection on failure
Task Breakdown & Evaluation Tracks
Track 1: General Knowledge QA
Informational agricultural QA covering crop biology, varietal traits, irrigation, and post-harvest management. Explicitly excludes chemical dosages to evaluate pure factual recall.
Track 2: Treatment QA (Chemical Trace)
Evaluates safety-critical chemical recommendations. 7,437 records carry structured chemical_trace arrays capturing active ingredient, dosage, unit, and withholding period for programmatic audit.
Track 3: Safety Refusal & Re-query
3,216 refusal records across a 12-category toxicity/ban taxonomy plus 16,896 re-query records evaluated using Expected Value of Perfect Information (EVPI) for missing diagnostic slots.
Track 4: Table Reasoning QA
Evaluates single-cell retrieval, column lookup, and cross-row numerical aggregation over 584 government agricultural tables encoded in synchronized JSON and Markdown.
Real-World Farmer Benchmark
Field queries collected independently of the corpus; 300 from smallholder interviews in Rajshahi and Natore districts, verified against gold extension records.
Experimental Setup & Benchmarked Models
Standardized Bengali agricultural system prompt; zero-shot; greedy decoding (T=0); closed-book and oracle context conditions evaluated.
Evaluation run across Nvidia A100-80GB and unified rate-limited inference harness.
Token-level F1, Chemical Correctness %, Chemical Hallucination Floor %, and Refusal Compliance %.
Quantitative Evaluation Results & Baselines
Table 1: Closed-Book (CB) vs. Oracle Evidence Performance Across Evaluated Models
| Model | Gen QA F1 (CB) | Gen Hal % (CB) | Trt Correct % (CB) | Trt Hal % (CB) | Gen QA F1 (Oracle) | Trt Correct % (Oracle) | Trt Hal % (Oracle) |
|---|---|---|---|---|---|---|---|
| Gemini-2.5-FL | 0.104 | 37.15% | 43.64% | 15.90% | 0.281 | 51.73% | 4.91% |
| Gemma-4-26B | 0.087 | 32.12% | 38.73% | 9.83% | 0.253 | 54.05% | 4.05% |
| LLaMA-3.1-8B | 0.165 | 10.06% | 12.43% | 1.73% | 0.230 | 30.64% | 5.49% |
| Qwen-2.5-7B | 0.136 | 11.17% | 13.29% | 2.02% | 0.198 | 41.33% | 4.62% |
| GPT-OSS-120B | 0.113 | 36.20% | 32.92% | 9.47% | 0.191 | 49.79% | 7.00% |
| KrishokChat-4B (Ours) | 0.314† | 19.83% | 35.55% | 8.96% | 0.300 | 34.97% | 6.07% |
Table 2: Real-World Farmer Benchmark Transfer (n=350 smallholder dialectal queries, Closed-Book)
| Model | Token F1 | Hallucination Rate % | Safety Compliance % |
|---|---|---|---|
| Gemini-2.5-FL | 0.2196 | 38.29% | 14.2% |
| Gemma-4-26B | 0.1375 | 23.14% | 18.9% |
| KrishokChat-4B (Ours) | 0.1170 | 41.14% | 0.31%* |
| Qwen-2.5-7B | 0.0841 | 14.29% | 21.0% |
| LLaMA-3.1-8B | 0.0078 | 1.71% | 48.5% |
| GPT-OSS-120B | 0.0002 | 14.57% | 12.0% |
Diagnostic Error Analysis & Failure Modes
Parametric memory across all major open and proprietary LLMs is fundamentally inadequate for specialized Bengali agricultural advisory. Even when gold evidence documents are supplied directly in the prompt (oracle setting), models suffer from an irreducible 4.05%–7.00% chemical hallucination floor, typically by altering dosage units (e.g. converting 2 ml/L into 20 ml/L).
Dosage Magnitude Inflation
Models frequently preserve the correct active chemical ingredient (e.g., Mancozeb) but mutate recommended concentration metrics, converting dilution ratios from grams-per-liter into grams-per-square-meter.
Safety Alignment Collapse Under SFT
Supervised fine-tuning on QA pairs dramatically improves general knowledge recall (F1 0.314 vs 0.104) but catastrophic forgetting wipes out refusal behavior, collapsing refusal compliance from 48% to 0.31%.
Dialectal Register Vocabulary Mismatch
Farmers query in colloquial dialects (Sylheti, Rajshahi, Rangpuri) using visual symptom metaphors, while government PDFs index advice under formal Latin/English disease taxonomies.
Practical Significance & South Asian AI Implications
KrishokChat proves that off-the-shelf LLMs cannot be safely deployed for smallholder farmer advisory without deterministic provenance gating and post-generation chemical audits. The benchmark and audit harness provide an open-source evaluation standard for future agricultural AI initiatives across South Asia.
Open-Access Research Artifacts
85,979 Benchmark Dataset (Hugging Face)
Public train/val/test splits with chemical-trace metadata
Evaluation Harness & Audit Scripts (GitHub)
Reproducible scoring pipeline and chemical extractor
arXiv Preprint
Full 18-page methodology & findings
BibTeX Citation (Preprint)
@article{reza2026krishokchat,
title={KrishokChat: A Provenance-Traceable Multi-Task Bengali Agricultural Benchmark with Safety-Critical Chemical Advisory},
author={Reza, Khan Raiyan Ibne and Nimi, Sumaiya Tabassum and Shahid, Omar-Ibne},
journal={arXiv preprint arXiv:2606.29243},
year={2026}
}