Research monograph & bibliography
Publications & empirical benchmarks.
Five human-verified benchmarks and preprints across low-resource Bengali NLP, RAG retrieval failure diagnostics, tutoring reasoning traces, and multimodal document understanding.
Academic Integrity & Preprint Transparency Notice
In adherence to strict academic honesty, all 5 research papers listed below are preprints or under active peer review (including arXiv-indexed manuscripts). We have zero published or accepted conference/journal papers claimed. Every metric, dataset split, and code checkpoint is accompanied by direct repository links and downloadable manuscripts.
5
Research Preprints
All preprints / under review
114,000+
Evaluated Instances
Across 4 major domains
13
National Institutes
BRRI, BARI, DAE, SRDI, Vumi
6
Regional Dialects
Standard, Sylheti, Chittagong...
Complete Preprint Bibliography (5 Papers)
Chronological ArchiveKrishokChat: A Provenance-Traceable Multi-Task Bengali Agricultural Benchmark with Safety-Critical Chemical Advisory
Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi, Omar-Ibne Shahid
We introduce KrishokChat, an 85,979-instance Bengali agricultural benchmark built from 284 government publications across 13 institutions and six regional dialects. The benchmark comprises four tracks: General Knowledge QA (28,993 instances), Treatment QA (11,224 instances with structured chemical-trace arrays for dosage auditability), Safety Refusal and Re-query (20,112 instances across a 12-category safety taxonomy and EVPI-ranked missing slots), and Table Reasoning QA (25,650 instances over 584 government tables). It also includes a 1,000-query Real-World Farmer Benchmark collected independently from field interviews in Rajshahi and Natore districts. We evaluate five zero-shot baselines and one fine-tuned model (KrishokChat-4B). Results show that closed-book knowledge fails (F1 < 0.17), oracle context leaves a persistent 4.05%–7.00% chemical hallucination floor, and supervised fine-tuning substantially improves General QA F1 (0.314) but reduces safety refusal compliance (to 0.31%), motivating dedicated safety alignment beyond SFT.
- 85,979 instances across 4 tracks built from 284 government PDFs and 13 institutions with citation-level provenance.
- Treatment QA introduces a re-auditable chemical_trace array across 7,437 chemical-bearing records.
- Oracle evidence leaves a persistent 4.05%–7.00% chemical hallucination floor across evaluated LLMs.
Where Does Retrieval Fail? Evaluating RAG Architectures for Agricultural Advisory
Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi
Retrieval quality in RAG systems is commonly reported as a single aggregate score, which can hide large differences across query types and language conditions. We study this problem in Bengali agricultural advisory, where farmer queries are often colloquial while official advisory documents use formal scientific terminology. We construct a test collection of 1,000 queries and 2,882 knowledge nodes extracted from 284 official Bangladeshi agricultural publications (with 19,768 entities and 17,501 factual triples), and evaluate five retrieval architectures and six embedding models across three controlled language conditions (BN→BN, EN→BN, EN→EN). Results show that Hybrid RRF achieves the highest overall R@10 (0.539), while dense retrieval exhibits a sharp register divide: R@10 collapses to 0.093 on colloquial farmer queries but reaches 0.970 on formal safety queries due to a severe symptom-to-entity lexical gap (96.4% Jaccard < 0.10). Furthermore, cross-lingual querying causes BM25 to collapse by 99% (R@10: 0.506 → 0.004) while dense retrieval falls only 8% (0.464 → 0.425).
- Hybrid RRF (Gemini + BM25) achieves the highest overall retrieval score in Bengali (R@10=0.539, MRR=0.551).
- Dense retrieval exhibits a bimodal register failure: R@10 is 0.093 on colloquial farmer queries vs. 0.970 on formal safety queries.
- Symptom-to-entity register gap: 96.4% of queries have Jaccard < 0.10 with gold documents; only 3.5% of gold entity names appear verbatim in farmer queries.
TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model
Khan Raiyan Ibne Reza, Research Collaborators
Bangla-English tutoring requires more than producing a correct translation: learners also need explanations of grammar differences, awareness of their likely errors, and targeted practice. We present TRACE-BN, a curriculum-guided dataset of 4,099 structured tutoring traces for Bangla-speaking learners of English at the CEFR A1–A2 level across 13 NCTB grammar units and 15 situational domains. Each trace encodes seven multi-task fields: word-level glosses, literal translation, natural translation, Bangla contrastive grammar notes, a common learner mistake, a practice question, and its answer. We transfer this multi-component tutoring behavior to Qwen3-0.6B using LoRA with 4-bit quantization (378.3 MB GGUF) for offline, on-device deployment. On 432 held-out examples, schema validity improves from 85.4% to 95.8%, chrF++ rises from 15.28 to 34.77 (+19.49 gain), and BLEU increases from 4.52 to 21.03, supported by dual automated judges and a 100-trace educator audit (4.71/5.00 overall rating).
- 4,099 curriculum-guided 7-field structured tutoring traces based on NCTB Classes 9–10 English curriculum (CEFR A1–A2).
- Transfers structured tutoring behavior to a sub-1B model (Qwen3-0.6B) deploying at 378.3 MB GGUF (75.47 tokens/sec).
- Schema validity increases from 85.4% to 95.8%; chrF++ improves by +19.49 (15.28 → 34.77) and BLEU from 4.52 to 21.03.
ChitraMiti: A Benchmark for Cross-Modal Consistency in Bengali Geometric Reasoning
Khan Raiyan Ibne Reza, Research Collaborators
Vision-language models perform well on multimodal reasoning benchmarks, but evaluation remains thin for low-resource languages and for geometry problems that require jointly reading diagrams and text. We introduce ChitraMiti-12.8k, a synthetic benchmark of 12,874 Bengali planar geometry problems paired with 15-attribute structured descriptions, and NCTB-500, a complementary set of 500 expert-curated geometry problems extracted directly from official Bengali NCTB textbooks (Classes 6–10; 78.6% hard). Using a three-phase protocol (Diagram-only Phase A, Diagram+Description Phase B, Description-only Phase C), we show that Phase B and Phase C are statistically equivalent (TOST α=0.05, ±5% margin). However, diagnostic adversarial spatial swaps reveal severe confirmation bias: models catch text-diagram contradictions only 6%–18% of the time, blindly following manipulated descriptions. QLoRA fine-tuning on ChitraMiti-12.8k improves in-distribution accuracy (up to 34.2%) and transfers 60%–65% of absolute gains to real textbook diagrams in NCTB-500.
- Pairs 12,874 synthetic planar geometry problems (ChitraMiti-12.8k) with 500 real textbook problems (NCTB-500).
- 15-attribute structured description schema captures complete geometric relationships (TOST equivalence confirmed, p < 0.01).
- VLM confirmation bias: models detect adversarial spatial contradictions only 6%–18% of the time, blindly trusting text over vision.
KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records
Khan Raiyan Ibne Reza, Research Collaborators
Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs (U+09F4–U+09F9), no mainstream font, and no OCR coverage. We introduce KhatianDoc, a four-task benchmark built from 107 real RS Khatian records obtained through the Vumi (Land) Office of Munshiganj district, spanning symbol recognition (Task 1, 95 crops), base-16 arithmetic conversion (Task 2, 53 targets), structured field extraction (Task 3, 261 rows), and legal document QA over 1,634 QA pairs (Task 4). Ground truth was transcribed by hand, verified line-by-line by a land-law practitioner, and anonymized via positional tokens. Evaluating six frontier multimodal LLMs (8B to 72B+) reveals that five QA categories (39.3% of evaluation set) return zero correct answers across every model, and all models score worse than a constant-mean baseline on base-16 arithmetic (MAE > 0.40 vs. 0.237 baseline), demonstrating a complete capability void rather than partial competence.
- First machine-checkable benchmark for Bengali RS Khatian records and Ana-Ganda-Kora-Kranti-Til base-16 fractions.
- 107 real Vumi-office records, 1,634 QA pairs, 100% human-transcribed and verified line-by-line by a land-law practitioner.
- Hard reasoning floor: 5 complex legal QA categories (39.3% of evaluation set) score 0% across all six frontier multimodal LLMs.
Four Systemic Failures of Frontier LLMs in Low-Resource Bengali
Synthesizing findings across our 5 benchmarks evaluating 12 open and proprietary model families (Gemini, Gemma, Qwen, LLaMA, GPT-4o):
1. The Symptom-to-Entity Register Gap
In real-world Bengali queries, 96.4% exhibit a Jaccard overlap < 0.10 with official documents. Farmers use colloquial symptom metaphors, while dense retrieval collapses to 0.093 recall unless bridged with sparse BM25 fusion.
2. Persistent Chemical Hallucination Floor
Supplying oracle evidence reduces factual errors but leaves a persistent 4.05%–7.00% dosage error floor in chemical recommendations, proving that generation must be gated by post-retrieval chemical audits.
3. VLM Confirmation Bias Over Visuals
When descriptions contradict geometric diagrams, vision-language models catch the error only 6%–18% of the time, statistically relying on language priors rather than diagram pixels (TOST equivalence confirmed).
4. Total Capability Void in Base-16 Records
Every evaluated model scores 0% on 39.3% of legal land deed QA categories due to unindexed Ana-Ganda Unicode glyphs (U+09F4–U+09F9), performing worse than a constant-mean statistical baseline on arithmetic.
Access Curated Hugging Face Datasets & Leaderboards
Download our verified Bengali benchmarks, training splits, and evaluation harnesses to test your own models against these failure modes.
For Graduate Admissions Committees & Peer Reviewers
If you are reviewing our submissions or evaluating our academic background for graduate admissions, we are glad to provide raw evaluation logs, unredacted research manuscripts, or custom benchmark runs on request.