Benchmarks & evaluation
Benchmarks that expose hidden failure modes.
I build benchmarks to expose failure modes that aggregate scores hide. Each one isolates a specific problem — retrieval mismatch, safety failure, visual-text inconsistency, or structured behavior in small models.
Most model evaluations report one number. The benchmarks below report the conditions under which systems fail — the query types, language conditions, document genres, and task categories an aggregate score hides.
How to read these: each sweep answers one failure-mode question — e.g., "can any retriever bridge the farmer-register gap?" Rows marked KEY are the headline result; OURS marks the proposed system or method. Full methodology lives in the linked paper.
Evaluation sweeps
5 benchmark projectsTRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model
Can a sub-1B model internalize multi-component tutoring behavior — translation, contrastive grammar, learner-error diagnosis, and practice — from structured supervision alone? Evaluated on 432 held-out traces.
| Model / Baseline | Metric | Result |
|---|---|---|
| Qwen3-0.6B (base, untuned)OURS | Schema Validity | 85.4% |
| Qwen3-0.6B (TRACE-BN tuned)KEY | Schema Validity | 95.8% |
| Qwen3-0.6B (base, untuned)OURS | chrF++ | 15.28 |
| Qwen3-0.6B (TRACE-BN tuned)KEY | chrF++ | 34.77 (+19.49) |
| Qwen3-0.6B (TRACE-BN tuned)OURS | BLEU | 21.03 |
| Tune → GGUF (378.3 MB, llama.cpp) | Edge Throughput | 75.47 tok/s |
Where Does Retrieval Fail? — RAG Architecture Comparison (BN→BN)
Which retriever is best for native Bengali agricultural queries, and does the answer hold across query types and language conditions? Verified results from arXiv:2608.14886.
| Model / Baseline | Metric | Result |
|---|---|---|
| Hybrid RRF (BM25 + Dense, k=60)KEY | R@10 | 0.539 (best) |
| BM25 (Okapi k1=1.5, b=0.75) | R@10 | 0.506 |
| ColBERT (BGE-M3, 512-token) | R@10 | 0.487 |
| Dense (Gemini-embedding-001) | R@10 | 0.464 |
| Dense on farmer queries | R@10 | 0.093 |
| Dense on safety queriesKEY | R@10 | 0.970 |
| BM25 EN→BN (cross-lingual) | R@10 | 0.004 (99% drop) |
KrishokChat — Closed-Book Knowledge & Chemical Hallucination Floor
Is parametric knowledge enough for safe agricultural advisory in Bengali? Findings from arXiv:2606.29243 across five zero-shot baselines and one fine-tuned model.
| Model / Baseline | Metric | Result |
|---|---|---|
| Best zero-shot baseline (Gemini-2.5-FL) | General QA Token F1 (closed-book) | 0.104 |
| KrishokChat-4B (LoRA SFT)KEY | General QA Token F1 (closed-book) | 0.314† |
| All baselines | General QA Token F1 | < 0.17 |
| Oracle-conditioned models | Treatment QA hallucination floor | 4.05–7.00% |
| KrishokChat-4B after SFT | Refusal compliance | 0.31% (alignment gap) |
ChitraMiti — Three-Phase Cross-Modal Consistency Protocol
Do VLMs verify text against diagrams, or rely on language priors? Verified results from arXiv:2609.12509 across five VLMs.
| Model / Baseline | Metric | Result |
|---|---|---|
| Phase B vs Phase C (all 5 VLMs)KEY | TOST equivalence | Statistically equivalent (±5% margin) |
| Adversarial spatial swap detection | Detection rate | 6–18% |
| Best zero-shot (Gemma-4-26B, NCTB-500) | Phase B accuracy | 45.2% |
| QLoRA FT-Qwen3.5-4BKEY | ChitraMiti-1k accuracy | 12.7 → 34.2% |
| FT gain transfer to NCTB-500KEY | Absolute gain carried over | 60–65% |
KhatianDoc — Capability Void on Bengali Legal Land Records
Can any frontier multimodal LLM read Bengali RS Khatian land records with Ana-Ganda base-16 fractions? Verified results from arXiv:2609.03597 (NLLP @ EMNLP 2026).
| Model / Baseline | Metric | Result |
|---|---|---|
| All 6 models (6/6 fail)KEY | QA categories at 0% | 39.3% of eval set |
| Constant-mean baseline | Base-16 arithmetic MAE | 0.237 |
| All models that emit numbers | Base-16 arithmetic MAE | 0.40–0.42 (worse than baseline) |
| Gemini 2.5 FL (after metric self-audit)KEY | Document QA EM | 13.33 → 21.67% |
| Ground truthKEY | Verification | 100% human + land-law review, zero OCR |
Nirapod Probaho — Multi-Agent Geospatial Triage Latency
System-level measurements for the disaster-response platform: does multi-agent orchestration meaningfully cut triage latency versus a sequential script?
| Model / Baseline | Metric | Result |
|---|---|---|
| Sequential single-agent script | Triage latency | 14.2s |
| CrewAI multi-agent pipeline | Triage latency | 3.4s |
| PostGIS spatial query optimization | Routing execution | 42ms |
| MapLibre GL vector tile pipeline | Client FPS (60Hz target) | 59.8 FPS |
How the benchmarks are run
Deterministic decoding
Zero temperature (T=0) greedy decoding across baseline LLM evaluations, eliminating sampling noise for reproducible token traces.
Statistical testing
BCa bootstrap resampling (10,000 iterations) with paired Wilcoxon signed-rank tests and Holm-Bonferroni correction.
Traceable to papers
Each table corresponds to a specific paper, linking directly to its methodology, data splits, and released assets.
Compute environment & evaluation harness
Evaluation GPUs
Nvidia A100-SXM4 (80GB) & Dual RTX 4090 (24GB VRAM)
Inference Engines
vLLM 0.6.4 (TensorRT-LLM) & OpenRouter unified gateway
Edge Quantization
Unsloth 4-bit LoRA + llama.cpp GGUF Q4_K_M export
Scoring Libraries
SymPy 1.13, FAISS 1.8.0, SacreBLEU, Rouge-Score, ANLS
Benchmark your model against these test splits
Evaluation manifests, scoring scripts, and test-split queries are shared with research labs and reviewers working on low-resource and South Asian NLP.