Evaluation matrix & sweeps
Benchmark evaluation registry.
Reproducible baselines and diagnostic sweeps across pedagogical reasoning, dialectal retrieval, tabular grounding, and multimodal geometry.
Deterministic Protocol
Zero temperature (T=0) greedy decoding across all baseline LLM evaluations, eliminating sampling noise and guaranteeing reproducible token traces.
Statistical Significance
BCa bootstrap resampling (10,000 iterations) with paired Wilcoxon signed-rank tests and Holm-Bonferroni correction for multi-hypothesis validation.
Preprint-Traceable
Every table corresponds strictly to our 5 research preprints under review, linking directly to underlying methodology and data splits.
Active Evaluation Sweeps
5 Diagnostic RegistriesTRACE-BN: Multi-Turn Bengali Tutoring Reasoning Benchmark
Evaluating multi-turn conversational reasoning, pedagogical step verification, and code-mixed Bangla-English tutoring dialogues comparing foundation LLMs and reasoning chains.
| Model / Baseline | Metric | Result |
|---|---|---|
| GPT-4o (Zero-Shot Chain-of-Thought) | Pedagogical Accuracy | 74.2% |
| Claude 3.5 Sonnet (CoT) | Pedagogical Accuracy | 78.6% |
| Gemini 1.5 Pro (Bengali Context) | Pedagogical Accuracy | 76.4% |
| TRACE-BN Fine-Tuned Llama 3 (8B) | Pedagogical Accuracy | 82.1% |
| Code-Mixed Error Localization | F1 Score | 0.842 |
| Step Verification Precision | Precision@1 | 89.3% |
Where Does Retrieval Fail? Diagnostic Sweeps on Low-Resource Bengali RAG
Empirical failure boundary analysis measuring retrieval recall@k, semantic drift, and ranking degradation across standard Bengali, regional dialects, and technical register shifts.
| Model / Baseline | Metric | Result |
|---|---|---|
| BM25 Standard (k1=1.2, b=0.75) | MRR@10 (Dialectal) | 0.362 |
| BM25 Dialect-Tuned (k1=2.2, b=0.4) | MRR@10 (Dialectal) | 0.518 |
| Dense mE5-small (FAISS Index) | MRR@10 (Dialectal) | 0.564 |
| Multilingual BGE-M3 (Dense) | MRR@10 (Dialectal) | 0.602 |
| Hybrid Reciprocal Rank Fusion (BM25 + mE5) | MRR@10 (Dialectal) | 0.674 |
| Dialect Normalizer + Hybrid RRF | Recall@5 | 0.789 |
KrishokChat Dual-Retriever Evaluation Sweep
Evaluating retrieval recall@k and mean reciprocal rank across Bengali agricultural advisory queries comparing BM25, dense embeddings (mE5-small), and hybrid Reciprocal Rank Fusion (RRF).
| Model / Baseline | Metric | Result |
|---|---|---|
| BM25 (k1=1.2, b=0.75 default) | Recall@5 | 0.41 |
| BM25 Optimized (k1=2.2, b=0.4 tuned) | Recall@5 | 0.54 |
| Dense Retriever (mE5-small + FAISS) | Recall@5 | 0.58 |
| Hybrid RRF (BM25 + mE5 Fusion) | Recall@5 | 0.63 |
| Wheat 11-Class Disease Vision Sweep | Top-1 Accuracy | 44/50 = 88% |
| 437-Image Crop Library Sweep | Coverage Recall | 436/437 = 99.8% |
Bengali Chemical Advisory Safety Refusal Benchmark
Diagnostic safety refusal evaluation measuring false-positive advisory rates and hazardous pesticide dosage rejection across 6 regional dialects and 12 agricultural chemical categories.
| Model / Baseline | Metric | Result |
|---|---|---|
| Standard LLM Zero-Shot | Hazard Refusal Rate | 52.4% |
| Fine-Tuned Bengali LLM | Hazard Refusal Rate | 78.1% |
| KrishokChat Multi-Stage Safety Sink | Hazard Refusal Rate | 97.6% |
| Dialectal Safety Normalizer | Refusal Precision | 99.1% |
| Prohibited Compound Detection | F1 Score | 0.984 |
ChitraMiti: Multimodal Geometric Diagram Reasoning Benchmark
Cross-modal geometric problem-solving pairing high-resolution diagram parsing with step-by-step Bengali theorem proofs and Euclidean construction steps.
| Model / Baseline | Metric | Result |
|---|---|---|
| Tesseract OCR + LLM Pipeline | Theorem Parsing Accuracy | 38.2% |
| pix2tex ViT Transformer | LaTeX Proof Accuracy | 79.5% |
| GPT-4V Multimodal (Bengali) | Geometric Step Accuracy | 71.0% |
| ChitraMiti Unified Vision-Reasoner | Overall Proof Accuracy | 84.3% |
Nirapod Probaho: Multi-Agent Geospatial Triage Latency Sweep
Benchmarking autonomous agent response latency, spatial pathfinding execution time, and volunteer dispatch routing across dynamic flood polygons.
| Model / Baseline | Metric | Result |
|---|---|---|
| Sequential Single-Agent Script | Triage Latency | 14.2s |
| CrewAI Multi-Agent Pipeline | Triage Latency | 3.4s |
| PostGIS Spatial Query Optimization | Routing Execution | 42ms |
| MapLibre GL Vector Tile Pipeline | Client FPS (60hz) | 59.8 FPS |
Hardware Infrastructure & Evaluation Harness
Evaluation GPUs
Nvidia A100-SXM4 (80GB) & Dual RTX 4090 (24GB VRAM)
Inference Engines
vLLM 0.6.4 (TensorRT-LLM) & OpenRouter unified gateway
Edge Quantization
Unsloth 4-bit LoRA + llama.cpp GGUF Q4_K_M export
Scoring Libraries
SymPy 1.13, FAISS 1.8.0, SacreBLEU, Rouge-Score, ANLS
Want to benchmark your model against these test splits?
We openly share evaluation manifests, scoring scripts, and test-split queries with research labs and reviewers working on low-resource and South Asian NLP.