ChitraMiti: A Benchmark for Cross-Modal Consistency in Bengali Geometric Reasoning
Abstract
Vision-language models perform well on multimodal reasoning benchmarks, but evaluation remains thin for low-resource languages and for geometry problems that require jointly reading diagrams and text. We introduce ChitraMiti-12.8k, a synthetic benchmark of 12,874 Bengali planar geometry problems paired with 15-attribute structured descriptions, and NCTB-500, a complementary set of 500 expert-curated geometry problems extracted directly from official Bengali NCTB textbooks (Classes 6–10; 78.6% hard). Using a three-phase protocol (Diagram-only Phase A, Diagram+Description Phase B, Description-only Phase C), we show that Phase B and Phase C are statistically equivalent (TOST α=0.05, ±5% margin). However, diagnostic adversarial spatial swaps reveal severe confirmation bias: models catch text-diagram contradictions only 6%–18% of the time, blindly following manipulated descriptions. QLoRA fine-tuning on ChitraMiti-12.8k improves in-distribution accuracy (up to 34.2%) and transfers 60%–65% of absolute gains to real textbook diagrams in NCTB-500.
Key Empirical Receipts & Quantitative Highlights
- Pairs 12,874 synthetic planar geometry problems (ChitraMiti-12.8k) with 500 real textbook problems (NCTB-500).
- 15-attribute structured description schema captures complete geometric relationships (TOST equivalence confirmed, p < 0.01).
- VLM confirmation bias: models detect adversarial spatial contradictions only 6%–18% of the time, blindly trusting text over vision.
- QLoRA fine-tuning on synthetic ChitraMiti-12.8k transfers 60%–65% of absolute accuracy gains to real NCTB-500 textbook scans.
- Hard composite problems (78.6% of NCTB-500) remain difficult across all evaluated 4B–26B vision-language models.
Problem Context & Real-World Motivation
Modern vision-language models (VLMs) achieve high benchmark scores on standard multimodal datasets, but low-resource languages such as Bengali are rarely tested on complex spatial reasoning. More critically, existing benchmarks fail to test whether VLMs actually inspect diagram pixels or merely exploit textual language priors. In secondary education geometry, answering a proof requires verifying that visual diagram angles match the written premise. ChitraMiti evaluates this cross-modal verification integrity.
Principal Research Contributions
Constructed ChitraMiti-12.8k, containing 12,874 planar geometry problems paired with 15-attribute structured spatial relational schemas in Bengali.
Created NCTB-500, a benchmark of 500 manually curated, expert-verified geometry exam problems from official Bangladeshi textbooks (Classes 6–10).
Designed a Three-Phase Evaluation Protocol (Diagram-only, Diagram+Text, Text-only) utilizing Two One-Sided Tests (TOST) for equivalence testing.
Pioneered adversarial spatial perturbation tests proving that VLMs suffer from acute confirmation bias, blindly trusting text descriptions over visual geometry.
Dataset Construction, Taxonomy & Verification
12,874 synthetic items (ChitraMiti-12.8k) + 500 authentic textbook scans (NCTB-500)
ChitraMiti: 11,231 train, 643 validation, 1,000 test. NCTB-500: held-out evaluation set.
Planar Geometry, Multi-Step Theorem Proofs, Cross-Modal Consistency
Official NCTB Mathematics textbooks (Classes 6–10) and filtered DeepVision geometry corpus
3 bilingual mathematics educators audited 200 items (Cohen κ = 0.81 for descriptions, 0.78 for translations)
Task Breakdown & Evaluation Tracks
Phase A (Diagram-Only Reasoning)
Models receive only the raw diagram image and the question prompt, testing pure visual spatial reasoning.
Phase B (Diagram + Structured Description)
Models receive both diagram pixels and full 15-attribute text specifications (vertex coordinates, angle degrees, parallel lines).
Phase C (Description-Only Reasoning)
Diagram pixels are removed entirely; models must solve the geometry problem solely from structured relational text.
Adversarial Spatial Swap Diagnostics
Single spatial constraints (e.g. parallel lines or right angles) are falsified in text to test if models spot the contradiction.
Experimental Setup & Benchmarked Models
Zero-shot deterministic generation (T=0); maximum generation length 64 tokens; SymPy symbolic matching + LLM judge.
Distributed evaluation using vLLM and OpenRouter inference APIs.
Mathematical Accuracy %, Two One-Sided Tests (TOST) equivalence margin ±5%, and Adversarial Detection %.
Quantitative Evaluation Results & Baselines
Table 1: Accuracy (%) Across Three-Phase Evaluation Protocol (ChitraMiti-1k vs. Real NCTB-500)
| Model | ChitraMiti Phase A (Vis) | ChitraMiti Phase B (Both) | ChitraMiti Phase C (Text) | NCTB-500 Phase A | NCTB-500 Phase B | NCTB-500 Phase C |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | 27.0% | 29.8% | 29.1% | 33.0% | 39.6% | 36.6% |
| Gemma-4-26B | 28.7% | 34.0% | 34.0% | 35.0% | 45.2% | 42.8% |
| LLaMA-3.2-11B | 13.8% | 16.2% | 17.5% | 16.8% | 21.6% | 22.0% |
| Gemini 2.5 Flash | 17.9% | 21.3% | 19.9% | 21.8% | 28.4% | 25.0% |
| GPT-4o-mini | 19.4% | 21.0% | 21.8% | 27.6% | 28.0% | 27.4% |
Table 2: Adversarial Contradiction Detection (Baseline Unperturbed vs. Swapped Description)
| Model | Base Detection Rate | Adversarial Detection Rate | Performance Collapse (Δ) |
|---|---|---|---|
| Qwen3-VL-8B | 31.2% | 18.0% | -13.2% |
| Gemma-4-26B | 31.8% | 14.0% | -17.8% |
| LLaMA-3.2-11B | 16.8% | 6.0% | -10.8% |
| Gemini 2.5 Flash | 19.6% | 13.0% | -6.6% |
| GPT-4o-mini | 21.0% | 12.5% | -8.5% |
Diagnostic Error Analysis & Failure Modes
Statistical analysis confirms Phase B (image+text) and Phase C (text-only) are statistically equivalent under TOST testing (p < 0.05). When textual descriptions are present, VLMs essentially ignore diagram pixels. Under adversarial perturbations where the text asserts an angle is 90° while the diagram clearly shows 45°, models identify the contradiction only 6%–18% of the time, hallucinating along with the flawed text.
Text-Over-Vision Confirmation Bias
When text contradicts visual evidence, models prioritize linguistic fluency and blindly solve based on the false text rather than reporting inconsistency.
Composite Theorem Breakdown
On NCTB-500, 65.0% of problems involving auxiliary construction lines or circle theorems are failed by every single evaluated model.
Practical Significance & South Asian AI Implications
Warns researchers that high multimodal VLM benchmark scores often reflect language reasoning on descriptions rather than genuine visual understanding, establishing the need for adversarial verification benchmarks in educational AI.
Open-Access Research Artifacts
ChitraMiti-12.8k Benchmark (Hugging Face)
12,874 synthetic geometry problems with 15-attribute schemas
NCTB-500 Official Textbook Benchmark
500 verified real textbook problem scans
Evaluation Harness & SymPy Judge
Three-phase evaluation code
BibTeX Citation (Preprint)
@article{reza2026chitramiti,
title={ChitraMiti: A Benchmark for Cross-Modal Consistency in Bengali Geometric Reasoning},
author={Reza, Khan Raiyan Ibne and others},
journal={Preprint},
year={2026}
}