KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records
Abstract
Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs (U+09F4–U+09F9), no mainstream font, and no OCR coverage. We introduce KhatianDoc, a four-task benchmark built from 107 real RS Khatian records obtained through the Vumi (Land) Office of Munshiganj district, spanning symbol recognition (Task 1, 95 crops), base-16 arithmetic conversion (Task 2, 53 targets), structured field extraction (Task 3, 261 rows), and legal document QA over 1,634 QA pairs (Task 4). Ground truth was transcribed by hand, verified line-by-line by a land-law practitioner, and anonymized via positional tokens. Evaluating six frontier multimodal LLMs (8B to 72B+) reveals that five QA categories (39.3% of evaluation set) return zero correct answers across every model, and all models score worse than a constant-mean baseline on base-16 arithmetic (MAE > 0.40 vs. 0.237 baseline), demonstrating a complete capability void rather than partial competence.
Key Empirical Receipts & Quantitative Highlights
- First machine-checkable benchmark for Bengali RS Khatian records and Ana-Ganda-Kora-Kranti-Til base-16 fractions.
- 107 real Vumi-office records, 1,634 QA pairs, 100% human-transcribed and verified line-by-line by a land-law practitioner.
- Hard reasoning floor: 5 complex legal QA categories (39.3% of evaluation set) score 0% across all six frontier multimodal LLMs.
- Arithmetic collapse: all models perform worse on base-16 conversion than a context-free mean baseline (MAE > 0.40 vs 0.237).
- Positional-token anonymization preserves multi-hop referential distinctions while protecting personal ownership privacy.
Problem Context & Real-World Motivation
Over 80% of civil litigation in Bangladesh stems from disputed land tenure, where ownership shares are documented in historical RS Khatian records. These deeds do not record shares in decimals; they utilize Ana-Ganda-Kora-Kranti-Til—a pre-colonial base-16 fractional calculation system utilizing obsolete Unicode glyphs (U+09F4–U+09F9). These symbols have zero OCR support, no font standardization, and are absent from all frontier LLM tokenizers. Deploying generic document AI on these records creates catastrophic legal errors.
Principal Research Contributions
Pioneered the first machine-checkable benchmark for Bengali RS Khatian land title deeds and base-16 Ana-Ganda calculations.
Sourced 107 authentic administrative records directly from the Vumi (Land) Office of Munshiganj district, Kumariya Mouza.
Conducted 100% human transcription audited line-by-line by a certified land-law practitioner (zero machine OCR used in ground truth).
Developed positional-token privacy anonymization ([PERSON_1], [PERSON_2]) preserving multi-hop inheritance co-reference without exposing private citizen data.
Executed a transparent metric self-audit that exposed and corrected prompt refusal evaluation bugs in published metrics.
Dataset Construction, Taxonomy & Verification
107 full-page deeds, 95 isolated symbol crops, 261 structured rows, 1,634 document QA pairs
Standardized evaluation set of 300 stratified questions (73.7% simple, 26.3% complex)
Bengali Legal Document OCR, Historical Base-16 Arithmetic, Title Deed QA
Official Revisional Survey (RS) Khatians from Munshiganj District Vumi Office
Double-pass human transcription verified by a practicing land-law advocate; complete consensus
Task Breakdown & Evaluation Tracks
Task 1: Symbol Recognition
Evaluates character error rate (CER) and exact match on cropped historical Ana-Ganda fractional glyphs.
Task 2: Base-16 Fraction Conversion
Tests mathematical conversion from archaic base-16 fraction strings into modern decimal representations.
Task 3: Structured Field Extraction
Extracts full ownership tables, share ratios, and plot identification numbers from unconstrained page scans.
Task 4: Legal Document QA
Multi-hop legal reasoning, counterfactual checks, and inheritance share validations over deed records.
Experimental Setup & Benchmarked Models
Zero-shot prompting via OpenRouter; exact token limits (30 for T1-T2, 500 for T3, 200 for T4); secure-original evaluation.
Single OpenRouter API management layer with deterministic retries.
Character Error Rate (CER), Exact Match (EM), Row-Level F1, Mean Absolute Error (MAE), and ANLS.
Quantitative Evaluation Results & Baselines
Table 1: Multimodal LLM Performance Across All Four KhatianDoc Tasks
| Model | T1 Symbol CER ↓ | T1 Exact Match ↑ | T2 Decimal Exact ↑ | T3 Row F1 ↑ | T4 DocQA ANLS ↑ | T4 DocQA EM ↑ |
|---|---|---|---|---|---|---|
| Gemini 2.5 FL | 93.29% | 0.00% | 2.11% | 5.16% | 21.12% | 13.33% |
| Qwen2.5-VL-72B | 80.91% | 1.05% | 6.32% | 11.97% | 5.88% | 0.34% |
| Qwen3-VL-8B | 76.89% | 0.00% | 2.11% | 17.48% | 19.54% | 13.00% |
| Llama 4 Scout | 90.91% | 1.05% | 0.00% | 6.43% | 17.84% | 11.33% |
| Gemma 4 26B | 84.70% | 0.00% | 5.26% | 25.84% | 4.68% | 0.33% |
| GPT-4o Mini | 96.56% | 0.00% | 1.05% | 0.10% | 3.48% | 0.00% |
Table 2: Five Legal QA Categories at 100% Failure Rate Across All 6 Models (39.3% of Evaluation Set)
| Reasoning Category | Number of Queries (n) | Models Scoring 0.00% | Failure Pattern |
|---|---|---|---|
| fraction_share | 39 | 6 / 6 (100%) | Inability to map Ana marks to fractional shares |
| legal_fraction_math | 20 | 6 / 6 (100%) | Arithmetic collapse on base-16 multi-owner summation |
| counterfactual_check | 20 | 6 / 6 (100%) | Hallucinated affirmative claims when property is unrecorded |
| conditional_filtering | 20 | 6 / 6 (100%) | Failure to filter rows by owner status and plot type |
| multi_hop_reasoning | 19 | 6 / 6 (100%) | Inability to connect owner names across page boundaries |
Diagnostic Error Analysis & Failure Modes
KhatianDoc documents not merely a performance lag, but the total absence of capability in frontier multimodal LLMs. On Task 2 base-16 arithmetic, a dumb baseline that always predicts the dataset constant mean (0.3935) achieves MAE = 0.237, while every single frontier LLM achieves an MAE between 0.40 and 0.42, showing their outputs are completely decorrelated from the ground truth.
Bizarre Glyph Substitution Patterns
When encountering the Ana mark (৴), Gemini 2.5 FL substitutes the Bengali Rupee symbol (U+09F2), while GPT-4o Mini outputs ancient Brahmi script glyphs (U+11156).
Script Incoherence Under Constraints
Despite explicit instructions to respond in Bengali script, GPT-4o Mini output Western Arabic numerals on 97 of 107 records.
Hard Reasoning Floor on Legal Math
118 out of 300 questions (39.3%) received zero correct answers from any of the six frontier models evaluated.
Practical Significance & South Asian AI Implications
Highlights the severe legal risks of applying out-of-the-box multimodal models to civil government records in the Global South, emphasizing the need for specialized character-level encoders.
Open-Access Research Artifacts
KhatianDoc Redacted Dataset (Hugging Face)
107 anonymized high-res deed scans with 1,634 QA pairs
Annotation Protocols & Legal Verification Guidelines
Methodology documentation
Manuscript (Preprint / ACL Review)
Double-blind evaluation report
BibTeX Citation (Preprint)
@article{reza2026khatiandoc,
title={KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records},
author={Reza, Khan Raiyan Ibne and others},
journal={Preprint},
year={2026}
}