Family Historian: multilingual search over a family archive
Reads 9,350 handwritten and printed Arabic and English pages with seven local OCR models, then answers questions about them, citing the source page for every answer.
Names are pseudonymised throughout. Every person named in a figure, caption or screenshot has been replaced with a consistent pseudonym.
- Context
- Independent project
- Year
- 2026
- Stack
- Python, vLLM, Vision LLMs, BGE-M3 + BM25, FastAPI, Cloudflare Tunnel
9,350
Arabic and English pages transcribed
handwritten and printed
50→76%
disputed handwritten-Arabic words read right
vote alone vs. with the AI judge, 66 gold-set words
0.54→0.93
right page in the top 10 results
known-item search, 28 test questions
Overview
Problem
A family archive holds 9,350 scanned pages in Arabic and English, printed and handwritten. A single OCR (text-recognition) model makes confident mistakes on handwritten Arabic, and a search tool only helps if every answer points back to the page it came from.
Approach
- Transcription. Seven vision models, run locally with vLLM, read every page. Their readings are lined up word by word and put to a weighted vote. Each model's weight comes from how accurately it read a gold set of 151 lines transcribed by hand.
- Disputes. 314,531 words were still disputed after the vote. For these, an AI judge (Claude Haiku 5.5) looks at the page and picks between the candidate readings. I picked it from six judges tested on the gold set.
- Search. Each question is matched two ways: by meaning (BGE-M3 embeddings) and by exact words (BM25 keyword search). A reranker then reorders the top 30 pages.
- Answers. Answers stream back with numbered citations, each opening the scan beside its transcription. The bilingual web app (FastAPI) also has a people directory and a timeline.
Results
The models agree on 79% of the archive's 1.86 million words. On disputed handwritten-Arabic words, the right reading won 50% of the time with the vote alone and 76% with the judge. In search, questions about one specific page found it among the first ten results 93% of the time, up from 54% with the old index. Recall@10, the share of relevant pages in the top ten, rose from 0.39 to 0.72 across all 60 test questions.
What I built
- Transcribed 9,350 pages with seven local vision models on vLLM, combined by a word-level vote whose weights are calibrated on 151 hand-transcribed lines.
- Sent the 314K words still disputed to an AI judge chosen from six tested on the gold set, raising correct picks on handwritten Arabic from 50% to 76%.
- Built hybrid search (BGE-M3 + BM25 + reranker) that puts the right page in the top 10 for 93% of known-item questions, up from 54%, with streamed, cited answers.
Results in detail
Scan to cited answer
Counts read from the project's run outputs on the DGX Spark
1Scanned pages
9,350 page images, Arabic and English, printed and handwritten
originals never modified; working copies turned upright first
2Seven local OCR models read every page
Qwen3.6-35B-A3B, PaddleOCR-VL-1.6, dots.mocr, Gemma 4 26B-A4B, Chandra OCR 2, Nanonets-OCR2-3B, DeepSeek-OCR-2
served one at a time with vLLM on one GPU
3Word-level vote
readings aligned word by word, weighted by each model's measured accuracy per page type
weights calibrated on 151 hand-transcribed gold lines
agreement only counts across different model families
314,531 words still disputed after the vote
4AI judge
Claude Haiku 5.5 looks at the page and picks between the candidate readings
chosen from 6 models tested on the gold set; names and dates judged first
304,387 words settled by the judge; 14,201 left for a person
5Hybrid index
14,613 chunks: BGE-M3 vectors plus BM25 keywords, fused by rank
BM25 also indexes the other readings of uncertain words
6Rerank
bge-reranker-v2-m3 reorders the top 30 pages
7Cited, streamed answer
the answer streams in with numbered citations; each opens the page viewer on the scan
Settling the words the models disagree on
Disputed words inside the hand-transcribed gold lines, scored against the person's reading
A disputed word is scored only when the person's reading is among the candidates (194 of 216). Judges compared: Claude Sonnet 5.5 82.0%, Claude Haiku 5.5 77.8%, GPT-6 Luna 75.3%, Gemini 3.8 Flash 74.2%, DeepSeek V4 Flash 60.3%, GLM-5.3 Flash 17.5%. Haiku was chosen on cost.
The gold lines are small and were also used to fit the vote weights, so these figures are indicative.
Search: old index against the new one
60 test questions (41 English, 19 Arabic), scored on the top 10 pages returned
What each part adds
| Configuration | Recall@10 | Known item in top 10 |
|---|---|---|
| Old index (multilingual-e5, earlier transcripts) | 0.393 | 0.536 |
| BM25 keywords only | 0.430 | 0.464 |
| BM25 + alternative spellings + older English translation | 0.513 | 0.607 |
| BGE-M3 vectors only | 0.548 | 0.750 |
| BGE-M3 + BM25, fused by rank | 0.556 | 0.714 |
| BGE-M3 vectors + reranker | 0.662 | 0.857 |
| BGE-M3 + BM25 + reranker (deployed) | 0.716 | 0.929 |
With 60 questions, differences below about 0.03 are noise. Questions were built semi-automatically and spot-checked by hand.
Technical details
The full list of measured results behind this project.
Show all 21 measurements
| Measurement | Value |
|---|---|
| Pages transcribed | 9,350 |
| OCR models that read every page | 7 (Qwen3.6-35B-A3B, PaddleOCR-VL-1.6, dots.mocr, Gemma 4 26B-A4B, Chandra OCR 2, Nanonets-OCR2-3B, DeepSeek-OCR-2) |
| Handwriting line model (extra voter) | Baseer on 2,265 handwritten-Arabic pages, 85,400 line crops |
| Words in the transcribed archive | 1,857,930 |
| Words the models agree on | 79.0% (1,468,558) |
| Words disputed after the vote | 314,531 |
| Words settled by the AI judge | 304,387 (16.4%) |
| Words still flagged for a person | 14,201 (0.8%); a further 70,784 (3.8%) were read by one model only |
| Gold set | 151 hand-transcribed lines (110 by the archive owner, 41 earlier) |
| Disputed handwritten-Arabic words chosen correctly | 50.0% by the vote alone, 75.8% with the AI judge (n = 66) |
| Disputed words chosen correctly, all page types | 64.9% by the vote alone, 77.8% with the AI judge (n = 194) |
| Judge candidates tested on the gold set | 6 (Claude Sonnet 5.5 82.0%, Claude Haiku 5.5 77.8%, GPT-6 Luna 75.3%, Gemini 3.8 Flash 74.2%, DeepSeek V4 Flash 60.3%, GLM-5.3 Flash 17.5%) |
| AI judge API spend | $31.72 over three runs |
| Consensus character error rate (in-sample) | handwritten Arabic 18.0%, handwritten English 7.1%, printed Arabic 6.5%, printed English 1.9% |
| Known-item questions with the page in the top 10 | 0.536 old index, 0.929 new (28 questions) |
| Recall@10, all 60 questions | 0.393 old, 0.716 new |
| Mean reciprocal rank, all questions | 0.496 old, 0.781 new |
| Mean reciprocal rank, Arabic questions | 0.300 old, 0.749 new (19 questions) |
| Recall@10, name questions | 0.400 old, 0.693 new (14 questions) |
| Search index | 14,613 chunks, 346 MB on disk |
| Search latency with reranking | 0.37 s median per query on the GPU |
Limitations (10)
- The gold set is small (151 lines) and was used both to fit the vote weights and to compare the judges, so every accuracy figure here is in-sample. A second batch of 120 gold lines was prepared but has not been transcribed, so there is no held-out check yet.
- The 50% to 76% result is measured on 66 disputed handwritten-Arabic words, not on whole pages. On handwritten English (13 words) the judge did worse than the vote, and printed English has only 5 scored words.
- The vote-alone baseline in the judge comparison was printed by the comparison script and is not saved in its result files; the judges' own scores are.
- About 85,000 words are not confirmed: 14,201 still flagged and 70,784 read by only one model. The viewer marks them with a dotted underline and shows the other readings.
- The search test set has 60 questions, built semi-automatically from the archive and spot-checked by the author. Differences below about 0.03 are noise. Topic questions count the archivists' folder for that topic as the right answer.
- Answers are written by a hosted model (Claude Haiku 5.5 through OpenRouter, zero-data-retention endpoints only), so retrieved passages leave the machine on every question.
- BM25 also indexes an older machine English translation of each page. It helps English questions reach Arabic pages but comes from an earlier, less accurate reading.
- The live site is for the family. A visitor without access sees only a welcome page, so it is not a public demo.
- No comparison with a classical OCR engine (Tesseract, Kraken, Google Vision) was run.
- The code is not public.