Chunking Arabic legal text for RAG (and why BM25 collapses on clitics)

Most retrieval failures I have seen in Arabic legal systems are not model failures. They happen earlier, in tokenization and chunking, and they are quiet. The system returns an answer, cites nothing useful, and no error is thrown. This note is about why that happens with Arabic statute text, and what I do about it. It draws on the work behind the Saudi Labor Law RAG case study.

Why Arabic retrieval is genuinely hard

English retrieval gets a lot of forgiveness because word boundaries are mostly whitespace and morphology is shallow. Arabic offers neither.

The first problem is clitics. Arabic attaches short function words directly onto the following word as prefixes: و (and), ف (so/then), ل (for/to), ب (with/by), ك (like), the definite article ال, and the future marker س. These stack. So the single lemma “worker” (عامل) appears in a corpus as العامل (the worker), للعامل (for the worker), وللعامل (and for the worker), بالعامل, and so on. To a whitespace tokenizer these are all distinct surface tokens that share no lexical key.

Three more issues compound it:

Diacritics. Short vowels (َ ُ ِ ّ) are optional in modern text. The same word appears with and without them, and a naive index treats the two forms as unrelated.
Orthographic variants. Alef forms (أ إ آ ا), taa marbuta versus haa (ة / ه), and alef maqsura versus yaa (ى / ي) are written inconsistently across sources. Ministry PDFs are not consistent even within one document.
Direction and script mechanics. Right-to-left text with embedded Latin numerals and article references extracts out of PDFs with reordering artifacts and stray control characters, which corrupt token boundaries before you even reach normalization.

And underneath all of this, Modern Standard Arabic (MSA) is the language of statute, while users often ask in a dialectal register. The query and the corpus are not drawn from the same distribution.

Why BM25 collapses on clitic-heavy queries

BM25 scores a document by term-frequency and inverse-document-frequency over exact token matches. It has no notion that للعامل and العامل are the same lemma. When a user asks a question phrased with clitics — which is the normal case, not the edge case — the query tokens simply do not match the surface tokens in the relevant article, even though the article is exactly what they want.

The failure is worse than a miss, because it is silent. BM25 still returns its top-k; those documents are just the wrong ones that happened to share some unprefixed token. The generator then produces a fluent answer grounded in irrelevant passages. You get an ungrounded answer that looks grounded. In a legal setting that is the most dangerous failure mode there is, because the surface form of the output is indistinguishable from a correct one.

You can confirm this quickly. Take a set of real questions, run them through lexical-only retrieval, and inspect whether the cited article is actually the governing one. On clitic-heavy phrasings the hit rate drops out from under you while the system reports nothing wrong.

The fix, part one: morphology-aware normalization

Before anything is indexed, both corpus and query pass through the same normalization pipeline. I use CAMeL Tools for the Arabic-specific work. The essential steps:

Strip diacritics and the tatweel elongation character.
Normalize alef, yaa, and taa-marbuta variants to canonical forms.
Segment clitics so that وللعامل becomes its proto-tokens (و + ل + العامل), and normalize the definite article, so the lemma is recoverable.

The rule that matters most: the query and the corpus must go through byte-identical normalization. A normalization asymmetry is its own leakage — you index one canonical form and search with another, and recall silently degrades. I keep normalization in a single shared function that both ingestion and query time call, precisely so they cannot drift apart.

def normalize_ar(text: str) -> str:
    text = dediac_ar(text)          # remove short vowels + tatweel
    text = normalize_alef_ar(text)  # أ إ آ -> ا
    text = normalize_alef_maksura_ar(text)  # ى -> ي
    text = normalize_teh_marbuta_ar(text)   # ة -> ه
    return text  # clitic segmentation applied by the tokenizer downstream

The fix, part two: clause-level chunking that preserves hierarchy

Legal answers are only useful if they point to the governing article. That constraint drives chunking more than token counts do.

I chunk statute at clause level, and every chunk carries its structural coordinates — chapter, article number, sub-clause — as metadata that survives into retrieval and into the citation. A fixed-size sliding window, the default in most RAG tutorials, is actively harmful here: it splits an article mid-sentence, merges the tail of Article 74 with the head of Article 75, and destroys the one thing the user needs, which is knowing which article applies.

The tradeoff is granularity. Chunk too finely and a clause loses the context that disambiguates it; a definition three clauses up no longer travels with the clause that relies on it. Chunk too coarsely and a retrieved passage spans several articles, blurring which one governs and diluting the embedding. For statute I bias toward the clause with its article header attached, and let retrieval breadth plus reranking recover the surrounding context rather than baking it into oversized chunks.

The fix, part three: hybrid retrieval and reranking

Dense retrieval solves the clitic problem that BM25 cannot, because a multilingual embedding maps للعامل and العامل to nearby vectors regardless of surface form — and maps an English query near the Arabic article that answers it. I use a strong multilingual embedding (BGE-M3 or multilingual-e5-large) for that reason.

But dense retrieval alone gives up something lexical search is good at: exact matches on article numbers, defined terms, and rare tokens. So I run both and fuse them with reciprocal rank fusion (RRF), which combines ranked lists by summing 1 / (k + rank) across retrievers without needing to calibrate their scores against each other.

def rrf(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranked in rankings:
        for rank, doc_id in enumerate(ranked):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return scores  # sort descending, take top-n into the reranker

The fused candidate set then goes through a cross-encoder reranker (a BGE reranker), which scores each query–passage pair jointly and recovers precision at the top of the list. On one engagement’s bilingual corpus, moving from lexical-only retrieval to this normalized hybrid-plus-rerank stack took recall@10 from roughly 0.61 to roughly 0.89 — a self-reported result on that data, not a general benchmark.

Tradeoffs worth stating plainly

Multilingual embedding versus per-language index. One multilingual index keeps the architecture simple and enables cross-language retrieval, so an English question can pull the Arabic article. A per-language index with language-specialized embeddings can sharpen recall, at the cost of routing logic and duplicated infrastructure. I default to the multilingual route and lean on reranking to buy back precision.
Chunk granularity for statute. Clause-level chunking maximizes citation fidelity and is the right default for law. It costs you when an answer requires reasoning across several articles, which is where retrieval breadth and reranking, rather than bigger chunks, do the work.
RRF constant and candidate depth. These are cheap to tune and worth tuning against your own labeled queries rather than inheriting defaults.

Limitations

Every number above is a retrieval-quality number, and retrieval quality is not answer correctness. A high recall@10 can sit next to a wrong answer, because the generator can still misread a correctly retrieved passage, or miss that the retrieved article was amended. Retrieval metrics tell you the right text was in front of the model; they say nothing about what the model then did with it.

For anything legal, that gap has to be closed by people. I treat citation enforcement and refusal-on-out-of-corpus as core features, add faithfulness and context-precision evaluation per language, and keep a human in the loop. Amended and superseded articles need explicit corpus versioning, or the system will confidently cite stale law. None of this is legal advice, and it should not be deployed as if it were.