Mathieu Ajaka

Case study

A Bilingual Arabic–English Assistant for Saudi Labor Law

Personal build Open-source personal build. A closely related professional engagement (anonymized) is described under "Related professional context."

A retrieval assistant that answers Saudi Labor Law questions in Arabic or English, cites the articles it relies on, and declines questions outside its corpus — with clause-level chunking, hybrid retrieval, reranking, and a bilingual evaluation set.

  • RAG
  • Arabic NLP (CAMeL, BGE-M3)
  • Hybrid retrieval + rerank
  • FastAPI
  • Bilingual evaluation

Evidence: Open-source build + public reference architecture · 120-query bilingual evaluation set (self-reported) · CV: related 2024 professional engagement (anonymized)

HR teams and legal advisors in Saudi Arabia answer labor-law questions all day — from long Arabic statutes and ministry circulars that most tools handle badly. I built an assistant that answers in Arabic or English, cites the specific articles it relies on, and declines questions the corpus doesn’t cover. This page shows the architecture, the evaluation approach, and the limitations that matter for anything legal-adjacent.

Context

Saudi Labor Law, ministry circulars, and platform policies (Qiwa/Musaned-style workforce compliance) are long, Arabic-first, and frequently cross-referenced. English-speaking management and Arabic-speaking staff need the same answer to the same question, with a pointer to the actual article — not a confident paragraph with no source.

Problem

  • Generic chatbots hallucinate legal specifics and cite nothing.
  • Arabic retrieval is genuinely hard: clitic-heavy queries (the prefixes و/ف/ل and friends), diacritics, and right-to-left text break naive lexical search.
  • The two languages must reach parity — an answer shouldn’t be better in English than in Arabic.

Intended users

An Arabic-speaking HR manager, a bilingual legal advisor, and an English-speaking (often expatriate) manager — all asking the same underlying questions.

Constraints & requirements

  • Answers must cite the article(s) they rely on.
  • The system must decline questions outside the corpus rather than guess.
  • Language-symmetric quality across Arabic and English.
  • A path to enterprise deployment (data residency, on-prem) for regulated buyers.

How the system works

flowchart TD
  subgraph Ingest["Ingestion (offline)"]
    PDF["Saudi Labor Law corpus<br/>Arabic PDFs"] --> NORM["Normalize<br/>RTL · diacritics · tokenization<br/>(CAMeL Tools)"]
    NORM --> CHUNK["Clause-level chunking<br/>preserves article/citation hierarchy"]
    CHUNK --> EMB["Multilingual embeddings<br/>BGE-M3 / e5"]
    EMB --> IDX[("Vector + lexical index")]
  end
  subgraph Query["Query (online)"]
    Q["User question<br/>Arabic or English"] --> HYB["Hybrid retrieval<br/>dense + BM25 (RRF)"]
    IDX --> HYB
    HYB --> RR["Reranker<br/>(BGE)"]
    RR --> GEN["Grounded generation<br/>citation-enforced"]
    GEN --> ANS["Answer + article citations<br/>or 'out of corpus'"]
  end
  GEN -. eval taps .-> EVAL["Evaluation<br/>RAGAS · retrieval metrics · human review"]
Architecture — offline ingestion normalizes and chunks Arabic statute into a hybrid index; online queries run hybrid retrieval, reranking, and citation-enforced generation, with evaluation taps throughout.
  • Ingestion. Arabic PDFs are normalized (RTL handling, diacritic and orthographic normalization, morphology-aware tokenization with CAMeL Tools) and chunked at clause level so that a retrieved passage carries its article/citation hierarchy.
  • Retrieval. Hybrid dense + lexical retrieval (multilingual embeddings — BGE-M3 / multilingual-e5-large — fused with BM25 via reciprocal rank fusion), followed by a BGE reranker.
  • Generation. The model answers only from retrieved context and is required to cite the articles; out-of-corpus questions are refused.

AI / retrieval strategy & tradeoffs

  • Multilingual embedding vs per-language index. A strong multilingual embedding keeps one index and supports cross-language retrieval; a per-language index can sharpen recall at the cost of complexity. The build uses the multilingual route with reranking to recover precision.
  • Chunk granularity. Statute rewards clause-level chunks (citation fidelity) over long windows (which blur which article actually applies).
  • Open-weight vs API models. For a public demo, hosted/open models are fine; for a regulated enterprise, open-weight models deployed privately address data-residency constraints — at some quality tradeoff to manage.

Evaluation

Retrieval quality was measured on a 120-query bilingual evaluation set across 8 legal domains, reporting Hit@5 = 0.93 and MRR@10 = 0.81 (self-reported on that set). The right way to read these: they say the correct article tends to be retrieved near the top — they do not certify that generated answers are legally correct. A production evaluation should add RAGAS faithfulness/answer-relevance/context-precision per language, LLM-as-judge with human spot-checks, and an explicit failure-mode analysis (dialect phrasing, cross-article reasoning, amended-law versioning).

Security & deployment

Containerized as a FastAPI service with a public reference architecture. For enterprise use, the design supports a Postgres/pgvector variant and private or on-prem/air-gapped deployment so the corpus and queries never leave the client’s environment — with no-PII logging and an audit trail of what was retrieved for each answer.

My contribution

I designed and built the retrieval pipeline, the Arabic normalization and chunking, the hybrid-retrieval-plus-rerank stack, and the evaluation set and harness. The open-source build and its reference architecture are my own work.

Limitations

Future work

Add per-language RAGAS + LLM-as-judge with human review as a release gate; corpus versioning for amended law; and a demo recording (Arabic and English) in place of a public live endpoint, to avoid exposing an unmetered LLM service.