Mathieu Ajaka

Case study

Agentic Workflows for a Saudi AI Agency

Professional work (client accounts anonymized) Professional work. My employer is named; individual client accounts are kept anonymous. Metrics are self-reported and measured on the accounts' own data.

As sole technical lead at a Riyadh AI agency, I built LangGraph-orchestrated agentic workflows with human-in-the-loop checkpoints, model routing, and an evaluation gate on every change — across Arabic and English pipelines for enterprise accounts.

  • LangGraph
  • Human-in-the-loop
  • Model routing
  • RAGAS + LLM-as-judge
  • vLLM · AWS
  • CI/CD eval gates

Evidence: CV: AI Solutions Architect, WAJEEH Agency (2025–present) · Self-reported metrics on client data

Context

I’m the sole technical hire and de facto technical lead at an AI agency in Riyadh serving Saudi enterprise clients across customer-experience, brand, and regulatory domains. I own the technical side end to end: scoping each client’s problem, designing the system, leading delivery, and running production operations — while mentoring junior engineers on delivery standards.

Problem

The agency’s clients needed AI to do multi-step content and CX work across Arabic and English — reading a brief, retrieving from brand and regulatory material, drafting, and routing for approval — reliably enough to put in front of enterprise stakeholders. A single prompt couldn’t do that, and an unsupervised agent acting freely on regulated, brand-sensitive material was a non-starter. The work needed orchestration, oversight, and evaluation.

Intended users & constraints

Client-side marketing, CX, and compliance teams — many working primarily in Arabic, with executive stakeholders reviewing in English. Constraints: outputs touching regulated or executive-facing contexts required human sign-off; Arabic quality had to match English; and cost had to stay controlled across many accounts.

What I designed and built

I designed the work as LangGraph-orchestrated agentic workflows: explicit state machines with tool-use routing and human-in-the-loop (HITL) checkpoints, running across Arabic and English pipelines, for roughly 12 Saudi enterprise accounts.

flowchart TD
  REQ["Client request / brief<br/>Arabic or English"] --> ROUTE["LangGraph state machine<br/>tool-use routing"]
  ROUTE --> RET["Retrieval over<br/>brand & regulatory corpora"]
  ROUTE --> TOOLS["Tools / APIs"]
  RET --> DRAFT["Draft output"]
  TOOLS --> DRAFT
  DRAFT --> MODEL{"Model routing"}
  MODEL -->|templated| LOCAL["Local open-weight<br/>Llama 3.1 8B / Mistral 7B"]
  MODEL -->|regulated / exec-facing| FRONTIER["Frontier API<br/>Claude Sonnet / GPT-4o"]
  LOCAL --> HITL["Human-in-the-loop<br/>approval checkpoint"]
  FRONTIER --> HITL
  HITL -->|approved| OUT["Delivered output"]
  HITL -->|rejected| ROUTE
  EVAL["Evaluation gate<br/>RAGAS · LLM-as-judge · golden set"] -. blocks promotion .-> ROUTE
Agentic workflow — a LangGraph state machine routes retrieval and tool use, chooses a model by sensitivity, and stops at a human approval checkpoint; an evaluation gate blocks promotion of any change that regresses quality.

The workflow reads the brief, retrieves from the account’s brand and regulatory corpora, drafts through a routed model, and stops at a human checkpoint before anything regulated or executive-facing is delivered. Reported effect: campaign execution moved from around three days to under six hours of workflow time. Across the accounts, client handoffs were consistent, with no handoff failures I’m aware of over that period.

AI approach — the hard problem: Arabic retrieval

The most instructive fix was a retrieval failure. Default BM25 collapsed on clitic-heavy Arabic queries (the attached prefixes و/ف/ل and friends) over regulatory and brand corpora — it was silently returning poor context, which produced confident but ungrounded outputs. I rebuilt retrieval with a CAMeL Tools morphology-aware chunker, BGE-M3 dense retrieval, and reciprocal rank fusion. On the account’s own material, recall@10 moved from 0.61 to 0.89 (self-reported).

Model routing & cost

I built a model-routing layer to match model to task: local open-weight models (Llama 3.1 8B / Mistral 7B) for templated content, and frontier APIs (Claude Sonnet / GPT-4o) for regulated and executive-facing outputs. Reported effect: roughly 68% lower per-token cost versus an API-only baseline. Serving used vLLM with AWQ 4-bit quantization on an AWS g5.xlarge (A10G); benchmarked against TGI, vLLM held time-to-first-token around 420 ms median and p95 under 950 ms at concurrency 4–8.

Evaluation & reliability

I established an evaluation discipline where none existed:

  • RAGAS and LLM-as-judge with calibration caught two retrieval-drift episodes before they reached a client.
  • A QLoRA adapter release gate (4-bit NF4, rank 32, alpha 64) evaluated adapters on a 180-query held-out set and rejected two that passed on voice/style but regressed factual grounding.
  • CI/CD as the enforcement mechanism: GitHub Actions re-evaluates against a golden set on every change; ECR stores versioned images; promotion to the EC2 environment is blocked until RAGAS thresholds pass, with no manual override to production.

Security & deployment

Regulated and brand-sensitive material stayed under control through the routing design (local models for the bulk of work) and human sign-off on sensitive outputs. Deployment was containerized on AWS with versioned images and a gated promotion path.

My contribution

Sole technical owner: I scoped the client problems, designed the orchestration and evaluation architecture, built the retrieval, routing, and serving, set up the CI/CD evaluation gate, and ran production operations — while mentoring junior engineers.

Limitations

Future work

Extend the evaluation gate with per-language RAGAS breakdowns as standard, and formalize the tool-integration layer so new client workflows can be composed from a shared, permissioned tool set.