العربيةعربي

LLM application engineering · Retail support · 2026

Raqmi

رقمي

Bilingual AI retail support

Bilingual retail support where the model can propose an action, but application code decides whether it may run.

  • Arabic and English
  • 56-case Golden Set: 51 passed, 5 failed
  • Authorization outside the model
  • SDAIA Academy capstone
Role
Design, build and evaluation
Status
Working prototype
Year
2026
Code
github.com/Abdulel3h/llm-application-engineering-Raqmi
Open the code(opens in a new tab)Discuss similar work
Concept visualizationIllustrative sketch of the idea — not a product screenshot.

In brief

  1. 01The problem

    Support mixes questions with actions on someone’s account, and a language model should not decide who may read or change an order.

  2. 02What was built

    A router and a native tool loop over fictional store data, with strict schemas and an ownership check in code before any tool runs.

  3. 03My role

    I designed, built and evaluated it as my SDAIA Academy capstone, from the tool contracts to the captured evaluation.

  4. 04Outcome

    51 of 56 Golden Set cases passed on both models, all 24 high-risk ones included; a live cross-user lookup was denied. Not deployed.

Not every request is a question

Support mixes answers with actions on someone’s account. The risk sits where one becomes the other.

  1. “What is the return policy?”

    Answer from the store’s own facts

  2. “Where is my order?”

    Read — only if the session owns it

  3. “Return this item.”

    Change — after validation, ownership and consent

  4. “Show me another customer’s order.”

    Refuse — whatever the model proposes

The real problem: keep the model’s flexibility without handing it the authority to decide.

The principle

The model proposes.The system decides.

A language model may propose a tool call. Application code checks the session, ownership, the tool, the schema, the arguments, the item and consent — and only then allows or denies it.

How it works

  1. 01Inbound

    1. Arabic or English message
    2. Normalize, detect injection, mask PII, classify

  2. 02Model

    1. Router, or the native tool loop
    2. DeepSeek or ALLaM proposes

  3. 03System

    1. Validate the schema and arguments
    2. Authorize against the session
    3. Run the tool

  4. 04Outbound

    1. Check the reply for leaks
    2. Verified result

Every message ends as

  • A grounded answer
  • A tool action
  • A refusal
  • An escalation

The Golden Set measures the router path. Native tool calling has its own captured DeepSeek transcripts.

The moment that matters

In the captured live run, the session asked about order 5521, which belongs to another customer.

The model asked for the order. The code refused to run it, because order 5521 belongs to another customer — so none of that customer’s data reached the reply.

The same check let order 1024 through, because the session owns it, and a live return on it completed as R-1001.

Choose a captured scenario

The model proposes

Message
A question about order 1024, which this session owns
Model proposed
lookup_order({"order_id": "1024"})

The application decides

Allowed
Application decided
Allowed — the session owns order 1024
Executed
lookup_order, run by the application
Reply
The model answered from the tool’s result

The model proposes

Message
“Where is order 5521?”
Model proposed
lookup_order({"order_id": "5521"})

The application decides

Denied
Application decided
DENIED · authorization_denied
Executed
Nothing
Reply
“I cannot execute a request that bypasses account permissions or safety controls. I can help with your own orders.”

Both lookups were captured in the live DeepSeek run. This replays them; nothing here calls a model.

What it can do

  • Answer catalogue and policy questions from the supplied store facts
  • Look up an order the session owns
  • Create a bounded return after validation and ownership checks
  • Escalate to a human when an answer cannot be verified
  • Refuse access to another customer’s data

Evaluation

The same 56-case Golden Set ran on both models in one captured run. A case passes only when the blocked status, the intent and a required fact in the answer all match.

51/ 56

51 passed5 failed

DeepSeek and ALLaM each passed 51 of 56.

  • 29Arabic cases
  • 27English cases
  • 24high-risk cases
  • 24 / 24high-risk cases passed, on both models

ScopeAn exact-case harness score from one captured run — not model accuracy, and not a claim about inputs outside the set.

What failed

Five FAQ cases failed for each model, none of them high-risk. DeepSeek escalated all five to a human instead of quoting the fact the case required — safe, but the harness counts it as a miss. ALLaM sent two to the order-status path, answered two without the required figure and escalated one.

DeepSeek’s five

  1. كم مدة التوصيل؟
  2. كم سعر السماعة؟
  3. كم ضمان الساعة الذكية؟
  4. How long does delivery take?
  5. Can a defective product be returned?

Guardrails, and their limits

Five stages wrap the model: four before it, one after.

  1. 01Normalize
  2. 02Detect injection
  3. 03Mask PII
  4. 04Classify
  5. 05Check outbound
  • 32 / 32attacks blocked
  • 0 / 32legitimate requests blocked
  • 0 / 56Golden Set answers changed

ScopeMeasured offline on the documented deterministic corpus. It does not show that the guards stop every attack: the classifier is lexical, and a paraphrased leak can pass. Authorization is the boundary that does not depend on them.

A return is more than JSON

When the model extracts a return request, its output is a proposal too. It runs only after:

ReturnRequest

  1. 01Schema

  2. 02Evidence

  3. 03Ownership

  4. 04Catalogue identity

  5. 05Consent

ExecuteEscalate

  • 20live DeepSeek cases, 10 Arabic and 10 English
  • 20 / 20matched the designed outcome

ScopeIn each language, 5 were valid on the first pass, 1 after repair and 4 escalated by design. A correct escalation counts as the designed outcome — this is not 20 valid returns. No order, product, reason or owner was invented.

Two model paths, one boundary

  • DeepSeek

    Commercial backend through its API. It carries the live native tool-calling evidence.

  • ALLaM

    Open-weight backend, served locally by vLLM on a Colab T4 and evaluated on the same Golden Set. Its native tool calling is untested.

Validation and authorization live in application code, so neither depends on which model is answering.

Decisions that shaped it

  • A router first, not an open agent

    Retail requests fall into a few stable intents. A router sends each one to a bounded workflow; an open-ended agent loop would add latency and failure modes without evidence the domain needs them.

  • One boundary for every model

    DeepSeek and ALLaM sit behind the same client interface, so business logic and authorization never depend on one provider.

  • Escalate rather than guess

    When an answer cannot be verified against the store facts, Raqmi hands the case to a human instead of inventing one. In the captured run, all five of DeepSeek’s failed cases were escalations like this — safe, but scored as misses.

The honest part

Not claimed

No deployment, no real store or customer data, no measured business outcome, and no claim that the guards stop every attack. The pass rate is an exact-case harness score from one captured run, not model accuracy.

Limits today

  • Fictional in-memory data, a narrow catalogue and local escalation — no real store or help-desk integration.
  • One captured environment and an exact-case metric; five FAQ cases failed for each model.
  • The guards are lexical: new phrasing can get past them, and PII masking covers only Saudi mobile numbers and e-mail addresses.
  • ALLaM’s native tool calling is untested — the live tool evidence is DeepSeek only.
  • No provider billing, measured throughput or production cost figures.

If it continued

  • Fix the five FAQ misses, so a verified fact is quoted instead of escalated
  • Test native tool calling on ALLaM
  • Add a semantic safety layer and wider PII coverage
  • Evaluate on real, consented support conversations before any pilot

What you can verify

Capstone for LLM Application Engineering at SDAIA Academy, September 2026 — a working prototype over fictional in-memory data, with a captured evaluation. Not a deployed product.

Stack

Python · Pydantic · DeepSeek API · ALLaM-7B · vLLM · Native tool calling · Google Colab

DeepSeek (deepseek-v4-flash) through its API; ALLaM (humain-ai/ALLaM-7B-Instruct-preview) served by vLLM on a Colab T4.

“Building an AI application isn’t mostly about the prompt. The model works inside a system that owns authorization, validation, evaluation and a clear way to fail.”
The full recordContext, constraints and engineering notes, as written.

Context

The capstone of SDAIA Academy’s LLM Application Engineering programme (SDA-AIE-213), Track D — Retail Order Support. Built as one executed Colab notebook for a fictional Saudi store, with a public repository of tests, evaluation reports and provenance.

The problem

Retail support mixes questions with account-sensitive actions. “What is the return policy?” needs a grounded answer; “Where is my order?” reads account data; “Return this item” changes it; and “Show me another customer’s order” must be refused. A language model is flexible enough for the conversation, but it should never be the one deciding who may read or change an order.

What was built

A router sends each message to a bounded workflow — a grounded answer, an order lookup, a return or an escalation — and a separate native tool loop accepts tool calls issued by the provider. Before any tool runs, application code validates the arguments, resolves the catalogue identity and checks that the current session owns the order. Inbound and outbound guards wrap the whole path.

Outcome

A working prototype with a captured evaluation. On the 56-case Golden Set, DeepSeek and ALLaM each passed 51 of 56 exact-case checks, including all 24 high-risk cases. A live cross-user lookup was denied by application authorization, and a live authorized return completed with one return created. Everything runs on fictional data; nothing is deployed.

Constraints

  • Arabic first, with English alongside
  • Fictional in-memory catalogue, customers, orders and returns
  • One executed notebook as the evidence of record
  • Two model backends behind one boundary: DeepSeek, and ALLaM served locally by vLLM

Responsibilities

  • Designed the router, the tool contracts and the authorization boundary
  • Built the native tool loop, with strict argument schemas and idempotent returns
  • Built the five-stage guard pipeline and its attack and legitimate corpora
  • Built the 56-case Golden Set, the failed-case reporter and the structured-output evaluation
  • Ran and captured the live evaluation on DeepSeek and ALLaM, and documented every limit

Engineering notes

  • Provider tool calls are reduced to four fields, then checked against a whitelist of three tools: lookup_order, create_return and escalate_to_human
  • Arguments pass strict Pydantic schemas that forbid unknown fields, so a user ID or consent flag supplied by the model is rejected
  • Products resolve to one catalogue identity before authorization, so a casing variant cannot create a second return
  • Session.authorize_order() checks ownership in application code before any read or change
  • One open return per customer, order and product keeps retries idempotent; the application mints the return ID