GenAI System Design & Case Studies

What makes GenAI system design different

Classic system design asks you to move and store data reliably at scale. GenAI system design asks all of that plus how to put a component in the middle of the request path that behaves nothing like a database or a microservice. A large language model (LLM) is:

• Probabilistic. The same input can give different outputs. Even at temperature 0, outputs can drift across provider model versions, hardware and batch composition.
• Confidently fallible. It can produce fluent, plausible and wrong content (hallucination). Nothing in the output reliably says "I am unsure".
• Slow and variable in latency. Latency grows with the number of output tokens. A 500-token answer can take 5 to 20 seconds, against 5 to 50 ms for a typical API call.
• Expensive per call. Each request costs money in proportion to the tokens it uses, so cost is a live design constraint rather than a line in next year's budget.
• Instructable by anyone. The same channel carries data and instructions, so untrusted text (a web page, an email, a PDF) can try to take control of it (prompt injection).
• Hard to test. There is rarely a single correct output, so unit tests give way to evaluation sets, rubric-based grading and statistical comparison.

What stays the same

Everything on the general system design page still applies: load balancing, caching, queues, databases, sharding, idempotency, rate limiting, observability and failure isolation. This page assumes those basics and covers only what is new or different when a model sits in the path. When an interviewer asks about the non-model parts (storing chat history, streaming over WebSockets, queueing batch jobs), answer them the same way you would in any system design interview.

The four archetypes you will be asked to design

A repeatable framework for GenAI design interviews

You get 35 to 60 minutes. A fixed structure stops you from rambling and makes sure you touch everything the rubric expects. Say the steps out loud at the start ("I'll clarify, set metrics, look at data, choose an approach, draw the architecture, then cover evaluation, safety, and cost and latency") so the interviewer can steer.

1. Clarify the use case and users (3 to 5 min) Who are the users (internal experts or anonymous public)? What task are they doing, and what do they do today without AI? What is the input (text, voice, documents, images), what is the output (answer, action, structured record), and how often is it used? Is a human in the loop? Which languages? Which regulations apply (financial, health, privacy)?
2. Define success metrics (3 min) Pick one business metric (deflection rate, analyst hours saved, conversion), a few quality metrics (groundedness, task success, extraction F1), guardrail metrics (hallucination rate, unsafe output rate, PII leak rate) and operational metrics (p95 time to first token, cost per request, availability). State targets, for example "p95 TTFT under 1.5 s, groundedness at least 95%, cost under 2 cents per conversation".
3. Understand the data (3 to 5 min) Where does the knowledge live, how fresh must it be, who is allowed to see what, how much is there, what is its quality, and are there labelled examples? The answers decide between RAG and fine-tuning, and they shape the access-control design.
4. Choose the approach (3 min) Walk the ladder: prompt only, then prompt with RAG, then fine-tuning, then an agent, then multiple agents. Justify every step up with a requirement the step below cannot meet.
5. Draw the high-level architecture (10 min) Client, API gateway, orchestrator, retrieval, model gateway, tools, memory, guardrails, observability and feedback. Trace one request end to end with rough latencies.
6. Deep-dive two or three components (10 min) Pick the riskiest ones, or ask the interviewer which they prefer: usually retrieval quality, agent control flow, or serving and scaling.
7. Evaluation (5 min) Offline golden sets, LLM-as-judge with calibration, human review, online A/B tests, regression gates in CI.
8. Safety, security and compliance (3 to 5 min) Prompt injection, PII, tenant isolation, harmful content, audit logs, data residency.
9. Cost and latency (3 to 5 min) Back-of-envelope token cost, GPU count if self-hosted, latency budget per stage, caching and routing.
10. Iteration and trade-offs (2 min) What ships in v1, what comes later, what you would monitor, and the main risks you accepted.

The clarifying-question checklist

Metrics vocabulary to use

The GenAI application stack

Almost every production LLM application, whatever its domain, is made of the same layers. Learn this reference architecture once and adapt it to each question.

                ┌──────────────────────────────────────────────────────────┐
  Users ──▶     │  Client / UI  (web, mobile, IDE plugin, voice, Slack)    │
                │  streaming (SSE / WebSocket), feedback buttons, citations│
                └───────────────────────────┬──────────────────────────────┘
                                            ▼
                ┌──────────────────────────────────────────────────────────┐
                │  API gateway: authN, tenant id, rate limits, quotas      │
                └───────────────────────────┬──────────────────────────────┘
                                            ▼
   ┌──────────────┐   ┌─────────────────────────────────────────────┐   ┌──────────────┐
   │ Input        │──▶│  ORCHESTRATOR (app logic / agent runtime)   │──▶│ Output       │
   │ guardrails   │   │  prompt templates, routing, workflow graph, │   │ guardrails   │
   │ PII, inject, │   │  tool loop, retries, budgets, state         │   │ schema, PII, │
   │ topic, abuse │   └───┬───────────┬────────────┬────────────┬───┘   │ safety, cite │
   └──────────────┘       │           │            │            │       └──────────────┘
                          ▼           ▼            ▼            ▼
                  ┌───────────┐ ┌──────────┐ ┌───────────┐ ┌────────────────────┐
                  │ Retrieval │ │  Tools   │ │  Memory   │ │   MODEL GATEWAY    │
                  │ vector +  │ │ APIs,SQL │ │ session,  │ │ routing, fallback, │
                  │ keyword,  │ │ search,  │ │ long-term │ │ caching, keys,     │
                  │ rerank,   │ │ code exec│ │ profile   │ │ cost metering      │
                  │ ACL filter│ └──────────┘ └───────────┘ └─────────┬──────────┘
                  └─────┬─────┘                                      │
                        ▼                                            ▼
                ┌──────────────┐                     ┌──────────────────────────────┐
                │ Ingestion    │                     │ Hosted APIs  │ Self-hosted   │
                │ parse, chunk,│                     │ (frontier)   │ vLLM / TGI    │
                │ embed, index │                     │              │ GPU cluster   │
                └──────────────┘                     └──────────────────────────────┘

   Cross-cutting:  Observability (traces, token counts, cost, latency, eval scores)
                   Prompt / config registry  ·  Eval pipeline  ·  Feedback store  ·  Audit log

Layer by layer

Tracing one request end to end

1. Request arrives The gateway authenticates the user, attaches tenant and role claims, and checks the token quota (about 5 ms).
2. Input guardrails A fast classifier screens for prompt injection, abuse and off-topic requests, and PII is masked if policy requires it (10 to 50 ms, can run in parallel with retrieval).
3. Context building The orchestrator loads the session summary, rewrites the query using the history, and calls retrieval with an access-control filter (50 to 200 ms).
4. Model call The model gateway picks a model, checks the semantic cache, and streams tokens back (TTFT 200 to 800 ms, then 30 to 100 tokens/s).
5. Tool loop (if agentic) The model asks for a tool, the orchestrator validates the arguments and permissions, executes, and feeds the result back. Each round trip adds a model call.
6. Output guardrails Schema validation, citation checks, PII and safety filters. For streamed output these run on chunks or on sentence boundaries.
7. Respond and log Stream to the user. Write the trace (prompt version, model, tokens, cost, latency, retrieved document IDs) and the feedback hooks.

Choosing the approach: prompt, RAG, fine-tune or agent

The most important decision in a GenAI design is how much machinery to use. Every step up the ladder adds capability but also cost, latency, failure modes and operational burden.

  Complexity / cost / risk ──────────────────────────────────────────▶

  ┌───────────┐   ┌────────────┐   ┌──────────────┐   ┌───────────┐   ┌─────────────┐
  │ Prompting │──▶│  + RAG     │──▶│ + Fine-tune  │──▶│ + Agent   │──▶│ Multi-agent │
  │ zero/few  │   │ ground on  │   │ style, format│   │ tools,    │   │ specialised │
  │ shot, CoT │   │ private /  │   │ domain skill,│   │ multi-step│   │ roles,      │
  │ structured│   │ fresh data │   │ small model  │   │ planning  │   │ handoffs    │
  └───────────┘   └────────────┘   └──────────────┘   └───────────┘   └─────────────┘
   hours           days-weeks        weeks             weeks           months
   Move right only when a concrete requirement fails on the left.

Decision table

They are not exclusive. A common mature pattern is a fine-tuned small model that consumes retrieved context: fine-tune for format, citation style and domain vocabulary, and use RAG for the facts. See fine-tuning and prompt engineering.

Workflow or agent?

A workflow is a fixed graph of steps where the LLM fills in specific nodes (classify, then retrieve, then answer, then validate). An agent lets the LLM choose the next step in a loop. Workflows are predictable, testable and cheap. Agents are flexible but can loop, waste tokens and take unexpected actions. Pick a workflow when the path is known, and pick an agent when the path depends on what is discovered along the way (debugging, research, multi-hop questions). Many good designs are hybrids: a workflow with one bounded agentic step. See agentic AI.

  Common agentic patterns

  Router          query ─▶ [classifier LLM] ─▶ path A | path B | path C
  Prompt chain    step1 ─▶ step2 ─▶ gate ─▶ step3
  Parallelise     ┌─▶ worker 1 ─┐
                  ├─▶ worker 2 ─┼─▶ aggregate / vote
                  └─▶ worker 3 ─┘
  Orchestrator-   planner ─▶ spawns sub-tasks dynamically ─▶ synthesiser
    workers
  Evaluator-      generator ─▶ critic ─▶ (revise) ─▶ critic ─▶ accept
    optimiser
  ReAct loop      think ─▶ act(tool) ─▶ observe ─▶ think ... ─▶ final answer
                  (max_steps, max_tokens, max_cost, timeout)

Model selection, routing, cascades and fallback

No single model is best for every request. Production systems use a portfolio of models and send each request to the cheapest one that can handle it well enough.

Dimensions for choosing a model

Routing strategies

  Cascade with verification

  request ─▶ [small model] ─▶ answer + signals ─▶ [verifier] ── pass ──▶ respond
                                                    │
                                                  fail (low logprob, schema error,
                                                    │   groundedness < t, "unsure")
                                                    ▼
                                              [large model] ─▶ respond
  Expected cost = C_small + P(escalate) × C_large
  Worth it when  P(escalate) < 1 − C_small / C_large   (roughly)

Confidence signals for escalation

• Mean or minimum token log-probability of the answer (low means unsure).
• Self-consistency: sample N answers and escalate if they disagree.
• Structured-output validation failed, or a tool call had invalid arguments.
• A groundedness check (natural language inference or an LLM judge) finds unsupported claims.
• The model explicitly abstains ("I don't have enough information").
• The input features: long, multi-part or domain-flagged queries.

Serving and inference optimization

If you self-host, serving efficiency decides your cost and latency. To discuss it credibly you need to understand the two phases of LLM inference and what limits each one. For model internals see transformers and LLMs.

Prefill versus decode

The KV cache

During generation, every layer stores the key and value vectors of every previous token so they are not recomputed. This cache grows linearly with sequence length and batch size, and it is usually what limits how many concurrent requests fit on a GPU.

Paged attention (KV blocks)

A naive engine reserves one contiguous KV buffer per request sized for the maximum sequence length. A 2k-token request on an 8k reservation wastes 75% of that allocation, and freeing a short request leaves a hole that a longer one cannot fill (fragmentation). Paged attention (the idea behind vLLM) stores KV in fixed-size blocks and maps logical token ranges to physical blocks through a block table, the same way an operating system maps virtual pages to RAM.

• On-demand growth. Allocate the next block when the sequence crosses a boundary, instead of reserving T_max up front.
• Prefix sharing. Two requests with the same first N tokens can point at the same physical blocks (copy-on-write if one of them later diverges).
• Preemption. A scheduler can swap a block to CPU memory and bring it back, because the mapping is already indirect.
• Cost. Attention kernels must gather K/V through the block table. That is why this needs a custom kernel, not a stock matmul.

The optimization toolbox

Speculative decoding economics

Verification of k draft tokens is one target forward pass (the weights are read once). If each draft token is accepted with probability α, the number of tokens you keep from that pass, including the bonus token the target produces after a rejection, is a truncated geometric series.

INT8 versus INT4 (weights)

Serving engines

Worked example: GPU sizing and throughput

Scenario. Serve an 8B-parameter model in BF16 for an internal assistant. Peak traffic is 50 requests per second, with an average of 1,500 input tokens and 300 output tokens per request. The target is p95 TTFT under 1 s and at least 25 tokens/s per user. The GPU is one 80 GB card with about 3.35 TB/s memory bandwidth and about 990 TFLOPS of dense BF16 compute.

1. Weights 8B parameters × 2 bytes = 16 GB.
2. KV budget The engine reserves about 90% of 80 GB, which is 72 GB. Subtract 16 GB of weights and about 4 GB for activations and overhead, leaving about 52 GB for the KV cache.
3. Tokens that fit 52 GB ÷ 128 KB per token ≈ 400,000 tokens. At 1,800 tokens per sequence (1,500 in plus 300 out), that is about 220 concurrent sequences on memory alone.
4. Decode step time Each step reads the weights once (16 GB) plus the KV of every active sequence. With a batch of 64 at an average of about 1,650 cached tokens: 64 × 1,650 × 128 KB ≈ 13.8 GB. Total read per step ≈ 29.8 GB. At 3.35 TB/s that is about 8.9 ms in theory, or about 14 ms at a realistic ~60% efficiency.
5. Decode throughput 64 sequences ÷ 0.014 s ≈ 4,500 tokens/s per GPU, about 70 tokens/s per user, which comfortably beats 25 tokens/s.
6. Demand Output: 50 req/s × 300 tokens = 15,000 tokens/s, so about 15,000 ÷ 4,500 ≈ 3.3 GPUs for decode. Prefill: 50 × 1,500 = 75,000 tokens/s × 2 × 8×10⁹ FLOPs per token = 1.2 PFLOP/s. At about 40% utilisation (~400 TFLOPS effective), that is about 3 GPUs' worth of compute.
7. Total Around 6 to 7 GPUs' worth of work. Add 30 to 50% headroom for bursts, failures and p95 targets, giving roughly 9 to 10 GPUs, run as independent replicas behind a load balancer.
8. Check concurrency with Little's law Each request lasts about 0.4 s of TTFT plus 300 ÷ 70 ≈ 4.3 s of decode, roughly 4.7 s in total. So 50 req/s × 4.7 s ≈ 235 concurrent sequences, or about 24 per GPU across 10 GPUs. That is well inside the KV capacity, so the design is compute and bandwidth bound rather than memory bound.
9. Levers FP8 weights halve the weight reads, which roughly doubles decode throughput at small batch sizes. Prefix caching of a shared 1,000-token system prompt removes about two-thirds of prefill compute. Together these could bring the fleet down to about 5 GPUs.

Quick memory table for weights

Mixture-of-experts (MoE) models activate only a few experts per token. Compute per token looks like a small model's, but all expert weights must sit in memory. Size memory by total parameters and compute by active parameters.

Autoscaling self-hosted models

• Scale on queue depth, KV cache utilisation or in-flight tokens, not CPU. GPU utilisation alone is misleading.
• Cold starts are slow (pulling tens of GB of weights and loading them onto GPUs takes minutes). Keep warm minimum replicas, pre-pull images, and cache weights on local NVMe.
• Serve many LoRA adapters on one base model (multi-LoRA) instead of a separate deployment per tenant or task.
• Send overflow to a hosted API during spikes (burst to cloud) rather than over-provisioning GPUs for the peak.

Latency budgeting and streaming

Users judge responsiveness mainly by how quickly something appears and how smoothly it flows, not by total generation time. Design the latency budget stage by stage.

The metrics

Example budget: RAG chat, p95 TTFT target of 1.2 s

  Stage                              p95 budget   Notes
  ─────────────────────────────────  ──────────   ─────────────────────────────────
  Network + gateway + auth               60 ms    keep-alive, regional endpoint
  Input guardrails (parallel)            40 ms    small classifier, runs alongside ▼
  Query rewrite (small LLM)             200 ms    skip for first-turn queries
  Hybrid retrieval (BM25 + vector)       80 ms    parallel, ANN index in memory
  Rerank top-50 → top-6 (cross-enc.)    120 ms    GPU, batched
  Prompt assembly                        10 ms
  LLM queue + prefill (4k tokens)       600 ms    prefix cache on system prompt
  ─────────────────────────────────  ──────────
  TTFT total                           ~1,110 ms   then stream at ~50 tok/s
  Output guardrails                     streamed   sentence-level, async where safe

Latency levers

• Stream everything. Use server-sent events or WebSockets. Show "searching documents..." status events during retrieval and tool steps.
• Parallelise. Run guardrails, retrieval and user-profile lookups concurrently, and fan out tool calls that do not depend on each other.
• Shrink prompts. Fewer, better chunks (reranking lets you send 5 instead of 20). Summarise old conversation turns. Reuse the prefix cache.
• Shrink outputs. Output length dominates. Ask for concise answers, set max_tokens, and use structured output rather than prose where a machine consumes it.
• Smaller or faster models for intermediate steps (routing, rewriting, grading), keeping the big model only for the final answer.
• Speculative execution. Start likely retrievals before the user finishes typing, or prefetch the next step.
• Semantic caching for frequent questions: embed the query, and if a cached query is similar enough, return the cached answer in milliseconds.
• Avoid unnecessary agent loops. Each tool round trip costs another TTFT plus generation.
• Regional deployment close to users, and provisioned throughput for predictable tail latency on hosted APIs.

Streaming and guardrails: the tension

If you stream tokens straight to the user, you cannot fully check the output before they see it. Options:

Cost modeling and optimization

With hosted models you pay per token. With self-hosted models you pay per GPU-hour whether or not the GPUs are busy. Either way, an interviewer expects you to produce a monthly estimate in a couple of minutes and then cut it.

Worked example: customer-support assistant

Assumptions (illustrative prices). 1 million conversations a month with 4 turns each, so 4 million model calls. Each call has a 1,500-token system prompt with policies and examples, 1,000 tokens of conversation history, 2,000 tokens of retrieved context and a 100-token user message, for 4,600 input tokens, plus 250 output tokens. The large model costs $3 per million input tokens and $15 per million output tokens. Cached input costs 10% of the normal price. The small model costs $0.15 and $0.60 per million.

The result is about a 75% reduction without changing the user experience, provided evaluation confirms quality on the routed segment. Supporting costs are usually small by comparison. Embedding 10 million chunks of 500 tokens once is 5 billion tokens, roughly $100 at typical embedding prices. Vector storage is 10M × 1,024 dimensions × 4 bytes ≈ 41 GB raw, which product quantization (16 one-byte codes per vector) shrinks to about 160 MB.

Hosted versus self-hosted break-even

Ten 80 GB GPUs at an illustrative $2.50 an hour cost about 10 × 2.5 × 730 ≈ $18,000 a month before engineering time. For the workload above, well-optimised hosted usage costs about the same, so self-hosting is not justified on cost alone. It wins when traffic is high and steady (utilisation above about 50%), when a fine-tuned small model replaces a large hosted one, or when compliance forbids sending data outside.

The cost-reduction checklist

1. Measure first Attribute tokens and cost by feature, tenant, prompt version and model. Most spend usually sits in one or two features.
2. Cut output tokens They are the most expensive. Ask for concise answers, use structured output, and set max_tokens.
3. Cut input tokens Shorter system prompts, fewer and better chunks, summarised history, and no duplicated instructions.
4. Cache Provider prompt caching or engine prefix caching for static prefixes, a semantic cache for repeated questions, and an exact-match cache for deterministic calls (classification with temperature 0).
5. Route and cascade Small models for easy or intermediate steps.
6. Batch Use provider batch APIs (often about half price, results within hours) for offline work such as document enrichment, evaluation runs and nightly summaries.
7. Distil or fine-tune Replace a large model on a narrow high-volume task.
8. Cap agents Put maximum steps and a maximum token budget on every agent run. A runaway loop can cost more than a thousand normal requests.
9. Quotas Per-user and per-tenant token budgets with alerts, so cost blowups surface in hours rather than on the invoice.

Scalability and reliability

LLM dependencies fail differently from ordinary services. Provider outages, rate-limit responses (HTTP 429), sudden latency spikes, context-length errors, malformed JSON and silent model updates are all normal. Your system must degrade gracefully. General patterns (circuit breakers, bulkheads, queues) are covered on the system design page; here is how they apply to models.

Failure modes and responses

Rate limiting that understands tokens

Requests per second is the wrong unit. One request may use 100 tokens and another 100,000. Limit on tokens per minute per user, tenant and API key, estimate tokens before the call, then reconcile with actual usage afterwards. Give each tenant a guaranteed slice plus a shared burst pool so one noisy tenant cannot starve the rest.

Queues for heavy and batch work

  Interactive path:   client ─▶ API ─▶ orchestrator ─▶ model (stream)      sync, low latency

  Async path:         client ─▶ API ─▶ [job queue] ─▶ workers ─▶ model / batch API
                                  │                    │
                                  └── job id ◀─────────┴──▶ results store ─▶ webhook / poll

  Priority lanes:     P0 interactive  │  P1 near-real-time agents  │  P2 batch enrichment
                      (preempt lower lanes when GPUs are saturated)

Long document processing, bulk extraction, report generation and evaluation runs belong on queues with retries, dead-letter queues and idempotent workers. Interactive traffic should never wait behind a batch job on the same GPUs, so use separate pools or priority scheduling.

Multi-provider failover

• Keep a prompt variant per model family that has been evaluated. Prompts do not transfer perfectly between models.
• Normalise tool-calling and structured-output formats in the gateway so the orchestrator is provider-agnostic.
• Test failover regularly (chaos drills). An untested fallback usually fails when you need it.
• Check the compliance consequences of failover. Failing over to a provider in another region may break residency rules, so the fallback list must be policy-aware.

Graceful degradation ladder

1. Full experience Frontier model, full retrieval, agent tools.
2. Reduced Smaller model, fewer chunks, tools disabled.
3. Minimal Search results or FAQ links without generation.
4. Handoff Route to a human queue with the context attached, or show an honest "temporarily unavailable" message.

Data pipelines and feedback loops

Two data flows keep a GenAI product healthy. The ingestion pipeline keeps knowledge fresh and correctly permissioned, and the feedback loop turns production behaviour into better prompts, evaluation sets and training data.

Ingestion pipeline for RAG

  Sources (SharePoint, Confluence, S3, DBs, tickets, email)
       │   connectors: full sync + change-data-capture / webhooks
       ▼
  [Parse] PDF/HTML/Office ─▶ text + layout + tables + images (OCR where needed)
       ▼
  [Clean] dedupe, boilerplate removal, language detect, PII tagging
       ▼
  [Chunk] structure-aware (headings, sections, clauses), 300-800 tokens, overlap
       ▼
  [Enrich] metadata: source, owner, ACL groups, date, doc type, section path;
           optional summaries, keywords, questions-this-answers
       ▼
  [Embed] batch embedding model (versioned)   ─┐
       ▼                                        ├─▶ vector index + BM25 index
  [Index] upsert by chunk id; tombstone deletes ─┘   + metadata / ACL store
       ▼
  [Validate] sample retrieval tests, chunk counts, freshness lag alerts

• Incremental updates. Hash content per chunk and re-embed only what changed. Propagate deletes quickly, because a deleted document that can still be retrieved is a compliance incident.
• Permission sync. Access-control lists change more often than content. Sync group membership separately and filter at query time.
• Embedding versioning. Changing the embedding model means re-embedding everything. Build the new index alongside the old one, evaluate it, and switch with an alias (blue/green).
• Freshness SLO. For example, "95% of document edits searchable within 15 minutes". Monitor the lag.
• Quality gates. Detect parse failures (empty text, garbled tables) before they poison retrieval.

The feedback flywheel

   ┌──────────────── production traffic ────────────────┐
   │                                                     ▼
 [users] ─▶ [app] ─▶ traces: prompt, context, output, tools, latency, cost
   ▲                                                     │
   │               explicit: thumbs, ratings, edits, escalations
   │               implicit: re-ask, copy, abandon, accepted suggestion
   │                                                     ▼
   │                       [triage] auto-cluster failures, LLM judge scoring,
   │                                sample for human review
   │                                                     ▼
   │        ┌────────────────────┬────────────────────────┬──────────────┐
   │        ▼                    ▼                        ▼              ▼
   │   new eval cases      prompt / retrieval fixes    fine-tune data  KB gaps
   │        │                    │                        │           (missing docs)
   └────────┴──── CI eval gate ──┴──── canary / A-B ──────┘

• Explicit feedback is sparse (often 1 to 5% of interactions) and skewed towards negatives. Use it for triage, not as a quality metric by itself.
• Implicit signals are plentiful but noisy: a user re-phrasing the same question, copying the answer, accepting a code suggestion, or escalating to a human.
• Human review queues with clear rubrics turn sampled traces into labelled data. Domain experts are expensive, so use active learning to send them the most informative cases.
• Consent and privacy. Using customer conversations for training needs a lawful basis and usually PII scrubbing. Many enterprise contracts forbid it.
• Synthetic data. Use strong models to generate question-answer pairs from documents to bootstrap evaluation sets before real traffic exists. Have humans review a sample to check quality.

LLMOps: prompts, evaluation pipelines, monitoring and experiments

LLMOps is MLOps adapted to systems whose behaviour is driven by prompts, retrieval configuration and third-party models as much as by trained weights. Every one of those is a versioned artefact that can break production.

What to version

Evaluation pipeline

  developer edits prompt / config
          │
          ▼
  [CI job]  run golden set (200-2,000 cases) against candidate + baseline
          │   ├─ deterministic checks: JSON schema, required fields, regex, length
          │   ├─ reference metrics: exact match, F1, ROUGE/BERTScore where suitable
          │   ├─ LLM-as-judge: correctness, groundedness, tone (rubric, calibrated)
          │   ├─ safety suite: jailbreaks, injection, PII, toxicity
          │   └─ cost + latency per case
          ▼
  [Gate]  block merge if any metric regresses beyond threshold (per slice!)
          ▼
  [Canary] 1-5% traffic, online metrics + sampled judge scores
          ▼
  [A/B]   business metric with guardrail metrics ─▶ full rollout / rollback

Evaluation methods

More depth on metrics and safety testing is on the LLM evaluation and safety page.

Production monitoring

• Operational: request rate, error rate by type, TTFT and TPOT percentiles, tokens in and out, cost per request and per tenant, cache hit rates, GPU KV utilisation and queue depth.
• Quality: sampled LLM-judge scores (groundedness, relevance), refusal rate, fallback and escalation rate, output length distribution, schema failure rate.
• Drift: input drift (topic clusters and languages change, new products launch), retrieval drift (share of queries with low top-1 similarity, which signals missing content), and model drift (provider changes). Embedding-based clustering of queries shows new intents.
• Safety: guardrail trigger rates, jailbreak attempts, PII detections, and abuse by particular users or keys.
• Alerts on sudden shifts (refusal rate doubling, groundedness dropping, cost per request spiking), not only on errors.

A/B testing GenAI changes

• Randomise by user or conversation, not by request, so a single conversation does not mix variants.
• Choose a primary metric (resolution rate) and guardrail metrics (CSAT, escalation, cost, latency, safety incidents).
• LLM effects are often small and noisy, so calculate the sample size in advance. Use offline evaluation to filter candidates first and A/B test only the promising ones.
• Watch for novelty effects in user-facing features, and for interaction effects when several prompt experiments run at once.

Security, privacy and compliance

An LLM application widens the attack surface. Any text the model reads can try to instruct it, any tool it holds can be misused, and every prompt and log may hold sensitive data. The ruling principle: never rely on the model to enforce security. Enforce it in deterministic code around the model.

Threat model

Access control in RAG: filter before retrieval

  user ─▶ gateway (SSO token: user_id, tenant_id, groups=[eng, emea])
                │
                ▼
  retrieval gateway builds query:
     vector_search(q_emb, k=50,
        filter = tenant_id == "acme" AND acl_groups ∩ {eng, emea} ≠ ∅
                 AND classification ≤ user.clearance)
                │
                ▼   only authorised chunks can ever enter the prompt
  reranker ─▶ LLM ─▶ answer (citations only to authorised docs)

If a junior engineer asks about executive salary bands, the right outcome is that retrieval returns zero documents, because the filter is applied inside the database query. Telling the model "refuse if salary data appears" is not access control: the data would already be in the context, where an injection or a mistake could reveal it. Post-filtering after retrieval is also weaker, because it can leak through counts, timing or reranker behaviour.

PII handling patterns

Tenant isolation options

Compliance checklist

• Audit logging: who asked what, which documents were retrieved, which model and prompt version answered, which tools ran with which arguments, and who approved them. Logs must be tamper-evident and retained per policy.
• Data residency: the model endpoint, vector store, logs and backups all in the required region, including failover targets.
• Retention and deletion: honour deletion requests across source store, chunks, embeddings, caches, logs and any fine-tuning data. This is much harder once data is inside model weights, which is another reason to prefer RAG for personal data.
• Explainability: in regulated decisions (credit, fraud, clinical), the LLM should produce evidence and suggestions, while a deterministic, auditable layer or a human makes the decision.
• Provider terms: no training on your data, zero or limited retention, enterprise agreements, certifications.
• AI governance: model cards, risk assessments, human oversight for high-risk uses, incident response plans.

Guardrails and hallucination control by design

Guardrails are the checks and constraints around the model that keep inputs, outputs and actions inside policy. Hallucination control is mostly architecture: ground the model, constrain it, verify it, and make abstaining acceptable.

          ┌────────────── INPUT RAILS ──────────────┐
  user ──▶│ size limit │ PII mask │ injection clf │ topic / policy │──▶ orchestrator
          └─────────────────────────────────────────┘
                                       │
          ┌────────────── PROCESS RAILS ────────────┐
          │ retrieval ACL │ tool allow-list │ arg validation │ approval gates │
          │ step / token / cost budgets │ sandboxed execution                  │
          └─────────────────────────────────────────┘
                                       │
          ┌────────────── OUTPUT RAILS ─────────────┐
  user ◀──│ schema validate │ groundedness / citation check │ safety clf │ PII │
          │ confidence threshold ─▶ abstain / escalate to human              │
          └─────────────────────────────────────────┘

Hallucination-reduction techniques, ranked by impact

1. Ground with retrieval Put the authoritative text in the context and instruct the model to answer only from it.
2. Allow abstention Explicitly permit "I don't know" or "not found in sources", and reward it in evaluation. Models hallucinate most when forced to answer.
3. Require citations Have the model cite chunk IDs for each claim, then verify in code that the cited chunk exists and supports the claim.
4. Constrain output Use JSON schemas, enumerations and constrained decoding for extraction, so the model can only pick valid codes or categories.
5. Verify Use a natural language inference model or an LLM judge to check each claim against the context (groundedness score), plus deterministic checks (does the account number exist? does the ICD code exist in the ontology?).
6. Extract, then reason For high-stakes work, first extract quoted spans (verbatim evidence), then reason only over the extracted evidence.
7. Self-consistency Sample several answers and treat disagreement as a signal of low confidence.
8. Lower temperature for factual tasks. This reduces variance but does not create correctness.
9. Human in the loop for anything below a confidence threshold or above a risk threshold.

Confidence scores that actually mean something

Many case studies (clinical, fraud) demand confidence scores. A raw model statement such as "I am 90% sure" is not calibrated. Better sources of confidence:

• Token log-probabilities of the chosen label or span (for classification and extraction).
• Agreement across N samples or across different models.
• Retrieval evidence strength: the similarity of the supporting chunk, the number of independent sources.
• A separate calibrated classifier (for example logistic regression on these features) trained on labelled outcomes, checked with reliability diagrams and expected calibration error.
• Then set thresholds per action: auto-accept above one level, send to human review in the middle band, reject or abstain below.

Worked designs I: assistants, search and moderation

Each design below follows the framework: requirements, architecture, component choices with trade-offs, evaluation and risks. In an interview you will not have time for every detail. Pick the two or three deep-dives that match the risk of the problem.

Design 1: Enterprise document Q&A (RAG)

Prompt. "Build an assistant that answers employees' questions from 5 million internal documents (policies, wikis, tickets, PDFs), with citations, for 50,000 employees."

Requirements.

• Functional: natural-language Q&A with citations, follow-up questions, filters (department, date), "I don't know" when the answer is not in the documents.
• Non-functional: respect document permissions, content fresh within 15 minutes, p95 TTFT under 1.5 s, about 20 QPS at peak, groundedness of at least 95%, all data inside the company cloud region.
• Scale: 5M documents ≈ 50M chunks. At 1,024 dimensions in float32 that is about 200 GB of raw vectors, so use quantization (scalar INT8 about 50 GB, or product quantization) or a disk-based index.

  INGESTION (async)                                      QUERY PATH (sync)
  ───────────────────                                    ─────────────────────────────
  connectors (wiki, drive,                user ─▶ SSO gateway (user, groups)
  tickets) + CDC                                   │
     │                                             ▼
     ▼                                   input rails (injection, PII)
  parse/OCR ─▶ structure-aware                     │
  chunking ─▶ metadata+ACL                         ▼
     │                                   query rewriter (small LLM: resolve
     ▼                                   pronouns from history, expand acronyms)
  embed (batch) ──────────┐                        │
     │                    ▼                        ▼
     └──────▶ [vector index] ◀── hybrid search ── [BM25 index]
              + [ACL/metadata]      with ACL filter; top-50 each, RRF fuse
                                                   │
                                                   ▼
                                     cross-encoder rerank ─▶ top 6-8 chunks
                                                   │
                                                   ▼
                                     LLM (grounded prompt, cite [doc_id])
                                                   │
                                                   ▼
                              citation verifier + groundedness check ─▶ stream
                                                   │
                                                   ▼
                                   traces + feedback ─▶ eval set / KB-gap report

Evaluation. A golden set of 500+ questions with answers and source documents, built from real search logs plus synthetic questions and reviewed by subject-matter experts. Retrieval recall@10 of at least 90%. Faithfulness and answer correctness via a calibrated judge. Permission test suite: users with various roles asking about restricted content must get zero leaks. Online: thumbs up rate, re-ask rate, "no answer" rate by topic (which reveals content gaps).

Risks. Stale or conflicting documents (prefer the newest, show dates, flag conflicts). Permission sync lag. Poor PDF parsing of tables. Users trusting uncited claims. Indirect injection through editable wiki pages.

Design 2: Customer-support agent

Prompt. "Automate first-line support for an e-commerce company: order status, returns, refunds, product questions, 200,000 conversations a day, chat and email."

Requirements. Resolve 50% or more without a human (containment), never make an unauthorised refund, hand off smoothly with full context, keep brand tone, p95 TTFT under 1 s in chat, and meet GDPR-style privacy rules.

  customer ─▶ channel adapter (chat / email / WhatsApp) ─▶ auth / identity (order lookup token)
                                   │
                                   ▼
                     intent + sentiment + language classifier (small model)
                     ├── FAQ / policy ────────▶ RAG over help center ─▶ answer
                     ├── account action ──────▶ AGENT (bounded tool loop)
                     │                           tools: get_order, track_shipment,
                     │                           create_return (idempotent),
                     │                           issue_refund (≤ $50 auto, else approval)
                     ├── angry / legal / VIP ─▶ human queue (with summary + context)
                     └── out of scope ────────▶ polite deflection
                                   │
                                   ▼
               output rails: policy check (no promises beyond policy), PII, tone
                                   │
                                   ▼
               conversation store ─▶ QA sampling ─▶ eval + prompt fixes

Evaluation. Simulated conversations (an LLM playing customer personas, including adversarial ones) scored on task success, policy compliance and tone. Tool-call accuracy, meaning the correct tool with correct arguments. Online: containment rate, CSAT, escalation rate, repeat-contact rate within 7 days (a hidden failure signal), refund error rate.

Risks. Promising refunds that are not in policy (a well-known class of real incidents). Social engineering. Looping conversations that frustrate customers. Tone failures with upset users. Handoffs that lose context.

Design 3: Code assistant (IDE copilot)

Prompt. "Design an in-IDE assistant with inline completions, chat about the codebase, and multi-file edits, for 5,000 developers."

Requirements. Inline completion p95 under 300 ms end to end (it must feel instant). Chat TTFT under 1.5 s. Repository-aware context. Proprietary code must not leave approved boundaries. Suggestions must not introduce vulnerabilities or licence-restricted code.

  IDE plugin
   ├─ inline completion ─▶ context builder (cursor prefix/suffix, open tabs,
   │                        recently edited, imported symbols) ─▶ small FIM model
   │                        (fill-in-the-middle, 1-8B, self-hosted, spec. decoding)
   │                        debounce + cancel on keystroke ─▶ ghost text
   │
   └─ chat / edit ─▶ orchestrator
                       ├─ code index: symbol graph (tree-sitter / LSP) + embeddings
                       │   of functions + grep; incremental re-index on save
                       ├─ retrieval: symbols referenced, callers/callees, similar code
                       ├─ large model (plan + edit as diffs)
                       ├─ tools: read_file, search, run_tests, lint (sandboxed)
                       └─ apply diff ─▶ user review ─▶ accept / reject (feedback)

Evaluation. Offline: pass@k on held-out repository tasks with unit tests, exact match and edit similarity for completions. Online: acceptance rate, characters retained after 30 seconds and 10 minutes (whether the suggestion survived), latency, and whether developers undo accepted suggestions.

Risks. Leaking secrets from the code into prompts (scan and exclude secret files), insecure code patterns (run a static analyser on suggestions), licence-restricted snippets (duplication filter against public code), over-trust by junior developers, and prompt injection through repository files or dependencies.

Design 4: Meeting summarizer

Prompt. "After every video meeting, produce a summary, decisions, and action items with owners, and let users ask questions about past meetings."

Requirements. Summary within 2 minutes of the meeting ending. Speaker attribution. Action items with owner and due date. Multilingual. Consent and recording rules. Access limited to invitees. 1 million meetings a day across tenants.

  meeting audio ─▶ streaming ASR + diarization (who spoke when) ─▶ transcript store
                                                           │ (speaker map from calendar)
                                                           ▼
          [queue] meeting.ended event ─▶ summarization worker
                       │  1. clean transcript (fillers, crosstalk)
                       │  2. chunk by topic segments (≈ 10-15 min each)
                       │  3. map: per-segment notes (small/mid model, parallel)
                       │  4. reduce: merge into summary, decisions, action items
                       │     (structured JSON: owner, task, due, evidence_timestamp)
                       │  5. verify: each action item has a supporting quote
                       ▼
          results store ─▶ email / chat post ─▶ user edits (feedback)
                       │
                       └▶ index transcript chunks (ACL = invitees) ─▶ Q&A over meetings (RAG)

Evaluation. Human-rated summaries on a rubric (coverage, accuracy, conciseness). Action-item precision and recall against human annotations. Faithfulness (claims supported by the transcript). ASR word error rate and diarization error rate, because errors upstream cascade. Online: edit rate on summaries, share and open rates.

Risks. Hallucinated decisions ("the team agreed to..." when nobody did). Attribution errors. Sensitive topics (HR, legal) being summarised and emailed widely. Consent and legal recording rules by jurisdiction. Top-of-hour load spikes.

Design 5: AI search engine (answer engine)

Prompt. "Build a web-scale answer engine: the user asks a question and gets a synthesised, cited answer with sources."

Requirements. Fresh information (minutes to hours), citations for every claim, TTFT under 1.5 s, a very high query volume (thousands of QPS), resistance to SEO spam and injection from web pages, low cost per query.

  query ─▶ query understanding (small LLM): intent, freshness need, rewrite into
           1-4 search queries, decide: answer directly / search / clarify
              │
              ▼
     ┌────────┴────────┬──────────────────┐
     ▼                 ▼                  ▼
  web index         news/fresh index    verticals (weather, stocks, sports APIs)
  (BM25 + dense,    (minutes-old)
   quality scores)
     └────────┬────────┴──────────────────┘
              ▼
   fetch + extract top pages (cached), passage split ─▶ rerank passages
              │   (spam / quality / injection filter on page text)
              ▼
   answer LLM: synthesise with inline [n] citations; stream
              │
              ▼
   citation check (each [n] supports the sentence) ─▶ UI: answer + source cards
              │
   cache popular queries (TTL depends on freshness class)

Evaluation. Answer correctness on factual question sets, citation precision (does the source support the sentence?) and citation recall (are claims cited?), freshness tests on breaking events, human side-by-side comparisons, click-through on sources, and follow-up query rate.

Risks. Confidently summarising misinformation or satire. Taking traffic from publishers (legal and ecosystem issues). Injection via web pages. Stale caches on breaking news. Costs growing with query volume.

Design 6: Content moderation platform (general)

Prompt. "Moderate text, images and short video for a platform with 500 million posts a day, in under a second for text."

Requirements. Policy categories (hate, harassment, self-harm, sexual content, violence, spam, misinformation). Per-market policy variants. Very high throughput at low cost. Appeals. Transparency reports. Human reviewers for borderline cases. This design is expanded in the media case study below with sarcasm and context handling.

  post ─▶ [tier 0] hash match (known bad images/text), rules, spam heuristics   ~1 ms
             │ pass
             ▼
          [tier 1] small multilingual classifiers per category (distilled)     ~10-30 ms
             │   scores ─▶ clear allow (most traffic) / clear block
             │   uncertain band (≈ 1-5%)
             ▼
          [tier 2] LLM policy reasoner: post + thread context + author history
             │    + policy text (RAG over policy docs) ─▶ label + rationale  ~0.5-2 s
             │   still uncertain / high impact
             ▼
          [tier 3] human review queue (priority by reach and severity)
             │
             ▼
          decisions ─▶ enforcement (remove, downrank, label, warn) + appeals
             └─▶ labelled data ─▶ retrain tier-1 classifiers; policy eval sets

Key trade-offs. The cascade keeps average cost close to that of the tier-1 classifier while reserving LLM reasoning for the hard few percent. Thresholds set the balance between precision and recall per category: self-harm and child-safety categories favour recall, borderline political speech favours precision to avoid over-removal. An LLM that reads the written policy can adapt to policy changes quickly, without retraining, but it is slower and costs more per item.

Evaluation. Per-category precision and recall on a stratified golden set, with measured human-reviewer agreement, evaluation sliced by language and dialect (for fairness), prevalence of violating content that users actually see, appeal overturn rate, and reviewer throughput.

Risks. Adversarial evasion (misspellings, coded language, text in images). Bias against dialects. Reviewer wellbeing. Regulatory transparency obligations. Latency spikes during viral events.

Worked designs II: recommendation, analytics, voice, on-device and document extraction

These designs combine LLMs with classic ML systems, databases, speech and edge hardware. The recurring lesson: the LLM is usually one stage in a pipeline, not the whole pipeline.

Design 7: Personalised recommendation with LLMs

Prompt. "Improve recommendations for a large e-commerce or streaming catalogue using LLMs, including conversational discovery ('something like X but lighter')."

Requirements. Recommendations under 100 ms for feeds (so no LLM on the hot path per item). Conversational mode tolerates about 1 to 2 s. Catalogue of 10 million items. Cold-start items and users. Explanations ("because you watched...").

  OFFLINE (LLM as feature factory)                    ONLINE
  ──────────────────────────────────                  ─────────────────────────────────
  item text/reviews ─▶ LLM: attributes, tags,          user ─▶ candidate generation
     themes, summary ─▶ item embeddings                        (two-tower ANN, co-visit,
  user history ─▶ LLM: interest profile summary                 LLM-embedding similarity)
     (periodic, cheap batch API)                                 │ ~1,000 candidates
  synthetic queries for cold-start items                         ▼
                                                           ranking model (GBDT / DNN,
                                                           features incl. LLM tags) ~100
                                                                 │
  CONVERSATIONAL MODE                                            ▼
  "cozy mystery, not too dark" ─▶ LLM parses to            re-rank: diversity, business
   structured filters + semantic query ─▶ retrieval ─▶      rules ─▶ top 20 ─▶ feed
   ranking ─▶ LLM writes short explanation for top 5
   (grounded in item attributes only)

Evaluation. Offline: recall@k, nDCG and coverage on held-out interactions. Online A/B: click-through, conversion, watch time, long-term retention, and diversity and novelty. For explanations: faithfulness to item attributes, measured by a judge.

Risks. Latency if the LLM is put on the per-item path. Feedback loops that narrow what users see. Popularity bias. Invented item features in explanations. Privacy of inferred interests (sensitive categories such as health).

Design 8: Text-to-SQL analytics agent

Prompt. "Let business users ask questions in plain English over the company data warehouse (2,000 tables) and get correct numbers and charts."

Requirements. Correct answers (a wrong number is worse than none). Respect row and column-level security. Read-only. Explain the query. Handle ambiguity by asking. Query cost limits on the warehouse. Answers in under 20 s.

  question ─▶ intent + ambiguity check ("revenue" = gross or net? which region?)
                 │  ask clarifying question if needed
                 ▼
  schema retrieval: embed table/column descriptions + semantic layer (metric
  definitions, joins, certified tables) + few-shot NL→SQL examples ─▶ top 5-10 tables
                 │
                 ▼
  SQL generation LLM (dialect-specific prompt, only retrieved schema)
                 │
                 ▼
  static validation: parse SQL, read-only (SELECT only), allowed tables,
  LIMIT enforced, cost estimate via EXPLAIN ─── fail ──▶ repair loop (≤ 3)
                 │
                 ▼
  execute as the USER's identity (warehouse RLS/CLS applies), timeout
                 │ error ─▶ feed error back to LLM ─▶ retry (≤ 2)
                 ▼
  result sanity checks (empty? absurd magnitude?) ─▶ LLM: narrative + chart spec
                 ▼
  UI: answer, chart, the SQL, definitions used, "certified metric" badge

Evaluation. Execution accuracy (the result set matches the gold query's result) on a few hundred real questions per business domain, not just string match on SQL. Clarification appropriateness. Cost per query on the warehouse. Online: user corrections, analyst spot checks.

Risks. Plausible but wrong joins (fan-out that double-counts), ambiguous metrics, expensive full-table scans, injection through column comments or data values, and users treating uncertified answers as official figures.

Design 9: Voice assistant

Prompt. "Build a voice assistant for a consumer app (or phone support line) that feels natural: fast replies, barge-in, tool use."

Requirements. Voice-to-voice latency under about 800 ms to first audio, interruption (barge-in) handling, noisy environments, multilingual, tool calls (bookings, account lookups), and consent and recording rules.

  mic ─▶ VAD (voice activity) ─▶ streaming ASR (partial transcripts) ──┐
                     ▲                                                   │ end-of-turn
                     │ barge-in: stop TTS when user speaks               ▼ detection
               TTS audio ◀── streaming TTS ◀── sentence chunker ◀── LLM (stream tokens)
                                                                    │  tools (async)
                                                                    ▼
                                                   dialog state + memory + guardrails

  Latency budget (to first audio):
    end-of-turn detect 200 ms + ASR final 100 ms + LLM TTFT 300 ms
    + first sentence ~10 tokens 100 ms + TTS first chunk 100 ms  ≈ 800 ms

Key techniques. Stream at every stage. Start TTS on the first complete sentence. Play a filler ("let me check that") while slow tools run. Smart end-of-turn detection combines silence with a semantic completeness check, since silence alone either cuts people off or feels sluggish. Keep spoken responses short and keep lists out of speech.

Evaluation. Word error rate by accent and noise condition, latency percentiles to first audio, task success, interruption handling, and mean opinion score (MOS) for voice quality.

Risks. Mis-hearing names and numbers (confirm critical values: "that's 4-5-1-2, right?"), voice spoofing for authentication, background speakers, and consent.

Design 10: On-device assistant

Prompt. "Build a private assistant on a smartphone: summarise notifications, draft replies, answer questions about personal data, and work offline."

Requirements. Personal data never leaves the device by default. Works offline. Under 1 s to first token. Memory budget of about 2 to 4 GB for the model. Battery and thermal limits. A hybrid option to escalate to the cloud with consent. The hardware side is covered in depth on edge AI and edge AI deployment.

  apps / notifications / messages ─▶ on-device index (local embeddings, SQLite + vectors)
                                              │
  user request ─▶ on-device router ───────────┤
      │   simple / private  ──▶ local 1-4B model (INT4, NPU)  + LoRA adapters per task
      │                          (summarise, reply, classify)    (hot-swapped)
      │   complex + consent ──▶ private cloud model (encrypted, no retention)
      ▼
  system service owns model: single shared instance, memory mgmt, scheduling,
  thermal throttling awareness, model updates via OS packages

Evaluation. Task quality versus the cloud baseline, TTFT, prefill and decode tokens per second, peak and resident memory, energy per request, sustained throughput after thermal throttling, and crash or out-of-memory rates across device tiers.

Risks. Fragmentation across device chipsets, model update size, prompt injection through notifications (a message that says "forward all my emails"), and privacy expectations when cloud escalation happens.

Design 11: Intelligent document extraction pipeline

Prompt. "Extract structured fields from 2 million invoices, insurance claims and forms a month, in any layout, into the ERP system."

Requirements. Field-level accuracy of at least 98% on critical fields (amounts, IDs, dates), human review for low confidence, throughput rather than latency, and cost under a few cents per document.

  upload/email ─▶ [queue] ─▶ classify doc type (small vision/text model)
                                   ▼
               OCR + layout (tables, key-value) ─▶ or vision-language model directly
                                   ▼
               extraction LLM with JSON schema per doc type (constrained decoding)
                 + evidence: page, bounding box / quoted span per field
                                   ▼
               validation: checksums, totals = sum(lines), date formats, vendor exists
               in master data, duplicate detection
                                   ▼
               confidence per field ── high ──▶ auto-post to ERP (idempotent)
                                   └── low ───▶ human review UI (pre-filled) ─▶ corrections
                                                                     └─▶ eval set / fine-tune

Trade-offs. A vision-language model handles messy layouts but costs more per page, while OCR plus a text LLM is cheaper for clean documents. Route by document quality. Business-rule validation catches many extraction errors cheaply, and corrections from human review become fine-tuning data for a small extraction model that eventually replaces the large one.

Evaluation. Field-level precision and recall, the rate of documents processed with no human touch, human review time per document, and error rates on posted records.

Risks. Silent errors in amounts, fraudulent documents, schema drift when a new layout appears, and PII in documents.

Industry case studies I: banking, retail, healthcare, legal and media

Industry case-study questions describe a business context, a problem, a challenge, an "agentic" requirement and a hard constraint (no hallucination, explainability, multilingual input, long documents, low latency). The hard constraint is where the marks are. Each case below gets a full design: requirements, architecture, data flow, model choices, agentic layer, guardrails, evaluation and trade-offs.

Case 1: Banking fraud narrative intelligence

Context. A large bank processes millions of transactions a day. Fraud analysts write free-text investigation notes describing suspicious patterns, but the fraud models use only structured transaction data, so the insight in those notes is never reused.

Goal. Extract entities (accounts, devices, IPs, merchants, locations) from the notes, identify recurring fraud patterns, and turn them into candidate real-time rules. An agent reads the notes, infers new patterns and suggests rule updates. Constraints: no invented fraud patterns, and every suggestion must be explainable to regulators.

Requirements.

• Functional: entity and relation extraction from notes, linking to transaction records, pattern clustering, rule suggestion with evidence, analyst review workflow, rule simulation before deployment.
• Non-functional: all processing inside the bank's boundary (self-hosted models), full audit trail, pattern discovery within hours (not per transaction), and a real-time scoring path that stays deterministic and runs in milliseconds.

  analyst notes (case mgmt)        transactions / device / login events
         │                                   │
         ▼                                   ▼
  [1] PII-safe ingestion ──────▶ [2] extraction (self-hosted LLM + NER, JSON schema)
                                   entities: account, device_id, ip, merchant, geo,
                                   amount, modus operandi; each with source span
                                          │
                                          ▼
                     [3] entity resolution ─▶ link to structured records (IDs verified)
                                          │
                                          ▼
                     [4] fraud knowledge graph (accounts─devices─merchants─cases)
                                          │
        ┌─────────────────────────────────┼────────────────────────────────┐
        ▼                                 ▼                                ▼
  [5] pattern mining             [6] PATTERN AGENT (planner)         analyst RAG Q&A
  clustering of note embeddings      tools: query_graph, query_txn_stats,  ("similar cases?")
  + graph motifs (shared device      backtest_rule, fetch_case_notes
  across N accounts in 24h)          output: candidate rule + evidence
        └─────────────▶ candidates ◀──┘   + backtest metrics
                                          │
                                          ▼
                     [7] rule simulation on 90 days of history (precision, recall,
                         alert volume, $ saved) ─▶ [8] analyst + model-risk approval
                                          │ approved
                                          ▼
                     [9] deterministic rules engine (real-time, ms) ─▶ monitored

Data flow.

1. Ingest New and updated case notes stream in from case management. Customer names are pseudonymised, and account and device IDs are kept because they are the join keys.
2. Extract A self-hosted LLM, constrained to a JSON schema, extracts entities, relations ("device X used by accounts A, B"), the modus operandi category, and the verbatim span supporting each field. A fine-tuned token-classification model handles the high-volume entity types cheaply.
3. Verify and link Every extracted ID is checked against transaction and device databases. Any ID that does not exist is discarded, which stops invented entities at the source.
4. Graph Verified entities and relations are added to a knowledge graph with case IDs as provenance.
5. Discover Clustering of embeddings finds note clusters with similar methods, and graph mining finds structural patterns (one device shared across many new accounts, mule chains).
6. Agent proposes The agent turns a cluster into a candidate rule, for example "flag transfers over a threshold from accounts under 7 days old that share a device fingerprint with 3 or more other accounts". It calls backtest_rule and attaches the metrics and the supporting case IDs.
7. Approve and deploy Analysts and model-risk reviewers approve, edit or reject. Approved rules are deployed to the deterministic engine with versioning and a shadow period.

Guardrails for "no hallucinated patterns".

• A pattern is only proposed if it is supported by at least N independent cases and by transaction statistics. The agent must cite case IDs and query results, and the orchestrator checks that the cited records exist.
• Rules are expressed in a constrained rule language (a grammar or schema) and validated by a parser. Free-text rules are never deployed.
• Every rule must pass a backtest threshold (for example precision of at least 30% at an acceptable alert volume) before a human sees it.
• Humans approve everything. The agent has no write access to the production rules engine.

Explainability. Each deployed rule carries a lineage record: source cases, extracted spans, graph evidence, backtest results, approver and version. The rule itself is human-readable logic, so the regulator sees deterministic criteria rather than a model's opinion.

Evaluation. Extraction precision and recall per entity type against analyst-labelled notes. Entity-linking accuracy. Pattern novelty (not duplicating existing rules). Accepted-rule rate. Backtest and live precision of deployed rules. Fraud losses prevented. False-positive customer friction.

Trade-offs. Human approval slows response to new attack patterns (hours rather than seconds), which is accepted for regulatory reasons, and a fast-track "temporary rule with a 48-hour expiry" path can close the gap. Self-hosting costs more engineering than an API but is mandatory for this data. A knowledge graph adds build effort but gives the evidence chains regulators want.

Case 2: Retail customer sentiment-to-action engine

Context. An e-commerce platform receives reviews, chat logs and social media complaints. It already has overall sentiment scores, but they do not lead to action and are not linked to operations such as returns and logistics.

Goal. Aspect-based sentiment analysis (which aspect, what polarity), mapping complaints to operational issues, and automated workflows. For example, a delivery-delay spike should trigger a logistics audit, and quality complaints about a product should alert the supplier team. Twist: multilingual and code-mixed text, such as Hindi-English mixing written in Latin script ("delivery bahut late thi, product accha hai").

Requirements. Process about 1 million texts a day in near-real time (minutes). Aspect taxonomy tied to operations (delivery speed, packaging, product quality, size and fit, refund process, agent behaviour). Linking to order, SKU, seller, warehouse and courier. Trend and anomaly detection. Workflow triggers with deduplication. Dashboards.

  reviews ─┐
  chats ───┼─▶ [stream] ─▶ language ID + code-mix detection ─▶ normalization
  social ──┘                (per-token lang tags)             (transliteration,
                                                               slang lexicon, emoji)
                                   │
                                   ▼
         [ABSA model] multilingual encoder fine-tuned (aspect, opinion span,
          polarity, intensity) ── low confidence ──▶ LLM fallback (few-shot, JSON)
                                   │
                                   ▼
         entity linking: order_id / SKU / seller / courier / warehouse (from metadata
          or text) ─▶ enriched event store (aspect, polarity, entity, time)
                                   │
                                   ▼
         aggregation + anomaly detection (per aspect × entity, rolling baselines,
          e.g. "delivery_delay for courier C in city Y +240% vs 7-day baseline")
                                   │
                                   ▼
         ACTION AGENT: reads anomaly + sample texts ─▶ decides playbook
           ├─ logistics_audit(courier, region)     ├─ supplier_alert(SKU, evidence)
           ├─ pause_listing(SKU) [needs approval]  └─ customer_outreach(segment)
                                   │
                                   ▼
         ticketing / workflow system (dedup, owner, SLA) ─▶ outcome tracked ─▶ feedback

Data flow.

1. Ingest and normalise Detect language at the token level, transliterate Romanised Hindi to a consistent form (or keep it Romanised if the model was trained that way), expand slang and handle emoji as sentiment signals.
2. Aspect-based analysis A fine-tuned multilingual encoder (trained on labelled code-mixed data, part of it synthetic from an LLM and reviewed by humans) outputs aspect, opinion span, polarity and intensity. Cases below the confidence threshold go to an LLM with few-shot examples and a strict JSON schema.
3. Link to operations Join each record to order and SKU metadata (from chat or review context), which gives the seller, warehouse and courier.
4. Detect Aggregate by aspect and entity over sliding windows and flag statistically significant spikes against the baseline (not single complaints).
5. Act The agent chooses a playbook, writes an evidence summary with sample quotes, and opens a deduplicated ticket for the right team. High-impact actions (pausing a product listing) need human approval.
6. Close the loop Track whether the complaint rate fell after the action, and feed the outcomes back to tune thresholds and playbooks.

Guardrails. Tickets are deduplicated by entity and issue within a time window. Rate limits apply per team. Sarcasm and review spam are detected (sudden bursts of similar reviews from new accounts). PII is removed from evidence quotes. Every action carries its evidence and can be undone.

Evaluation. Aspect F1 and polarity accuracy, sliced by language (English, Hindi, code-mixed) and channel. Anomaly detection precision (share of triggered audits that confirmed a real issue) and time to detection. Business: time from issue to action, reduction in the related return rate, CSAT.

Trade-offs. Near-real time rather than real time keeps cost down, since a few minutes of delay is fine for operations. A fixed aspect taxonomy keeps things actionable, and a periodic LLM clustering job over "other" complaints finds new aspects to add.

Case 3: Healthcare clinical decision augmentation

Context. Doctors write unstructured clinical notes. Important symptoms are buried in text, and downstream models have no structured representation to work from.

Goal. Extract symptoms, diagnoses, medications (with dose and frequency), allergies and procedures. Map them to clinical ontologies (ICD for diagnoses, plus standard vocabularies for medications and lab tests). Generate patient summaries. An agent monitors records and suggests missing tests and possible diagnoses. Risk: zero tolerance for invented facts, and confidence scores are required.

Requirements. On-premises or in a compliant cloud with health-data agreements. Complete audit trail. Clinician in the loop for every suggestion. Latency: extraction within minutes of a note being signed, summary on demand within 10 s. Integration with electronic health records through standard health-data APIs.

  EHR notes, labs, meds, vitals (FHIR-style API)
         │
         ▼
  [1] de-identify for model training paths; identified only in secure runtime
         │
         ▼
  [2] clinical NER (fine-tuned clinical encoder): problem, drug, dose, route,
      frequency, test, procedure, anatomy  + context detection:
      NEGATION ("no chest pain"), UNCERTAINTY ("rule out"), TEMPORALITY (history
      vs current), EXPERIENCER (patient vs family history)
         │
         ▼
  [3] concept normalization: candidate generation from ontology index (lexical +
      embedding) ─▶ LLM/cross-encoder picks code from CANDIDATES ONLY ─▶ code +
      confidence (calibrated) + source span
         │
         ▼
  [4] structured patient record (problems, meds, allergies, timeline) + provenance
         │
    ┌────┴─────────────────────────────┬──────────────────────────────────┐
    ▼                                  ▼                                  ▼
  [5] summary generator            [6] MONITORING AGENT                 downstream
  (extract-then-summarize:         triggers: new note / lab result      models (risk
   every sentence cites span;      tools: get_labs, get_meds,           scores)
   NLI check vs source)            guideline_RAG (curated guidelines),
                                   drug_interaction_db, ontology_lookup
                                   output: "Consider HbA1c: diabetes on
                                   problem list, no HbA1c in 12 months
                                   [guideline §x], confidence 0.92"
                                        │
                                        ▼
                               clinician inbox / EHR sidebar ─▶ accept / dismiss
                               (reason) ─▶ feedback + audit

Data flow.

1. Capture A signed note or new result triggers processing through the health-record integration.
2. Extract with context Clinical NER finds mentions, and assertion detection marks negated, uncertain, historical and family-history mentions. "Denies chest pain" must not become a chest-pain finding. This is the single most important clinical NLP detail.
3. Normalise Candidate codes come from the ontology index. The model chooses among valid candidates (constrained selection) and returns a calibrated confidence and the evidence span. Invented codes are impossible by construction.
4. Structure The patient timeline stores every fact with provenance (note ID, character offsets, author, time).
5. Summarise Evidence sentences are selected first, then an abstractive summary is written with a citation per sentence. An entailment checker verifies each sentence, and unsupported sentences are dropped or flagged.
6. Suggest The agent compares the structured record with curated clinical guidelines retrieved from a vetted corpus, and uses deterministic rules for gaps where possible ("diabetic patient, no eye exam in 12 months"). It uses the LLM to explain and rank, and outputs suggestions with guideline citations and confidence.
7. Clinician decides Suggestions appear as non-interruptive advisories. Accept and dismiss decisions (with a reason) are logged and fed back.

Guardrails. Every generated statement links to source evidence, and anything unlinked is removed. Codes must exist in the ontology version in use. Confidence thresholds decide whether an item is shown, shown with a warning, or held back. Drug-interaction checks use an authoritative database, not the LLM. No suggestion is ever applied automatically to orders. Suggestion volume per patient is capped to avoid alert fatigue.

Confidence scores. Combine ranker probability, extraction model probability, agreement between two methods (rules versus model), and evidence count. Calibrate against clinician-labelled data with reliability curves, and report calibration error. Show confidence to clinicians as bands (high, medium, low) rather than false-precision decimals.

Evaluation. Entity and assertion F1 per type against clinician annotations. Coding accuracy at the top-1 and top-3 levels. Summary faithfulness (the share of sentences fully supported, target near 100%) and omission rate of critical facts (allergies, active medications). Suggestion acceptance rate and clinician-rated usefulness. Prospective shadow-mode studies before go-live. Fairness across patient demographics.

Trade-offs. Higher precision thresholds reduce harmful noise but miss some useful suggestions, and in clinical tools missing an item is preferred to overwhelming clinicians. Extractive summaries are safer but less readable than free abstractive ones. Regulatory classification as a medical device may apply if the system suggests diagnoses, which affects validation scope.

Case 4: Legal contract intelligence

Context. A law firm handles thousands of contracts. Manual review is slow and inconsistent between reviewers.

Goal. Extract clauses (termination, liability, indemnity, confidentiality, governing law, assignment, change of control, payment terms), identify risky terms, and compare them with the firm's standard templates (playbook). An agent reviews contracts, flags deviations and suggests rewritten clauses. Complexity: highly domain-specific language and long documents (100+ pages) with cross-references and defined terms.

Requirements. Clause-level accuracy with exact page and paragraph references. Deterministic, repeatable results (temperature 0, pinned versions). Client confidentiality with matter-level access control and ethical walls. Review of a 100-page contract in under 10 minutes. Output as a redline document in the lawyers' word processor.

  contract (DOCX/PDF) ─▶ parse layout: sections, numbering, headings, tables,
                          schedules; OCR if scanned
                                   │
                                   ▼
          structure tree: Article ▸ Section ▸ Clause ▸ sub-clause (+ page refs)
          defined-terms extractor ("Losses" means ...) ─▶ glossary
          cross-reference resolver ("subject to Section 12.3")
                                   │
                                   ▼
          clause classifier (fine-tuned encoder / LLM) ─▶ clause type per node
                                   │
                                   ▼
          for each clause type: retrieve playbook position (standard, fallback,
          walk-away) from firm template library (RAG, per practice area)
                                   │
                                   ▼
          REVIEW AGENT (per clause, parallel):
            inputs: clause text + resolved defined terms + cross-refs + playbook
            1. compare ─▶ deviation? which attributes (cap amount, carve-outs,
               notice period, mutuality)
            2. risk score (rubric: high/med/low) + reasoning + quoted evidence
            3. suggest redline: minimal edit toward playbook fallback language
                                   │
                                   ▼
          document-level checks: missing mandatory clauses, conflicting clauses,
          uncapped liability + broad indemnity combination
                                   │
                                   ▼
          lawyer UI: issues list ranked by risk ─▶ tracked-changes export
          lawyer accept / edit / reject ─▶ feedback to playbook + eval set

Handling long documents. Do not paste 100 pages into one prompt and hope. Parse the structure, process clause by clause in parallel, and supply each clause with its resolved dependencies: the definitions of capitalised terms it uses and the text of the sections it references. A long-context model can run a final document-level pass over a condensed map (clause summaries plus key terms) for interactions between clauses, such as a liability cap undermined by an indemnity carve-out.

Guardrails. Every flagged issue quotes the exact clause text with its location, and quotes are verified by string match against the source. Suggested language comes from the approved library or is marked "novel, needs partner review". Matter-level access control and ethical walls are enforced at retrieval. Outputs are labelled as draft review assistance, not legal advice.

Evaluation. Clause extraction F1 per type (recall matters most, since a missed indemnity clause is the costliest error). Deviation detection precision and recall against senior-lawyer reviews. Redline acceptance rate. Review time saved. Consistency: the same contract reviewed twice gives identical flags. Evaluate with lawyer-annotated contracts across contract types (supply, licensing, NDAs, employment).

Trade-offs. A clause-level pipeline is more engineering than a single long-context call but is more accurate, cheaper, parallel and traceable. Recall-oriented thresholds create more false flags for lawyers to dismiss, which is acceptable because the cost of a miss is far higher.

Case 5: Media content moderation at scale (context and nuance)

Context. A social platform moderates user-generated content. Rule-based filters fail on nuanced toxicity, and they miss sarcasm, context and intent.

Goal. Detect hate speech, misinformation and harmful sarcasm, taking conversational context into account. An agent escalates borderline cases and learns from moderator feedback. Advanced requirement: real-time moderation at low latency.

Requirements. Decisions on text posts and comments within about 200 ms p95 before or just after publishing, with deeper review asynchronously within minutes. Context-aware: the parent post, the thread and the target of a reply. Per-region policy. Appeals. High recall on severe harm, high precision on speech-sensitive categories.

  new post/comment
       │
       ▼
  SYNC PATH (≤ 200 ms)                                   ASYNC PATH (seconds-minutes)
  ─────────────────────                                  ──────────────────────────────
  hash / blocklist / spam rules ─▶ block obvious         sampled + uncertain + reported
       │                                                 items + viral-velocity items
       ▼                                                        │
  context builder: parent post, last k thread msgs,             ▼
  target user, author risk features (cached)              LLM POLICY REASONER
       │                                                  input: item + full thread +
       ▼                                                  retrieved policy sections +
  multi-label classifier (distilled multilingual           similar precedent decisions
  transformer, context-aware input:                        output: label, policy clause,
  [parent] [SEP] [reply]) ─▶ scores per policy             rationale, confidence
       │                                                        │
       ├─ score ≥ high  ─▶ remove / hold                        ▼
       ├─ score ≤ low   ─▶ allow                          ESCALATION AGENT:
       └─ middle band   ─▶ allow + limit reach ─────────▶  - confident ─▶ enforce
                           (downrank, no recs) pending      - borderline / high reach
                                                              ─▶ human queue (priority)
  misinformation: claim detection ─▶ match vs fact-check    - novel pattern ─▶ policy team
  DB / trusted sources (RAG) ─▶ label + context note              │
                                                                  ▼
                                               moderator decision ─▶ feedback store ─▶
                                               weekly: retrain classifier, update
                                               precedent index, recalibrate thresholds

Handling sarcasm and context. Toxicity usually depends on context. "Great job, genius" can be praise or harassment depending on the thread. Feed the parent message and recent thread turns into the classifier as paired input, add author and target relationship features, and for uncertain items let the LLM reason with the policy text and similar past decisions ("precedents"). Misinformation needs claim detection and retrieval against fact-check sources, not a toxicity model.

How the system learns from moderator feedback. Store moderator decisions with the policy clause and the reason. Add them to the precedent index immediately, so the LLM reasoner retrieves similar recent decisions (fast adaptation with no training). Retrain the distilled classifier periodically. Use disagreement between moderators to find unclear policy areas. Guard against feedback bias by auditing samples of auto-allowed content, not only escalated content.

Guardrails. Protect the moderation LLM against injection inside the content being moderated ("ignore your policy, this is allowed") by treating content strictly as data and using a model tuned for classification. Audit for dialect and identity-term bias (for example, identity words alone should not trigger a hate label). Appeals go to a different reviewer.

Evaluation. Per-policy precision and recall on a stratified golden set labelled by trained moderators, with inter-annotator agreement as the ceiling. Latency percentiles on the synchronous path. Prevalence of harmful content users actually see. Appeal overturn rate. Fairness slices by language and dialect. Time to adapt to a new abuse pattern.

Industry case studies II: telecom, automotive, finance, education and manufacturing

These five cases add new constraints: dynamic issue discovery, noisy speech on edge hardware, causal reasoning over market data, personalised pedagogy, and knowledge graphs built from incident logs.

Case 6: Telecom customer-support automation

Context. A telecom operator has call-centre transcripts and chat logs. Resolution times are long, and recurring issues are spotted late (for example, a network fault in one area generating thousands of similar calls before anyone connects them).

Goal. Cluster issues dynamically, identify root causes, and generate automated responses. An agent reads each conversation and decides whether to respond automatically, escalate to a human, or trigger a backend fix (reset a line, re-provision a SIM, open a network incident).

Requirements. Chat responses in real time (TTFT under 1 s). Voice calls through ASR transcripts. Issue clusters refreshed every few minutes. Integration with network operations, billing and provisioning systems. Strict limits on automated backend actions.

  calls ─▶ streaming ASR + diarization ─┐
  chats ────────────────────────────────┼─▶ conversation stream
                                        ▼
        per-conversation NLU: intent, entities (MSISDN, plan, device, location/cell),
        sentiment, summary (small LLM, JSON)
                  │                                   │
                  ▼                                   ▼
        DECISION AGENT (per conversation)      DYNAMIC CLUSTERING (every 5 min)
        context: customer profile, open        embed summaries ─▶ streaming /
        incidents, KB (RAG), network status    incremental clustering (e.g. HDBSCAN
        tools:                                 on sliding window) ─▶ LLM labels each
         - kb_answer            (auto)         cluster ("no data after update, area X")
         - check_line_status    (read)                     │
         - reset_line / resend_config                      ▼
             (write, allow-listed, idempotent)   ROOT-CAUSE CORRELATOR: join cluster
         - open_incident        (write, dedup)   with network alarms, deployments,
         - escalate_to_human    (handoff)        outages by cell / region / device
        policy: auto only if intent in allow-        │
        list AND confidence ≥ t AND no risk      incident opened / linked; proactive
        flags (legal threat, churn intent,       message to affected customers;
        vulnerable customer)                     agent auto-answers "known issue"
                  │
                  ▼
        response / action / handoff ─▶ outcome (resolved? repeat call in 7 days?)

Data flow.

1. Understand Every conversation (a live chat, or a call transcript streamed from ASR) is summarised into structured fields: intent, device, location, plan, sentiment.
2. Decide The decision agent checks known incidents first. If the issue matches an open incident, it replies with the status and expected fix time. Otherwise it answers from the knowledge base, runs a permitted diagnostic or fix, or escalates.
3. Cluster Summaries from the last N hours are embedded and clustered incrementally. New or fast-growing clusters are labelled by an LLM and scored by growth rate.
4. Find root causes The correlator links clusters to network alarms, recent software rollouts and device models by region and time. It proposes a hypothesis with evidence ("87% of the cluster uses device model M, which received firmware F yesterday").
5. Act at scale A confirmed incident triggers a proactive notification and a scripted response for all agents, human and AI. Repeat contacts drop.

Guardrails. No automated action on accounts showing fraud or vulnerability flags. Retention offers only from an approved catalogue. Customer identity verified before account actions. Transcripts redacted of card numbers. Cluster-driven proactive messages reviewed by a human before mass sending.

Evaluation. Automated resolution rate, the correctness of the respond, escalate or fix decision against expert labels, repeat-contact rate, average handling time, cluster quality (human-rated purity), time from first complaint to incident detection, and root-cause hypothesis precision.

Trade-offs. A lower automation threshold means more containment but more wrong actions. Start conservative and loosen per intent as evidence builds. A shorter clustering window detects faster but is noisier.

Case 7: Automotive voice command understanding

Context. In-car assistants process spoken commands. Commands are ambiguous and context awareness is weak. The classic edge case: "make it cooler". Does it mean the cabin temperature, or the music?

Goal. Understand intent from noisy speech (road noise, several passengers, music), keep the conversation context, and have an agent that tracks user preferences and adapts commands dynamically.

Requirements. Voice to action in under about 500 ms for vehicle controls. Works without connectivity (tunnels, rural roads), so the core runs on the vehicle. Safety first: distraction limits and no unsafe actions while driving. Multi-speaker awareness (driver versus passenger zone). Multilingual, with accents.

  microphone array ─▶ beamforming + echo cancel (remove car audio) + noise suppress
          │                          speaker zone detection (driver / passenger)
          ▼
  wake word ─▶ on-device streaming ASR (small, quantized; domain vocabulary boost:
               "defrost", contact names, POIs)  ─▶ n-best hypotheses + confidence
          │
          ▼
  ON-DEVICE NLU / small LLM (INT4 on car SoC NPU) ─▶ structured intent:
     {domain, action, slots, confidence}
          │  uses CONTEXT STATE:
          │   - vehicle signals: cabin temp 27°C, AC on, media playing (jazz), speed
          │   - dialog state: last domain = climate (2 turns ago)
          │   - user profile: driver prefers 21°C, dislikes loud bass
          ▼
  AMBIGUITY RESOLVER: score candidate interpretations
     "make it cooler":  climate.decrease_temp (cabin hot, last domain climate) 0.82
                        media.change_style   (music just changed)            0.11
     ─▶ if margin large: act + brief confirm ("Lowering to 22 degrees")
     ─▶ if margin small: ask ("Temperature or music?")
          │
          ▼
  vehicle action API (safety policy layer: allowed while driving? limits?)
          │
          ▼
  PREFERENCE AGENT: logs outcome + corrections ("no, I meant the music")
     ─▶ updates per-driver preference model (on-device; synced encrypted if opted-in)
  cloud path (when connected): complex queries, navigation search, general Q&A

Resolving "make it cooler". Treat it as ranking interpretations using context features: current cabin temperature against the driver's preferred temperature, whether the climate system was mentioned recently, whether the music just changed, the time of day, and the individual driver's history of what "cooler" meant. Act when the top interpretation clearly wins, and confirm briefly out loud so the user can correct it. Ask when the margin is small. Learn from corrections: if this driver says "cooler" and then corrects to music twice, their prior shifts.

Guardrails. A safety policy layer outside the model decides which actions are allowed while moving. Value ranges are enforced (the temperature stays within limits). Safety-critical driving functions are never controlled by voice. Only the driver zone can issue certain commands. Responses stay short to limit distraction.

Evaluation. Word error rate by noise condition (speed, windows open, music volume) and accent. Intent accuracy and slot F1. Ambiguity resolution accuracy on a curated ambiguous-command set. Clarification rate. Latency percentiles on target hardware. Task completion, and correction rate as an implicit signal.

Case 8: Financial market intelligence with NLP and RAG

Context. Analysts track news, earnings call transcripts, filings and research reports by hand, which is slow and subjective.

Goal. A RAG system over the financial corpus that extracts insights about stock movements. An agent monitors news streams, detects anomalies and correlates them with market data. Twist: tell causation from correlation.

Requirements. Minute-level freshness for news. Point-in-time correctness (no look-ahead: an answer about 2 March must not use documents published later). Citations. Numbers pulled from source tables rather than generated. Compliance rules (no material non-public information, recorded research). Analyst-facing, not automated trading.

  news wires, filings, transcripts, reports ─▶ ingest (timestamp = publish time)
          │                                        │
          ▼                                        ▼
  NLP enrichment: entity linking to tickers    market data: prices, volumes,
  (company ─▶ ticker ─▶ sector), event type    returns, factor/index returns,
  (earnings, M&A, guidance, lawsuit, exec      options implied vol
  change), sentiment per entity, numeric
  extraction (guidance figures + units)
          │
          ▼
  indexes: vector + BM25 + time-partitioned + entity filters
          │
  ┌───────┴───────────────────────────┬───────────────────────────────────────────┐
  ▼                                   ▼                                           ▼
  ANALYST RAG Q&A                    MONITORING AGENT                          reports
  "why did X fall on Tue?"           1. detect anomaly: abnormal return =        (daily brief)
  retrieval filtered to              actual − expected (market/sector model),
  [t−7d, t], entity X                 z-score > 3, or volume spike
  answers with citations +           2. retrieve news for ticker in window
  numbers from tables                  BEFORE and around the move
                                     3. causal-evidence scoring (below)
                                     4. write alert: move, candidate drivers
                                        ranked, evidence, confidence, caveats

Causation versus correlation: how the agent reasons.

• Temporal precedence. The candidate cause must be published before the move started, measured to the minute. News that follows the move is often explaining it after the fact.
• Abnormal returns, not raw returns. Subtract the market and sector moves (an event-study approach). If the whole sector fell, a company-specific headline is probably not the cause.
• Specificity. Does the news name this company and relate to value (guidance cut, regulatory action)? Is it new information, or did it appear earlier?
• Magnitude and consistency. Compare with historical reactions to similar events. Check whether peers mentioned in the same news also moved.
• Alternative explanations. Index rebalancing, options expiry, large block trades, macro releases at the same time.
• Output language. Report "likely driver, with evidence" and a confidence level. Never claim proven causation, and list confounders. The LLM writes the narrative, while the statistics come from deterministic code.

Guardrails. Point-in-time retrieval enforced by filters. Every number traced to a source cell. Disclaimers that the output is research support and not advice. No personalised investment recommendations. Separation of restricted data. Full logs for compliance review.

Evaluation. Retrieval recall on questions about past events whose drivers are known. Numeric accuracy (exact match against sources). Precision of driver attribution against analyst judgement on historical moves. Time to alert. Analyst usefulness ratings. Backtests of alert quality free of look-ahead.

Case 9: Education adaptive learning assistant

Context. Learners use a chat-based learning product. Content is static and not personalised.

Goal. Analyse learner questions, detect knowledge gaps, and adapt the learning path. An agent tracks progress, adjusts difficulty dynamically and recommends content.

Requirements. Pedagogically sound: guide learners rather than hand them answers. Grounded in the curriculum. Age-appropriate safety. Privacy for minors. Explainable progress for teachers and parents. Costs that work at the scale of millions of students.

  learner message ─▶ safety filter (age-appropriate, self-harm signals ─▶ escalate)
          │
          ▼
  question analysis (small LLM, JSON): topic, skill ids (mapped to curriculum
  knowledge graph), question type (conceptual / procedural / homework-dump),
  misconception signals ("thinks 1/3 + 1/4 = 2/7")
          │
          ▼
  LEARNER MODEL (per learner, per skill): mastery probability
     updated by evidence (knowledge tracing: correct/incorrect attempts,
     hints used, time) ─▶ gap = prerequisite skill with low mastery
          │
          ▼
  TUTOR AGENT
    tools: curriculum_RAG (approved content), generate_practice(skill, difficulty),
           grade_answer (rubric + solver/verifier for math), update_mastery,
           recommend_next (policy over knowledge graph)
    behaviour: Socratic hints before solutions; difficulty target ≈ 70-85%
               expected success (zone of proximal development)
          │
          ▼
  response + practice item ─▶ learner attempt ─▶ graded ─▶ learner model updated
          │
          ▼
  teacher dashboard: mastery map, misconceptions, flagged struggles

Guardrails. Do not simply complete graded homework: detect homework-dump requests and switch to guided mode. Strict content safety for minors. Escalate wellbeing signals to a human. Collect minimal data with parental consent and set retention limits. Avoid labelling students in discouraging ways.

Evaluation. Learning outcomes (pre- and post-test gains against a control group), mastery prediction accuracy (does the learner model predict the next attempt?), hint helpfulness ratings, correctness of generated content, engagement and retention, and fairness across student groups.

Case 10: Manufacturing incident report intelligence

Context. Factories produce text-based incident logs. Nothing is learned systematically from past failures, so the same incidents repeat.

Goal. Extract causes, actions and outcomes from reports, and build a knowledge graph. An agent reads each new incident, suggests preventive actions and flags recurring patterns.

Requirements. Handle shorthand, jargon, several languages and poor grammar written by technicians. Link to equipment master data and maintenance systems. Explainable suggestions based on past incidents. Safety incidents always go to humans. On-premises deployment at plant level is common.

  incident logs, maintenance work orders, sensor alarms, shift notes
          │
          ▼
  normalize: abbreviation expansion (plant glossary), language, equipment tag
  linking (e.g. "P-101 pump" ─▶ asset id)
          │
          ▼
  EXTRACTION (LLM + schema, few-shot per plant):
     {asset, component, failure_mode, symptoms, root_cause, cause_category
      (human / machine / material / method / environment), actions_taken,
      outcome, downtime_min, severity}  each with source span + confidence
          │
          ▼
  KNOWLEDGE GRAPH
     (Asset)─has_component─▶(Component)─failed_with─▶(FailureMode)
     (Incident)─caused_by─▶(Cause)─mitigated_by─▶(Action)─resulted_in─▶(Outcome)
     + taxonomy alignment (failure-mode codes), dedupe of similar causes
          │
          ├──────────────────────────────────────────┐
          ▼                                          ▼
  PREVENTION AGENT (on new incident)          PATTERN MONITOR (daily)
  1. extract + link new incident              recurrence: same asset + failure
  2. graph query: similar incidents            mode ≥ N in window; cross-plant
     (same component/failure mode) + vector    similarity; cause trends
     similarity on narratives                 ─▶ flags to reliability engineer
  3. which actions led to good outcomes?
  4. suggest preventive actions ranked by
     evidence (k past cases, success rate)
  5. create draft work order ─▶ engineer approves

Guardrails. Suggestions cite past incident IDs. Actions touching safety systems need engineer approval, and the agent drafts work orders but never executes them. Uncertain extractions are flagged for review. Safety-severity incidents trigger mandatory human workflows regardless of the model's output.

Evaluation. Extraction F1 for cause and action fields against engineer labels. Entity-linking accuracy. Graph quality (duplicate node rate). Suggestion acceptance rate. Repeat-incident rate and mean time between failures over months (the business outcome). Downtime reduction.

Senior-level trade-offs and failure patterns

At senior and staff level, the interview is less about knowing components and more about judgement: what to build first, what to leave out, how to reduce risk, and how to explain trade-offs to stakeholders.

The core trade-off axes

Phased delivery plan (say this in interviews)

1. Week 0 to 2: prototype Hosted frontier model, simple RAG, 50 hand-written test cases, internal users only. Goal: prove the value and find the failure modes.
2. Month 1 to 2: MVP Golden set of 300 or more cases, guardrails, tracing, feedback buttons, access control, cost dashboards. Limited beta with a human fallback.
3. Month 3 to 6: scale Routing and caching for cost, CI evaluation gates, A/B testing, SLOs, failover, and a fine-tuned small model for the high-volume sub-task.
4. Ongoing Feedback flywheel, drift monitoring, red-teaming, periodic model re-selection as new models ship.

Failure patterns seen in production

Quick revision

• A GenAI system is a distributed system with a slow, costly, probabilistic, instructable component in the request path. Design to contain it.
• Framework: clarify use case and users, then metrics, data, approach, architecture, deep-dives, evaluation, safety, cost and latency, iteration.
• State success metrics early: one business metric, quality metrics, guardrail metrics and operational SLOs.
• Approach ladder: prompting, then RAG, then fine-tuning, then agent, then multi-agent. Climb only when a requirement forces it.
• RAG supplies knowledge that changes, with citations and permissions. Fine-tuning supplies behaviour, format, style and a cheaper small model.
• Workflows are predictable and testable. Agents are flexible but compound errors: five 95%-reliable steps give about 77% end to end.
• Reference stack: UI, API gateway, orchestrator, retrieval, tools, memory, model gateway, serving, guardrails, observability, LLMOps.
• The model gateway centralises routing, failover, caching, keys, cost metering and data policy.
• Routing sends each request to the cheapest adequate model. A cascade escalates on low confidence. A fallback chain covers outages.
• Cascade expected cost = C_small + p_escalate × C_large.
• Prefill is compute-bound and sets TTFT. Decode is memory-bandwidth-bound and sets tokens per second.
• KV bytes per token = 2 × layers × KV heads × head dimension × bytes. That is about 128 KB for an 8B model and 320 KB for a 70B model in FP16 with grouped-query attention.
• Weight memory ≈ parameters × bytes per parameter (2 for BF16, 1 for FP8 or INT8, 0.5 for INT4).
• Prefill FLOPs ≈ 2 × parameters × input tokens. Decode step time ≈ (weight bytes + active KV bytes) / bandwidth.
• Continuous batching adds and removes sequences at every step, giving several times the throughput of static batching.
• Paged attention stores the KV cache in blocks, removing fragmentation and fitting more concurrent sequences.
• Paged attention stores KV in fixed-size blocks mapped by a block table. Contiguous waste is (T_max − T_actual) × KV per token; paged waste is at most one block per sequence. Shared blocks enable prefix caching.
• Prefix caching reuses the KV of identical prompt prefixes (saves about P / (P + U) of prefill). Put static content first; a timestamp at the top kills the cache.
• Speculative decoding: expected tokens per target pass = (1 − α^k+1) / (1 − α); speed-up ≈ that / (1 + k · c). Helps most at low batch sizes when draft acceptance is high.
• INT8 / FP8 weights are about 2× smaller than BF16 and usually near-lossless. INT4 is about 4× smaller and faster for decode, but quality loss is larger on reasoning and maths; always re-evaluate and keep sensitive layers higher.
• MoE models: size memory by total parameters and compute by active parameters.
• Little's law: concurrency = arrival rate × time in system. Use it to check KV capacity.
• End-to-end latency ≈ pre-processing + TTFT + (N_out − 1) × TPOT. The first token is already in TTFT; output length still dominates long answers.
• Stream responses, parallelise independent steps, shrink prompts and outputs, and design to p95 rather than the average.
• Cost per request = input tokens × input price + output tokens × output price. Output tokens usually cost several times more.
• Conversation history is re-sent every turn, so summarise or window it.
• Cheapest cost levers first: caching and prompt trimming. Then routing, batch APIs, distillation, self-hosting.
• Self-hosting pays off at high, steady utilisation, or when compliance requires it.
• Rate-limit by tokens, not just requests, with per-tenant guaranteed slices plus a shared burst pool.
• Retry with exponential backoff and jitter. Never retry write tools without idempotency keys.
• Fallback prompts must be evaluated per model, and failover targets must respect data residency.
• The ingestion pipeline must handle incremental updates, fast deletes, permission sync and embedding version switches.
• The feedback flywheel: production failures become evaluation cases, then fixes, then CI gates, then canary releases.
• Version prompts, models, retrieval configuration, tool schemas, guardrail policies and evaluation sets.
• Calibrate LLM-as-judge against human labels, and watch for length, position and self-preference bias.
• Gate releases on per-slice metrics, because averages hide regressions in a language or a customer tier.
• Never rely on the model for security. Enforce permissions, limits and approvals in code.
• RAG access control means filtering inside the vector query before retrieval, never instructing the model to refuse.
• Indirect prompt injection arrives through documents, web pages and emails. Treat that content as data and apply least privilege.
• Semantic cache keys must be scoped by tenant and permission, or they leak data.
• Hallucination control: ground, allow abstention, cite, constrain, verify, and route to a human.
• "Zero hallucination" is achieved by making unverified output non-actionable, not by claiming the model never errs.
• In regulated decisions the LLM produces evidence and suggestions, while deterministic rules or humans decide.
• Industry cases are mostly extraction plus analytics plus workflow. Use cheap specialised models for the bulk and LLMs for the hard remainder.
• Agents should suggest, and policy gates plus human approval should execute consequential actions.

Glossary

A/B test

A controlled online experiment that randomly splits users between variants to measure the effect on business and guardrail metrics.

Abstention

The model declining to answer when evidence is insufficient. It is designed in to reduce hallucination.

Agent

An LLM-driven loop that chooses and calls tools, observes the results and decides the next step towards a goal.

API gateway

The entry point that handles authentication, tenant resolution, rate limits and quotas before requests reach the application.

Aspect-based sentiment analysis (ABSA)

Sentiment measured per aspect of a product or service (delivery, quality, price) rather than for the text as a whole.

Assertion detection

Clinical NLP task that marks whether a mention is negated, uncertain, historical or about someone other than the patient.

Audit log

A tamper-evident record of who requested what, which data was used, which model answered and which actions were taken.

Backtest

Running a candidate rule or strategy against historical data to estimate its precision, recall and impact before deployment.

Barge-in

A user interrupting a voice assistant while it speaks. The system must stop speech output and listen.

Batch API

An asynchronous, discounted way to submit many model requests that complete within hours.

Block table

The mapping from a sequence's logical KV-token ranges to physical KV blocks in paged attention, analogous to a page table.

Canary release

Rolling a change out to a small share of traffic first and monitoring it before a full rollout.

Cascade

A routing pattern that tries a cheap model first and escalates to a stronger one when confidence checks fail.

Chunking

Splitting documents into retrievable pieces, ideally along structural boundaries, with sizes tuned for retrieval quality.

Circuit breaker

A mechanism that stops calling a failing dependency for a period and routes to a fallback instead.

Code-mixing

Mixing two or more languages within one sentence or conversation, often in a non-native script.

Constrained decoding

Restricting generated tokens so the output must match a grammar, regular expression or JSON schema.

Continuous batching

A serving technique that adds and removes sequences from the running batch at every decode step to keep the GPU busy.

Cross-encoder reranker

A model that scores a query and a passage together for precise relevance, used on a short candidate list.

Data residency

A requirement that data is stored and processed only in specified geographic regions.

Decode

The generation phase of inference that produces one token per step. It is limited by memory bandwidth.

Diarization

Identifying who spoke when in an audio recording.

Distillation

Training a smaller model to imitate a larger one on a task, to cut cost and latency.

Draft acceptance rate (α)

The per-token probability that the target model accepts a speculative draft token. It sets the expected tokens per target pass.

Drift

Change over time in inputs, retrieved content or model behaviour that degrades quality.

Embedding router

A router that matches a query embedding against example-query profiles for each route using cosine similarity.

Entity resolution

Linking extracted mentions to canonical records (accounts, assets, tickers) and merging duplicates.

Evaluation set (golden set)

A curated, versioned collection of inputs with expected outputs or rubrics, used to measure and gate changes.

Fallback chain

An ordered list of alternative models or behaviours used when the primary fails.

Fill-in-the-middle (FIM)

A code model that completes text between a given prefix and suffix, used for inline completions.

Groundedness (faithfulness)

The degree to which every claim in an output is supported by the provided context.

Guardrails

Checks and constraints on inputs, processing and outputs that keep an LLM system within policy.

Hybrid search

Combining keyword (BM25) and dense vector retrieval, often fused by reciprocal rank fusion.

Idempotency key

A unique request identifier that makes repeated calls perform an action only once.

Indirect prompt injection

Malicious instructions hidden in content the model reads (documents, web pages, emails) rather than typed by the user.

Knowledge graph

A network of entities and typed relationships that supports structured queries and explainable reasoning.

KV cache

Stored attention keys and values for previous tokens, reused at each decode step. It grows with sequence length and batch size.

Least privilege

Giving each user, agent or tool only the minimum permissions its task requires.

Little's law

Average concurrency equals arrival rate multiplied by average time in the system.

LLM-as-judge

Using a strong model with a rubric to grade outputs. It must be calibrated against human labels.

LLMOps

Practices for versioning, evaluating, deploying, monitoring and improving LLM applications.

LoRA

Low-rank adaptation: fine-tuning small adapter matrices instead of all weights, so many task adapters can share one base model.

Map-reduce summarisation

Summarising chunks in parallel (map) and then merging the partial summaries (reduce).

Mixture of experts (MoE)

An architecture that activates a subset of expert sub-networks per token. Memory scales with total parameters and compute with active parameters.

Model gateway

An internal proxy that gives applications one interface to many models, with routing, failover, caching and metering.

Orchestrator

Application logic that builds prompts, calls retrieval and tools, runs agent loops and enforces budgets.

Paged attention

Managing the KV cache in fixed-size blocks, like virtual memory pages, to avoid fragmentation and share prefixes. Waste is at most one block per sequence.

Prefill

The inference phase that processes all prompt tokens in parallel. It is compute-bound and determines TTFT.

Prefix caching (prompt caching)

Reusing computed KV state for identical prompt prefixes across requests, lowering latency and cost.

Product quantization (PQ)

Compressing vectors by splitting them into sub-vectors, each encoded as a centroid index, to shrink index memory.

Pseudonymisation

Replacing identifiers with consistent placeholders, with the mapping stored securely so values can be restored.

Quantization

Representing weights, activations or KV in lower precision (FP8, INT8, INT4) to cut memory and speed up inference.

RAG

Retrieval-augmented generation: fetching relevant external content at inference time and conditioning the answer on it.

Reciprocal rank fusion (RRF)

Merging ranked lists by summing 1 ÷ (k + rank) scores for each item.

Semantic cache

A cache that returns a stored answer when a new query's embedding is close enough to a previous one.

Speculative decoding

A draft model proposes several tokens, and the target model verifies them in one pass, speeding up decoding without changing the output distribution.

Tenant isolation

Guaranteeing that one customer's data, context and cache cannot reach another customer.

Tensor parallelism

Splitting each layer's matrix operations across GPUs to fit large models and reduce per-token latency.

Time per output token (TPOT)

The average time between generated tokens during streaming. It is the inverse of tokens per second.

Time to first token (TTFT)

The time from sending a request to receiving the first generated token, covering queueing and prefill.

Token-based rate limiting

Limiting usage by tokens per minute rather than requests, reflecting the real cost and load.

vLLM

An open-source LLM serving engine built on paged attention and continuous batching, with an OpenAI-compatible API.

Interview questions