LLM APIs, Structured Outputs & Tool Calling

The big picture: what an LLM API really is

A large language model is a next-token predictor: given a sequence of tokens, it produces a probability distribution over the next token, picks one, appends it, and repeats. An LLM API wraps that loop behind an HTTP endpoint. You send JSON describing the conversation and generation settings; the provider tokenizes it, runs the model on its GPUs, decodes the output and returns JSON containing the generated message and a usage report.

Because the heavy lifting happens on someone else's hardware, an API call looks like any other web-service call from your code's point of view. That is the key mental shift for interviews: the LLM is a remote, slow, probabilistic, metered dependency. Everything else on this page follows from those four adjectives.

Four properties that drive every design decision

Where the API sits in an application

  User / upstream system
          |
          v
  +--------------------+     build prompt, pick model, set params
  |  Your application  |----------------------------------------------+
  |  (orchestration)   |                                              |
  +--------------------+                                              v
     ^      |   ^                                           +-------------------+
     |      |   | tool results                              |   LLM provider    |
     |      |   |                                           |  (hosted API) or  |
     |      v   |                                           |  local server     |
     |  +------------+   validate / execute                 |  (vLLM, Ollama)   |
     |  | Tools, DBs,|<-----------------------------------  +-------------------+
     |  | APIs, RAG  |   tool calls proposed by model                 |
     |  +------------+                                                |
     +---------------- text / JSON / tool calls + token usage --------+

The three ways to get at a model

Anatomy of a request and response

Every chat-style LLM API call has the same skeleton: a model identifier, a list of messages, optional generation parameters, and optional tools or response format. The response contains one or more choices (each with a message and a finish_reason) and a usage block with token counts.

Message roles

Under the hood, the provider flattens these structured messages into one token sequence using the model's chat template, with special tokens marking where each role starts and ends. The model was fine-tuned on exactly that template, which is why roles work: they are learned formatting conventions, not a separate channel.

messages = [system, user, assistant, user]
      |
      v  chat template (model-specific special tokens)
<|start|>system  You are a concise tutor.<|end|>
<|start|>user    What is RAG?<|end|>
<|start|>assistant RAG means retrieval-augmented...<|end|>
<|start|>user    Give an example.<|end|>
<|start|>assistant            <-- generation starts here

Your first call (OpenAI-compatible Python SDK)

import os
from openai import OpenAI

# The key is read from the environment - never hard-code it.
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
MODEL = os.environ.get("LLM_MODEL", "gpt-4o-mini")

resp = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a concise tutor. Answer in one sentence."},
        {"role": "user", "content": "Explain gradient descent."},
    ],
    temperature=0.2,
    max_tokens=100,
)

choice = resp.choices[0]
print(choice.message.content)          # the generated text
print(choice.finish_reason)            # "stop", "length", "tool_calls", "content_filter"
print(resp.usage.prompt_tokens,        # input tokens billed
      resp.usage.completion_tokens,    # output tokens billed
      resp.usage.total_tokens)

The same call against other providers

import os
from openai import OpenAI

# Any OpenAI-compatible server: only base_url, key and model change.
groq = OpenAI(base_url="https://api.groq.com/openai/v1",
              api_key=os.environ["GROQ_API_KEY"])
local = OpenAI(base_url="http://localhost:11434/v1",   # Ollama on this machine
               api_key="unused-for-local")              # local servers ignore the key

for client, model in [(groq, "llama-3.3-70b-versatile"), (local, "llama3.2")]:
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Say hi in five words."}],
    )
    print(model, "->", r.choices[0].message.content)

import os
import anthropic

# Anthropic native SDK: system is a top-level field and max_tokens is required.
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
msg = client.messages.create(
    model=os.environ.get("ANTHROPIC_MODEL", "claude-sonnet-4-20250514"),
    max_tokens=300,
    system="You are a concise tutor.",
    messages=[{"role": "user", "content": "Explain gradient descent in one sentence."}],
)
print(msg.content[0].text)
print(msg.usage.input_tokens, msg.usage.output_tokens, msg.stop_reason)

import os
from openai import AzureOpenAI

# Azure OpenAI: same SDK, but "model" is your DEPLOYMENT name and api_version is required.
az = AzureOpenAI(
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_version=os.environ.get("AZURE_OPENAI_API_VERSION", "2024-10-21"),
)
r = az.chat.completions.create(
    model=os.environ["AZURE_OPENAI_DEPLOYMENT"],
    messages=[{"role": "user", "content": "Hello"}],
)
print(r.choices[0].message.content)

Raw HTTP: what the SDK hides

import os, requests

resp = requests.post(
    "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
             "Content-Type": "application/json"},
    json={"model": "gpt-4o-mini",
          "messages": [{"role": "user", "content": "Ping"}],
          "max_tokens": 5},
    timeout=30,
)
resp.raise_for_status()
data = resp.json()
print(data["choices"][0]["message"]["content"], data["usage"])

Conversation memory: the API is stateless

The model remembers nothing between calls. To hold a conversation you append each user turn and each assistant reply to a list and resend the whole list every time. That means every turn re-bills all previous turns as input tokens, and the conversation eventually hits the context window. (Some newer "stateful" endpoints can store history server-side and reference it by an ID, but they still process those tokens; the cost model is the same.)

history = [{"role": "system", "content": "You are a helpful assistant."}]

def chat(user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    r = client.chat.completions.create(model=MODEL, messages=history)
    reply = r.choices[0].message.content
    history.append({"role": "assistant", "content": reply})   # keep memory
    return reply

print(chat("My name is Asha."))
print(chat("What is my name?"))   # works only because history was resent

Reading finish_reason

Generation parameters: the control panel

At each step the model outputs a logit (an unnormalized score) for every token in its vocabulary. Softmax turns logits into probabilities, and the decoding strategy picks one token. Generation parameters reshape that choice or decide when to stop.

The parameters you must know

Top-k vs top-p on one example

Next-token probabilities after "The pet I adopted was a":
dog 0.28 | cat 0.22 | bird 0.15 | fish 0.10 | horse 0.08 | rabbit 0.06 | turtle 0.04 | ...

top_k = 4   -> keep {dog, cat, bird, fish}, renormalize, sample
top_p = 0.85 -> cumulative: 0.28, 0.50, 0.65, 0.75, 0.83, 0.89 -> keep first 6
               (dynamic: a confident model keeps few tokens, an unsure one keeps many)

Decoding strategies in one table

Presets that work

PRESETS = {
    "extraction":    dict(temperature=0,   max_tokens=500),
    "classification":dict(temperature=0,   max_tokens=5, logprobs=True, top_logprobs=3),
    "code":          dict(temperature=0.2, max_tokens=1500),
    "chat":          dict(temperature=0.7, top_p=0.95, max_tokens=800),
    "creative":      dict(temperature=1.0, top_p=0.95, frequency_penalty=0.3, max_tokens=1200),
}

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Write a tagline for a coffee shop."}],
    **PRESETS["creative"],
)

Using logprobs as a confidence score

import math

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "system", "content": "Reply with exactly one word: Positive, Negative or Neutral."},
              {"role": "user", "content": "The delivery was late but the food was great."}],
    temperature=0, max_tokens=2, logprobs=True, top_logprobs=3,
)
first = r.choices[0].logprobs.content[0]
print("label:", first.token, "p =", round(math.exp(first.logprob), 3))
for alt in first.top_logprobs:
    print("  alt", alt.token, round(math.exp(alt.logprob), 3))
# Route low-confidence items (p < 0.7, say) to a stronger model or a human.

Tokens and pricing math

Models do not see characters or words; they see tokens, sub-word pieces produced by a tokenizer (usually byte-pair encoding or SentencePiece). Common words are one token; rare words, code, numbers and many non-English scripts split into several. APIs bill, rate-limit and cap context in tokens.

Rules of thumb (English, modern tokenizers)

• 1 token ≈ 4 characters ≈ 0.75 words; 1 word ≈ 1.3 tokens; 1,000 words ≈ 1,300-1,400 tokens.
• Code, JSON, URLs, numbers and whitespace-heavy text are more token-dense than prose.
• Many non-Latin scripts cost 1.5-4x more tokens per word than English on the same tokenizer.
• Each model family has its own tokenizer, so the same text gives different counts on different providers.
• Chat formatting adds a few overhead tokens per message; tool schemas and images also count as input tokens.

Counting tokens locally

import tiktoken   # tokenizer library for OpenAI-family models

enc = tiktoken.get_encoding("o200k_base")    # encoding used by recent OpenAI models
text = "Tokenization splits text into sub-word units."
ids = enc.encode(text)
print(len(ids), ids[:8])
print([enc.decode([i]) for i in ids])        # see the pieces, note leading spaces

# For open models, use the model's own tokenizer:
# from transformers import AutoTokenizer
# tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
# print(len(tok.apply_chat_template(messages, tokenize=True)))

How pricing works

• Prices are quoted per million tokens (sometimes per thousand), separately for input and output.
• Output tokens typically cost 3-5x input tokens because each one requires a full sequential forward pass, while input tokens are processed in parallel.
• Cached input tokens (prompt caching) are discounted, often 50-90%.
• Batch APIs typically give around 50% off in exchange for asynchronous completion within hours.
• Reasoning tokens produced by thinking models are billed as output tokens even when you never see them.
• Images, audio and tool definitions are converted to input tokens; audio output may be priced separately.

Worked example: a support chatbot

Assume illustrative prices of $2.50 per million input tokens, $10.00 per million output tokens, and cached input at $1.25 per million. (Real prices change often; always check the provider's price page.)

1. Per request System prompt 1,200 tokens, retrieved context 2,000 tokens, chat history 600 tokens, user message 200 tokens: 4,000 input tokens. Answer: 300 output tokens.
2. Uncached cost 4,000 × $2.50/1M = $0.0100 input; 300 × $10/1M = $0.0030 output; total $0.0130 per request.
3. Monthly volume 50,000 conversations/day × 4 turns = 200,000 requests/day ≈ 6 million requests/month → 6M × $0.013 = $78,000/month.
4. Add prompt caching If the 1,200-token system prompt is cached, it costs $1.25/1M instead of $2.50/1M: saving 1,200 × $1.25/1M = $0.0015 per request, or about $9,000/month (a 11.5% cut).
5. Trim context Retrieving 3 chunks instead of 6 (1,000 instead of 2,000 tokens) saves another $0.0025 per request, about $15,000/month.
6. Right-size the model Routing 70% of easy questions to a model priced 10x lower cuts the remaining bill by about 63%. Model choice is usually the biggest lever.

def request_cost(inp, out, cached=0, reasoning=0,
                 p_in=2.50, p_out=10.00, p_cached=1.25):
    """Prices in USD per 1M tokens."""
    return ((inp - cached) * p_in + cached * p_cached + (out + reasoning) * p_out) / 1_000_000

per_req = request_cost(4000, 300)
print(f"per request: ${per_req:.4f}")                  # $0.0130
print(f"per month:   ${per_req * 6_000_000:,.0f}")     # $78,000
print(f"cached:      ${request_cost(4000, 300, cached=1200) * 6_000_000:,.0f}")

# Track real usage from every response instead of estimating:
# u = resp.usage; cost = request_cost(u.prompt_tokens, u.completion_tokens,
#     cached=getattr(u.prompt_tokens_details, "cached_tokens", 0))

Levers to reduce cost (in rough order of impact)

1. Pick the smallest model that passes your eval and route by difficulty (cheap model first, escalate on low confidence).
2. Cap and shape output with max_tokens, "answer in N bullets", and structured output (JSON with only needed fields).
3. Shrink input: fewer, better retrieved chunks; summarize history; remove boilerplate; compress few-shot examples.
4. Cache: stable prefix first to hit provider prompt caching; exact or semantic response cache for repeated questions.
5. Batch offline workloads through a batch API for the discount.
6. Control reasoning effort on thinking models (low/medium/high) and avoid them for simple tasks.

Context-window management

The context window is the maximum number of tokens the model can attend to in one call, input plus output combined. Windows range from a few thousand tokens on small local models to hundreds of thousands or more on frontier models. Exceeding it causes an error (or silent truncation on some servers), and even within the limit, very long prompts cost more, run slower and can reduce accuracy because models attend unevenly across long inputs (the "lost in the middle" effect).

Budgeting the window

|<--------------------------- context window (e.g. 128k tokens) --------------------------->|
| system + tools | few-shot | retrieved docs | summary of old turns | recent turns | user | reserved for output |
|    fixed       |  fixed   |   variable     |      compact         |   sliding    |      |    max_tokens       |

Always reserve room for the answer: available_input = context_limit - max_tokens - safety_margin.

Strategies

Code: token-aware sliding window with summary

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
CONTEXT_LIMIT, MAX_OUT, MARGIN = 16_000, 1_000, 500

def n_tokens(msgs):
    # ~4 tokens of chat-format overhead per message is a safe approximation
    return sum(len(enc.encode(m["content"] or "")) + 4 for m in msgs)

def summarize(turns):
    text = "\n".join(f'{m["role"]}: {m["content"]}' for m in turns)
    r = client.chat.completions.create(
        model="gpt-4o-mini", temperature=0, max_tokens=300,
        messages=[{"role": "system", "content":
                   "Summarize this conversation. Keep names, numbers, decisions and open questions."},
                  {"role": "user", "content": text}])
    return r.choices[0].message.content

def fit_history(system, turns):
    budget = CONTEXT_LIMIT - MAX_OUT - MARGIN
    if n_tokens([system] + turns) <= budget:
        return [system] + turns
    # Summarize the oldest half, keep recent turns verbatim.
    half = len(turns) // 2
    summary = {"role": "system", "content": "Earlier conversation summary: " + summarize(turns[:half])}
    kept = turns[half:]
    while kept and n_tokens([system, summary] + kept) > budget:
        kept = kept[1:]                     # final safety: drop oldest recent turns
    return [system, summary] + kept

Code: map-reduce summarization of a long document

def chunks(text, size=3000, overlap=200):
    ids = enc.encode(text)
    step = size - overlap
    for i in range(0, len(ids), step):
        yield enc.decode(ids[i:i + size])

def summarize_long(doc: str) -> str:
    partials = []
    for c in chunks(doc):                                   # MAP
        r = client.chat.completions.create(model="gpt-4o-mini", temperature=0, max_tokens=250,
            messages=[{"role": "user", "content": f"Summarize the key points:\n\n{c}"}])
        partials.append(r.choices[0].message.content)
    joined = "\n\n".join(partials)                          # REDUCE
    r = client.chat.completions.create(model="gpt-4o-mini", temperature=0, max_tokens=500,
        messages=[{"role": "user", "content": f"Merge these partial summaries into one coherent summary:\n\n{joined}"}])
    return r.choices[0].message.content

Streaming responses

By default the API returns only when generation is complete. With stream=True, the server sends partial results as they are produced using server-sent events (SSE): a long-lived HTTP response with Content-Type: text/event-stream where each event is a line starting with data: carrying a JSON "delta". The stream ends with a sentinel such as data: [DONE] or a final "message stop" event.

Latency vocabulary

• TTFT (time to first token): network + queueing + prefill (processing the whole prompt). Grows with prompt length.
• TPOT / ITL (time per output token / inter-token latency): decode speed, e.g. 20-150 tokens/second depending on model and hardware.
• Total latency ≈ TTFT + output_tokens × TPOT. Halving output length roughly halves generation time.

What the wire looks like

HTTP/1.1 200 OK
Content-Type: text/event-stream

data: {"choices":[{"delta":{"role":"assistant","content":""}}]}

data: {"choices":[{"delta":{"content":"Gradient"}}]}

data: {"choices":[{"delta":{"content":" descent"}}]}

data: {"choices":[{"delta":{},"finish_reason":"stop"}],"usage":{...}}

data: [DONE]

Code: streaming with the Python SDK

stream = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Explain SSE in three sentences."}],
    stream=True,
    stream_options={"include_usage": True},   # final chunk carries token usage
)

parts, usage = [], None
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        piece = chunk.choices[0].delta.content
        parts.append(piece)
        print(piece, end="", flush=True)      # render incrementally
    if chunk.usage:                           # only on the last chunk
        usage = chunk.usage
full_text = "".join(parts)
print("\n", usage)

Code: relaying a stream to a browser (FastAPI)

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI
import os, json

app = FastAPI()
aclient = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])

@app.post("/chat")
async def chat(body: dict):
    async def events():
        stream = await aclient.chat.completions.create(
            model=MODEL, messages=body["messages"], stream=True)
        async for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                yield f"data: {json.dumps({'text': chunk.choices[0].delta.content})}\n\n"
        yield "data: [DONE]\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")

Streaming gotchas

• Structured output: partial JSON is not parseable until complete. Either buffer, or use a partial-JSON parser to render fields progressively.
• Tool calls arrive as fragments: the function name first, then argument strings in pieces keyed by index. Accumulate per index before parsing.
• Moderation: if you must filter output, you either buffer sentences before display or accept that users may briefly see content you later retract.
• Proxies and load balancers may buffer responses; disable buffering and raise idle timeouts for SSE routes.
• Client disconnects: cancel the upstream request when the user navigates away, otherwise you still pay for the full generation.
• Errors mid-stream arrive after a 200 status; handle error events and incomplete streams, not just HTTP codes.

Concurrency, async and batch processing

An LLM call spends almost all its time waiting on the network and the provider's GPUs. Doing 1,000 calls one after another wastes that waiting time. Three tools fix this: async concurrency (many requests in flight from one process), batching inside a prompt (several items per request), and provider batch APIs (upload a file of requests, collect results hours later at a discount).

Code: bounded async concurrency with retries

import asyncio, os
from openai import AsyncOpenAI

aclient = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"], max_retries=3, timeout=60)
SEM = asyncio.Semaphore(16)          # stay under your requests-per-minute limit

async def classify(text: str) -> str:
    async with SEM:
        r = await aclient.chat.completions.create(
            model="gpt-4o-mini", temperature=0, max_tokens=3,
            messages=[{"role": "system", "content": "Answer Positive, Negative or Neutral."},
                      {"role": "user", "content": text}])
        return r.choices[0].message.content.strip()

async def main(texts):
    results = await asyncio.gather(*(classify(t) for t in texts), return_exceptions=True)
    for t, res in zip(texts, results):
        print("ERROR" if isinstance(res, Exception) else res, "|", t[:40])

asyncio.run(main(["Loved it", "Terrible service", "It was fine"] * 10))

Note return_exceptions=True: one failed item should not cancel the other 999.

Batching several items into one prompt

from pydantic import BaseModel
from typing import Literal

class Item(BaseModel):
    id: int
    sentiment: Literal["positive", "negative", "neutral"]

class Batch(BaseModel):
    results: list[Item]

texts = ["Loved it", "Terrible service", "It was fine", "Never again"]
numbered = "\n".join(f"{i}. {t}" for i, t in enumerate(texts))
r = client.beta.chat.completions.parse(          # helper path: check current docs
    model="gpt-4o-mini", temperature=0, response_format=Batch,
    messages=[{"role": "system", "content": "Classify every numbered text. Return one result per id."},
              {"role": "user", "content": numbered}])
batch = r.choices[0].message.parsed
assert sorted(x.id for x in batch.results) == list(range(len(texts)))   # verify nothing was skipped

Packing items amortizes the system prompt and instructions across many items, but larger batches increase the risk of skipped or shuffled items and a single failure affects all of them. Keep batches modest (10-50 short items) and always verify IDs.

Provider batch APIs

1. Prepare a JSONL file where each line is one full request with a unique custom_id.
2. Upload the file and create a batch job with a completion window (commonly 24 hours).
3. Poll the job status or receive a webhook.
4. Download the results file and join on custom_id (results are not guaranteed to be in order).
5. Retry the failed lines only.

import json

with open("requests.jsonl", "w", encoding="utf-8") as f:
    for i, text in enumerate(texts):
        f.write(json.dumps({
            "custom_id": f"row-{i}",
            "method": "POST", "url": "/v1/chat/completions",
            "body": {"model": "gpt-4o-mini", "temperature": 0, "max_tokens": 3,
                     "messages": [{"role": "user", "content": f"Sentiment of: {text}"}]},
        }) + "\n")

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
job = client.batches.create(input_file_id=batch_file.id,
                            endpoint="/v1/chat/completions", completion_window="24h")
print(job.id, job.status)     # later: client.batches.retrieve(job.id); download output_file_id

Rate limits, retries and error handling

Providers protect shared GPU capacity with rate limits, typically several at once: requests per minute (RPM), tokens per minute (TPM, counting input plus the max_tokens you request), requests or tokens per day, and sometimes concurrent-request limits. Exceeding any returns HTTP 429 Too Many Requests, often with a Retry-After header and x-ratelimit-remaining-* headers telling you how much budget is left.

Error taxonomy: retry or not?

Exponential backoff with jitter

import random, time
import openai

RETRYABLE = (openai.RateLimitError, openai.APITimeoutError,
             openai.APIConnectionError, openai.InternalServerError)

def call_with_retries(fn, *, max_attempts=6, base=1.0, cap=40.0, **kwargs):
    for attempt in range(max_attempts):
        try:
            return fn(**kwargs)
        except RETRYABLE as e:
            if attempt == max_attempts - 1:
                raise
            retry_after = None
            resp = getattr(e, "response", None)
            if resp is not None and resp.headers.get("retry-after"):
                retry_after = float(resp.headers["retry-after"])
            delay = max(retry_after or 0, random.uniform(0, min(cap, base * 2 ** attempt)))
            time.sleep(delay)
        except openai.BadRequestError:
            raise                                  # never retry a malformed request

resp = call_with_retries(client.chat.completions.create, model=MODEL,
                         messages=[{"role": "user", "content": "hi"}], max_tokens=20)

Most official SDKs already retry connection errors, 408, 409, 429 and 5xx a couple of times with backoff (configurable via max_retries and timeout). Libraries such as tenacity give declarative retry policies. Do not stack three layers of retries (SDK + your wrapper + a job queue) or one failure multiplies into dozens of calls.

Idempotency

A retry after a timeout may duplicate work: the first request may have succeeded server-side even though you never saw the response. For pure text generation that only wastes money, but for side effects (sending an email, charging a card, writing a record via a tool) it can do real harm. Make side-effecting operations idempotent: generate an idempotency key per logical operation, pass it to downstream APIs that support it, and record completed keys so a replay becomes a no-op. Some LLM and batch endpoints also accept client-provided request IDs; where they do not, deduplicate on your side.

Client-side rate limiting and resilience patterns

• Token bucket on your side, sized to your TPM/RPM, so you smooth traffic before the provider rejects it.
• Priority queues: interactive traffic first, background jobs fill spare capacity.
• Timeouts on every call (connect and read), sized to expected output length.
• Circuit breaker: after repeated failures, stop calling for a cool-down period and serve a fallback.
• Fallback chain: primary model, then same model in another region/deployment, then a different provider or smaller model, then a graceful canned response.
• Request larger quotas or provisioned throughput for predictable high volume.
• Lower max_tokens: many providers count requested max tokens against TPM up front.

class Router:
    """Try deployments in order; skip ones whose circuit is open."""
    def __init__(self, targets):              # [(client, model), ...]
        self.targets = targets
        self.open_until = {}

    def complete(self, messages, **kw):
        last_err = None
        for client, model in self.targets:
            if time.time() < self.open_until.get(model, 0):
                continue
            try:
                return client.chat.completions.create(model=model, messages=messages, timeout=30, **kw)
            except RETRYABLE as e:
                self.open_until[model] = time.time() + 30   # open circuit for 30 s
                last_err = e
        raise RuntimeError("all LLM targets failed") from last_err

Prompt caching and response caching

There are two very different kinds of caching around LLMs. Prompt (prefix) caching happens inside the provider: if a new request starts with exactly the same token prefix as a recent one, the model reuses the already-computed attention key/value state for that prefix, so you pay less for those input tokens and get a faster time to first token. Response caching happens in your system: if you have already answered this (or a very similar) request, return the stored answer without calling the model at all.

Prompt caching: how to get hits

• Put stable content first (system prompt, tool definitions, few-shot examples, large reference documents), and variable content (user message, timestamps, IDs) last.
• Keep the prefix byte-identical: even a changed date string or reordered tools at the top invalidates everything after it.
• Caches have a minimum size (commonly around 1,024 tokens) and expire after minutes of inactivity; high-traffic prefixes stay warm.
• Some providers cache automatically (discount shows up as cached_tokens in usage); others require explicit cache-control markers on content blocks and may charge a small premium to write the cache.
• Caching discounts only input tokens; output is billed normally.

# Anthropic-style explicit cache breakpoint on a large, stable system block
msg = anthropic_client.messages.create(
    model=os.environ.get("ANTHROPIC_MODEL", "claude-sonnet-4-20250514"),
    max_tokens=500,
    system=[
        {"type": "text", "text": "You answer questions about the policy manual below."},
        {"type": "text", "text": POLICY_MANUAL_TEXT,           # tens of thousands of tokens
         "cache_control": {"type": "ephemeral"}},             # marker name/TTL: check current docs
    ],
    messages=[{"role": "user", "content": user_question}],
)
print(msg.usage)   # shows cache creation vs cache read input tokens

Response caching: exact and semantic

import hashlib, json, time

CACHE = {}          # use Redis or similar in production
TTL = 3600

def cache_key(model, messages, **params):
    blob = json.dumps({"m": model, "msgs": messages, "p": params}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()

def cached_completion(model, messages, **params):
    if params.get("temperature", 1) > 0.3:           # only cache near-deterministic calls
        return client.chat.completions.create(model=model, messages=messages, **params)
    k = cache_key(model, messages, **params)
    hit = CACHE.get(k)
    if hit and time.time() - hit[0] < TTL:
        return hit[1]
    r = client.chat.completions.create(model=model, messages=messages, **params)
    CACHE[k] = (time.time(), r)
    return r

Structured outputs: getting reliable JSON

Free text is great for humans and terrible for software. The moment an LLM's output feeds a database, an API, a UI component or another program, you need output that parses (valid JSON syntax) and conforms (right fields, types, enums, ranges). Structured outputs are the set of techniques that get you there, from polite prompting to mathematically guaranteed constrained decoding.

Why "just ask for JSON" breaks

• The model wraps JSON in markdown fences: ```json ... ```, so json.loads fails at character 0.
• It adds prose: "Sure! Here is the JSON you asked for: {...}".
• Syntax errors: trailing commas, single quotes, unescaped newlines, comments.
• Truncation at max_tokens leaves an unclosed object.
• Schema drift: "age": "forty", missing required keys, renamed keys (userName vs user_name), an enum value you never allowed.
• Behaviour is intermittent: fine in greedy tests, broken 2% of the time in production at higher temperature or on unusual inputs.

The reliability ladder

Whatever level you use, validate after generation. Constrained decoding guarantees shape, not truth: a perfectly typed {"invoice_total": 1250.0} can still be the wrong number.

Pydantic: the schema and the validator in one

Pydantic models are Python classes with type hints. They validate data at construction, coerce safe conversions (the string "10" becomes the integer 10), raise precise ValidationErrors for unsafe ones, and generate JSON Schema via model_json_schema(), which is exactly what APIs need for tools and strict mode. A plain @dataclass does none of this: it happily stores age="10" as a string and your code crashes later on age + 1.

from pydantic import BaseModel, Field, ValidationError, field_validator
from typing import Literal, Optional
from enum import Enum
import datetime

class Sentiment(str, Enum):
    positive = "positive"
    neutral = "neutral"
    negative = "negative"

class Aspect(BaseModel):
    aspect: str = Field(description="Product aspect, e.g. 'battery life'")
    sentiment: Sentiment
    quote: Optional[str] = Field(None, description="Supporting quote from the review, verbatim")

class ReviewAnalysis(BaseModel):
    product: str = Field(description="Product being reviewed")
    overall: Sentiment
    rating: int = Field(ge=1, le=5, description="Predicted star rating 1-5")
    aspects: list[Aspect]
    would_recommend: Optional[bool] = None
    review_date: Optional[datetime.date] = Field(None, description="ISO date if stated")

    @field_validator("product")
    @classmethod
    def not_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("product must not be empty")
        return v.strip()

# Coercion vs rejection
print(ReviewAnalysis.model_validate({"product": "X1", "overall": "positive", "rating": "4", "aspects": []}).rating)  # 4 (int)
try:
    ReviewAnalysis.model_validate({"product": "X1", "overall": "great", "rating": 7, "aspects": []})
except ValidationError as e:
    print(e.errors()[0]["loc"], e.errors()[0]["msg"])   # ('overall',) Input should be 'positive', ...

print(ReviewAnalysis.model_json_schema()["required"])    # schema generated for the API

Pydantic's default ("lax") mode converts "10" to 10 but rejects "13.4" for an int field, because converting it would silently lose information, and rejects "ten" because it is not numeric at all. Use strict=True on a model or field when even safe coercion is unacceptable.

Level 3: JSON mode

import json

r = client.chat.completions.create(
    model=MODEL, temperature=0,
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content":
         "Extract the review as JSON with keys: product (string), overall "
         "(positive|neutral|negative), rating (integer 1-5), aspects (list)."},   # the word JSON must appear
        {"role": "user", "content": "The Acme blender is powerful and quiet. Five stars."},
    ],
)
data = json.loads(r.choices[0].message.content)     # syntactically valid...
review = ReviewAnalysis.model_validate(data)        # ...but still validate the schema

JSON mode guarantees parseable JSON but not your keys or types. Many providers require the word "JSON" to appear in the messages when JSON mode is on, and output can still be cut off by max_tokens.

Level 5: strict JSON Schema structured outputs

# Option A: SDK helper that converts a Pydantic model to a strict JSON Schema and parses the result
# Path name moves between SDK versions (beta.parse vs chat.completions.parse): check current docs.
r = client.beta.chat.completions.parse(
    model=MODEL, temperature=0,
    response_format=ReviewAnalysis,
    messages=[{"role": "system", "content": "Analyze the product review."},
              {"role": "user", "content": review_text}],
)
msg = r.choices[0].message
if msg.refusal:                               # safety refusals are reported separately
    print("refused:", msg.refusal)
else:
    analysis: ReviewAnalysis = msg.parsed     # typed object, already validated

# Option B: raw JSON Schema
schema = {
    "type": "object",
    "properties": {
        "city": {"type": "string"},
        "unit": {"type": "string", "enum": ["C", "F"]},
        "days": {"type": "integer"},
    },
    "required": ["city", "unit", "days"],     # strict mode: every property listed as required
    "additionalProperties": False,            # strict mode: no extra keys
}
r = client.chat.completions.create(
    model=MODEL,
    response_format={"type": "json_schema",
                     "json_schema": {"name": "forecast_request", "strict": True, "schema": schema}},
    messages=[{"role": "user", "content": "Weather for Pune for the next 3 days in Celsius"}],
)

Strict modes typically support a subset of JSON Schema: all fields required (model optional values as a union with null), additionalProperties: false, limited support for keywords such as pattern, format, minimum, and limits on nesting depth and total properties. The first request with a new schema may be slower while the provider compiles it into a grammar; it is cached afterwards.

Level 4: forcing a tool call as a structured-output mechanism

from typing import List

class Package(BaseModel):
    name: str
    author: str

class Packages(BaseModel):
    packages: List[Package]

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Who created pydantic and FastAPI?"}],
    tools=[{"type": "function", "function": {
        "name": "record_packages",
        "description": "Record Python packages and their primary authors.",
        "parameters": Packages.model_json_schema(),      # no hand-written schema
    }}],
    tool_choice={"type": "function", "function": {"name": "record_packages"}},   # force it
)
args = r.choices[0].message.tool_calls[0].function.arguments   # a JSON string
result = Packages.model_validate_json(args)

Here the "function" is never executed; it exists only so the model fills in its arguments. This was the standard trick before native strict schemas existed and is still useful on providers that support tools but not json_schema response formats (the Anthropic pattern for structured output is exactly this: define a tool, force it, read its input).

Level 2: the validate-and-repair loop (works with any model)

import json, re
from pydantic import ValidationError

FENCE = re.compile(r"```(?:json)?\s*(.*?)```", re.S)

def extract_json(text: str) -> str:
    m = FENCE.search(text)
    if m:
        return m.group(1).strip()
    start, end = text.find("{"), text.rfind("}")   # fall back to outermost braces
    return text[start:end + 1] if start != -1 and end > start else text

def structured_call(model_cls, messages, max_attempts=3, **kw):
    msgs = list(messages)
    schema = json.dumps(model_cls.model_json_schema())
    msgs[0] = {"role": "system",
               "content": msgs[0]["content"] + f"\nReturn ONLY JSON matching this schema:\n{schema}"}
    for attempt in range(max_attempts):
        r = client.chat.completions.create(model=MODEL, messages=msgs, temperature=0, **kw)
        raw = r.choices[0].message.content
        try:
            return model_cls.model_validate_json(extract_json(raw))
        except (ValidationError, ValueError) as e:
            msgs += [{"role": "assistant", "content": raw},
                     {"role": "user", "content":
                      f"Your JSON was invalid:\n{e}\nReturn corrected JSON only, no explanation."}]
    raise RuntimeError(f"no valid {model_cls.__name__} after {max_attempts} attempts")

Feeding the exact validation error back is powerful because Pydantic errors name the field, the problem and the bad input. Cap the attempts, log every failure (they are your eval data), and consider a deterministic repair step (for example a JSON-repair library) before spending another model call.

Instructor and similar libraries

Libraries such as Instructor wrap a provider client so you pass response_model=YourPydanticClass and get back a validated object; under the hood they generate the schema, choose a mode (tools, JSON mode or strict schema), parse, validate, and automatically retry with the validation error. They work across many providers, including local OpenAI-compatible servers.

import instructor
from openai import OpenAI

iclient = instructor.from_openai(OpenAI(api_key=os.environ["OPENAI_API_KEY"]))

class SearchQuery(BaseModel):
    rewritten_query: str = Field(description="Expanded, keyword-rich version of the user query")
    start_date: datetime.date
    end_date: datetime.date
    domains_allow_list: list[str] = Field(description="Trusted domains to restrict search to")

q = iclient.chat.completions.create(
    model=MODEL,
    response_model=SearchQuery,
    max_retries=2,                       # re-asks with the ValidationError text on failure
    messages=[{"role": "system", "content": f"You rewrite search queries. Today is {datetime.date.today()}."},
              {"role": "user", "content": "recent advancements in AI safety"}],
)
print(q.model_dump(mode="json"))         # ready to send to a search API

Pydantic is the core validation layer and is provider-independent (it is also what frameworks like FastAPI use for request validation). Instructor is a convenience layer on top. Now that major APIs support native strict schemas, Instructor is optional, but it remains handy for multi-provider code, retries and streaming partial objects.

Schema design tips (they are prompts too)

• Field names and descriptions are read by the model. rewritten_query with a description beats q.
• Use enums / Literal for closed sets (labels, categories, units) so invalid values are impossible under strict mode.
• Nest related fields (DateRange{start, end}) instead of flat pairs; the relationship becomes explicit.
• Make uncertainty expressible: Optional fields or a "unknown" enum value. If every field is required and non-null, the model is forced to invent values when the input lacks them.
• Order fields for reasoning: since generation is left-to-right, put a short reasoning or evidence field before the answer field if you want the model to think first; put it after if you only want an explanation.
• Keep schemas small and shallow; very large schemas cost input tokens on every call and hurt accuracy.
• Version your schemas and keep parsing code tolerant of added optional fields.

Constrained decoding: how guarantees are enforced

Constrained (or structured, grammar-guided) decoding changes which tokens are allowed at each generation step. Before sampling, the decoder sets the logits of every token that would make the output invalid to −∞. Since e^−∞ = 0, those tokens get zero probability and can never be chosen. The model still picks among the allowed tokens using its normal preferences, so you keep quality and diversity while making invalid output impossible.

Step by step

1. Compile the constraint (regex, JSON Schema, context-free grammar) into an automaton: a finite-state machine for regular languages, a pushdown automaton for nested structures like JSON.
2. Track state: after each emitted token, advance the automaton (for example "inside a string value for key age, expecting a digit").
3. Compute the mask: the set of vocabulary tokens whose characters keep the automaton in a valid state. Efficient engines precompute token masks per state.
4. Mask logits: add −∞ to disallowed tokens, then apply temperature/top-p and sample as usual.
5. Control termination: allow the end-of-sequence token only when the automaton is in an accepting state (all required fields present, brackets closed).

Schema: {"name": string, "age": integer}

generated so far        allowed next tokens (simplified)
""                      {
{                       "name"   (required key must come first in this grammar)
{"name":                space, "
{"name": "Ada           any string chars, "
{"name": "Ada",         "age"
{"name": "Ada", "age":  digits, space      <-- letters masked to -inf
{"name": "Ada", "age": 36   digits, }
{"name": "Ada", "age": 36}  EOS only

A LogitsProcessor by hand (Hugging Face Transformers)

With local models you have direct access to logits, so you can write constraints yourself. A LogitsProcessor is called at every step with the tokens so far and the score tensor, and returns modified scores.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, LogitsProcessor, LogitsProcessorList

name = "HuggingFaceTB/SmolLM2-360M-Instruct"     # small enough for a laptop CPU
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

class AllowList(LogitsProcessor):
    def __init__(self, allowed_ids):
        self.allowed = torch.tensor(sorted(set(allowed_ids)))
    def __call__(self, input_ids, scores):
        mask = torch.full_like(scores, float("-inf"))   # hard mask: probability exactly 0
        mask[:, self.allowed] = 0
        return scores + mask

digits = [tok.encode(c, add_special_tokens=False)[0] for c in "0123456789 "]
digits.append(tok.eos_token_id)                        # always allow stopping

inputs = tok("The powers of 2 are:", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20, do_sample=False,
                     logits_processor=LogitsProcessorList([AllowList(digits)]))
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# e.g. " 1 2 4 8 16 32 64 128" -- no words, no commas

Regex constraints with partial matching

To enforce a pattern such as a URL or date, you must allow a token if the text so far plus that token could still become a match. Python's built-in re cannot answer that; the third-party regex package supports partial=True.

import regex

pattern = regex.compile(r" ?\d{4}-\d{2}-\d{2}$")    # leading optional space: tokens often start with one

class RegexMask(LogitsProcessor):
    def __init__(self, pattern, tok, prompt_len):
        self.p, self.tok, self.start = pattern, tok, prompt_len
        self.vocab = {i: tok.decode([i]) for i in range(len(tok))}   # precompute once
    def __call__(self, input_ids, scores):
        so_far = self.tok.decode(input_ids[0, self.start:])
        allowed = [self.tok.eos_token_id] if self.p.fullmatch(so_far) else []
        for i, s in self.vocab.items():
            if self.p.match(so_far + s, partial=True):
                allowed.append(i)
        mask = torch.full_like(scores, float("-inf"))
        mask[:, allowed] = 0
        return scores + mask

prompt = "The moon landing happened on"
inputs = tok(prompt, return_tensors="pt")
rm = RegexMask(pattern, tok, inputs["input_ids"].shape[1])
out = model.generate(**inputs, max_new_tokens=8, logits_processor=[rm])

This naive version scans the whole vocabulary (tens of thousands of tokens) with a regex at every step, which is slow. Production engines (Outlines, XGrammar, llguidance, llama.cpp grammars) precompile the pattern into an automaton and precompute, for each automaton state, which tokens are valid, making masking nearly free.

Why hand-written token masks fall short

What you can use where

import requests
# Ollama structured output with a JSON Schema generated from Pydantic
r = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Extract: Ada Lovelace, born 1815, mathematician"}],
    "format": {"type": "object",   # Ollama format / schema field: check current docs
               "properties": {"name": {"type": "string"}, "born": {"type": "integer"},
                              "occupation": {"type": "string"}},
               "required": ["name", "born", "occupation"]},
    "stream": False,
}, timeout=120)
print(r.json()["message"]["content"])      # JSON string matching the schema

# vLLM OpenAI-compatible server: guided decoding through extra_body
vllm = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
r = vllm.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Is this review positive or negative? 'Great value.'"}],
    extra_body={"guided_choice": ["positive", "negative"]},   # field name varies: check current docs
)
# Newer vLLM versions also accept response_format json_schema; confirm the server version's flag.

Side effects of constraints

• Quality can drop if the grammar forces an unnatural format or forbids the model from expressing "I don't know". Include escape hatches.
• Reasoning suppression: forcing an immediate answer field removes the model's chance to think; add a reasoning field first or let a reasoning model think before structured output.
• Token boundary effects: forcing characters that split tokens unnaturally can slightly distort probabilities; good engines minimize this.
• Latency: grammar compilation on first use; per-step masking is cheap in modern engines.

Function and tool calling

Tool calling (function calling) lets a model request that your code run a function. You describe available tools with names, descriptions and JSON Schemas for their parameters. When the model decides a tool is needed, instead of answering in prose it returns a structured tool call: the tool name and a JSON arguments string. Your application validates the arguments, executes the real function, and sends the result back in a tool message. The model then either calls more tools or writes the final answer. The model never executes anything; it only proposes.

The loop

  user question
       |
       v
  [LLM call with tools] ---- finish_reason = "stop" ----> final answer
       |
       | finish_reason = "tool_calls"
       v
  assistant message with tool_calls [{id, name, arguments-JSON}, ...]
       |
       v
  YOUR CODE: parse JSON -> validate -> authorize -> execute (maybe in parallel)
       |
       v
  append assistant message + one "tool" message per call (matching tool_call_id)
       |
       +------------------ loop back to LLM call (with a max-iterations cap)

Complete, runnable example

import json, math, os
from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
MODEL = os.environ.get("LLM_MODEL", "gpt-4o-mini")

# 1. Real Python functions, with Pydantic models for their arguments
class LcmArgs(BaseModel):
    a: int = Field(description="First positive integer")
    b: int = Field(description="Second positive integer")

class WeatherArgs(BaseModel):
    city: str = Field(description="City name, e.g. 'Pune'")
    unit: str = Field("C", pattern="^(C|F)$", description="C or F")

def compute_lcm(a: int, b: int) -> dict:
    return {"lcm": abs(a * b) // math.gcd(a, b)}

def get_weather(city: str, unit: str = "C") -> dict:
    return {"city": city, "temp": 24 if unit == "C" else 75, "unit": unit, "sky": "clear"}  # stub

REGISTRY = {
    "compute_lcm": (compute_lcm, LcmArgs, "Compute the least common multiple of two integers."),
    "get_weather": (get_weather, WeatherArgs, "Get the current weather for a city."),
}

# 2. Tool schemas generated from the Pydantic models
TOOLS = [{"type": "function",
          "function": {"name": n, "description": d, "parameters": m.model_json_schema()}}
         for n, (_, m, d) in REGISTRY.items()]

def run_tool(name: str, raw_args: str) -> str:
    if name not in REGISTRY:
        return json.dumps({"error": f"unknown tool {name}"})
    fn, argmodel, _ = REGISTRY[name]
    try:
        args = argmodel.model_validate_json(raw_args)        # validate model-produced args
    except ValidationError as e:
        return json.dumps({"error": "invalid arguments", "details": e.errors()})
    try:
        return json.dumps(fn(**args.model_dump()))
    except Exception as e:                                   # report failures to the model
        return json.dumps({"error": str(e)})

# 3. The agent loop
def ask(question: str, max_steps: int = 5) -> str:
    messages = [{"role": "system", "content": "Use tools when they help. Be concise."},
                {"role": "user", "content": question}]
    for _ in range(max_steps):
        r = client.chat.completions.create(model=MODEL, messages=messages,
                                           tools=TOOLS, tool_choice="auto")
        msg = r.choices[0].message
        if not msg.tool_calls:
            return msg.content
        messages.append(msg)                                 # keep the assistant's tool request
        for call in msg.tool_calls:                          # may be several (parallel calls)
            result = run_tool(call.function.name, call.function.arguments)
            messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
    return "Stopped: too many tool steps."

print(ask("What is the LCM of 84 and 120, and what's the weather in Pune?"))

Tool choice options

Parallel tool calls

Modern models can return several tool calls in a single assistant message ("weather in Pune" and "weather in Delhi"). Execute independent calls concurrently, then return one tool message per call ID before the next model request. Missing or mismatched tool_call_ids cause 400 errors on the next call.

import asyncio

async def run_calls_concurrently(tool_calls):
    loop = asyncio.get_running_loop()
    results = await asyncio.gather(*[
        loop.run_in_executor(None, run_tool, c.function.name, c.function.arguments)
        for c in tool_calls])
    return [{"role": "tool", "tool_call_id": c.id, "content": res}
            for c, res in zip(tool_calls, results)]

Provider differences at a glance

Hosted "built-in tools" (web search, code interpreter, file search) run on the provider's side, which is the exception to "tools run on your machine". Custom function tools always run in your code. The Model Context Protocol (MCP) standardizes how applications discover and call tools exposed by separate servers, so the same tool server can plug into many clients; it is not required for basic tool calling. See Agentic AI for multi-step agents.

Designing good tools

• Descriptions are prompts. Say what the tool does, when to use it, when not to, and what each parameter means with examples and units.
• One tool, one capability, with clear verb-noun names (search_orders, get_order_status). Many overlapping tools confuse selection; accuracy tends to drop beyond a few dozen tools, so group or route them.
• Constrain parameters with enums, formats and ranges; prefer IDs over free text for entities.
• Return compact, structured results (only the fields the model needs), with clear error messages the model can act on ("order not found; ask the user for the order number").
• Make write-tools idempotent and require confirmation for irreversible actions.
• Cap loop iterations and total tool-call budget to prevent runaway agents.

Security of tools: the model is an untrusted caller

Treat every tool call as if an anonymous user on the internet typed those arguments, because through prompt injection (malicious instructions hidden in a web page, email, document or tool result) an attacker effectively can.

Multimodal inputs: images, documents and audio

Multimodal models accept more than text. Images and PDF pages are converted into visual tokens by a vision encoder; audio is either transcribed first or fed to an audio-native model. In the API, a user message's content becomes a list of parts (text part, image part, audio part) instead of a single string.

Images

import base64

def image_part(path: str, detail: str = "auto") -> dict:
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    mime = "image/png" if path.lower().endswith(".png") else "image/jpeg"
    return {"type": "image_url", "image_url": {"url": f"data:{mime};base64,{b64}", "detail": detail}}

r = client.chat.completions.create(
    model=MODEL,                                  # must be a vision-capable model
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Extract the invoice number, date and total as JSON."},
        image_part("invoice.jpg", detail="high"),
    ]}],
    response_format={"type": "json_object"},
)
print(r.choices[0].message.content)

# Anthropic-style image block
msg = anthropic_client.messages.create(
    model=os.environ.get("ANTHROPIC_MODEL", "claude-sonnet-4-20250514"), max_tokens=500,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": b64_string}},
        {"type": "text", "text": "What does this chart show?"},
    ]}])

• Cost: image tokens scale with resolution (images are tiled); a "low detail" mode uses a small fixed budget. Downscale images to what the task needs.
• Uses: document and receipt extraction, chart and screenshot understanding, UI testing, visual QA, content moderation, accessibility alt-text.
• Limits: small text, precise counting, spatial reasoning and exact coordinates remain error-prone; verify critical numbers.
• PDFs: some APIs accept PDFs directly (text plus page images); otherwise extract text with a parser and send page images only where layout matters.

Audio

# Transcribe, then summarize
with open("meeting.mp3", "rb") as f:
    transcript = client.audio.transcriptions.create(model="whisper-1", file=f)  # model id: check current docs
summary = client.chat.completions.create(
    model=MODEL, temperature=0.2,
    messages=[{"role": "user", "content": f"Summarize action items:\n\n{transcript.text}"}])

# Text to speech (model / voice / write helper names change: check current docs)
speech = client.audio.speech.create(model="tts-1", voice="alloy", input="Your order has shipped.")
speech.write_to_file("reply.mp3")

Generating images

Image generation is a separate endpoint family (text-to-image and image editing) with per-image pricing by size and quality, not the chat endpoint. Treat generated media with the same safety, licensing and provenance care as text.

Embeddings APIs

An embedding is a fixed-length vector of floats representing the meaning of a piece of text (or image). Texts with similar meaning land close together, measured by cosine similarity or dot product. Embedding endpoints are cheaper and faster than generation, and they power semantic search, RAG retrieval, clustering, deduplication, recommendation, classification with few labels, and semantic caching.

import numpy as np

docs = ["How do I reset my password?",
        "Refund policy for damaged items",
        "Steps to change account login credentials"]

def embed(texts, model="text-embedding-3-small"):
    r = client.embeddings.create(model=model, input=texts)       # batch many texts per call
    return np.array([d.embedding for d in r.data], dtype=np.float32)

D = embed(docs)
q = embed(["I forgot my login"])[0]
D_n = D / np.linalg.norm(D, axis=1, keepdims=True)
q_n = q / np.linalg.norm(q)
scores = D_n @ q_n
for i in np.argsort(-scores):
    print(round(float(scores[i]), 3), docs[i])     # password and credentials docs rank highest

# Local, free alternative
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vecs = m.encode(docs, normalize_embeddings=True)

Practical rules

• Never mix models: vectors from different embedding models (or versions) live in different spaces. Changing model means re-embedding the corpus.
• Dimensions trade quality for storage and speed; some models support shortened vectors (Matryoshka-style truncation) via a dimensions parameter.
• Batch inputs (hundreds per request) and respect per-input token limits; chunk long documents first.
• Asymmetric search: some models expect prefixes or task types ("query:" vs "passage:"); follow the model card.
• Evaluate retrieval (recall@k, MRR) on your own data rather than relying on leaderboard ranks alone.
• Cache embeddings by content hash; they are deterministic for a given model version.

Choosing a model and serving it: hosted, open-weight and local

Model selection is a multi-objective trade-off between quality, latency, cost, context length, modalities and features (tools, strict schemas, vision), privacy and compliance, and operational burden. No single model wins everywhere; mature systems route between several.

Decision factors

Model tiers and routing

A router (rules, a small classifier, or a cheap LLM call) sends each request to the cheapest tier likely to succeed; a cascade tries a cheap model first and escalates when validation fails or confidence is low.

Open-weight vs hosted proprietary

Many enterprises go hybrid: hosted APIs for low-risk or prototype workloads, self-hosted or private-cloud models for sensitive data, all behind one internal gateway.

Local and self-hosted serving options

Calling local servers

import requests
# Ollama native API (after: ollama pull llama3.2)
r = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.2",
    "messages": [{"role": "system", "content": "Be concise."},
                 {"role": "user", "content": "Three advantages of running LLMs locally?"}],
    "stream": False,
    "options": {"temperature": 0.2, "num_ctx": 8192},     # context size is configurable
}, timeout=120)
print(r.json()["message"]["content"])

# vLLM: start a server (shell):  vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 16384
from openai import OpenAI
vllm = OpenAI(base_url="http://localhost:8000/v1", api_key=os.environ.get("VLLM_API_KEY", "unused"))
r = vllm.chat.completions.create(model="Qwen/Qwen2.5-7B-Instruct",
                                 messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)

# Transformers in-process with a chat template
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

name = "HuggingFaceTB/SmolLM2-360M-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32, device_map="auto")
messages = [{"role": "system", "content": "You are concise."},
            {"role": "user", "content": "List 3 benefits of open-source LLMs."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=150, do_sample=True, temperature=0.7)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))   # decode only new tokens

Hardware sizing rules of thumb

Quantization shrinks memory and speeds up inference with a modest quality loss that grows at very low bit-widths. Local models do not learn from your prompts: inference never changes weights. Adapting them requires explicit fine-tuning, and the resulting weights stay wherever you save them.

Common use cases with code

Most business applications of LLM APIs reduce to a handful of task shapes. For each, the pattern is the same: a clear instruction, the input clearly delimited, the right output format (usually a schema), a low temperature for deterministic tasks, and validation. Prompt-writing technique itself is covered in depth in Prompt Engineering.

1. Classification (single and multi-label)

from typing import Literal
from pydantic import BaseModel, Field

class Ticket(BaseModel):
    reasoning: str = Field(description="One short sentence explaining the labels")
    tags: list[Literal["auth", "billing", "technical", "urgent"]]
    priority: Literal["low", "medium", "high"]

def classify_ticket(text: str) -> Ticket:
    r = client.beta.chat.completions.parse(   # helper path: check current docs
        model="gpt-4o-mini", temperature=0, response_format=Ticket,
        messages=[{"role": "system", "content":
                   "Tag the support ticket. Use only the allowed tags. 'urgent' means the user is blocked."},
                  {"role": "user", "content": f"<ticket>{text}</ticket>"}])
    return r.choices[0].message.parsed

print(classify_ticket("I can't log in and never got the password reset email."))

Few-shot examples (a handful of labelled inputs) sharpen borderline cases such as mixed sentiment. For very high volume with stable labels, a fine-tuned small classifier or an embedding + logistic-regression model can be far cheaper than an LLM per item.

2. Information extraction

from typing import Optional
import datetime

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    invoice_date: Optional[datetime.date] = Field(None, description="null if not present")
    currency: str = Field(description="ISO 4217 code, e.g. INR, USD")
    total: float
    line_items: list[dict] = []

def extract_invoice(text: str) -> Invoice:
    r = client.beta.chat.completions.parse(   # helper path: check current docs
        model=MODEL, temperature=0, response_format=Invoice,
        messages=[{"role": "system", "content":
                   "Extract fields exactly as written. Use null for missing values. Never guess."},
                  {"role": "user", "content": text}])
    inv = r.choices[0].message.parsed
    # Semantic check the schema cannot express:
    items_total = sum(float(i.get("amount", 0)) for i in inv.line_items)
    if inv.line_items and abs(items_total - inv.total) > 0.01:
        raise ValueError("line items do not sum to total; route to human review")
    return inv

Note: strict schema modes need concrete item types; replace list[dict] with a nested LineItem model when using strict parsing.

3. Summarization

def summarize(text: str, audience: str = "busy executive", bullets: int = 3) -> str:
    r = client.chat.completions.create(
        model="gpt-4o-mini", temperature=0.3, max_tokens=250,
        messages=[{"role": "system", "content":
                   f"Summarize for a {audience} in exactly {bullets} bullets. "
                   "Use only facts from the text. No preamble."},
                  {"role": "user", "content": f'"""\n{text}\n"""'}])
    return r.choices[0].message.content

Summaries can still introduce facts not in the source. For high-stakes use, ask for quotes or source spans per bullet and check them, or use an evaluator (see LLM Evaluation & Safety).

4. Grounded question answering

def answer_from_context(question: str, context: str) -> str:
    r = client.chat.completions.create(
        model=MODEL, temperature=0,
        messages=[{"role": "system", "content":
                   "Answer ONLY from the context. If the answer is not in the context, reply exactly: "
                   "'I don't know based on the provided information.' Quote the supporting sentence."},
                  {"role": "user", "content": f"<context>\n{context}\n</context>\n\nQuestion: {question}"}])
    return r.choices[0].message.content

policy = "Employees get $500/year for training. Requests must be filed 30 days before purchase."
print(answer_from_context("Can I claim a $200 course I bought yesterday?", policy))

5. Generation and rewriting

def product_description(name: str, features: list[str], tone: str = "friendly") -> str:
    r = client.chat.completions.create(
        model=MODEL, temperature=0.8, max_tokens=180, n=3,     # 3 candidates in one call
        messages=[{"role": "user", "content":
                   f"Write a {tone} 60-word product description for {name}. Features: {', '.join(features)}."}])
    return [c.message.content for c in r.choices]            # let a human or a ranker pick

6. Translation and localization

def translate(text: str, target: str, glossary: dict[str, str]) -> str:
    terms = "\n".join(f"- {k} => {v}" for k, v in glossary.items())
    r = client.chat.completions.create(
        model=MODEL, temperature=0,
        messages=[{"role": "system", "content":
                   f"Translate into {target}. Preserve placeholders like {{name}} and HTML tags exactly. "
                   f"Use this glossary:\n{terms}\nReturn only the translation."},
                  {"role": "user", "content": text}])
    return r.choices[0].message.content

7. Code generation and transformation

import subprocess, tempfile, pathlib

def generate_function(spec: str, tests: str) -> str:
    r = client.chat.completions.create(
        model=MODEL, temperature=0.2,
        messages=[{"role": "system", "content": "Write a single Python function. Return only code, no fences."},
                  {"role": "user", "content": spec}])
    code = r.choices[0].message.content
    # Verify by running tests in an isolated sandbox (container with no network or secrets in production)
    with tempfile.TemporaryDirectory() as d:
        p = pathlib.Path(d, "test_gen.py")
        p.write_text(code + "\n\n" + tests, encoding="utf-8")
        ok = subprocess.run(["python", "-m", "pytest", "-q", str(p)], capture_output=True, timeout=60).returncode == 0
    return code if ok else None

8. Text-to-SQL and data questions

Give the schema (tables, columns, a few sample rows), ask for a single read-only query, validate it with a SQL parser, allow-list tables, run it with a read-only role and a row limit, then let the model explain the result. For numeric analysis, have the model write code or SQL rather than doing arithmetic in its head.

What LLM APIs are not good at

• Precise numeric forecasting (for example stock prices): use statistical or ML models; an LLM can explain results or generate the analysis code.
• Exact arithmetic over large tables: delegate to code or SQL via tools.
• Facts after the training cutoff: supply them through retrieval or tools.
• Guaranteed determinism or 100% instruction compliance: they follow instructions probabilistically, so validate.

LLM frameworks and SDKs

You can build everything on this page with a provider SDK plus Pydantic. Frameworks add abstractions for common patterns: provider switching, prompt templates, retrieval pipelines, agents, memory, tracing. They speed up prototypes and can add indirection, version churn and debugging difficulty. Choose based on what you actually need.

# Minimal provider-agnostic layer you can own instead of a framework
from dataclasses import dataclass

@dataclass
class LLMTarget:
    base_url: str | None
    key_env: str
    model: str

TARGETS = {
    "fast":  LLMTarget(None, "OPENAI_API_KEY", "gpt-4o-mini"),
    "local": LLMTarget("http://localhost:11434/v1", "OLLAMA_KEY", "llama3.2"),
}

def complete(tier: str, messages, **kw):
    t = TARGETS[tier]
    c = OpenAI(base_url=t.base_url, api_key=os.environ.get(t.key_env, "unused"))
    return c.chat.completions.create(model=t.model, messages=messages, timeout=60, **kw)

Observability: logging, tracing and usage tracking

LLM features fail in quiet ways: a prompt change drops accuracy by 5%, a model update doubles output length, one tenant burns half the budget, an agent loops ten times. Without per-call telemetry you find out from the invoice or from angry users. Observability for LLMs means capturing, for every call, what went in, what came out, how long it took, how many tokens it used, what it cost, and how good it was, linked into traces across multi-step workflows.

What to capture per call

Code: a thin instrumented wrapper

import json, logging, time, uuid

log = logging.getLogger("llm")

def traced_completion(feature: str, prompt_version: str, trace_id: str | None = None, **kwargs):
    span = {"trace_id": trace_id or str(uuid.uuid4()), "span_id": str(uuid.uuid4()),
            "feature": feature, "prompt_version": prompt_version,
            "model": kwargs.get("model"), "params": {k: v for k, v in kwargs.items()
                                                     if k not in ("messages", "tools")}}
    t0 = time.perf_counter()
    try:
        r = client.chat.completions.create(**kwargs)
        u = r.usage
        cached = getattr(getattr(u, "prompt_tokens_details", None), "cached_tokens", 0) or 0
        span.update(status="ok", finish_reason=r.choices[0].finish_reason,
                    input_tokens=u.prompt_tokens, cached_tokens=cached,
                    output_tokens=u.completion_tokens,
                    cost_usd=request_cost(u.prompt_tokens, u.completion_tokens, cached=cached),
                    provider_request_id=getattr(r, "_request_id", None))
        return r
    except Exception as e:
        span.update(status="error", error_type=type(e).__name__, error=str(e)[:300])
        raise
    finally:
        span["latency_ms"] = round((time.perf_counter() - t0) * 1000)
        log.info(json.dumps(span))           # ship to your log/trace backend

Tracing multi-step workflows

trace: "answer_support_question"  (total 4.2 s, $0.011)
 ├─ span: classify_intent        model=small   120 in / 5 out    0.3 s
 ├─ span: retrieve_docs          vector search                   0.1 s
 ├─ span: generate_answer        model=mid     3.1k in / 280 out 2.9 s   finish=tool_calls
 │   └─ span: tool get_order_status                              0.4 s
 ├─ span: generate_answer (2)    model=mid     3.5k in / 190 out 0.5 s   finish=stop
 └─ span: validate_output        pydantic ok

OpenTelemetry has emerging semantic conventions for generative-AI spans (model, token counts, operation), and dedicated LLM observability tools (open-source and commercial) provide prompt/response viewers, cost dashboards, datasets built from production traces, and evaluation hooks.

Dashboards and alerts worth having

• Cost per day, per feature, per tenant; cost per successful task.
• Tokens per request (input and output) distribution; sudden jumps signal prompt bloat or verbose model updates.
• p50/p95/p99 latency and TTFT; error rate by status code; 429 rate; retry count.
• Schema validation failure rate, refusal rate, finish_reason=length rate.
• Cache hit rate (prompt cache and response cache).
• Quality: sampled automated evals, user feedback, escalation-to-human rate.
• Agent health: tool calls per task, loop-limit hits, tool error rate.

Security and privacy

An LLM integration adds three new risk surfaces: secrets (API keys that spend money and access data), data flows (prompts carrying personal or confidential data to a third party), and model-driven behaviour (prompt injection, unsafe tool use, leaking system prompts or other users' data). Each needs deliberate controls.

Key management

• Never hard-code keys in source, notebooks, front-end code or mobile apps. Read them from environment variables locally and from a secrets manager in deployed environments.
• Never ship provider keys to browsers or mobile clients; route calls through your backend, which authenticates users and enforces quotas.
• Use separate keys per environment and service, with project-level spend limits and rate limits.
• Rotate keys regularly and immediately on suspected exposure; revoke leaked keys first, investigate second.
• Add secret scanning to repositories and CI, and keep .env files in .gitignore.
• Prefer workload identity (cloud IAM roles, managed identities) over long-lived static keys where the platform supports it.

# .env (never committed; listed in .gitignore)
# OPENAI_API_KEY=your-key-here

import os
from dotenv import load_dotenv      # pip install python-dotenv (local development only)

load_dotenv()
api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
    raise RuntimeError("OPENAI_API_KEY is not set")   # fail fast; never print the key itself
client = OpenAI(api_key=api_key)

Data privacy and PII

• Know where data goes: with hosted APIs, prompts are processed (and possibly temporarily stored for abuse monitoring) on the provider's infrastructure. Review data-usage terms: business/enterprise tiers typically do not train on your data by default; retention windows vary; zero-data-retention options may be available for eligible use cases.
• Minimize: send only fields the task needs. The model rarely needs full names, account numbers or addresses to classify a ticket.
• Redact or pseudonymize PII before sending (replace with placeholders like <PERSON_1>), and re-insert afterwards if needed.
• Residency and compliance: choose regional endpoints and providers with required certifications; sign data processing agreements; for regulated data consider private-cloud or self-hosted models.
• Encryption: TLS in transit (standard), encryption at rest for stored logs, caches and vector stores.
• Access control on retrieval: RAG must filter documents by the requesting user's permissions before they reach the prompt.
• Follow organizational policy: use only approved providers and endpoints; do not paste confidential material into consumer chat tools.

import re

PATTERNS = {
    "EMAIL": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"),
    "PHONE": re.compile(r"\+?\d[\d\s-]{8,}\d"),
    "CARD":  re.compile(r"\b(?:\d[ -]?){13,16}\b"),
}

def redact(text: str):
    mapping, n = {}, 0
    for label, pat in PATTERNS.items():
        for m in set(pat.findall(text)):
            n += 1
            ph = f"<{label}_{n}>"
            mapping[ph] = m
            text = text.replace(m, ph)
    return text, mapping          # send redacted text; keep mapping server-side only

safe, mapping = redact("Contact asha@example.com or +91 98765 43210 about card 4111 1111 1111 1111")
print(safe)   # Contact <EMAIL_1> or <PHONE_2> about card <CARD_3>
# Regexes miss many cases; production systems use dedicated PII detection (NER-based) tools.

(In real Python source the placeholder is written with plain angle brackets, e.g. f"<{label}_{n}>"; they appear escaped here only because this is a web page.)

Model-level threats

Production checklist and reference architecture

Moving from a notebook demo to a production LLM feature is mostly ordinary software engineering applied to an unusual dependency. The checklist below is what reviewers and interviewers expect you to have thought through. For end-to-end system design (capacity, multi-region, RAG and agents at scale) see GenAI System Design.

Reference architecture

Client ──> Your API (authn, per-user quota, input size limits)
              │
              ▼
        LLM Service layer
        ├─ prompt registry (versioned templates, schemas, tool defs)
        ├─ router (task -> model tier), response cache
        ├─ guardrails: input checks, PII redaction, injection heuristics
        ├─ LLM gateway: keys, retries+backoff, timeouts, fallbacks, rate limiter, budgets
        │        │
        │        ├──> Provider A (primary)   ├──> Provider B / region 2 (fallback)
        │        └──> Self-hosted vLLM (sensitive data)
        ├─ tool executor: schema validation, authz, sandbox, idempotency keys
        ├─ output validation: Pydantic, business rules, moderation
        └─ telemetry: traces, tokens, cost, eval samples ──> dashboards + alerts

Checklist

1. Requirements Define the task, success metric, latency budget (p95), cost ceiling per request and per month, and data classification.
2. Model choice Shortlist, evaluate on a representative labelled set, pick the cheapest model meeting the bar; pin the model version; plan re-evaluation on upgrades.
3. Prompts as code Version prompts, schemas and tool definitions in source control; review changes; run regression evals in CI.
4. Structured outputs Use strict schemas where available, validate with Pydantic, implement bounded repair, handle refusals and truncation.
5. Parameters Explicit temperature, max_tokens on every call, stop sequences where useful, seed for tests.
6. Context management Token budgeting, history trimming/summarization, retrieval limits, reserved output space.
7. Reliability Timeouts, retries with exponential backoff and jitter (only on retryable errors), circuit breaker, fallback models/regions, graceful degradation message.
8. Throughput Bounded concurrency, client-side rate limiting, batch API for offline work, quota increases or provisioned capacity for peaks.
9. Latency Streaming for user-facing text, prompt caching via stable prefixes, smaller models for simple steps, parallelize independent calls.
10. Cost Per-request cost logging, budgets and alerts per feature/tenant, caching, routing, output limits.
11. Tools Least privilege, argument validation, authorization in code, confirmations for irreversible actions, sandboxing, iteration caps, idempotency.
12. Security Secrets manager, no client-side keys, key rotation, secret scanning, per-user quotas, output sanitization.
13. Privacy Approved providers only, data processing agreement, retention settings, PII minimization and redaction, regional processing, tenant isolation in caches, logs and indexes.
14. Safety Input and output moderation appropriate to the domain, refusal handling, human escalation paths.
15. Observability Traces with prompt version, model, tokens, cost, latency, finish_reason, validation results; dashboards; alerting.
16. Evaluation Offline eval set, online sampling with automated judges and user feedback, A/B tests for prompt or model changes.
17. Operations Runbooks for provider outages, key leaks, cost spikes; track provider deprecation dates; load test with realistic prompts.

Quick revision

• An LLM API call sends a model name, a list of role-tagged messages and parameters, and returns generated message(s), a finish_reason and token usage. Chat Completions is the common teaching shape; some providers also have a Responses-style API — check current docs for field names.
• Roles: system (or developer) sets rules, user is the input, assistant is prior model output or few-shot examples, tool carries function results.
• Roles are flattened into one token sequence by the model's chat template; they are learned conventions, not separate channels.
• APIs are stateless: conversation memory means resending history every call, so input cost grows with each turn.
• Temperature divides logits before softmax; near 0 is focused, higher is more random.
• Top-p keeps the smallest token set reaching cumulative probability p; top-k keeps a fixed count.
• Temperature 0 is not a determinism guarantee; GPU numerics, batching and model updates cause variation.
• max_tokens (or max_completion_tokens / max_output_tokens on some APIs) caps output, bounding cost and latency; finish_reason "length" means truncation. Confirm the accepted name for your model.
• Frequency penalty scales with repeat count, presence penalty is a flat penalty once a token appears.
• logprobs give per-token confidence, useful for classification thresholds and routing.
• logit_bias adjusts specific token IDs; -100 effectively bans a token.
• 1 token is about 4 English characters or 0.75 words; tokenizers differ by model and language.
• Input and output tokens are priced separately; output is usually several times more expensive.
• Reasoning tokens are billed as output even when hidden.
• Cost = uncached input x input price + cached input x cached price + output x output price.
• Biggest cost levers: model choice and routing, output length, input trimming, caching, batch APIs.
• Context window covers input plus output; reserve space for the answer.
• Context strategies: truncation, sliding window, rolling summary, retrieval, map-reduce, structured state.
• Streaming uses server-sent events; it cuts time to first token, not total generation time.
• Latency is roughly TTFT plus output tokens times per-token time; shorter outputs are faster.
• Async with a semaphore gives concurrency without blowing rate limits; batch APIs trade hours of latency for about half price.
• Rate limits come as RPM, TPM, daily quotas and concurrency; exceeding them returns 429.
• Retry 429, timeouts and 5xx with exponential backoff plus jitter, honoring Retry-After; never retry 400/401/403 or quota exhaustion.
• Make side-effecting operations idempotent so retries cannot double-charge or double-send.
• Prompt caching reuses the provider's computed prefix; put stable content first and keep it byte-identical.
• Response caching skips the model; semantic caches risk false hits and must be tenant-scoped.
• Structured-output ladder: prompt, validate-and-retry, JSON mode, forced tool call, strict JSON Schema.
• JSON mode guarantees parseable JSON, not your schema; strict schema mode guarantees both, for a supported schema subset.
• Pydantic validates, safely coerces and generates JSON Schema; dataclasses do none of this.
• Validation catches wrong shape; only evaluation and grounding catch wrong values.
• Constrained decoding masks invalid tokens to minus infinity at every step using an automaton compiled from a schema, regex or grammar.
• Hosted APIs do not expose logits for custom processors; local engines (Transformers, vLLM, llama.cpp, Ollama) support grammars or schemas.
• Tool calling: the model proposes a function name and JSON arguments; your code validates, executes and returns a tool message; loop until a final answer.
• Custom tools run in your environment, not on the provider's servers.
• tool_choice can be auto, none, required or a specific function; forcing a function is a structured-output trick.
• Parallel tool calls need one tool message per tool_call_id before the next request.
• Treat model tool calls as untrusted input: validate, authorize in code, least privilege, confirm irreversible actions, sandbox code.
• Multimodal content is a list of parts; image tokens scale with resolution.
• Embeddings map text to vectors; compare with cosine similarity; never mix embedding models in one index.
• Choose models by eval quality, latency, cost, context, features, governance and licensing; route easy tasks to small models.
• Ollama is easy local running; vLLM is high-throughput serving with PagedAttention and continuous batching; llama.cpp runs quantized models on modest hardware.
• Weight memory is roughly parameters times bytes per parameter; 4-bit quantization makes a 7B model fit in about 4-5 GB.
• Keys live in environment variables or a secrets manager, never in code or client apps; rotate on exposure.
• Minimize and redact PII, check provider retention and training terms, and log with redaction and access control.
• Observe every call: prompt version, model, tokens, cost, latency, finish_reason, validation outcome, quality signals.

Glossary

Assistant message

A message with role assistant containing earlier model output, a hand-written example reply, or tool-call requests.

Batch API

An endpoint that accepts a file of many requests, processes them asynchronously within a time window, and usually charges less.

Beam search

A deterministic decoding method that keeps the k most probable partial sequences at each step; rarely exposed by chat APIs.

Chat template

The model-specific format, with special tokens, that turns a list of role-tagged messages into one token sequence.

Circuit breaker

A resilience pattern that stops calling a failing dependency for a cool-down period and serves a fallback instead.

Constrained decoding

Restricting which tokens may be generated at each step so the output must satisfy a regex, grammar or JSON Schema.

Context window

The maximum number of tokens, input plus output, a model can process in a single call.

Continuous batching

A serving technique that adds and removes sequences from the GPU batch at every decoding step, maximizing throughput.

Cosine similarity

The cosine of the angle between two vectors; the standard closeness measure for embeddings.

Embedding

A fixed-length numeric vector representing the meaning of text or other content, used for search and similarity.

Exponential backoff

A retry policy in which the wait before each retry grows exponentially, usually with a cap.

finish_reason

The field explaining why generation stopped: stop, length, tool_calls or content_filter (names vary by provider).

Frequency penalty

A parameter that lowers a token's logit in proportion to how many times it has already appeared.

Function calling

Another name for tool calling: the model outputs a function name and JSON arguments for your code to execute.

GGUF

A file format for quantized model weights used by llama.cpp and Ollama.

Grammar (GBNF)

A formal description of allowed output syntax; llama.cpp uses the GBNF notation for grammar-constrained generation.

Greedy decoding

Always choosing the single most probable next token.

Idempotency key

A unique identifier for a logical operation so repeated requests with the same key are applied only once.

Indirect prompt injection

Malicious instructions hidden in content the model reads (web pages, emails, documents, tool results) rather than typed by the user.

Instructor

A library that returns validated Pydantic objects from LLM calls, handling schema generation, parsing and retries.

Jitter

Randomness added to retry delays so many clients do not retry at the same moment.

JSON mode

An API option that guarantees syntactically valid JSON output but not conformance to a particular schema.

JSON Schema

A standard vocabulary for describing the structure, types and constraints of JSON documents.

KV cache

Stored attention keys and values for already-processed tokens, reused to avoid recomputation; the basis of prompt caching.

Logit

The raw, unnormalized score a model assigns to each vocabulary token before softmax.

logit_bias

An API parameter that adds a fixed bias to the logits of specified token IDs to encourage or ban them.

LogitsProcessor

A Hugging Face Transformers hook that modifies the score tensor at each generation step, used for custom constraints.

logprobs

Log-probabilities of generated tokens (and optionally top alternatives) returned by the API, usable as confidence scores.

max_tokens

The cap on the number of tokens the model may generate in one response. Some APIs use max_completion_tokens or max_output_tokens instead; check current docs.

Model Context Protocol (MCP)

An open protocol for exposing tools, resources and prompts from servers to LLM applications in a standard way.

Ollama

A local model runner that downloads quantized open models and serves them through a REST API.

OpenAI-compatible API

An endpoint that accepts the Chat Completions request format, letting the same client code target different providers or local servers.

PagedAttention

vLLM's technique for storing the KV cache in fixed-size pages, reducing memory fragmentation and increasing concurrency.

Parallel tool calls

Multiple tool calls returned in a single assistant message, which can be executed concurrently.

PII

Personally identifiable information, such as names, emails, phone numbers, addresses and account numbers.

Presence penalty

A parameter that applies a flat logit penalty to any token that has already appeared, encouraging new topics.

Prompt caching

Provider-side reuse of computation for a repeated prompt prefix, reducing input cost and time to first token.

Prompt injection

An attack in which crafted input causes the model to ignore its instructions or take unintended actions.

Pydantic

A Python library that validates data against type-annotated models, coerces safe conversions and generates JSON Schema.

Quantization

Storing model weights at lower numeric precision (8-bit, 4-bit) to save memory and speed up inference.

Rate limit

A cap on requests or tokens per unit time; exceeding it returns HTTP 429.

Reasoning tokens

Hidden intermediate tokens a reasoning model generates before its answer; billed as output tokens.

Responses API

A newer provider endpoint family (names vary) alongside Chat Completions, with different field names for the same ideas: messages, tools, usage, streaming. Check current docs before coding against it.

Retry-After

An HTTP response header telling the client how long to wait before retrying.

Semantic cache

A response cache keyed by embedding similarity rather than exact text match.

Server-sent events (SSE)

A one-way HTTP streaming format where the server pushes "data:" lines; used for token streaming.

Stop sequence

A string that ends generation as soon as the model produces it.

Strict mode

A structured-output setting that enforces exact conformance to a supplied JSON Schema via constrained decoding.

Structured outputs

Techniques and API features that make model output machine-readable and schema-conformant.

System message

The highest-priority instruction message setting persona, rules and format.

Temperature

A value that divides logits before softmax, controlling randomness in sampling.

Time to first token (TTFT)

The delay between sending a request and receiving the first generated token.

Token

The sub-word unit models read and write; the unit of billing, rate limits and context length.

Token bucket

A rate-limiting algorithm in which requests consume tokens that refill at a fixed rate.

Tokenizer

The component that converts text to token IDs and back; each model family has its own.

Tool message

A message carrying the result of an executed tool, linked to the originating call by its ID.

tool_choice

A parameter controlling whether the model may, must, or must not call tools, or must call a specific one.

Top-k sampling

Sampling only from the k most probable next tokens.

Top-p (nucleus) sampling

Sampling only from the smallest set of tokens whose cumulative probability reaches p.

Usage

The response block reporting input, cached, output and reasoning token counts for billing and monitoring.

vLLM

An open-source high-throughput inference server for LLMs with an OpenAI-compatible API.

Interview questions