NLP Fundamentals

What NLP is and why language is hard

Natural Language Processing (NLP) is the field that builds systems which read, understand, transform or generate human language. It covers Natural Language Understanding (NLU: classify, extract, answer) and Natural Language Generation (NLG: summarize, translate, converse). Search engines, spam filters, autocomplete, voice assistants, machine translation and every chatbot you have used are NLP systems.

The core problem is simple to state: computers do arithmetic on numbers, but language arrives as a stream of characters. Everything in NLP is a strategy for building a numeric representation of text that keeps as much meaning as possible while staying cheap enough to compute.

Why language is hard for machines

The four eras of NLP

The canonical NLP pipeline

raw text
   │  normalize (Unicode, case, whitespace, HTML)
   ▼
clean text
   │  tokenize (words / subwords / characters)
   ▼
tokens ──▶ ids (vocabulary lookup)
   │  represent (one-hot, TF-IDF, embeddings, contextual encoder)
   ▼
vectors / matrix  [seq_len × dim]
   │  model (linear classifier, CRF, LSTM, Transformer)
   ▼
output (label, tag per token, span, generated text)
   │  post-process + evaluate (F1, BLEU, ROUGE, perplexity)
   ▼
product decision

Text preprocessing

Preprocessing converts raw strings into a consistent form so that the same meaning maps to the same tokens. How much you do depends on the model: classical models (bag-of-words, TF-IDF) benefit from aggressive cleaning; modern pretrained Transformers want the text nearly raw and come with their own tokenizer that must be used exactly as trained.

Normalization

• Unicode normalization (NFC/NFKC): make visually identical characters identical bytes, e.g. "é" as one code point vs "e" + combining accent; full-width digits to ASCII.
• Case folding: lowercase to merge "Apple"/"apple". Loses information (Apple the company vs apple the fruit), so NER and cased models often keep case.
• Noise removal: strip HTML tags (IMDB reviews contain <br />), URLs, markup; or replace them with placeholders like <URL>, <NUM>, <USER>.
• Contractions and spelling: "don't" → "do not", "gr8" → "great". Useful for noisy social text.
• Emoji and emoticon handling: often carry sentiment; map to words (":)" → "smile") instead of deleting.

Tokenization

Tokenization splits text into units (tokens) that become vocabulary entries. Three granularities exist:

Word tokenization is not just split(): punctuation ("end." vs "end"), contractions ("can't"), hyphens ("state-of-the-art"), numbers ("3.14"), and languages without spaces (Chinese, Japanese, Thai) all need rules or learned segmenters. Sentence segmentation ("Dr. Smith arrived. He sat.") is its own small problem.

Stop words

Stop words are very frequent function words ("the", "is", "of", "and") that carry little topical content. Removing them shrinks bag-of-words features and helps topic models and keyword search. But be careful: "not", "no", "never" are in many default stop lists, and deleting them destroys sentiment ("not good" becomes "good"). Modern neural models should not have stop words removed at all.

Stemming vs lemmatization

Other preprocessing steps

• Part-of-speech tags as features (needed for accurate lemmatization and for rule-based extraction).
• Negation handling for classical sentiment: prefix tokens after "not" until punctuation, e.g. "not good movie" → "not NOT_good NOT_movie".
• Deduplication and language identification before training; near-duplicate reviews inflate test scores.
• PII masking: replace emails, phone numbers, account numbers with typed placeholders before text leaves a secure boundary.

Python: a classical preprocessing pipeline

import re, unicodedata
import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
# nltk.download("stopwords"); nltk.download("wordnet"); nltk.download("punkt")

STOP = set(stopwords.words("english")) - {"not", "no", "nor", "never"}  # keep negations!
stem = PorterStemmer()
lemma = WordNetLemmatizer()

def clean(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)
    text = re.sub(r"<[^>]+>", " ", text)          # strip HTML like <br />
    text = re.sub(r"https?://\S+", " <URL> ", text)
    text = text.lower()
    text = re.sub(r"[^a-z0-9<>'\s]", " ", text)
    return re.sub(r"\s+", " ", text).strip()

def tokens(text: str, mode="lemma"):
    toks = [t for t in nltk.word_tokenize(clean(text)) if t not in STOP]
    if mode == "stem":
        return [stem.stem(t) for t in toks]
    return [lemma.lemmatize(t) for t in toks]

print(tokens("The movies were NOT bad<br /><br />Studies show... https://x.y"))
# ['movie', 'not', 'bad', 'study', 'show', 'url']

Text representation: one-hot, bag-of-words, n-grams, TF-IDF

Once text is tokenized, each document must become a fixed-length numeric vector (for classical models) or a sequence of vectors (for neural models). The simplest family of representations is sparse: vectors as long as the vocabulary, mostly zeros.

One-hot word vectors

Give each word in a vocabulary of size V an index. The word's vector has length V with a single 1 at its index. With V = 8: cat = [1,0,0,0,0,0,0,0], king = [0,0,1,0,0,0,0,0], queen = [0,0,0,1,0,0,0,0].

Problems: no notion of similarity, vectors as long as the vocabulary (100k+ dimensions, 99.99% zeros), and no representation for words unseen in training (OOV).

Bag-of-words (BoW)

A document vector is the sum (or binary OR) of its word one-hots: position j holds the count of word j (count BoW) or 1 if present (binary BoW). Order is discarded, so "dog bites man" equals "man bites dog".

n-grams

An n-gram is a run of n consecutive tokens. Adding bigrams ("not bad", "very good") and trigrams restores some local order and negation, at the cost of a much larger, sparser feature space. Character n-grams (e.g. 3–5 characters) are robust to typos and morphology and work well for language ID and noisy text.

TF-IDF

TF-IDF (term frequency × inverse document frequency) reweights counts so that words frequent in this document but rare across the corpus get high scores, and words that appear everywhere ("the") get low scores.

Worked TF-IDF example

Corpus of N = 3 documents: D1 "the cat sat on the mat", D2 "the dog sat on the log", D3 "cats and dogs". Use tf = raw count / document length and idf = ln(N / df) (natural log).

Reading the table: "cat" and "mat" appear once each but get the highest weight because they are unique to D1. "the" appears twice but is discounted because it is in two of three documents. A word present in every document gets idf = ln 1 = 0 and vanishes entirely. Notice also that "cat" and "cats" are different features unless you stem or lemmatize.

Measuring similarity between text vectors

Python: BoW, n-grams, TF-IDF and cosine search

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = ["the cat sat on the mat",
        "the dog sat on the log",
        "diffusion models generate images"]

bow = CountVectorizer(ngram_range=(1, 2))        # unigrams + bigrams
Xb = bow.fit_transform(docs)
print(Xb.shape, bow.get_feature_names_out()[:6])

tfidf = TfidfVectorizer(sublinear_tf=True, min_df=1)
X = tfidf.fit_transform(docs)                   # sparse (3 docs x vocab), rows L2-normalised
q = tfidf.transform(["cat on a mat"])
print(cosine_similarity(q, X).round(2))         # doc 0 scores highest, doc 2 scores 0

BM25: TF-IDF's search-engine cousin

BM25 is the ranking function behind most keyword search. It improves TF-IDF with term-frequency saturation (the 10th occurrence of a word adds less than the 2nd) and document-length normalization. Defaults k₁ ≈ 1.2–1.5 and b = 0.75. It remains the lexical half of hybrid retrieval in RAG systems (see RAG).

Word embeddings: word2vec, GloVe, FastText

A word embedding is a dense, learned vector (typically 100–300 dimensions for classic methods, 768–4096 inside modern Transformers) in which geometry encodes meaning: similar words point in similar directions. Compared with one-hot, a 300-dimensional embedding is roughly 170× more compact than a 50,000-dimensional one-hot vector, and every dimension is active.

The distributional hypothesis

"You shall know a word by the company it keeps." Words that occur in similar contexts tend to have similar meanings. "King" and "queen" both appear near "throne", "rules", "kingdom", "royal", so any method that predicts context will be pushed to give them similar vectors. No dictionary, thesaurus, grammar or spelling information is used; the signal is purely co-occurrence.

word2vec

word2vec trains a shallow network with one hidden (projection) layer on a fake prediction task. The learned weight matrix, not the predictions, is the product: row i of the input matrix is the embedding of word i. Multiplying a one-hot vector by this matrix simply selects that row (an embedding lookup).

Negative sampling

Instead of a V-way softmax, turn each (center, context) pair into a binary classification: real pair (label 1) vs k random "negative" words drawn from the corpus (label 0). k = 5–20 for small data, 2–5 for large data. Negatives are sampled from the unigram distribution raised to the 3/4 power, which boosts rare words slightly.

Other word2vec tricks: subsampling frequent words (randomly drop "the", "a" with probability growing with frequency), window size as a hyperparameter (small windows of 2–5 capture syntactic similarity such as "walk"~"run"; large windows of 5–10+ capture topical similarity such as "walk"~"park"), and dimension 100–300 chosen empirically as a balance of expressiveness and cost.

GloVe

GloVe (Global Vectors) builds a global word-word co-occurrence matrix X, where X_ij counts how often word j appears in the context of word i, then fits vectors so their dot product approximates the log co-occurrence. It combines the global statistics of matrix-factorization methods (like LSA) with the linear-structure benefits of word2vec.

FastText: subword embeddings

FastText represents each word as the sum of vectors for its character n-grams (typically 3–6 characters) plus the whole word. "where" with n = 3 becomes <wh, whe, her, ere, re> plus <where>, with < and > marking boundaries. Benefits:

• OOV words still get a vector from their n-grams ("unfriendliness" shares pieces with "friendly", "unhappiness").
• Morphologically rich languages (Finnish, Turkish, Hindi) and typos are handled much better.
• Pretrained 300-dimensional vectors exist for 150+ languages, which is why the IMDB case study below used them.

Embedding arithmetic

Typical cosine similarities in a good embedding space: cat–kitten ≈ 0.9, king–queen ≈ 0.85, happy–joyful ≈ 0.85, cat–algebra ≈ 0.1. Note that antonyms are often close ("hot"/"cold", "good"/"bad") because they occur in identical contexts; embeddings capture relatedness, not truth or polarity.

Static vs contextual embeddings

word2vec, GloVe and FastText are static: one vector per word type regardless of sentence. "Bank" in "withdraw cash at the bank" and "the boat drifted to the river bank" gets the identical vector, an average of all its senses. Training with CBOW does not change this; the context is used during training, but the output is still one vector per word.

Contextual embeddings compute a vector per token occurrence from the whole sentence. ELMo (2018) used a deep bidirectional LSTM language model and combined its layers; the two "bank" vectors come out clearly different (cosine around 0.5). BERT and every modern Transformer do the same with self-attention, and are now the default. The cost: an encoder forward pass per sentence instead of a table lookup.

Python: training and querying word2vec / FastText

from gensim.models import Word2Vec, FastText

sentences = [["the", "king", "rules", "the", "kingdom"],
             ["the", "queen", "rules", "the", "kingdom"],
             ["a", "man", "walks"], ["a", "woman", "walks"]]   # use millions of sentences in practice

w2v = Word2Vec(sentences, vector_size=100, window=5, min_count=1,
               sg=1,            # 1 = skip-gram, 0 = CBOW
               negative=5,      # negative samples
               sample=1e-3,     # subsample frequent words
               epochs=50)
print(w2v.wv.most_similar("king", topn=3))
print(w2v.wv.most_similar(positive=["king", "woman"], negative=["man"], topn=1))

ft = FastText(sentences, vector_size=100, window=5, min_count=1, min_n=3, max_n=6)
print(ft.wv["kingdoms"][:5])     # OOV word still gets a vector from its n-grams
# w2v.wv["kingdoms"] would raise KeyError: word2vec has no subword fallback

Using pretrained embeddings in a neural model

import torch, torch.nn as nn
import numpy as np

def build_embedding(vocab, kv, dim=300, freeze=False):
    """vocab: dict word->id; kv: gensim KeyedVectors (e.g. FastText)."""
    W = np.random.normal(0, 0.1, (len(vocab), dim)).astype("float32")
    W[0] = 0.0                                      # id 0 = <pad>
    hits = 0
    for w, i in vocab.items():
        if w in kv:
            W[i] = kv[w]; hits += 1
    print(f"coverage: {hits / len(vocab):.1%}")      # report how many words were found
    return nn.Embedding.from_pretrained(torch.tensor(W), freeze=freeze, padding_idx=0)

Sentence and document embeddings

Many tasks (semantic search, clustering, deduplication, RAG retrieval) need one vector for a whole sentence, paragraph or document so that texts with similar meaning score high on cosine similarity even when they share no words: "The cat sits on the mat" vs "A kitten rests on the rug" should be close; "Quarterly revenue grew 12%" should be far.

From simple to strong

1. Average (mean) pooling of static vectors Average the word2vec/FastText vectors of all tokens. Surprisingly decent baseline; ignores order and is dominated by frequent words.
2. TF-IDF-weighted average / SIF Weight each word vector by its idf (or by a/(a + p(w))), then remove the common principal component. Reduces the influence of "the", "is".
3. Doc2Vec (paragraph vectors) Learn a vector per document jointly with word vectors. Needs inference-time training for new documents; mostly historical now.
4. Raw BERT [CLS] or mean of BERT token outputs Contextual but not trained for similarity; raw BERT sentence vectors are often worse than averaged GloVe for semantic similarity because the space is anisotropic (all vectors crowd into a narrow cone).
5. SBERT (Sentence-BERT) A siamese/bi-encoder: the same BERT encodes each sentence, mean pooling gives the sentence vector, and the network is fine-tuned on sentence pairs (natural language inference, paraphrase) so that cosine similarity reflects meaning.
6. Modern embedding models Contrastively trained on hundreds of millions of (query, relevant passage) pairs with in-batch negatives and hard negatives: E5, BGE, GTE, Nomic, Jina, Cohere and OpenAI embedding APIs. Often support instructions/prefixes ("query: ", "passage: "), long inputs (8k+ tokens), multilingual text and Matryoshka truncation (use the first 256 of 1024 dimensions with a small quality loss).

Bi-encoder vs cross-encoder

Python: sentence embeddings and semantic search

from sentence_transformers import SentenceTransformer, CrossEncoder, util

bi = SentenceTransformer("all-MiniLM-L6-v2")          # 384-dim, fast
docs = ["The cat sits on the mat.",
        "A kitten rests on the rug.",
        "Quarterly revenue grew 12%."]
D = bi.encode(docs, normalize_embeddings=True)
q = bi.encode("where is the small cat sleeping?", normalize_embeddings=True)
scores = util.cos_sim(q, D)[0]
print(scores)                                         # kitten/cat high, revenue near 0

top = scores.argsort(descending=True)[:2]
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
print(ce.predict([("where is the small cat sleeping?", docs[i]) for i in top]))

Subword tokenization: BPE, WordPiece, Unigram, SentencePiece

Word-level vocabularies hit a wall: they either explode in size or map rare and new words to a single [UNK] token. Character-level models avoid OOV but produce very long sequences. Subword tokenization is the compromise used by essentially every modern language model: frequent words stay whole ("the", "movie"), rare words split into reusable pieces ("tokenization" → "token" + "ization").

Byte-Pair Encoding (BPE)

BPE starts from characters and repeatedly merges the most frequent adjacent pair into a new symbol, recording each merge rule in order, until the vocabulary reaches a target size (for example 32k, 50k, 100k or 200k). At inference, a new word is split into characters and the learned merges are replayed in the same order. Used by GPT-2/3/4 (byte-level BPE via tiktoken), Llama 3, RoBERTa, and many others.

Worked BPE example

Training corpus word counts: low: 5, lower: 2, newest: 6, widest: 3. Append an end-of-word marker _ so suffixes stay distinct.

Start (characters):
  l o w _        ×5
  l o w e r _    ×2
  n e w e s t _  ×6
  w i d e s t _  ×3

Count adjacent pairs (weighted by word frequency):
  e s : 6 + 3 = 9     s t : 6 + 3 = 9     t _ : 9
  l o : 5 + 2 = 7     o w : 7             w e : 2 + 6 = 8
  n e : 6   e w : 6   w _ : 5   e r : 2   r _ : 2   w i : 3  i d : 3  d e : 3

Merge 1: (e, s) → "es"    (tie with s t / t _ broken by first seen)
  n e w es t _ ×6      w i d es t _ ×3
Merge 2: (es, t) → "est"  count 9
  n e w est _ ×6       w i d est _ ×3
Merge 3: (est, _) → "est_" count 9
Merge 4: (l, o) → "lo"    count 7
Merge 5: (lo, w) → "low"  count 7
  low _ ×5   low e r _ ×2
Merge 6: (n, e) → "ne"    count 6 ... then (ne, w) → "new", (new, est_) → "newest_"

Vocabulary now: base characters + {es, est, est_, lo, low, ne, new, newest_, ...}

Encode unseen word "lowest":  l o w e s t _
  apply merges in order → low est_     → tokens: ["low", "est_"]

The unseen word "lowest" is represented perfectly by two meaningful pieces, which is the whole point. Vocabulary size is the main hyperparameter: bigger vocabularies produce shorter sequences (cheaper attention, more text per context window) but larger embedding and output matrices and more rarely-trained tokens.

Byte-level BPE

Instead of starting from Unicode characters (there are over 150,000), byte-level BPE starts from the 256 possible bytes of UTF-8. Every string, including emoji, any script and binary junk, can be encoded, so there is literally no unknown token. GPT-2 onward use this. The cost: characters from non-Latin scripts take 2–4 bytes and often end up as more tokens.

WordPiece

Used by BERT, DistilBERT, ELECTRA. Same bottom-up merging idea, but a pair is chosen by how much it increases training-data likelihood, approximately count(ab) / (count(a) × count(b)), rather than by raw count. That prefers merging pieces that are rarely seen apart. Continuation pieces are marked with ##: "playing" → ["play", "##ing"]. At inference it uses greedy longest-match-first from the left. Unknown characters become [UNK].

Unigram language model tokenization

Works top-down: start with a large candidate vocabulary, fit a unigram language model where each token has a probability, and iteratively remove the tokens whose removal hurts corpus likelihood the least, until the target size is reached. Encoding picks the most probable segmentation (Viterbi), and training can sample alternative segmentations (subword regularization), which makes models more robust to noise. Used by T5, ALBERT, XLNet, mBART, many multilingual models.

SentencePiece

SentencePiece is a library and a design choice rather than a separate algorithm: it trains BPE or Unigram directly on raw text, treating the space as a normal symbol (shown as ▁, "lower one-eighth block"). No language-specific pre-tokenizer is needed, which is essential for Chinese, Japanese and Thai, and decoding is exactly reversible. Llama 1/2, T5, mT5, XLM-R and Gemma use SentencePiece.

Which to pick in an interview: BPE is deterministic (replay merges in order) and is the GPT-family default. Unigram is probabilistic: Viterbi picks the most likely segmentation and training can sample alternatives (subword regularization), which helps noisy and multilingual text (T5, ALBERT). WordPiece is BERT's cousin of BPE that scores merges by likelihood gain rather than raw count. SentencePiece is the library that trains BPE or Unigram on raw bytes/text with space as a symbol.

Special tokens and tokenizer outputs

• BERT-style: [CLS] at the start (its output is used for classification), [SEP] between/after segments, [PAD], [MASK], [UNK].
• GPT-style: <|endoftext|>; chat models add role markers such as <|im_start|> or [INST]. A chat template must be applied exactly as in training.
• A Hugging Face tokenizer returns input_ids, attention_mask (1 = real token, 0 = padding) and, for pair tasks, token_type_ids. Use return_offsets_mapping=True to map tokens back to character spans (essential for NER and extraction).

Python: seeing tokenizers in action

import tiktoken
from transformers import AutoTokenizer

enc = tiktoken.get_encoding("cl100k_base")               # GPT-3.5/4 tokenizer
ids = enc.encode("Retrieval-augmented generation rocks!")
print(len(ids), [enc.decode([i]) for i in ids])
# 7 ['Retrieval', '-', 'augmented', ' generation', ' rocks', '!'] (illustrative)

bert = AutoTokenizer.from_pretrained("bert-base-uncased")
out = bert("Tokenization is unbelievably useful", return_offsets_mapping=True)
print(bert.convert_ids_to_tokens(out["input_ids"]))
# ['[CLS]', 'token', '##ization', 'is', 'unbelievably', 'useful', '[SEP]']
print(out["offset_mapping"])                            # (start, end) char spans per token

# Training your own BPE tokenizer
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel()
trainer = trainers.BpeTrainer(vocab_size=8000, special_tokens=["[UNK]", "[PAD]"])
tok.train(files=["corpus.txt"], trainer=trainer)

Why tokenization matters for LLM behaviour and cost

• Cost and context: APIs bill per token and context windows are measured in tokens. English averages about 1.3 tokens per word (roughly 4 characters per token); Hindi, Tamil, Arabic or Thai text can take 2–5× more tokens for the same content, so they cost more and fit less in the window.
• Arithmetic and spelling: numbers split unpredictably ("12345" might become "123" + "45"), and the model never sees individual letters of a multi-character token, which is why "count the r's in strawberry" was hard.
• Leading spaces matter: " hello" and "hello" are different tokens in GPT tokenizers; prompt formatting changes the token sequence.
• Glitch tokens: tokens present in the vocabulary but almost absent from training data can produce bizarre behaviour.
• Domain vocabulary: code, chemistry, or legal text can fragment heavily; adding domain tokens requires resizing and training the embedding matrix.

Language modeling basics: n-gram LMs and perplexity

A language model (LM) assigns a probability to a sequence of tokens, or equivalently predicts the next token given the previous ones. It is the engine behind autocomplete, speech recognition rescoring, spelling correction and, scaled up enormously, every LLM (see Transformers & LLMs).

n-gram language models

The Markov assumption truncates history to the previous n−1 tokens. Probabilities are maximum-likelihood count ratios:

Worked example. Corpus: "<s> I like cats </s>", "<s> I like dogs </s>", "<s> you like cats </s>". Then P(I | <s>) = 2/3, P(like | I) = 2/2 = 1, P(cats | like) = 2/3, P(</s> | cats) = 1. So P("I like cats") = 2/3 × 1 × 2/3 × 1 = 4/9 ≈ 0.44. But P("you like dogs") = 1/3 × 1 × 1/3 × 1 = 1/9, and any sentence containing an unseen bigram such as "cats like" gets probability exactly 0.

Smoothing: handling unseen n-grams

• Add-one (Laplace): (count + 1) / (count(context) + V). Simple, but moves too much probability to unseen events.
• Add-k: use a small k like 0.01 instead of 1.
• Backoff and interpolation: if a trigram is unseen, fall back to the bigram, then unigram; or mix them with weights λ.
• Kneser-Ney: the best classical method; the lower-order distribution uses "in how many different contexts does this word appear" rather than raw frequency ("Francisco" is frequent but almost only after "San").

Limits of n-gram LMs: storage grows explosively with n, they cannot share statistics between similar words ("cat" and "kitten" are unrelated symbols), and they cannot see beyond n−1 tokens. Neural LMs (first feed-forward, then RNN, then Transformer) solve all three by working with embeddings and learned context.

Perplexity

Perplexity (PPL) measures how surprised a model is by held-out text. It is the exponential of the average negative log-likelihood per token (the cross-entropy):

Example: if a model assigns probabilities 0.5, 0.25, 0.5, 0.25 to four test tokens, the average log-probability is (ln 0.5 + ln 0.25 + ln 0.5 + ln 0.25)/4 = −1.04, so PPL = e^1.04 ≈ 2.83: the model is as uncertain as a choice among about 2.8 options per token.

import math, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()

def perplexity(text):
    ids = tok(text, return_tensors="pt").input_ids
    with torch.no_grad():
        loss = lm(ids, labels=ids).loss        # mean cross-entropy over predicted tokens
    return math.exp(loss.item())

print(perplexity("The cat sat on the mat."))    # low (fluent)
print(perplexity("Mat the on sat cat the."))    # much higher (scrambled)

Sequence models: RNN, LSTM, GRU, seq2seq, attention and the move to Transformers

Bag-of-words discards order. Sequence models read tokens in order and maintain a running summary, so "not bad" can differ from "bad". This section is the conceptual bridge from classic NLP to Transformers; the network mechanics are covered in depth on the Deep Learning page.

Recurrent neural networks (RNN)

Backpropagation through time (BPTT) multiplies the gradient by the Jacobian ∂h_t/∂h_t−1 = diag(1 − h_t²) W_h once per step. Over hundreds of steps the product shrinks exponentially if its norm is below 1 (vanishing gradient: early words stop influencing learning) or blows up if above 1 (exploding gradient: NaN losses). If each step scales the gradient by about 0.99, after 100 steps 0.99¹⁰⁰ ≈ 0.37; with a factor of 0.9, 0.9¹⁰⁰ ≈ 0.00003.

LSTM

Tiny worked example (scalar, all weights 0.5, biases 0, h_t−1 = 0.5, x_t = 1.0, C_t−1 = 0.3): every gate input is 0.5 × 1.5 = 0.75, so f = i = o = σ(0.75) = 0.679; C̃ = tanh(0.75) = 0.635; C_t = 0.679 × 0.3 + 0.679 × 0.635 = 0.204 + 0.431 = 0.635; h_t = 0.679 × tanh(0.635) = 0.679 × 0.561 = 0.381.

A good example of the forget gate firing: in "Alice finished her talk. Bob then started his", the gender/subject information about Alice should be dropped when a new subject appears.

GRU

The Gated Recurrent Unit merges the cell and hidden state and uses two gates: an update gate z (how much of the old state to keep, like forget + input tied together) and a reset gate r (how much of the past to use when proposing the new state). About 25% fewer parameters than an LSTM, often similar accuracy, faster to train.

Bidirectional and stacked RNNs

A bidirectional RNN/LSTM runs one pass left-to-right and one right-to-left and concatenates the states, so each position sees both past and future context. Ideal for tagging (NER, POS) and classification where the full input is available; impossible for left-to-right generation. Stacking 2–3 layers adds abstraction. The combination (BiLSTM + CRF) was the state of the art for NER before BERT.

Sequence-to-sequence (encoder–decoder)

For tasks where input and output lengths differ (translation, summarization, question answering), an encoder RNN reads the source into a context vector and a decoder RNN generates the target token by token, feeding back its own previous output. During training the decoder receives the true previous token (teacher forcing); at inference it receives its own prediction, which causes exposure bias (errors compound).

The bottleneck: the entire source sentence must be squeezed into one fixed-size vector. Translation quality dropped sharply for sentences longer than about 20–30 words.

Attention

At each decoding step, the decoder state (the query) is compared with every encoder state (the keys); a softmax turns the scores into weights; the context is the weighted sum of encoder states (the values). Bahdanau (additive) attention scores with a small MLP; Luong (multiplicative) attention uses a dot product. Transformers use scaled dot-product attention:

Self-attention applies the same idea within one sequence: every token attends to every other token, so "it" in "The animal didn't cross the street because it was tired" can attend directly to "animal". Cross-attention lets decoder tokens attend to encoder tokens (the bridge between source and target in translation).

Why Transformers replaced RNNs

The trade-offs: attention is O(n²) in sequence length, and because self-attention is permutation-invariant, word order must be injected with positional encodings (sinusoidal, learned, or rotary). Three families emerged: encoder-only (BERT: understanding, classification, NER, embeddings), decoder-only (GPT, Llama: generation, chat), encoder-decoder (T5, BART: translation, summarization).

Python: an LSTM text classifier with the fixes that matter

import torch, torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence

class BiLSTMClassifier(nn.Module):
    def __init__(self, vocab_size, emb_dim=300, hidden=128, n_classes=2, emb=None):
        super().__init__()
        self.emb = emb or nn.Embedding(vocab_size, emb_dim, padding_idx=0)
        self.lstm = nn.LSTM(emb_dim, hidden, num_layers=1, batch_first=True,
                            bidirectional=True)
        self.drop = nn.Dropout(0.3)
        self.fc = nn.Linear(2 * hidden, n_classes)

    def forward(self, ids, lengths):
        x = self.drop(self.emb(ids))
        packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True,
                                      enforce_sorted=False)     # ignore padding
        _, (h, _) = self.lstm(packed)
        h = torch.cat([h[-2], h[-1]], dim=1)                     # last fwd + last bwd state
        return self.fc(self.drop(h))                             # raw logits

model = BiLSTMClassifier(vocab_size=20000)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
loss_fn = nn.CrossEntropyLoss()

def train_step(ids, lengths, y):
    model.train(); opt.zero_grad()
    loss = loss_fn(model(ids, lengths), y)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)      # essential for RNNs
    opt.step()
    return loss.item()

Classic NLP tasks

Almost every NLP problem reduces to one of a small number of output shapes. Recognizing the shape tells you the model head, the loss and the metric.

Text classification and sentiment analysis

Sentiment analysis classifies opinion polarity (positive/negative/neutral) or intensity (1–5 stars). Variants:

• Document-level: one label per review.
• Sentence-level: useful when reviews mix opinions.
• Aspect-based sentiment analysis (ABSA): extract (aspect, opinion, polarity) triples. "Battery life is great but the camera is blurry" → (battery life, great, positive), (camera, blurry, negative). Needed whenever sentiment must drive action.
• Emotion detection: anger, joy, fear, sadness; stance detection: for/against a target.

Hard cases: negation ("not bad"), contrast ("I expected to hate it, but..."), sarcasm ("Oh great, it broke on day one"), comparative sentences, domain shift ("unpredictable" is positive for a thriller plot, negative for a car's brakes). Approaches in order of power: lexicons (VADER, SentiWordNet) → TF-IDF + linear model → LSTM/CNN with pretrained embeddings → fine-tuned encoder (BERT, RoBERTa, DeBERTa) → zero/few-shot LLM prompting.

Named Entity Recognition (NER)

NER finds spans that name things and assigns a type: PERSON, ORG, LOC, DATE, MONEY, plus domain types (DRUG, SYMPTOM, ACCOUNT_ID, CLAUSE_TYPE). It is framed as token classification with a tagging scheme:

Token:   Maria   Lopez   visited  New    York   on  Monday
BIO:     B-PER   I-PER   O        B-LOC  I-LOC  O   B-DATE
BIOES:   B-PER   E-PER   O        B-LOC  E-LOC  O   S-DATE
(B = begin, I = inside, O = outside, E = end, S = single-token entity)

Models: CRF on hand features → BiLSTM-CRF → fine-tuned BERT token classifier (the default) → LLM extraction with a JSON schema for zero-shot or rare types. The CRF layer learns valid tag transitions (I-PER cannot follow B-LOC). With subword tokenizers, label only the first subword of each word (set others to -100 so the loss ignores them) and use offset mappings to recover character spans.

Encoder-only models are the natural fit for NER: each token needs full bidirectional context, the output is aligned one-to-one with the input, and inference is cheap. A decoder-only LLM can do it via prompting, but costs more and needs span grounding.

Part-of-speech (POS) tagging

Assign grammatical categories (noun, verb, adjective...; Universal Dependencies has 17 tags, Penn Treebank 45). Classic models: Hidden Markov Model with Viterbi decoding (emission P(word | tag) × transition P(tag | previous tag)), then CRFs, then BiLSTMs, now transformers at around 97–98% accuracy for English. Uses: lemmatization, rule-based extraction ("adjective + noun" phrases for aspect mining), feature engineering, linguistic analysis.

Parsing

• Constituency parsing builds a phrase-structure tree (S → NP VP) from a grammar.
• Dependency parsing links each word to its head with a labelled relation (nsubj, dobj, amod). "The camera takes blurry photos": takes → nsubj → camera, takes → dobj → photos, photos → amod → blurry. Great for extracting who-did-what-to-whom and linking opinion words to aspects. Measured by UAS (unlabelled attachment score) and LAS (labelled).
• Other structure tasks: coreference resolution (linking "she" to "Dr. Rao"), relation extraction (founded_by(Company, Person)), semantic role labeling (agent, patient, instrument).

Topic modeling (LDA)

Latent Dirichlet Allocation is a generative probabilistic model: each document has a topic mixture θ_d ~ Dirichlet(α), each topic k has a word distribution φ_k ~ Dirichlet(β), and each word is generated by picking a topic z ~ θ_d then a word w ~ φ_z. Inference by Gibbs sampling or variational Bayes. You choose the number of topics K; evaluate with topic coherence (for example C_v or NPMI) and human inspection. Alternatives: NMF on TF-IDF (fast, often cleaner topics), and BERTopic (sentence embeddings → UMAP → HDBSCAN clusters → class-based TF-IDF to name clusters), which handles short text like tweets and chat messages far better than LDA.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

cv = CountVectorizer(stop_words="english", min_df=5, max_df=0.5)
X = cv.fit_transform(complaints)                      # LDA wants raw counts, not TF-IDF
lda = LatentDirichletAllocation(n_components=8, random_state=0).fit(X)
vocab = cv.get_feature_names_out()
for k, comp in enumerate(lda.components_):
    print(k, [vocab[i] for i in comp.argsort()[-8:][::-1]])
doc_topics = lda.transform(X)                         # per-document topic mixture

Summarization

• Extractive: select the most important sentences verbatim (TextRank: a PageRank graph over sentence similarities; or a classifier per sentence). Faithful by construction, can be choppy.
• Abstractive: generate new sentences (BART, PEGASUS, T5, LLMs). Fluent and concise, but can hallucinate facts not in the source. Check faithfulness with entailment models or QA-based checks, not just ROUGE.
• Long documents: chunk and summarize hierarchically (map-reduce), or use long-context models; watch for "lost in the middle" effects.

Machine translation

Evolution: rule-based → phrase-based statistical MT (alignment models plus an n-gram LM) → RNN seq2seq with attention → Transformer encoder–decoder (the original 2017 Transformer was built for translation) → massively multilingual models (NLLB, M2M-100, mBART) and LLMs. Decoding uses beam search (keep the top-B partial translations at each step, typical B = 4–5) with length normalization. Evaluated with BLEU, chrF, COMET and human judgement.

Question answering

• Extractive QA (SQuAD-style): given question + passage, predict start and end token of the answer span with an encoder. Metrics: exact match and token-level F1.
• Open-domain QA: retrieve relevant passages first (BM25 or dense retrieval), then read. This retriever–reader pattern is the ancestor of RAG.
• Generative / closed-book QA: an LLM answers from parametric memory; fluent but hallucination-prone, hence retrieval grounding (see RAG).
• Related: natural language inference (entailment / contradiction / neutral), which powers zero-shot classification and faithfulness checking.

Other tasks worth naming

Python: modern task heads with Hugging Face pipelines

from transformers import pipeline

clf = pipeline("sentiment-analysis")                      # fine-tuned encoder
print(clf("The plot was predictable, but the acting was superb."))

ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")
print(ner("Priya joined Infosys in Bengaluru in March 2021."))
# [{'entity_group': 'PER', 'word': 'Priya', ...}, {'entity_group': 'ORG', ...}, {'entity_group': 'LOC', ...}]

zs = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
print(zs("My parcel arrived two weeks late.",
         candidate_labels=["delivery", "product quality", "billing"]))

qa = pipeline("question-answering")
print(qa(question="Where did Priya join?", context="Priya joined Infosys in Bengaluru."))

summ = pipeline("summarization", model="facebook/bart-large-cnn")
print(summ(long_article, max_length=80, min_length=30)[0]["summary_text"])

Evaluating NLP systems

Choosing the metric is a design decision, not an afterthought. Classification and extraction have a single correct answer, so precision/recall-style metrics apply. Generation has many acceptable answers, so metrics compare against references (BLEU, ROUGE), use embeddings (BERTScore), or use judges (humans, LLMs).

Classification and extraction metrics

• Accuracy is fine only for balanced classes (IMDB is 50/50, so random guessing scores 50%).
• Macro-F1 averages per-class F1 equally (protects minority classes); micro-F1 pools all decisions (dominated by frequent classes); weighted-F1 weights by support.
• ROC-AUC / PR-AUC: threshold-free ranking quality; PR-AUC is more informative when positives are rare (toxicity, fraud).
• Entity-level F1 for NER: a predicted entity counts only if both span boundaries and type match exactly (strict) or overlap (lenient).
• Exact match (EM) and token F1 for extractive QA.
• Cohen's kappa for inter-annotator agreement: if humans agree only 70% of the time, do not expect a model to reach 95%.

BLEU (translation, precision-oriented)

Worked example. Reference "the cat is on the mat" (r = 6), candidate "the cat is on mat" (c = 5).

• Unigrams: the, cat, is, on, mat all appear in the reference → p₁ = 5/5 = 1.0.
• Bigrams: "the cat", "cat is", "is on" match; "on mat" does not → p₂ = 3/4 = 0.75.
• Trigrams: "the cat is", "cat is on" match; "is on mat" does not → p₃ = 2/3. 4-grams: "the cat is on" matches, "cat is on mat" does not → p₄ = 1/2.
• BP = e^{1 − 6/5} = e^−0.2 = 0.819.
• BLEU = 0.819 × (1.0 × 0.75 × 0.667 × 0.5)^1/4 = 0.819 × 0.25^0.25 = 0.819 × 0.707 ≈ 0.58.

Why clipping matters: the candidate "the the the the" would get unclipped p₁ = 4/4; clipped, "the" counts at most twice (its count in the reference), giving 2/4. BLEU is designed for corpus-level scoring; sentence-level BLEU is noisy and needs smoothing. Use sacrebleu for reproducible, standardized tokenization.

ROUGE (summarization, recall-oriented)

• ROUGE-N = overlapping n-grams / n-grams in the reference (recall; precision and F variants are also reported).
• ROUGE-L uses the longest common subsequence (in-order, not necessarily contiguous), so it rewards correct word order without requiring exact phrase matches.
• Same example: unigram overlap 5 of the reference's 6 → ROUGE-1 recall = 0.83; LCS("the cat is on the mat", "the cat is on mat") = 5 → ROUGE-L recall = 5/6 = 0.83; ROUGE-2 recall = 3/5 = 0.6.

BLEU asks "how much of my output is correct?"; ROUGE asks "how much of the reference did I cover?", which is the right question for summaries.

METEOR, chrF and learned metrics

• METEOR: aligns candidate and reference words using exact matches, stems and synonyms (WordNet), computes a recall-weighted F-mean, then applies a fragmentation penalty for scrambled order. Correlates better with humans than BLEU at the sentence level.
• chrF: character n-gram F-score; robust for morphologically rich languages.
• BERTScore: embed every token of candidate and reference with a contextual model, greedily match each token to its most similar counterpart by cosine, and compute precision, recall and F1 over those similarities. Gives credit for "large" vs "big". Optional idf weighting and baseline rescaling.
• COMET, BLEURT: neural models trained to predict human quality ratings; the current standard for machine translation evaluation.
• LLM-as-a-judge: prompt a strong model with a rubric to score or compare outputs. Scales human-like judgement for open-ended text; check for position bias, verbosity bias and self-preference, and calibrate against human labels.
• Faithfulness metrics for summaries and RAG answers: NLI-based entailment (does the source entail each claim?), QA-based checks (QAGS, QuestEval), and citation precision.

import evaluate
from sklearn.metrics import classification_report
from seqeval.metrics import classification_report as ner_report

print(classification_report(y_true, y_pred, digits=3))           # P/R/F1 per class + macro

print(ner_report([["B-PER", "I-PER", "O"]], [["B-PER", "O", "O"]]))  # entity-level

bleu = evaluate.load("sacrebleu")
print(bleu.compute(predictions=["the cat is on mat"],
                   references=[["the cat is on the mat"]])["score"])
rouge = evaluate.load("rouge")
print(rouge.compute(predictions=["the cat is on mat"], references=["the cat is on the mat"]))
bs = evaluate.load("bertscore")
print(bs.compute(predictions=["a big dog"], references=["a large dog"], lang="en")["f1"])

End-to-end case study: IMDB sentiment with eight models

This case study compares four classical and four neural models on the same binary sentiment task. It is valuable precisely because the result is counter-intuitive: the simple models won. Every lesson here generalizes to real projects.

The dataset

Because classes are balanced, random guessing gives about 50% accuracy and an F1 near 0.5–0.67 depending on which class is guessed. Any model below roughly 0.55 F1 has learned essentially nothing.

Representation for the classical models: binary one-hot bag-of-words

Each review became a binary vector over the vocabulary (up to 50,000 words): 1 if the word appears, 0 otherwise. Frequency, rarity (no idf) and order are all ignored. "The film is not bad" and "the film is bad, not good" produce nearly identical vectors. Despite this, linear models do very well because sentiment is dominated by strong individual words.

Part 1: four classical models

Part 2: four neural models

The motivation: bag-of-words cannot distinguish "not bad" from "bad not." Neural models add dense embeddings, sequential context, bidirectional context and finally attention between all words.

1. Vanilla RNN (F1 0.568, 3.9M parameters) Reads left to right with one hidden state and classifies from the final state. Training loss barely moved, 0.7006 → 0.6987 over 10 epochs, which is essentially ln 2: the model guessed throughout. Diagnosis: vanishing gradients over 400-step sequences, no gradient clipping, and a final state computed after padding.
2. LSTM (F1 0.528, 4.6M parameters) Gates should fix vanishing gradients, and training loss did improve (0.693 → 0.569), yet test F1 was even lower than the RNN. Precision 0.694 but recall 0.426: it learned a skewed decision rule that over-predicted one class. Lesson: a falling training loss does not guarantee a good test metric; check the confusion matrix and the threshold.
3. BiLSTM + FastText embeddings (F1 0.811, 11.6M parameters) Two LSTMs (forward and backward) concatenated, initialized with 300-dimensional pretrained FastText vectors, so "brilliant" and "masterpiece" start near each other. Jumped about 24 F1 points over the RNN. But training loss crashed to 0.048 while precision was 0.764 and recall 0.865: classic overfitting that over-predicted the positive class.
4. Cross-attention classifier (F1 0.863, 3.8M parameters, best neural) Every token attends directly to every other token with softmax(QK^T/√d)V, so "not" can attend to "bad" with no sequential bottleneck and no vanishing gradient. Fewest parameters of the neural models, best neural result, still 1.2 points behind the linear models.

Review: "the film was not as bad as critics claimed"

Forward LSTM  →  the → film → was → not → as → bad ...   (sees "not" before "bad")
Backward LSTM ←  ... bad ← as ← not ← was ← film ← the   (sees "claimed", "critics" first)
Concatenate [h_fwd ; h_bwd] → classifier → POSITIVE

Attention: row for "bad" puts high weight on "not" and "critics claimed"
           → direct 1-hop connection, regardless of distance

Full results: all eight models on the test set

Note that Bernoulli NB (0.812) and BiLSTM + FastText (0.811) are essentially tied on F1 but fail in opposite directions: NB is precise but misses positives; the BiLSTM finds positives but raises many false alarms.

Key takeaways

1. Classical baselines are hard to beat Even binary bag-of-words with a linear model reached F1 0.875. Never assume neural means better; the baseline sets the bar every fancier model must clear.
2. Gradient clipping is not optional for RNNs on long text One line, clip_grad_norm_(model.parameters(), 1.0), plus shorter sequences (or packing), would likely have rescued the RNN and LSTM.
3. Training loss is not test performance The BiLSTM's training loss of 0.048 looked excellent and was a symptom of memorization. Use a validation set, early stopping, dropout and weight decay.
4. Pretrained embeddings matter enormously FastText initialization plus bidirectionality moved F1 from 0.568 to 0.811.
5. Attention removes the sequential bottleneck With the fewest parameters, the attention model beat every recurrent model.
6. Always inspect precision and recall, not just F1 Two models with the same F1 can have opposite failure modes that matter differently to the business.

How to push beyond 0.875

• TF-IDF with word 1–2 grams (and optionally character n-grams) instead of binary unigrams: typically around 0.89–0.91 on IMDB with a linear model; NB-SVM (Naive Bayes log-count-ratio features fed to an SVM) is a classic strong baseline around 0.91.
• Fine-tune a pretrained encoder (DistilBERT, RoBERTa, DeBERTa) for 2–3 epochs with learning rate around 2e-5: roughly 0.93–0.96 accuracy. Handle reviews longer than 512 tokens by head+tail truncation or chunk-and-average.
• Zero-shot or few-shot LLM prompting: strong without training data, but costlier per prediction and needs output parsing.

Python: the classical half end to end

from datasets import load_dataset
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.naive_bayes import BernoulliNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
import re

ds = load_dataset("imdb")
clean = lambda s: re.sub(r"<br\s*/?>", " ", s)
Xtr, ytr = [clean(t) for t in ds["train"]["text"]], ds["train"]["label"]
Xte, yte = [clean(t) for t in ds["test"]["text"]], ds["test"]["label"]

onehot = lambda n: CountVectorizer(binary=True, max_features=n)   # binary bag-of-words

pipe = Pipeline([("vec", onehot(20000)), ("clf", LogisticRegression(max_iter=2000))])
grid = GridSearchCV(pipe, {"vec__max_features": [20000, 50000],
                           "clf__C": [0.01, 0.1, 1.0]}, cv=5, scoring="f1", n_jobs=-1)
grid.fit(Xtr, ytr)
print(grid.best_params_, round(grid.best_score_, 4))
print(classification_report(yte, grid.predict(Xte), digits=4))

for name, clf in [("svm", LinearSVC(C=0.01)), ("bnb", BernoulliNB())]:
    p = Pipeline([("vec", onehot(50000)), ("clf", clf)]).fit(Xtr, ytr)
    print(name, classification_report(yte, p.predict(Xte), digits=4))

# Stronger baseline: TF-IDF uni+bigrams
strong = Pipeline([("vec", TfidfVectorizer(ngram_range=(1, 2), min_df=3, sublinear_tf=True)),
                   ("clf", LogisticRegression(C=4, max_iter=2000))]).fit(Xtr, ytr)
print(classification_report(yte, strong.predict(Xte), digits=4))

Python: the attention classifier (neural half, compact)

import torch, torch.nn as nn

class AttnClassifier(nn.Module):
    def __init__(self, vocab, d=128, heads=4, max_len=400, n_classes=2):
        super().__init__()
        self.tok = nn.Embedding(vocab, d, padding_idx=0)
        self.pos = nn.Embedding(max_len, d)                      # learned positions
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True, dropout=0.1)
        self.norm = nn.LayerNorm(d)
        self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        self.out = nn.Linear(d, n_classes)

    def forward(self, ids):
        pad = ids.eq(0)                                          # True where padding
        pos = torch.arange(ids.size(1), device=ids.device)
        x = self.tok(ids) + self.pos(pos)
        a, _ = self.attn(x, x, x, key_padding_mask=pad)          # every token looks at every token
        x = self.norm(x + a)
        x = self.norm(x + self.ff(x))
        x = x.masked_fill(pad.unsqueeze(-1), 0).sum(1) / (~pad).sum(1, keepdim=True)
        return self.out(x)                                       # mean-pooled logits

Multilingual and code-mixed text

Most of the world's users do not write in clean English. A production NLP system must cope with many languages, scripts, transliteration and switching between languages within a single sentence.

Challenges

• Scripts and segmentation: no spaces in Chinese, Japanese, Thai; complex scripts (Devanagari, Arabic right-to-left, Tamil) with combining characters; Unicode normalization becomes critical.
• Morphology: agglutinative languages (Turkish, Finnish, Tamil) create huge numbers of word forms; subwords or characters are essential.
• Low-resource languages: little labelled or even unlabelled data.
• Tokenizer fertility: English-heavy tokenizers split other languages into many more tokens per word, raising cost, shrinking effective context, and hurting quality. This is sometimes called the token tax.
• Code-mixing: "Delivery bahut late thi, totally disappointed" mixes Hindi and English in one sentence; "Hinglish" is often written in Roman script (transliterated), with non-standard, inconsistent spelling (bahut / bohot / bhot).
• Culture and pragmatics: politeness forms, sarcasm conventions and sentiment words differ by culture; translated labels may not carry the same intensity.

Approaches

1. Language identification first (fastText lid.176, CLD3), at the sentence or even token level for code-mixed text.
2. Normalization and transliteration Normalize spelling variants; optionally transliterate Romanized Hindi to Devanagari (or the reverse) so it matches the model's training data.
3. Multilingual pretrained encoders mBERT, XLM-RoBERTa, MuRIL / IndicBERT for Indian languages, LaBSE and multilingual E5/BGE for embeddings. They support zero-shot cross-lingual transfer: fine-tune on English labels, apply to Hindi or Spanish with reasonable accuracy.
4. Translate-train or translate-test Machine-translate training data into the target language, or translate incoming text into English and use an English model. Simple and effective, but translation loses slang and sentiment nuance.
5. Collect in-domain code-mixed labels Even a few thousand labelled Hinglish examples usually beat any zero-shot transfer. Use LLMs to pre-label and humans to verify.
6. Multilingual LLMs Prompt directly in the user's language or ask for output in a fixed schema regardless of input language (for example, always English aspect labels).

from transformers import AutoTokenizer
for name in ["bert-base-uncased", "xlm-roberta-base"]:
    t = AutoTokenizer.from_pretrained(name)
    for s in ["The delivery was very late", "Delivery bahut late thi", "डिलीवरी बहुत देर से आई"]:
        print(name, len(t.tokenize(s)), t.tokenize(s)[:8])
# The English-only tokenizer fragments Devanagari into many pieces or [UNK];
# the multilingual SentencePiece tokenizer keeps it compact.

Applying NLP inside GenAI systems

LLMs did not make classic NLP tasks disappear; they changed how the tasks are implemented. Information extraction, sentiment, NER, entity linking, classification and summarization are now frequently done by prompting an LLM with a schema, and the results feed databases, rules engines, dashboards, RAG indexes and agents. The fundamentals on this page are what let you design, evaluate and debug those systems.

Information extraction with LLMs

1. Define the schema Entity types, relation types, allowed enum values, required vs optional fields, and a null/unknown option so the model is not forced to invent values.
2. Prompt with definitions and examples Clear type definitions, 2–5 few-shot examples including hard negatives ("a city mentioned in a signature is not the incident location").
3. Constrain the output JSON mode, function/tool calling or grammar-constrained decoding so outputs always parse; validate with a schema library (for example Pydantic) and retry on failure.
4. Ground every value Ask for the exact supporting quote or character offsets for each field and verify the quote exists in the source. Reject or flag ungrounded values; this is the main defence against hallucinated entities.
5. Normalize and link Map extracted strings to canonical IDs (dates to ISO format, drug names to a medical ontology code, companies to a registry ID).
6. Evaluate like NER Field-level precision, recall and F1 against a human-labelled gold set; track per-field error rates and drift over time.
7. Hybridize for cost and reliability Regex and rules for rigid formats (IBANs, phone numbers, IP addresses), a fine-tuned small encoder for high-volume standard entities, and the LLM for long-tail, ambiguous or relational extraction.

from pydantic import BaseModel, Field
from typing import Literal, Optional
import json

class Entity(BaseModel):
    type: Literal["ACCOUNT", "DEVICE", "LOCATION", "AMOUNT", "DATE"]
    value: str
    evidence: str = Field(description="exact substring copied from the note")

class Extraction(BaseModel):
    entities: list[Entity]
    fraud_pattern: Optional[str] = None      # null when the note does not describe one

PROMPT = """Extract entities from the analyst note. Return JSON matching this schema:
{schema}
Rules: copy 'evidence' verbatim from the note; if unsure, omit the entity; never guess.
Note: {note}"""

def extract(note, llm):
    raw = llm(PROMPT.format(schema=Extraction.model_json_schema(), note=note))
    data = Extraction.model_validate(json.loads(raw))
    data.entities = [e for e in data.entities if e.evidence in note]   # grounding check
    return data

Aspect-based sentiment with LLMs

Document sentiment ("3 stars, negative") cannot drive action. Aspect-based sentiment extracts which aspect is being judged and how, so it can be routed:

"Phone is great but it came 5 days late and the box was crushed"
   │  LLM / fine-tuned ABSA model, fixed aspect taxonomy
   ▼
[{aspect: product_quality, polarity: positive, evidence: "Phone is great"},
 {aspect: delivery_delay,  polarity: negative, evidence: "came 5 days late"},
 {aspect: packaging,       polarity: negative, evidence: "box was crushed"}]
   │  aggregate by aspect, SKU, warehouse, time window
   ▼
delivery_delay spike at one warehouse → trigger logistics audit workflow

Design points: a fixed, versioned aspect taxonomy (with an "other" bucket reviewed periodically for new aspects), evidence spans, calibration against human labels, aggregation with volume thresholds before triggering actions, and multilingual/code-mixed support.

Entity linking and normalization

Entity linking maps a mention to a unique knowledge-base entry. "Paris" may be the city, a person or a hotel; "MI" may be myocardial infarction or Michigan. Pipeline: (1) candidate generation by alias dictionaries, fuzzy string matching and embedding search over entity descriptions; (2) disambiguation by comparing the mention's context with each candidate (cross-encoder or LLM choosing from a short list); (3) NIL detection when no candidate fits. Letting an LLM pick from retrieved candidates, instead of generating an ID from memory, prevents invented codes.

Where classic NLP shows up in GenAI stacks

LLM vs fine-tuned small model vs classical model

Quick revision

• NLP pipeline: normalize → tokenize → map to ids → represent as vectors → model → post-process and evaluate.
• Language is hard because of ambiguity, word order and composition, long-range dependencies, noise, world knowledge, and a long tail of rare words (Zipf's law).
• Aggressive cleaning (lowercase, stop words, stemming) helps bag-of-words models and hurts pretrained Transformers.
• Keep negations ("not", "no", "never") when removing stop words for sentiment tasks.
• Stemming chops suffixes by rule ("studies" → "studi"); lemmatization maps to a dictionary form using part of speech ("better" → "good").
• Distinct one-hot vectors always have cosine similarity 0, so one-hot encodes identity but no similarity.
• Bag-of-words ignores order: "dog bites man" equals "man bites dog"; n-grams restore some local order.
• TF-IDF = term frequency × log(N / document frequency); a term in every document gets idf 0; fit it on training data only.
• Cosine similarity compares direction and ignores length; on unit vectors it ranks identically to Euclidean distance and dot product.
• BM25 adds term-frequency saturation and length normalization to TF-IDF and is the standard lexical retriever.
• Distributional hypothesis: words in similar contexts have similar meanings; embeddings are learned purely from co-occurrence.
• CBOW predicts the center word from context (faster, frequent words); skip-gram predicts context from the center word (better for rare words).
• Negative sampling replaces the full-vocabulary softmax with k binary real-vs-random classifications, sampled from unigram^3/4.
• GloVe fits word vectors to log global co-occurrence counts; FastText sums character n-gram vectors, so it can embed unseen words.
• Embedding arithmetic works because consistent relations become roughly constant offset directions (king − man + woman ≈ queen).
• Static embeddings give one vector per word type; contextual models (ELMo, BERT) give one vector per occurrence and resolve polysemy.
• Antonyms are often close in embedding space because they share contexts.
• Raw BERT [CLS] vectors are poor for similarity; SBERT and contrastively trained embedding models fix this.
• Bi-encoders embed separately and scale to millions of documents; cross-encoders score pairs jointly and are used for re-ranking.
• BPE repeatedly merges the most frequent adjacent pair (deterministic, GPT default); WordPiece merges the pair that most increases likelihood; Unigram prunes a large vocabulary top-down and can sample segmentations.
• SentencePiece trains on raw text with the space as a symbol, so it needs no language-specific pre-tokenizer.
• Byte-level BPE starts from 256 bytes and never produces an unknown token.
• English averages about 1.3 tokens per word; many non-Latin-script languages need several times more tokens, raising cost.
• Always use the exact tokenizer, version and chat template the model was trained with.
• A language model factorizes P(sequence) into next-token probabilities by the chain rule.
• n-gram LMs use the Markov assumption and need smoothing (Laplace, backoff, Kneser-Ney) for unseen n-grams.
• Perplexity = exp(average negative log-likelihood per token); lower is better; a uniform model over V tokens has perplexity V; compare only across identical tokenizers.
• Vanilla RNNs suffer vanishing and exploding gradients through backpropagation through time; clip gradients.
• The LSTM's additive cell-state update with a forget gate near 1 lets gradients flow over long distances.
• GRU uses update and reset gates, fewer parameters than LSTM, similar accuracy.
• Bidirectional RNNs suit tagging and classification, not left-to-right generation.
• Seq2seq squeezes the source into one vector; attention lets the decoder look back at every encoder state.
• Scaled dot-product attention: softmax(QK^T / √d_k) V; the scaling prevents softmax saturation.
• Transformers replaced RNNs because of parallel training, 1-hop paths between any two tokens, and no fixed-size bottleneck; they need positional encodings.
• NER is token classification with BIO tags; evaluate with entity-level F1, not token accuracy.
• Aspect-based sentiment extracts (aspect, opinion, polarity) and is what makes sentiment actionable.
• LDA models documents as mixtures of topics and topics as distributions over words; BERTopic works better for short text.
• Extractive summarization is faithful but choppy; abstractive is fluent but can hallucinate.
• BLEU is clipped n-gram precision with a brevity penalty; ROUGE is n-gram or LCS recall.
• BERTScore matches tokens by contextual embedding similarity and rewards paraphrases; METEOR uses stems and synonyms.
• In the IMDB study, linear SVM and logistic regression (F1 0.875) beat all neural models; cross-attention was the best neural model (0.863).
• IMDB lessons: build baselines, clip gradients, do not trust training loss, use pretrained embeddings, and inspect precision and recall separately.
• Code-mixed text (for example Hinglish) needs language ID, transliteration handling, multilingual encoders and in-domain labels.
• LLM extraction needs a schema, constrained output, evidence spans verified against the source, normalization and F1 evaluation.
• Entity linking maps mentions to knowledge-base IDs via candidate generation plus contextual disambiguation, with a NIL option.

Glossary

Aspect-based sentiment analysis (ABSA)

Extracting which aspect of a product or service is discussed and the sentiment toward that aspect, rather than one label per document.

Attention

A mechanism that computes a weighted average of value vectors, with weights from query–key similarity, letting a model focus on relevant tokens.

Attention mask

A 0/1 array telling the model which positions are real tokens and which are padding (or, in causal models, which future positions are hidden).

Backpropagation through time (BPTT)

Training an RNN by unrolling it over time steps and backpropagating through every step.

Bag-of-words (BoW)

A document vector of word counts or presence flags that ignores word order.

Beam search

A decoding strategy that keeps the B most probable partial outputs at each step instead of only the single best.

BERTScore

A generation metric that matches candidate and reference tokens by cosine similarity of contextual embeddings and reports precision, recall and F1.

Bi-encoder

A model that embeds query and document independently so documents can be pre-indexed for fast similarity search.

BIO tagging

A labelling scheme for spans: B- begins an entity, I- continues it, O is outside any entity.

BLEU

A translation metric based on clipped n-gram precision (n = 1–4) combined geometrically and multiplied by a brevity penalty.

BM25

A lexical ranking function extending TF-IDF with term-frequency saturation and document-length normalization.

Byte-Pair Encoding (BPE)

A subword tokenizer that learns a vocabulary by repeatedly merging the most frequent adjacent symbol pair.

CBOW

Continuous bag of words: the word2vec variant that predicts a center word from its averaged context words.

Code-mixing

Switching between two or more languages within a sentence or conversation, often in a single script (for example Romanized Hindi with English).

Conditional random field (CRF)

A probabilistic sequence model that scores whole tag sequences, enforcing valid transitions; often placed on top of BiLSTMs for NER.

Contextual embedding

A token vector computed from the whole sentence, so the same word gets different vectors in different contexts.

Cosine similarity

The dot product of two vectors divided by the product of their lengths; measures the angle between them.

Cross-encoder

A model that reads a query and a document together and outputs a relevance score; accurate but cannot pre-compute document vectors.

Dependency parsing

Predicting a tree where each word points to its grammatical head with a labelled relation such as subject or object.

Distributional hypothesis

The idea that words occurring in similar contexts have similar meanings.

Document frequency (df)

The number of documents in a corpus that contain a given term.

Embedding

A dense, learned vector representing a token, word, sentence or document so that geometric closeness reflects similarity.

Entity linking

Mapping a text mention to a unique entry in a knowledge base or ontology.

Exposure bias

The mismatch between training with true previous tokens (teacher forcing) and inference with the model's own, possibly wrong, predictions.

F1 score

The harmonic mean of precision and recall.

FastText

A word-embedding method that represents words as sums of character n-gram vectors, enabling vectors for unseen words.

GloVe

Global Vectors: embeddings learned by fitting dot products to the logarithm of global word co-occurrence counts.

GRU

Gated recurrent unit: a recurrent cell with update and reset gates, lighter than an LSTM.

Hallucination

Generated content that is fluent but not supported by the source or by facts.

Inverse document frequency (idf)

log(N / df): a weight that is high for rare terms and zero for terms in every document.

Language model

A model that assigns probabilities to token sequences, usually by predicting the next token.

LDA

Latent Dirichlet Allocation: a generative topic model treating documents as mixtures of topics and topics as distributions over words.

Lemmatization

Reducing a word to its dictionary form using vocabulary and part of speech.

LSTM

Long short-term memory: a recurrent cell with forget, input and output gates and an additive cell state that preserves long-range information.

METEOR

A generation metric that aligns words by exact match, stem and synonym, weights recall over precision and penalizes fragmented order.

n-gram

A sequence of n consecutive tokens or characters.

Named entity recognition (NER)

Finding spans that name people, organizations, locations, dates and domain-specific entities, and labelling their types.

Negative sampling

Training embeddings by distinguishing real context words from a few randomly sampled words instead of computing a full softmax.

Normalization

Making text consistent: Unicode forms, casing, whitespace, markup removal, spelling variants.

One-hot vector

A vector as long as the vocabulary with a single 1 at the word's index.

Out-of-vocabulary (OOV)

A word not present in the model's vocabulary.

Perplexity

The exponential of the average per-token negative log-likelihood; the effective number of choices a language model is uncertain between.

Polysemy

One word having several related or unrelated meanings, such as "bank".

POS tagging

Assigning a part-of-speech category (noun, verb, adjective, and so on) to each token.

Positional encoding

Information about token position added to embeddings so an order-blind attention model can use word order.

Precision

The fraction of predicted positives (or extracted items) that are correct.

Recall

The fraction of true positives (or gold items) that the system found.

RNN

Recurrent neural network: processes a sequence one step at a time, updating a hidden state with shared weights.

ROUGE

A family of summarization metrics measuring n-gram (ROUGE-N) or longest-common-subsequence (ROUGE-L) overlap, emphasizing recall.

Self-attention

Attention where queries, keys and values all come from the same sequence, so every token can use every other token as context.

Sentence-BERT (SBERT)

A BERT model fine-tuned in a siamese setup so that pooled sentence vectors are comparable with cosine similarity.

SentencePiece

A tokenizer library that learns BPE or Unigram vocabularies directly from raw text, treating whitespace as an ordinary symbol.

Sentiment analysis

Classifying the opinion or emotional polarity expressed in text.

Seq2seq

An encoder–decoder architecture that maps an input sequence to an output sequence of possibly different length.

Skip-gram

The word2vec variant that predicts surrounding context words from the center word.

Smoothing

Techniques that reserve probability mass for unseen n-grams in count-based language models.

Stemming

Stripping affixes with heuristic rules to reach a crude root form that may not be a real word.

Stop words

Very frequent function words often removed in classical pipelines.

Subword tokenization

Splitting text into units between characters and words so frequent words stay whole and rare words decompose into known pieces.

Teacher forcing

Feeding the true previous output token to a decoder during training rather than its own prediction.

TF-IDF

Term frequency multiplied by inverse document frequency; weights words that are frequent in a document but rare in the corpus.

Token

The basic unit a model processes: a word, subword, character or byte.

Tokenizer fertility

The average number of tokens a tokenizer produces per word; higher fertility means higher cost and shorter effective context.

Topic coherence

A score of how semantically related the top words of a topic are, used to compare topic models.

Transliteration

Writing a language in another script, such as Hindi in Latin letters.

Unigram tokenization

A subword method that starts with a large vocabulary and prunes tokens whose removal least reduces corpus likelihood.

Vanishing gradient

Gradients shrinking exponentially as they propagate back through many steps or layers, preventing early inputs from being learned.

word2vec

A family of shallow neural methods (CBOW and skip-gram) that learn static word embeddings from local context windows.

WordPiece

BERT's subword tokenizer, which merges pairs that most increase training likelihood and marks continuations with ##.

Zero-shot cross-lingual transfer

Fine-tuning a multilingual model on one language and applying it to others without target-language labels.

Zipf's law

In natural language, the r-th most frequent word occurs about 1/r as often as the most frequent word, so most of the vocabulary is rare.

Interview questions