Python & Data Tools for ML

The big picture: why Python runs ML

Python itself is a slow, interpreted, dynamically typed language. It became the language of machine learning anyway because it is easy to read and because its scientific libraries push the expensive work into compiled code. When you write X @ W in NumPy, one Python instruction triggers millions of multiply-adds inside an optimized linear-algebra library (BLAS). When you call model(x) in PyTorch on a GPU, Python only launches kernels; the GPU does the arithmetic.

The layer cake

+------------------------------------------------------------------+
|  Your notebook / script / service  (Python: glue, control flow)  |
+------------------------------------------------------------------+
|  scikit-learn   PyTorch   Hugging Face   LLM SDKs                |  modelling
+------------------------------------------------------------------+
|  pandas (tables)   matplotlib / seaborn (plots)   SciPy          |  data + stats
+------------------------------------------------------------------+
|  NumPy ndarray  (typed, contiguous memory, vectorized ufuncs)    |  array core
+------------------------------------------------------------------+
|  C / C++ / Fortran / CUDA : BLAS, LAPACK, cuBLAS, cuDNN, Arrow   |  speed lives here
+------------------------------------------------------------------+

The standard ML workflow in Python

1. Load Read CSV, Parquet, SQL or JSON into a pandas DataFrame.
2. Explore (EDA) Check shape, dtypes, missing values, distributions, class balance, correlations and leakage risks.
3. Split Hold out a test set first (stratified for classification) so it stays untouched.
4. Preprocess Impute, encode categoricals, scale numerics, engineer features, all inside a pipeline fitted on training data only.
5. Train and tune Fit models, compare with cross-validation, tune hyperparameters with a search.
6. Evaluate Use metrics that match the business cost (recall for missed cancers, precision for spam, RMSE for prices), inspect errors, tune the decision threshold on validation data.
7. Ship Save the whole pipeline, version data and code, monitor in production.

Environments, packaging and Jupyter

Every project should live in its own isolated environment so that one project's numpy 1.26 cannot break another project's numpy 2.x. An environment is simply a folder containing a Python interpreter plus the packages installed for it.

venv + pip (built in)

# create and activate an isolated environment
python -m venv .venv
.venv\Scripts\activate            # Windows PowerShell / cmd
source .venv/bin/activate         # macOS / Linux

python -m pip install --upgrade pip
pip install numpy pandas scikit-learn matplotlib seaborn torch
pip freeze > requirements.txt     # pin exact versions
pip install -r requirements.txt   # recreate elsewhere
deactivate

The options compared

Packaging your own code

Modern projects describe themselves in a pyproject.toml file (name, version, dependencies, build backend). A folder with an __init__.py is a package; any .py file is a module. Installing your project in editable mode (pip install -e .) lets notebooks import your code while you keep editing it.

# pyproject.toml (minimal)
[project]
name = "churn_model"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["pandas>=2.0", "scikit-learn>=1.4"]

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

The guard if __name__ == "__main__": makes a file behave as both an importable module and a runnable script: the block runs only when the file is executed directly, not when it is imported. It is also required on Windows for multiprocessing and PyTorch DataLoader workers, because child processes re-import the main file.

Jupyter essentials

A notebook talks to a kernel, a live Python process that keeps variables in memory between cells. That is convenient for exploration and dangerous for reproducibility, because the order you ran cells in is invisible to a reader.

%pip install seaborn        # installs into the kernel's own environment
%timeit np.sort(arr)        # micro-benchmark a line
%%time                      # time a whole cell (must be the first line)
%load_ext autoreload
%autoreload 2               # re-import edited .py modules automatically
%matplotlib inline          # render plots below cells
df.head()                   # the last expression in a cell is displayed

Core Python: types, mutability and containers

In Python, every value is an object with a type, and variables are just names bound to objects. Assignment never copies data; it attaches another name to the same object. Whether that shared object can be changed in place (mutability) explains a large share of Python bugs and interview questions.

Built-in types at a glance

Names, identity and equality

a = [1, 2]
b = a              # same object, two names
b.append(3)
print(a)           # [1, 2, 3]  -- a changed too
print(a is b)      # True   (same identity)
print(a == [1, 2, 3], a is [1, 2, 3])   # True False  (equal value, different object)

x = 10
y = x
y += 1             # ints are immutable: y now names a NEW object
print(x, y)        # 10 11

== compares values (calls __eq__); is compares identity (same object in memory). Use is only for singletons: if x is None. CPython caches small integers (-5 to 256) and some strings, so is may accidentally appear to work on them; never rely on it.

Shallow versus deep copies

import copy
grid = [[0, 0], [0, 0]]
shallow = grid.copy()          # new outer list, SAME inner lists
deep = copy.deepcopy(grid)     # everything duplicated
grid[0][0] = 9
print(shallow[0][0], deep[0][0])   # 9 0

bad = [[0] * 2] * 2            # two references to ONE inner list
bad[0][0] = 1
print(bad)                     # [[1, 0], [1, 0]]
good = [[0] * 2 for _ in range(2)]

Container cost cheat sheet

Dict keys and set members must be hashable: immutable and with a stable __hash__. That is why a tuple can be a dict key but a list cannot. See Data Structures & Algorithms for how hash tables achieve O(1) average lookups.

Strings and f-strings

name, lr, acc = "resnet", 3e-4, 0.91234
print(f"{name:>8} lr={lr:.1e} acc={acc:.2%}")   #   resnet lr=3.0e-04 acc=91.23%
print(f"{acc=:.3f}")                            # acc=0.912   (self-documenting, 3.8+)
words = "the cat sat".split()                   # ['the', 'cat', 'sat']
print(" ".join(words).upper())                  # THE CAT SAT
# strings are immutable: building with += in a loop is O(n^2); collect then join

Truthiness and handy built-ins

Empty containers, 0, 0.0, "" and None are falsy; everything else is truthy. But note that NumPy arrays and pandas Series refuse to be used in an if directly (they raise "truth value is ambiguous"); use .any(), .all() or .empty.

from collections import Counter, defaultdict
labels = ["cat", "dog", "cat", "bird", "cat"]
print(Counter(labels).most_common(2))     # [('cat', 3), ('dog', 1)]
by_len = defaultdict(list)
for w in labels:
    by_len[len(w)].append(w)
print(dict(by_len))                       # {3: ['cat', 'dog', 'cat', 'cat'], 4: ['bird']}
for i, (w, n) in enumerate(zip(["a", "b"], [1, 2])):
    print(i, w, n)                        # 0 a 1 / 1 b 2
print(sorted(labels, key=len, reverse=True)[:2])   # ['bird', 'cat']

Floating point

print(0.1 + 0.2 == 0.3)          # False: binary floats cannot represent 0.1 exactly
import math
print(math.isclose(0.1 + 0.2, 0.3))   # True -- compare with a tolerance (np.isclose for arrays)
print(float("nan") == float("nan"))    # False: NaN is not equal to anything, even itself

Functions, comprehensions, closures and decorators

Functions in Python are ordinary objects: you can pass them as arguments, return them from other functions, and store them in dicts. That single fact enables callbacks (learning-rate schedulers), closures (factories that remember settings) and decorators (wrappers that add timing, caching or retries).

Arguments: positional, keyword, defaults, *args and **kwargs

def build_mlp(in_dim, *hidden, activation="relu", dropout=0.0, **extra):
    print(in_dim, hidden, activation, dropout, extra)

build_mlp(30, 64, 32, dropout=0.1, bias=False)
# 30 (64, 32) relu 0.1 {'bias': False}

def scale(x, /, factor=2, *, clip=None):   # x positional-only, clip keyword-only
    y = x * factor
    return min(y, clip) if clip is not None else y
print(scale(5, clip=8))        # 8

params = {"activation": "gelu", "dropout": 0.2}
build_mlp(10, 16, **params)    # unpack a dict into keyword arguments
shape = (2, 3)
rows, cols = shape             # tuple unpacking
first, *rest = [1, 2, 3, 4]    # first=1, rest=[2, 3, 4]

*args collects extra positional arguments into a tuple; **kwargs collects extra keyword arguments into a dict. At a call site, * and ** do the reverse: they spread a sequence or dict into arguments. Library code uses this heavily to forward options, for example model(**tokenizer_output) in Hugging Face.

Comprehensions

nums = range(10)
squares = [n * n for n in nums if n % 2 == 0]        # [0, 4, 16, 36, 64]
tok2id = {t: i for i, t in enumerate(["the", "cat"])} # {'the': 0, 'cat': 1}
id2tok = {i: t for t, i in tok2id.items()}            # invert a mapping
uniq_lens = {len(w) for w in ["a", "bb", "cc"]}       # {1, 2}
total = sum(n * n for n in nums)                      # generator expression: no list built
matrix = [[r * c for c in range(3)] for r in range(2)]  # [[0, 0, 0], [0, 1, 2]]
flat = [x for row in matrix for x in row]            # loops read left to right

Comprehensions are faster than equivalent append loops (less interpreter overhead) and clearer when short. If a comprehension needs nested conditions or side effects, use a normal loop. For numeric work on large data, neither beats a NumPy vectorized operation.

Lambda, map, filter, sorted with key

runs = [{"name": "a", "f1": 0.81}, {"name": "b", "f1": 0.87}]
best = max(runs, key=lambda r: r["f1"])        # {'name': 'b', 'f1': 0.87}
ranked = sorted(runs, key=lambda r: -r["f1"])
lengths = list(map(len, ["hi", "there"]))      # [2, 5]
long_words = list(filter(lambda w: len(w) > 2, ["a", "the", "cat"]))  # ['the', 'cat']

Scope (LEGB) and closures

Python resolves a name by searching Local, then Enclosing function, then Global (module), then Built-in scope. Assigning to a name inside a function makes it local unless declared global or nonlocal.

def make_scheduler(base_lr, decay):
    step = 0
    def next_lr():
        nonlocal step              # modify the enclosing variable
        lr = base_lr * decay ** step
        step += 1
        return lr
    return next_lr                 # the inner function "closes over" base_lr, decay, step

sched = make_scheduler(0.1, 0.5)
print(sched(), sched(), sched())   # 0.1 0.05 0.025

Decorators

import time, functools

def timer(func):
    @functools.wraps(func)                 # keep the original name and docstring
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.perf_counter() - start:.3f}s")
        return result
    return wrapper

def retry(times=3, exceptions=(ConnectionError,)):   # decorator factory: takes arguments
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions:
                    if attempt == times:
                        raise
                    time.sleep(2 ** attempt)       # exponential backoff
        return wrapper
    return decorator

@timer
def train_epoch():
    time.sleep(0.2)
train_epoch()                                  # train_epoch took 0.200s

@functools.lru_cache(maxsize=None)             # memoize pure functions
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(80))                                 # 23416728348467685, instantly

@timer above a definition is shorthand for train_epoch = timer(train_epoch). Built-in decorators you will meet constantly: @property, @staticmethod, @classmethod, @dataclass, @functools.lru_cache, @torch.no_grad(), and @app.get(...) in web frameworks.

Iterators and generators: streaming data lazily

An iterable is anything you can loop over (list, dict, file, range). An iterator is the object that actually produces the values one at a time and remembers where it is. A generator is the easiest way to write an iterator: a function containing yield. Generators compute values on demand, so they can stream a 50 GB log file or an infinite data source using almost no memory. This is exactly how data loaders feed mini-batches to a model.

The iterator protocol

nums = [10, 20]
it = iter(nums)          # calls nums.__iter__()
print(next(it))          # 10   (calls it.__next__())
print(next(it))          # 20
# next(it) now raises StopIteration -- a for loop catches this for you

class Countdown:
    def __init__(self, start):
        self.current = start
    def __iter__(self):
        return self
    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

print(list(Countdown(3)))   # [3, 2, 1]

Generators

def countdown(start):
    while start > 0:
        yield start          # pause here, hand out a value
        start -= 1

gen = countdown(3)
print(next(gen), list(gen))  # 3 [2, 1]
print(list(gen))             # []  -- a generator is exhausted after one pass

def batches(items, size):
    """Yield successive mini-batches without building them all at once."""
    for i in range(0, len(items), size):
        yield items[i:i + size]

print(list(batches(list(range(7)), 3)))   # [[0, 1, 2], [3, 4, 5], [6]]

def read_jsonl(path):
    import json
    with open(path, encoding="utf-8") as f:
        for line in f:                    # files are lazy iterators over lines
            yield json.loads(line)

# pipelines of generators: nothing runs until something consumes them
# valid = (r for r in read_jsonl("events.jsonl") if r.get("label") is not None)

Generator expressions and memory

import sys
squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))
print(sys.getsizeof(squares_list) > 8_000_000)   # True: about 8 MB of pointers
print(sys.getsizeof(squares_gen) < 500)          # True: a few hundred bytes, whatever n is

itertools and yield from

import itertools as it
print(list(it.islice(it.count(0, 5), 4)))              # [0, 5, 10, 15]
print(list(it.chain([1, 2], [3])))                     # [1, 2, 3]
grid = list(it.product([0.01, 0.1], ["relu", "gelu"]))  # hyperparameter grid
print(len(grid))                                       # 4
print(list(it.accumulate([1, 2, 3, 4])))               # [1, 3, 6, 10]  running sum
print(list(it.batched(range(5), 2)))                   # [(0, 1), (2, 3), (4,)]  (3.12+)

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)     # delegate to a sub-generator
        else:
            yield item
print(list(flatten([1, [2, [3, 4]], 5])))   # [1, 2, 3, 4, 5]

Classes, OOP and dunder methods

Object-oriented code bundles data (attributes) with behaviour (methods). You need it in ML because frameworks are built on it: every PyTorch model is a class that inherits from nn.Module, every custom dataset implements __len__ and __getitem__, and every scikit-learn estimator follows a class-based contract.

A class from scratch

class Neuron:
    activation = "relu"                     # class attribute: shared by all instances

    def __init__(self, weights, bias=0.0):  # initializer, runs on Neuron(...)
        self.weights = list(weights)        # instance attributes
        self.bias = bias

    def forward(self, inputs):
        z = sum(w * x for w, x in zip(self.weights, inputs)) + self.bias
        return max(0.0, z)

    def __call__(self, inputs):             # makes instances callable like functions
        return self.forward(inputs)

    def __repr__(self):                     # unambiguous developer view
        return f"Neuron(weights={self.weights}, bias={self.bias})"

n = Neuron([0.5, -0.2], bias=0.1)
print(n)                 # Neuron(weights=[0.5, -0.2], bias=0.1)
print(n([1.0, 2.0]))     # 0.19999999999999998  (0.2 up to float rounding)

self is simply the instance, passed automatically as the first argument when you call n.forward(...). __init__ is not a constructor in the strict sense (that is __new__); it initialises an already-created object.

Inheritance and super()

class Model:
    def __init__(self, name):
        self.name = name
    def predict(self, x):
        raise NotImplementedError

class Constant(Model):
    def __init__(self, name, value):
        super().__init__(name)      # run the parent's initializer
        self.value = value
    def predict(self, x):           # override (polymorphism)
        return [self.value] * len(x)

print(Constant("baseline", 1).predict([7, 8, 9]))   # [1, 1, 1]
print(isinstance(Constant("b", 0), Model))         # True

With multiple inheritance, Python follows the method resolution order (MRO, the C3 linearization, viewable via Cls.__mro__); super() means "next class in the MRO", not simply "my parent".

The dunder methods worth knowing

class Vocab:
    def __init__(self, tokens):
        self.itos = sorted(set(tokens))
        self.stoi = {t: i for i, t in enumerate(self.itos)}
    def __len__(self):
        return len(self.itos)
    def __getitem__(self, token):
        return self.stoi.get(token, -1)     # -1 for unknown tokens
    def __contains__(self, token):
        return token in self.stoi

v = Vocab("the cat sat on the mat".split())
print(len(v), v["cat"], v["dog"], "mat" in v)   # 5 0 -1 True

property, classmethod, staticmethod

class Temperature:
    def __init__(self, celsius):
        self.celsius = celsius

    @property
    def fahrenheit(self):                 # computed attribute, accessed without ()
        return self.celsius * 9 / 5 + 32

    @classmethod
    def from_fahrenheit(cls, f):          # alternative constructor, receives the class
        return cls((f - 32) * 5 / 9)

    @staticmethod
    def is_valid(c):                      # plain function namespaced in the class
        return c >= -273.15

t = Temperature.from_fahrenheit(212)
print(t.celsius, t.fahrenheit, Temperature.is_valid(-300))   # 100.0 212.0 False

Python has no truly private attributes. By convention _name means "internal, do not touch"; __name triggers name mangling to _ClassName__name to avoid clashes in subclasses.

Robust Python: typing, dataclasses, exceptions and context managers

Research code often starts as a notebook, but production ML code must be readable, checkable and safe when things fail. Four features do most of that work: type hints (document and statically check intent), dataclasses (tidy configuration objects), exceptions (explicit failure handling) and context managers (guaranteed cleanup).

Type hints

from typing import Optional, Union, Literal, Callable, Iterator, TypedDict, Protocol
import numpy as np

def top_k(scores: list[float], k: int = 5) -> list[int]:
    return sorted(range(len(scores)), key=scores.__getitem__, reverse=True)[:k]

def load(path: str, sep: Optional[str] = None) -> "pd.DataFrame": ...
Metric = Literal["accuracy", "f1", "recall"]        # only these strings allowed
Scorer = Callable[[np.ndarray, np.ndarray], float]  # a function type

class Sample(TypedDict):          # typed shape for a dict (e.g. JSON records)
    text: str
    label: int

class SupportsPredict(Protocol):  # structural typing: "anything with .predict"
    def predict(self, X: np.ndarray) -> np.ndarray: ...

print(top_k([0.1, 0.9, 0.4], k=2))   # [1, 2]

Since Python 3.10 you can write int | None instead of Optional[int]. Hints are ignored at runtime by the interpreter; libraries such as Pydantic and FastAPI read them to validate data, which is why LLM structured-output code relies on them (see LLM APIs).

Dataclasses for configs and records

from dataclasses import dataclass, field, asdict

@dataclass(frozen=True)              # frozen: fields cannot be reassigned
class TrainConfig:
    lr: float = 3e-4
    epochs: int = 10
    layers: list[int] = field(default_factory=lambda: [64, 32])  # safe mutable default
    seed: int = 42

cfg = TrainConfig(lr=1e-3)
print(cfg)           # TrainConfig(lr=0.001, epochs=10, layers=[64, 32], seed=42)
print(asdict(cfg)["epochs"], cfg == TrainConfig(lr=1e-3))   # 10 True
# cfg.lr = 0.1  -> raises FrozenInstanceError

@dataclass generates __init__, __repr__ and __eq__ from annotated fields. Use slots=True (3.10+) to save memory for millions of small objects. Choose Pydantic instead when you need runtime validation of untrusted input such as API payloads or LLM output.

Exceptions

import json

class DataValidationError(ValueError):
    """Raised when an input file fails schema checks."""

def load_config(path):
    try:
        with open(path, encoding="utf-8") as f:
            cfg = json.load(f)
    except FileNotFoundError:
        return {"lr": 1e-3}                         # sensible default
    except json.JSONDecodeError as e:
        raise DataValidationError(f"bad JSON in {path}") from e   # keep the cause
    else:
        return cfg                                  # runs only if no exception
    finally:
        print("config load attempted")              # always runs

• Catch the most specific exception you can handle; never write a bare except:, which also swallows KeyboardInterrupt.
• raise ... from e chains exceptions so the traceback shows the root cause.
• Python style is EAFP ("easier to ask forgiveness than permission"): try the operation and catch failure, rather than checking every precondition first.
• Use assert for internal sanity checks (shapes during development) but not for validating user input, because asserts are removed when Python runs with -O.

Context managers

import time
from contextlib import contextmanager

@contextmanager
def timed(label):
    start = time.perf_counter()
    try:
        yield                              # the body of the with-block runs here
    finally:
        print(f"{label}: {time.perf_counter() - start:.2f}s")

with timed("feature engineering"):
    time.sleep(0.1)                        # feature engineering: 0.10s

class Timer:                               # the same idea, class-based
    def __enter__(self):
        self.start = time.perf_counter()
        return self
    def __exit__(self, exc_type, exc, tb):
        self.elapsed = time.perf_counter() - self.start
        return False                       # False: do not swallow exceptions

with Timer() as t:
    sum(range(10**6))
print(t.elapsed < 1)                       # True

ML examples: with open(...), with torch.no_grad():, with torch.autocast("cuda"):, with pd.option_context("display.max_rows", 5):, with mlflow.start_run():.

Concurrency: the GIL, threads, processes and asyncio

CPython has a Global Interpreter Lock (GIL): only one thread executes Python bytecode at a time within a process. Threads therefore do not speed up pure-Python number crunching, but they do help when threads spend most of their time waiting (network, disk), because waiting threads release the GIL. Heavy NumPy, pandas and PyTorch operations also release the GIL while their C code runs. For CPU-bound pure-Python work, use multiple processes, each with its own interpreter and GIL.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import math, time

def fetch(url):                    # I/O-bound: mostly waiting
    time.sleep(0.5)
    return len(url)

def heavy(n):                      # CPU-bound pure Python
    return sum(math.isqrt(i) for i in range(n))

if __name__ == "__main__":         # required for processes on Windows and macOS
    urls = [f"doc{i}" for i in range(8)]
    with ThreadPoolExecutor(max_workers=8) as ex:
        sizes = list(ex.map(fetch, urls))       # about 0.5 s total instead of 4 s
    with ProcessPoolExecutor() as ex:
        totals = list(ex.map(heavy, [2_000_000] * 4))   # uses several CPU cores

asyncio for many API calls

import asyncio

async def call_llm(prompt, sem):
    async with sem:                         # cap concurrency (rate limits)
        await asyncio.sleep(0.2)            # stands in for an async HTTP call
        return prompt.upper()

async def main(prompts):
    sem = asyncio.Semaphore(5)
    return await asyncio.gather(*(call_llm(p, sem) for p in prompts))

print(asyncio.run(main(["hi", "hello", "hey"])))   # ['HI', 'HELLO', 'HEY']

In Jupyter an event loop is already running, so use await main(prompts) directly in a cell instead of asyncio.run.

NumPy: arrays, vectorization and broadcasting

A NumPy ndarray is a block of memory holding many values of one type, plus metadata describing its shape and how to step through memory (strides). Because every element has the same type and sits next to its neighbours, NumPy can hand the whole block to optimized C loops and CPU vector instructions. A Python list, by contrast, is an array of pointers to separate objects scattered around memory, each carrying its own type information.

Creating arrays and reading their metadata

import numpy as np
a = np.array([1, 2, 3, 4])
print(a.dtype, a.shape, a.ndim, a.itemsize, a.nbytes)   # int64 (4,) 1 8 32
print(np.array([1, 2.5]).dtype)        # float64  (upcast to a common type)
print(np.array([1, "a"]).dtype)        # <U21     (everything became a string!)

np.zeros((2, 3)); np.ones(4); np.full((2, 2), 7); np.eye(2)
print(np.arange(0, 1, 0.25))           # [0.   0.25 0.5  0.75]  (stop excluded)
print(np.linspace(0, 1, 5))            # [0.   0.25 0.5  0.75 1.  ]  (stop included)
x = np.arange(12).reshape(3, 4)        # reshape keeps the same data
print(x)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

dtypes that matter in ML

img = np.array([200, 100], dtype=np.uint8)
print(img + img)                         # [144 200]: 400 wrapped around modulo 256!
print(img.astype(np.int32) + img)        # [400 200]  cast before arithmetic
print(np.array([127], dtype=np.int8) + 1)   # [-128]  silent overflow in arrays
X32 = np.random.default_rng(0).normal(size=(1000, 100)).astype(np.float32)   # half the RAM of float64

Vectorization and ufuncs

A ufunc (universal function) applies an operation element-wise in compiled code: arithmetic operators, np.exp, np.log, np.maximum, comparisons. Replacing a Python loop with one ufunc call typically speeds things up 10 to 100 times.

x = np.random.default_rng(0).normal(size=1_000_000)

# slow: interpreted loop, one Python float object per element
out = [max(0.0, v) for v in x]

# fast: one call into C
relu = np.maximum(0, x)
sigmoid = 1 / (1 + np.exp(-x))
clipped = np.clip(x, -1, 1)
labels = np.where(x > 0, 1, 0)          # vectorized if/else

# in a notebook:
# %timeit [max(0.0, v) for v in x]       # on the order of 100 ms
# %timeit np.maximum(0, x)               # on the order of 1 ms

Indexing: basic, boolean and fancy

x = np.arange(12).reshape(3, 4)
print(x[1, 2])          # 6            row 1, column 2
print(x[:, 1])          # [1 5 9]      every row, column 1
print(x[1:, ::2])       # [[ 4  6] [ 8 10]]  rows 1.., every 2nd column
print(x[-1])            # [ 8  9 10 11] last row
print(x[x % 2 == 0])    # [ 0  2  4  6  8 10]  boolean mask -> 1-D result
print(x[[0, 2], [1, 3]])  # [ 1 11]  fancy: picks (0,1) and (2,3)
print(x[[0, 2]])        # rows 0 and 2
mask = (x > 2) & (x < 8)   # combine masks with & | ~ and parentheses, not and/or

Views versus copies

Basic slicing (start:stop:step) returns a view: a new array object that shares the original memory. Fancy (integer-list) and boolean indexing return copies. reshape returns a view when possible; .copy() always copies.

b = np.arange(6)
v = b[1:4]              # view
v[0] = 99
print(b)                # [ 0 99  2  3  4  5]   original changed!
c = b[[1, 2]]           # copy (fancy indexing)
c[0] = -1
print(b)                # [ 0 99  2  3  4  5]   unchanged
print(np.shares_memory(b, v), np.shares_memory(b, c))   # True False
safe = b[1:4].copy()    # explicit copy when you plan to modify

Axes and aggregation

The axis argument names the dimension that gets collapsed. For a table of shape (rows, columns), axis=0 aggregates down the rows and returns one value per column; axis=1 aggregates across columns and returns one value per row.

          axis=1  -->
        +----+----+----+----+
axis=0  |  0 |  1 |  2 |  3 |   sum(axis=1) ->  6
  |     +----+----+----+----+
  v     |  4 |  5 |  6 |  7 |                  22
        +----+----+----+----+
        |  8 |  9 | 10 | 11 |                  38
        +----+----+----+----+
sum(axis=0) -> 12   15   18   21

x = np.arange(12).reshape(3, 4)
print(x.sum(axis=0))                       # [12 15 18 21]  per column
print(x.sum(axis=1))                       # [ 6 22 38]     per row
print(x.mean(axis=1, keepdims=True).shape) # (3, 1)  keeps a broadcastable shape
print(np.argmax([[1, 9], [8, 2]], axis=1)) # [1 0]   index of the max in each row
print(np.argsort([30, 10, 20]))            # [1 2 0] indices that would sort
vals, counts = np.unique([3, 1, 3, 2], return_counts=True)   # [1 2 3] [1 1 2]

Broadcasting

When two arrays have different shapes, NumPy compares their shapes from the right. Two dimensions are compatible if they are equal or one of them is 1 (a missing leading dimension counts as 1). Size-1 dimensions are stretched virtually to match.

A      (8, 1, 6, 1)
B         (7, 1, 5)
Result (8, 7, 6, 5)

X      (100, 3)        data: 100 samples, 3 features
mean        (3,)        per-feature mean
X - mean  (100, 3)      centred data, no loop, no copy of mean

M = np.array([[1, 2, 3], [4, 5, 6]])
print(M - M.mean(axis=0))          # centre each column
# [[-1.5 -1.5 -1.5]
#  [ 1.5  1.5  1.5]]
col = np.array([[10], [20]])       # shape (2, 1)
row = np.array([1, 2, 3])          # shape (3,)
print(col + row)                   # shape (2, 3): [[11 12 13] [21 22 23]]

# standardize features (what StandardScaler does)
X = np.random.default_rng(0).normal(5, 2, size=(100, 3))
Xs = (X - X.mean(axis=0)) / X.std(axis=0)

# pairwise squared distances between points (n, d) and centroids (k, d)
P, C = np.random.rand(5, 2), np.random.rand(3, 2)
D = ((P[:, None, :] - C[None, :, :]) ** 2).sum(axis=-1)   # (5, 1, 2) - (1, 3, 2) -> (5, 3)

np.ones(3) + np.ones(4)
# ValueError: operands could not be broadcast together with shapes (3,) (4,)

Use arr[:, None] or np.newaxis or np.expand_dims to add a size-1 axis deliberately; use squeeze to remove them.

Combining and reshaping

X = np.array([[1., 2.], [3., 4.]])
np.concatenate([X, X], axis=0).shape   # (4, 2)  join along an existing axis
np.stack([X, X]).shape                 # (2, 2, 2)  create a new axis
np.hstack([X, X]).shape                # (2, 4)
X.reshape(-1)                          # flatten (view if possible); -1 means "infer"
X.ravel(); X.flatten()                 # ravel: view if possible; flatten: always a copy
X.T                                    # transpose is a view (not contiguous)

Linear algebra

X = np.array([[1., 2.], [3., 4.]])
print(X @ X)             # matrix product  [[ 7. 10.] [15. 22.]]
print(X * X)             # element-wise    [[ 1.  4.] [ 9. 16.]]

A = np.array([[3., 1.], [1., 2.]]); b = np.array([9., 8.])
print(np.linalg.solve(A, b))            # [2. 3.]   prefer solve over inv(A) @ b
print(np.linalg.norm([3, 4]))           # 5.0
print(np.linalg.eigh(A)[0].round(3))    # [1.382 3.618]  eigenvalues of a symmetric matrix
U, S, Vt = np.linalg.svd(np.random.rand(4, 3), full_matrices=False)  # basis of PCA
w, *_ = np.linalg.lstsq(np.c_[np.ones(4), [1, 2, 3, 4]], [3, 5, 7, 9], rcond=None)
print(w.round(3))                       # [1. 2.]   intercept 1, slope 2 (y = 1 + 2x)

# one dense neural-network layer
Xb = np.random.default_rng(0).normal(size=(4, 3))   # 4 samples, 3 features
W = np.random.default_rng(1).normal(size=(3, 2)); bias = np.array([0.1, -0.2])
Z = Xb @ W + bias        # (4,3) @ (3,2) = (4,2); bias broadcasts across rows

Statistics and a numerically stable softmax

d = [22, 18, 14, 10, 15, 20, 25, 12]
print(np.mean(d), np.median(d))                  # 17.0 16.5
print(np.var(d), np.var(d, ddof=1).round(3))     # 23.25 26.571  population vs sample
print(np.std(d).round(3))                        # 4.822
print(np.percentile([15, 18, 67, 100, 21, 50], [25, 50, 75]))   # [18.75 35.5  62.75]
print(np.quantile([15, 18, 67, 100, 21, 50], 1/3))              # 20.0 (linear interpolation)
print(np.nan == np.nan, np.mean([1, np.nan, 3]), np.nanmean([1, np.nan, 3]))   # False nan 2.0

def softmax(z, axis=-1):
    z = z - z.max(axis=axis, keepdims=True)      # subtracting the max prevents overflow
    e = np.exp(z)
    return e / e.sum(axis=axis, keepdims=True)
print(softmax(np.array([1000., 1001., 1002.])).round(3))   # [0.09  0.245 0.665]

Note that NumPy's var and std default to the population formula (ddof=0) while pandas defaults to the sample formula (ddof=1), so the same column can give slightly different answers in the two libraries. The statistics themselves are covered in Math & Statistics for ML.

Random numbers done right

rng = np.random.default_rng(42)        # modern Generator API, local and seedable
rng.normal(0, 1, size=(2, 3))          # Gaussian
rng.uniform(0, 1, size=5)
rng.integers(0, 10, size=5)            # high end excluded
rng.choice(["a", "b", "c"], size=4, p=[0.5, 0.3, 0.2])
idx = rng.permutation(100)             # shuffled indices for a manual split
train_idx, test_idx = idx[:80], idx[80:]

Prefer default_rng over the legacy global np.random.seed: a generator object can be passed around explicitly, so two parts of a program do not silently share and disturb one global random state.

pandas: working with tables

pandas wraps NumPy arrays with labels. A Series is a one-dimensional array with an index (row labels). A DataFrame is a collection of Series that share the same index, one per column, and each column can have its own dtype. Almost every operation aligns on labels, which is powerful (joins and arithmetic just work) and occasionally surprising (mismatched indexes produce NaN).

Series and DataFrame basics

import pandas as pd, numpy as np
s = pd.Series([10, 20, 30], index=["a", "b", "c"])
print(s["b"], s.iloc[-1], s[s > 15].index.tolist())   # 20 30 ['b', 'c']

df = pd.DataFrame({
    "name":   ["Asha", "Ben", "Chen", "Dia"],
    "age":    [34, np.nan, 29, 41],
    "dept":   ["ml", "web", "ml", None],
    "salary": [120, 80, 110, 95],
})
df.shape            # (4, 4)
df.dtypes           # name: str (object before pandas 3), age: float64, dept: str, salary: int64
df.head(); df.tail(2); df.sample(2, random_state=0)
df.info()           # dtypes, non-null counts, memory
df.describe()       # count, mean, std, min, quartiles, max for numeric columns
df.columns.tolist(); df.index

Notice that age became float64: a classic NumPy-backed integer column cannot hold NaN, so pandas upcasts it. Use the nullable dtype "Int64" (capital I) if you need integers with missing values; it displays missing entries as <NA>.

Loading and saving data

df = pd.read_csv("data.csv",
                 usecols=["id", "ts", "amount", "city"],   # read only what you need
                 dtype={"city": "category", "amount": "float32"},
                 parse_dates=["ts"],
                 na_values=["", "NA", "?"])
pd.read_parquet("data.parquet")        # columnar, typed, compressed: best for big data
pd.read_json("records.jsonl", lines=True)
pd.read_excel("sheet.xlsx", sheet_name="Q1")
pd.read_sql("SELECT * FROM orders", connection)
for chunk in pd.read_csv("huge.csv", chunksize=100_000):   # stream a file bigger than RAM
    process(chunk)
df.to_parquet("clean.parquet", index=False)
df.to_csv("out.csv", index=False)      # index=False avoids an extra unnamed column

Selecting: [], loc and iloc

print(df.loc[0:2, "salary"].tolist())    # [120, 80, 110]  labels 0..2 INCLUDED
print(df.iloc[0:2]["salary"].tolist())   # [120, 80]       positions 0,1 (2 excluded)
print(df.loc[df.age > 30, ["name", "salary"]])
#    name  salary
# 0  Asha     120
# 3   Dia      95

Filtering and creating columns

high = df[(df.salary > 90) & (df.dept == "ml")]       # & | ~ with parentheses
df[df.dept.isin(["ml", "data"])]
df[df.name.str.startswith("A")]
df.query("salary > 90 and dept == 'ml'")               # readable string filter
df["salary_k"] = df.salary * 1000                      # vectorized new column
df = df.assign(ratio=lambda t: t.salary / t.salary.max())  # chain-friendly
df["band"] = np.where(df.salary >= 100, "high", "normal")
df["tier"] = np.select([df.salary < 90, df.salary < 110], ["low", "mid"], default="top")
df = df.rename(columns={"salary": "pay"}).drop(columns=["salary_k"])
df.sort_values(["dept", "pay"], ascending=[True, False])
df.nlargest(2, "pay")

groupby: split, apply, combine

rows            split by key        apply (sum)      combine
Pune   250  -->  Pune:  250, 150  -->  400    --\
Delhi  400  -->  Delhi: 400, 100  -->  500    ---->  Delhi 500 | Mumbai 300 | Pune 400
Pune   150       Mumbai: 300      -->  300    --/
Mumbai 300
Delhi  100

sales = pd.DataFrame({
    "city":  ["Pune", "Delhi", "Pune", "Mumbai", "Delhi"],
    "sales": [250, 400, 150, 300, 100],
    "units": [5, 8, 3, 6, 2],
})
print(sales.groupby("city")["sales"].sum())
# city
# Delhi     500
# Mumbai    300
# Pune      400
# Name: sales, dtype: int64

print(sales.groupby("city").agg(total=("sales", "sum"), avg_units=("units", "mean")))
#         total  avg_units
# city
# Delhi     500        5.0
# Mumbai    300        6.0
# Pune      400        4.0

# transform returns a result aligned to the ORIGINAL rows (same length)
sales["share"] = sales.sales / sales.groupby("city")["sales"].transform("sum")
print(sales.share.tolist())      # [0.625, 0.8, 0.375, 1.0, 0.2]

sales.groupby("city", as_index=False)["sales"].sum()       # key stays a column
sales.groupby("city").filter(lambda g: g.sales.sum() > 350)  # keep whole groups
sales.groupby(["city", "units"]).size()                       # multi-key counts
sales.city.value_counts(normalize=True)                       # class proportions

Combining tables: merge, join, concat

left = pd.DataFrame({"k": [1, 2, 3], "a": ["x", "y", "z"]})
right = pd.DataFrame({"k": [2, 3, 4], "b": [True, False, True]})
print(pd.merge(left, right, on="k", how="left", indicator=True))
#    k  a      b     _merge
# 0  1  x    NaN  left_only
# 1  2  y   True       both
# 2  3  z  False       both

pd.merge(orders, customers, on="customer_id", how="left",
         validate="many_to_one")        # raises if customers has duplicate keys
pd.merge(a, b, left_on="uid", right_on="user_id", suffixes=("_a", "_b"))
a.join(b, how="inner")                  # join on the index
pd.concat([jan, feb], ignore_index=True)  # stack rows (same columns)
pd.concat([X_num, X_cat], axis=1)       # side by side (aligns on index!)

Reshaping: pivot, pivot_table, melt, crosstab

wide = pd.DataFrame({"id": [1, 2], "jan": [10, 20], "feb": [11, 21]})
long = wide.melt(id_vars="id", var_name="month", value_name="sales")   # wide -> long
print(long)
#    id month  sales
# 0   1   jan     10
# 1   2   jan     20
# 2   1   feb     11
# 3   2   feb     21
back = long.pivot(index="id", columns="month", values="sales")         # long -> wide

rev = pd.DataFrame({"region": ["N", "N", "S", "S"], "q": ["Q1", "Q2", "Q1", "Q2"], "rev": [10, 15, 7, 9]})
print(rev.pivot_table(index="region", columns="q", values="rev", aggfunc="sum", margins=True))
# q       Q1  Q2  All
# region
# N       10  15   25
# S        7   9   16
# All     17  24   41
pd.crosstab(rev.region, rev.q)          # frequency table of two categoricals

pivot fails if an index/column pair appears twice; pivot_table aggregates duplicates instead. stack/unstack move levels between the row index and the columns.

Missing values

df.isna().sum()                     # missing count per column
df.isna().mean().sort_values()      # missing fraction, worst last
df.dropna()                         # drop rows with ANY missing value
df.dropna(subset=["age"])           # only rows missing age
df.dropna(axis=1, thresh=int(0.7 * len(df)))   # keep columns at least 70% filled
df.fillna({"age": df.age.median(), "dept": "unknown"})
df.age.fillna(df.groupby("dept").age.transform("median"))   # group-aware imputation
ts.ffill(); ts.bfill(); ts.interpolate()   # time series: forward-fill, back-fill, interpolate
df["age_missing"] = df.age.isna().astype(int)   # missingness can itself be informative

For modelling, prefer imputing inside a scikit-learn pipeline (SimpleImputer) so the fill values are learned from the training fold only. Filling the whole DataFrame with the global median before splitting is a small but real form of leakage.

apply versus vectorized operations

# slow
df["bmi"] = df.apply(lambda r: r.weight / r.height ** 2, axis=1)
# fast (often 100x or more on large frames)
df["bmi"] = df.weight / df.height ** 2
# if you truly need a loop, itertuples is much faster than iterrows
for row in df.itertuples(index=False):
    pass

Strings, dates and times

names = pd.Series(["a b", "c"])
print(names.str.split().str.len().tolist(), names.str.upper().tolist())   # [2, 1] ['A B', 'C']
df.email.str.extract(r"@(\w+)\.")        # regex capture group to a column
df.text.str.contains("refund", case=False, na=False)

ev = pd.DataFrame({
    "ts": pd.to_datetime(["2026-01-01 09:00", "2026-01-01 17:00", "2026-01-02 10:00", "2026-01-04 12:00"]),
    "clicks": [3, 5, 2, 7]})
ev["dow"] = ev.ts.dt.day_name(); ev["hour"] = ev.ts.dt.hour   # Thursday 9, Thursday 17, ...
print(ev.set_index("ts").resample("D")["clicks"].sum().tolist())   # [8, 2, 0, 7]  gap day filled with 0
print(ev.clicks.rolling(2).mean().tolist())    # [nan, 4.0, 3.5, 4.5]
print(ev.clicks.shift(1).tolist())             # [nan, 3.0, 5.0, 2.0]  lag feature
print(ev.clicks.diff().tolist())               # [nan, 2.0, -3.0, 5.0]
pd.to_datetime(raw, format="%d/%m/%Y", errors="coerce")   # explicit format; bad values -> NaT
ev.ts.dt.tz_localize("UTC").dt.tz_convert("Asia/Kolkata")

Binning and categoricals

ages = [5, 17, 30, 64, 80]
print(pd.cut(ages, bins=[0, 18, 60, 120], labels=["child", "adult", "senior"]).tolist())
# ['child', 'child', 'adult', 'senior', 'senior']   fixed edges, (0, 18] includes 18
print(pd.qcut(pd.Series([1, 2, 3, 4, 100]), 2, labels=["lo", "hi"]).tolist())
# ['lo', 'lo', 'lo', 'hi', 'hi']                    equal-frequency bins

c = pd.Series(["lo", "hi", "lo"], dtype="category")
print(c.cat.categories.tolist(), c.cat.codes.tolist())   # ['hi', 'lo'] [1, 0, 1]
size = pd.Categorical(["M", "S", "L"], categories=["S", "M", "L"], ordered=True)   # ordered: S < M < L
pd.get_dummies(df.dept, prefix="dept", dtype=int)        # one-hot for quick analysis

A category column stores each distinct value once plus a small integer code per row, which slashes memory for repetitive strings (city, product, country) and speeds up groupby.

Memory optimisation

big = pd.DataFrame({"i": np.arange(1000, dtype="int64"), "f": np.random.rand(1000)})
print(big.memory_usage(deep=True).sum())    # 16132 bytes (index + 8 + 8 bytes per row)
big["i"] = pd.to_numeric(big.i, downcast="integer")   # int16 is enough for 0..999
big["f"] = big.f.astype("float32")
print(big.memory_usage(deep=True).sum())    # 6132 bytes, about 62% less

def shrink(df):
    for col in df.select_dtypes("integer"):
        df[col] = pd.to_numeric(df[col], downcast="integer")
    for col in df.select_dtypes("float"):
        df[col] = pd.to_numeric(df[col], downcast="float")
    for col in df.select_dtypes(["object", "string"]):
        if df[col].nunique() / len(df) < 0.5:      # repetitive text -> category
            df[col] = df[col].astype("category")
    return df

1. Measure df.info(memory_usage="deep") and df.memory_usage(deep=True).
2. Read less usecols, nrows for prototyping, filter early.
3. Type tightly Downcast numbers, category for low-cardinality strings, Arrow-backed strings.
4. Store columnar Parquet keeps dtypes and lets you read only some columns.
5. Stream or scale out chunksize, or move to Polars, DuckDB, Dask or Spark when data truly exceeds one machine's memory.

Copies, views and chained assignment

# WRONG: chained indexing. The first [] may return a copy, so the write is lost.
df[df.salary > 90]["salary"] = 0      # warning; original df unchanged
# RIGHT: one .loc call selects rows and column together
df.loc[df.salary > 90, "salary"] = 0
# taking a subset you intend to modify independently
sub = df[df.salary > 90].copy()

Older pandas emits SettingWithCopyWarning for ambiguous cases. With Copy-on-Write (the default behaviour from pandas 3.0), any derived object behaves like a copy, chained assignment never modifies the original, and pandas warns with a ChainedAssignmentError. The fix is the same in every version: one .loc assignment.

# "top 2 cities by total sales" and "top 1 row per group"
sales.groupby("city")["sales"].sum().nlargest(2)
sales.sort_values("sales", ascending=False).groupby("city").head(1)

Visualization: matplotlib and seaborn

matplotlib is the low-level plotting engine: a Figure is the canvas, and each Axes is one plot area on it with its own x-axis, y-axis, title and legend. seaborn sits on top and draws statistical plots straight from DataFrames with sensible defaults, colouring by a column with a single hue= argument. Pick the plot from the question you are asking, not from habit.

The object-oriented matplotlib pattern

import numpy as np, matplotlib.pyplot as plt
x = np.linspace(-6, 6, 200)
fig, axes = plt.subplots(1, 2, figsize=(10, 3.5))       # 1 row, 2 plots
axes[0].plot(x, 1 / (1 + np.exp(-x)), label="sigmoid")
axes[0].plot(x, np.tanh(x), label="tanh", linestyle="--")
axes[0].axhline(0, color="gray", lw=0.8)
axes[0].set(title="Activations", xlabel="z", ylabel="output")
axes[0].legend()
axes[1].hist(np.random.default_rng(0).normal(size=1000), bins=30, alpha=0.7)
axes[1].set_title("Histogram")
fig.tight_layout()
fig.savefig("plots.png", dpi=150)                       # or plt.show() in scripts

seaborn one-liners

import seaborn as sns
sns.histplot(df, x="mean radius", hue="diagnosis", kde=True)   # distribution by class
sns.boxplot(df, x="diagnosis", y="mean area")                  # spread and outliers
sns.scatterplot(df, x="mean radius", y="mean texture", hue="diagnosis")
sns.countplot(df, x="diagnosis")                               # class balance
sns.heatmap(df.corr(numeric_only=True), cmap="coolwarm", center=0)   # correlations
sns.pairplot(df[cols + ["diagnosis"]], hue="diagnosis")        # all pairs (small cols only)
sns.lineplot(history, x="epoch", y="loss", hue="split")        # learning curves

Which plot answers which question

Model diagnostic plots

from sklearn.metrics import ConfusionMatrixDisplay, RocCurveDisplay, PrecisionRecallDisplay
ConfusionMatrixDisplay.from_estimator(model, X_test, y_test)
RocCurveDisplay.from_estimator(model, X_test, y_test)
PrecisionRecallDisplay.from_estimator(model, X_test, y_test)

# learning curves from a PyTorch or Keras history
plt.plot(train_losses, label="train"); plt.plot(val_losses, label="val"); plt.legend()
# val loss rising while train keeps falling = overfitting: stop earlier or regularize

Exploratory data analysis (EDA) workflow

EDA is the structured interrogation of a dataset before modelling. Its goals are to understand what each column means, find data-quality problems, spot relationships worth modelling, and catch leakage before it inflates your metrics. Good EDA is a checklist you run every time, then a few targeted deep dives.

1. Understand the question What is the target, what is the unit of prediction (patient, transaction, day), and what data is available at prediction time?
2. Structure shape, dtypes, head(), info(). Are numbers stored as strings? Are dates parsed?
3. Quality Missing values per column, duplicates (df.duplicated().sum()), impossible values (negative ages), inconsistent categories ("NY" versus "New York").
4. Univariate describe(), histograms, value_counts(); note skew, outliers and scale differences.
5. Target Class balance for classification; distribution and skew for regression.
6. Bivariate Each feature against the target: grouped means, box plots, correlation with the target.
7. Multivariate Correlation heatmap, multicollinearity, a quick PCA scatter.
8. Leakage hunt Features that are near-perfect predictors, IDs correlated with the target, columns created after the outcome, duplicate entities across train and test.
9. Decide Write down the preprocessing each finding implies (impute, log-transform, encode, drop, stratify, group split).

A reusable EDA snippet

def quick_eda(df, target):
    print("shape:", df.shape)
    print("duplicates:", df.duplicated().sum())
    report = pd.DataFrame({
        "dtype": df.dtypes.astype(str),
        "missing_%": (df.isna().mean() * 100).round(1),
        "unique": df.nunique(),
    })
    print(report.sort_values("missing_%", ascending=False).head(15))
    print(df[target].value_counts(normalize=True).round(3))
    num = df.select_dtypes("number").drop(columns=[target], errors="ignore")
    print(num.describe().T[["mean", "std", "min", "max"]].round(2).head(10))
    print("skew (top):", num.skew().abs().sort_values(ascending=False).head(5).round(2).to_dict())
    corr = num.corrwith(df[target]).abs().sort_values(ascending=False)
    print("most target-correlated:", corr.head(5).round(3).to_dict())

Descriptive statistics in pandas

scores = pd.Series([22, 18, 14, 10, 15, 20, 25, 12])
scores.mean(), scores.median(), scores.mode().tolist()   # 17.0 16.5 [10, 12, 14, ...] (all tie)
scores.var(), scores.std()          # 26.571..., 5.154...  (sample, ddof=1 by default)
scores.var(ddof=0)                  # 23.25  population variance, matches NumPy's default
scores.quantile([0.25, 0.5, 0.75])  # quartiles
iqr = scores.quantile(0.75) - scores.quantile(0.25)
(scores - scores.mean()).abs().mean()   # mean absolute deviation: 4.25
scores.max() - scores.min()         # range: 15

The mean is pulled by outliers while the median is not, so a mean far above the median signals right skew (incomes, prices, counts). The IQR rule flags outliers below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR, which is exactly what box-plot whiskers show.

Outliers: detect, then decide

q1, q3 = df.amount.quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df.amount < q1 - 1.5 * iqr) | (df.amount > q3 + 1.5 * iqr)]
z = (df.amount - df.amount.mean()) / df.amount.std()     # |z| > 3 rule for roughly normal data
df["amount_log"] = np.log1p(df.amount)                    # compress a heavy right tail
df["amount_clip"] = df.amount.clip(upper=df.amount.quantile(0.99))   # winsorize

Do not delete outliers automatically. A 10-million transaction may be an error or exactly the fraud you want to catch. Investigate, then choose to fix, cap, transform, or keep and use a robust model or loss.

scikit-learn: the estimator API, pipelines and evaluation

scikit-learn succeeds because every object follows the same small contract. An estimator learns from data with fit(X, y). A transformer also has transform(X) (and the shortcut fit_transform). A predictor has predict(X), and classifiers usually add predict_proba(X) and score(X, y). Hyperparameters are set in the constructor; learned values get a trailing underscore (coef_, mean_, classes_). Once you know this contract, every model in the library is familiar.

The contract in five lines

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

scaler = StandardScaler()                 # hyperparameters go in the constructor
X_train_s = scaler.fit_transform(X_train) # learns mean_ and scale_ from TRAIN only
X_test_s = scaler.transform(X_test)       # re-uses train statistics: no leakage
clf = LogisticRegression(max_iter=1000).fit(X_train_s, y_train)
clf.predict(X_test_s); clf.predict_proba(X_test_s)[:, 1]; clf.score(X_test_s, y_test)

Preprocessing toolbox

Distance- and gradient-based models care about feature scale; tree-based models (decision trees, random forests, gradient boosting) do not. On the breast-cancer data, scaling moves kNN from 0.912 to 0.956 test accuracy and an RBF SVM from 0.904 to 0.974, while an unscaled random forest already reaches 0.965.

Pipelines and ColumnTransformer

import numpy as np, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

df = pd.DataFrame({
    "age":    [25, np.nan, 47, 35, 52, 23],
    "income": [40, 60, np.nan, 80, 120, 30],
    "city":   ["Pune", "Delhi", "Pune", np.nan, "Mumbai", "Delhi"],
    "bought": [0, 1, 1, 0, 1, 0],
})
X, y = df.drop(columns="bought"), df.bought
numeric, categorical = ["age", "income"], ["city"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))]), categorical),
])
model = Pipeline([("pre", preprocess), ("clf", LogisticRegression())])
model.fit(X, y)
print(model.named_steps["pre"].get_feature_names_out())
# ['num__age' 'num__income' 'cat__city_Delhi' 'cat__city_Mumbai' 'cat__city_Pune']
print(model.predict(pd.DataFrame({"age": [30], "income": [70], "city": ["Chennai"]})))
# [0]   unseen city "Chennai" is encoded as all zeros instead of crashing

Parameters of steps inside a pipeline are addressed as step__param, with nesting as deep as needed: clf__C, pre__num__impute__strategy. make_pipeline and make_column_transformer build the same objects with auto-generated step names. Calling set_output(transform="pandas") makes transformers return DataFrames with readable column names.

Splitting data correctly

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)   # stratify keeps class ratios

5-fold cross-validation (training data only; test set stays locked away)
fold 1:  [VAL ][train][train][train][train]
fold 2:  [train][VAL ][train][train][train]
fold 3:  [train][train][VAL ][train][train]
fold 4:  [train][train][train][VAL ][train]
fold 5:  [train][train][train][train][VAL ]
score = mean of 5 validation scores (report the std too)

Cross-validation

from sklearn.model_selection import StratifiedKFold, cross_val_score, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
y = 1 - y                                     # make malignant the positive class (1)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
res = cross_validate(pipe, X, y, cv=cv, scoring=["accuracy", "recall", "roc_auc"])
print({k: round(v.mean(), 3) for k, v in res.items() if k.startswith("test")})
# {'test_accuracy': 0.979, 'test_recall': 0.958, 'test_roc_auc': 0.995}

Why the pipeline matters: a leakage demonstration

from sklearn.feature_selection import SelectKBest, f_classif
rng = np.random.default_rng(0)
Xr = rng.normal(size=(100, 10_000))          # pure noise features
yr = rng.integers(0, 2, 100)                 # random labels: true accuracy is 50%

# WRONG: select the 20 "best" features using ALL rows, then cross-validate
leaky = SelectKBest(f_classif, k=20).fit_transform(Xr, yr)
print(cross_val_score(LogisticRegression(max_iter=1000), leaky, yr, cv=5).mean())   # 0.87 (!)

# RIGHT: selection happens inside each training fold
honest = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
print(cross_val_score(honest, Xr, yr, cv=5).mean())                                 # 0.48

The leaky version reports 87% accuracy on random noise, because the feature selector already "saw" the validation labels. Any step that learns from data, including scaling, imputation, feature selection, target encoding, oversampling (SMOTE) and PCA, must live inside the pipeline. Cross-validation estimates how well that whole procedure generalises; a bootstrap of a single test-set metric instead puts an error bar on a frozen model's score. Use CV to choose models, and a paired bootstrap (or McNemar) on the sealed test set to compare two finished models.

Hyperparameter search

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import loguniform

param_grid = {"logisticregression__C": [0.01, 0.1, 1, 10, 100],
              "logisticregression__class_weight": [None, "balanced"]}
grid = GridSearchCV(pipe, param_grid, cv=cv, scoring="recall", n_jobs=-1)
grid.fit(X_train, y_train)
grid.best_params_, grid.best_score_       # best combination and its mean CV score
best = grid.best_estimator_               # already refit on all training data
pd.DataFrame(grid.cv_results_)[["params", "mean_test_score", "std_test_score"]]

rand = RandomizedSearchCV(pipe, {"logisticregression__C": loguniform(1e-3, 1e3)},
                          n_iter=30, cv=cv, scoring="roc_auc", random_state=0, n_jobs=-1)

• Grid search tries every combination: cost multiplies with each parameter added. Randomized search samples a fixed budget and usually finds good regions faster when only a few parameters matter.
• HalvingGridSearchCV and libraries such as Optuna (Bayesian optimisation) spend more budget on promising candidates.
• best_score_ is optimistically biased because it was used for selection. Report final performance on the untouched test set, or use nested cross-validation (a search inside each outer fold) when the dataset is small.
• With multiple metrics, pass scoring={"rec": "recall", "prec": "precision"} and choose which one to refit.

Metrics: pick the one that matches the cost of errors

from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report, roc_auc_score, average_precision_score,
    mean_absolute_error, mean_squared_error, r2_score)
y_pred = best.predict(X_test)
y_proba = best.predict_proba(X_test)[:, 1]          # probability of class 1
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))
roc_auc_score(y_test, y_proba)                      # needs scores, not hard labels
f1_score(y_true_multi, y_pred_multi, average="macro")

Imbalance and decision thresholds

• class_weight="balanced" re-weights the loss so the rare class counts more; often the simplest effective fix.
• The default 0.5 threshold is arbitrary. Choose a threshold on validation data (or out-of-fold predictions) with precision_recall_curve, according to the business constraint such as "recall at least 0.95". Newer versions also provide TunedThresholdClassifierCV for this.
• Resampling (under-sampling, SMOTE from the imbalanced-learn library) must be applied only to training folds, inside a pipeline.
• Always compare against a trivial baseline: DummyClassifier() scores 0.627 on the breast-cancer data just by predicting the majority class.

A model zoo with one API

from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
models = {
    "logreg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
    "knn":    make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)),
    "svm":    make_pipeline(StandardScaler(), SVC(probability=True)),
    "tree":   DecisionTreeClassifier(max_depth=4, random_state=0),
    "forest": RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1),
    "gbm":    HistGradientBoostingClassifier(random_state=0),   # handles NaN natively
}
for name, m in models.items():
    s = cross_val_score(m, X_train, y_train, cv=cv, scoring="recall")
    print(f"{name:7s} recall {s.mean():.3f} +/- {s.std():.3f}")

Algorithm details (how trees split, what the SVM margin is, why kNN suffers in high dimensions) are covered in Machine Learning. Unsupervised tools follow the same API: PCA(n_components=2).fit_transform(X), KMeans(n_clusters=3, n_init=10).fit_predict(X).

Inspecting and saving models

coefs = pd.Series(best[-1].coef_[0], index=feature_names).sort_values(key=abs)   # linear models
importances = forest.feature_importances_                                         # trees
from sklearn.inspection import permutation_importance
pi = permutation_importance(best, X_test, y_test, scoring="recall", n_repeats=10, random_state=0)

import joblib
joblib.dump(best, "model.joblib")       # save the WHOLE pipeline, preprocessing included
model = joblib.load("model.joblib")     # same library versions required; never load untrusted files
model.predict(new_rows)

PyTorch fundamentals: tensors, autograd and the training loop

PyTorch rests on three ideas. Tensors are NumPy-like arrays that can live on a GPU. Autograd records the operations you perform on tensors that require gradients and, when you call backward(), applies the chain rule backwards through that record to compute every gradient. Modules (nn.Module) package parameters and a forward computation into reusable, nestable building blocks. Around these sit Dataset/DataLoader for feeding data and optimizers for updating weights.

Tensors

import torch, numpy as np
x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print(x.shape, x.dtype, x.device)     # torch.Size([2, 2]) torch.float32 cpu
print(torch.tensor([1, 2]).dtype)     # torch.int64 -- the default for class labels
print(x @ x, x * x)                   # matrix product / element-wise, same as NumPy
print(x.sum(dim=0), x.mean(dim=1))    # tensor([4., 6.]) tensor([1.5000, 3.5000])  dim = axis
torch.zeros(2, 3); torch.randn(4, 3); torch.arange(6); torch.linspace(0, 1, 5)

# reshaping
x.unsqueeze(0).shape                  # torch.Size([1, 2, 2])  add a batch dimension
x.permute(1, 0); x.T                  # reorder dimensions
x.T.reshape(4)                        # tensor([1., 3., 2., 4.])  copies if needed
# x.T.view(4) -> RuntimeError: view needs contiguous memory; use .contiguous() or reshape

# NumPy bridge: shares memory on CPU
a = np.ones(2)
t = torch.from_numpy(a)               # float64 tensor sharing a's buffer
a[0] = 5
print(t)                              # tensor([5., 1.], dtype=torch.float64)
back = t.numpy()                      # GPU tensors need .cpu() first; grad tensors need .detach()

Devices

device = ("cuda" if torch.cuda.is_available()
          else "mps" if torch.backends.mps.is_available()   # Apple silicon
          else "cpu")
model = model.to(device)                  # moves parameters in place
xb, yb = xb.to(device), yb.to(device)     # tensors return a NEW tensor; reassign!
# every tensor in one operation must be on the same device, or you get
# "Expected all tensors to be on the same device"

Autograd

x = torch.tensor(2.0, requires_grad=True)
y = x ** 3 + 2 * x            # dy/dx = 3x^2 + 2
y.backward()
print(x.grad)                 # tensor(14.)

w = torch.tensor(1.0, requires_grad=True)
for _ in range(2):
    loss = (w * 3) ** 2       # d/dw = 18w = 18
    loss.backward()
    print(w.grad)             # tensor(18.) then tensor(36.)  -- gradients ACCUMULATE
w.grad.zero_()                # this is why training loops call optimizer.zero_grad()

with torch.no_grad():         # no graph recorded: inference and manual updates
    w -= 0.1 * 18
frozen = y.detach()           # same data, cut from the graph (requires_grad=False)

Gradients accumulate by design, so that you can sum gradients over several small batches before one update (gradient accumulation). The price is that you must clear them each step.

Building models with nn.Module

import torch.nn as nn, torch.nn.functional as F

class MLP(nn.Module):
    def __init__(self, d_in, d_hidden=32, n_classes=2, p_drop=0.2):
        super().__init__()                      # must come first
        self.net = nn.Sequential(
            nn.Linear(d_in, d_hidden),          # weight (32, d_in), bias (32,)
            nn.ReLU(),
            nn.Dropout(p_drop),
            nn.Linear(d_hidden, n_classes),
        )
    def forward(self, x):
        return self.net(x)                      # raw logits: no softmax here

model = MLP(30)
print(sum(p.numel() for p in model.parameters()))   # 1058 = 30*32+32 + 32*2+2
print(model(torch.randn(5, 30)).shape)               # torch.Size([5, 2])
for name, p in model.named_parameters():
    print(name, tuple(p.shape))                      # net.0.weight (32, 30) ...

Assigning an nn.Module or nn.Parameter as an attribute registers it automatically, so model.parameters(), .to(device) and state_dict() all find it. A plain Python list of layers is not registered; use nn.ModuleList or nn.ModuleDict.

Losses and output layers

logits = torch.tensor([[2.0, 0.5, -1.0]])
print(nn.CrossEntropyLoss()(logits, torch.tensor([0])).item())   # 0.2413...  = -log softmax(logits)[0]
probs = F.softmax(logits, dim=1)       # only for reporting or inference, never before CrossEntropyLoss

Datasets and DataLoaders

from torch.utils.data import Dataset, DataLoader, TensorDataset

class TabularDataset(Dataset):
    def __init__(self, X, y):
        self.X = torch.as_tensor(X, dtype=torch.float32)
        self.y = torch.as_tensor(y, dtype=torch.long)
    def __len__(self):
        return len(self.y)
    def __getitem__(self, i):
        return self.X[i], self.y[i]           # one sample; DataLoader stacks them into batches

train_dl = DataLoader(TabularDataset(X_tr, y_tr), batch_size=32, shuffle=True,
                      num_workers=0, pin_memory=torch.cuda.is_available())
val_dl = DataLoader(TabularDataset(X_va, y_va), batch_size=256, shuffle=False)
xb, yb = next(iter(train_dl))
print(xb.shape, yb.shape)                     # torch.Size([32, 30]) torch.Size([32])

num_workers > 0 starts worker processes that prepare batches in parallel, which matters when each item needs disk reads or image decoding. For tensors already in memory, 0 is often fastest. A custom collate_fn handles variable-length samples such as padded text.

The training loop

for each epoch:
    model.train()
    for each batch:
        optimizer.zero_grad()      1. clear old gradients
        logits = model(xb)         2. forward pass (builds graph)
        loss = loss_fn(logits, yb) 3. scalar loss
        loss.backward()            4. backprop: fill p.grad
        optimizer.step()           5. update weights
    model.eval(); with torch.no_grad(): validate, log, checkpoint best

import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

torch.manual_seed(0)
X, y = load_breast_cancer(return_X_y=True); y = 1 - y           # 1 = malignant
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
sc = StandardScaler().fit(X_tr); X_tr, X_te = sc.transform(X_tr), sc.transform(X_te)

train_ds = TensorDataset(torch.tensor(X_tr, dtype=torch.float32), torch.tensor(y_tr, dtype=torch.long))
test_ds = TensorDataset(torch.tensor(X_te, dtype=torch.float32), torch.tensor(y_te, dtype=torch.long))
train_dl = DataLoader(train_ds, batch_size=32, shuffle=True)
test_dl = DataLoader(test_ds, batch_size=256)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = MLP(30).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(1, 31):
    model.train(); total = 0.0
    for xb, yb in train_dl:
        xb, yb = xb.to(device), yb.to(device)
        opt.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        opt.step()
        total += loss.item() * xb.size(0)      # .item() -> Python float, frees the graph
    if epoch % 10 == 0:
        model.eval(); correct = 0
        with torch.no_grad():
            for xb, yb in test_dl:
                correct += (model(xb.to(device)).argmax(1) == yb.to(device)).sum().item()
        print(f"epoch {epoch:2d}  train_loss {total/len(train_ds):.4f}  test_acc {correct/len(test_ds):.3f}")
# epoch 10  train_loss 0.0947  test_acc 0.974
# epoch 20  train_loss 0.0661  test_acc 0.974
# epoch 30  train_loss 0.0536  test_acc 0.974   (exact numbers vary by version and hardware)

In real projects, evaluate on a separate validation split for early stopping and model selection, and keep the test set for one final measurement, exactly as with scikit-learn.

train() versus eval(), no_grad versus inference_mode

For inference you need both: model.eval() and with torch.no_grad():.

Saving and loading

torch.save({"model": model.state_dict(), "optimizer": opt.state_dict(),
            "epoch": epoch, "config": {"d_in": 30}}, "checkpoint.pt")

ckpt = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)  # False: file also has epoch/config; only load trusted files
model = MLP(**ckpt["config"])
model.load_state_dict(ckpt["model"])
model.eval()

Save the state_dict (a dict of tensors) rather than pickling the whole model object, which ties the file to your exact class definitions and file paths. Recent PyTorch versions default torch.load(..., weights_only=True), which refuses non-tensor objects; a training checkpoint that also stores epoch or a config dict needs weights_only=False and must come from a trusted source. The safetensors format stores tensors without pickle and avoids that risk.

Useful extras

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # tame exploding gradients
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=30)   # call sched.step() each epoch
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):       # mixed precision on GPU
    loss = loss_fn(model(xb), yb)
model = torch.compile(model)                                         # graph compilation (PyTorch 2.x)
torch.manual_seed(0); np.random.seed(0); random.seed(0)              # reproducibility

Hugging Face basics: pipelines, tokenizers and Auto classes

The Hugging Face transformers library gives you thousands of pretrained models behind a uniform interface. Three layers of abstraction cover almost every use: pipeline() for a task in one line, AutoTokenizer to turn text into token IDs, and AutoModel... classes to load the right architecture for a checkpoint by name. The theory of what these models do (attention, tokenization, decoding) lives in Transformers & LLMs; this section is the Python mechanics.

from transformers import pipeline
clf = pipeline("sentiment-analysis")                  # downloads a default model on first use
print(clf(["Great battery life!", "Screen cracked in a week."]))
# [{'label': 'POSITIVE', 'score': 0.99...}, {'label': 'NEGATIVE', 'score': 0.99...}]
# other tasks: "zero-shot-classification", "ner", "question-answering",
#              "summarization", "text-generation", "feature-extraction"

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
name = "distilbert-base-uncased-finetuned-sst-2-english"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

batch = tok(["Great battery life!", "Screen cracked in a week."],
            padding=True, truncation=True, max_length=128, return_tensors="pt")
print(batch.keys())            # input_ids, attention_mask
with torch.no_grad():
    logits = model(**batch).logits          # shape (2, 2)
print(logits.softmax(-1).argmax(-1).tolist(), model.config.id2label)

• The attention_mask marks real tokens (1) versus padding (0) so padded positions are ignored.
• Models are ordinary PyTorch nn.Modules: .to(device), .eval(), no_grad and custom training loops all work. The Trainer class and the datasets library wrap the loop and data handling for fine-tuning.
• Load large models with torch_dtype=torch.bfloat16 and device_map="auto" to save memory; parameter-efficient fine-tuning (LoRA via the peft library) trains small adapter matrices instead of all weights.
• In restricted networks, download models through your organisation's approved mirror and load them from a local folder path with from_pretrained("path/to/model").

Calling LLM APIs from Python

Most LLM application code is ordinary Python: build a list of messages, send an HTTP request through an SDK, parse the response, handle errors and rate limits, and validate structured output. Many providers and self-hosted servers expose an OpenAI-compatible interface, so one client pattern works across them. Prompting, structured outputs, tool calling and evaluation are covered in depth in LLM APIs.

import os
from openai import OpenAI            # the same client works with OpenAI-compatible servers
from pydantic import BaseModel, ValidationError

client = OpenAI(api_key=os.environ["LLM_API_KEY"],          # never hard-code keys
                base_url=os.environ.get("LLM_BASE_URL"))    # your approved endpoint

resp = client.chat.completions.create(
    model=os.environ.get("LLM_MODEL", "gpt-4o-mini"),
    messages=[{"role": "system", "content": "You are a concise tutor."},
              {"role": "user", "content": "Explain overfitting in one sentence."}],
    temperature=0, max_tokens=80)
print(resp.choices[0].message.content, resp.usage.total_tokens)

class Review(BaseModel):
    sentiment: str
    rating: int

raw = '{"sentiment": "positive", "rating": 5}'     # e.g. a JSON-mode response
try:
    review = Review.model_validate_json(raw)
except ValidationError as e:
    print("retry or fall back:", e)

• Keep keys in environment variables or a secrets manager; never commit them or print them in notebooks.
• Wrap calls with timeouts and exponential-backoff retries on rate-limit and transient server errors.
• Use asyncio with a semaphore (see the concurrency section) to evaluate hundreds of prompts in parallel within rate limits.
• Cache responses keyed by (model, prompt, parameters) during development to save cost and make runs repeatable.
• Log token usage and latency per call; cost scales with input plus output tokens.

Performance and best practices

Speed work follows one rule: measure first, then fix the biggest bottleneck. In data code the usual culprits are Python-level loops over rows, wrong dtypes that bloat memory, re-reading or re-computing the same data, and moving data between CPU and GPU or between processes more than necessary.

Measure

# notebook
%timeit df.a * df.b                    # repeated timing of one line
%prun train_features(df)               # function-level profile
%load_ext line_profiler
%lprun -f build_features build_features(df)   # line-by-line (needs line_profiler)
%memit df = pd.read_csv("big.csv")     # peak memory (needs memory_profiler)

# script
import cProfile, pstats, tracemalloc
cProfile.run("main()", "prof.out")
pstats.Stats("prof.out").sort_stats("cumulative").print_stats(10)
tracemalloc.start(); run(); print(tracemalloc.get_traced_memory())   # (current, peak) bytes
# command line: python -m cProfile -s cumtime train.py ; py-spy for sampling live processes

The speed ladder (fastest first)

1. Avoid the work Filter early, read fewer columns, cache intermediate results (Parquet, joblib.Memory, lru_cache).
2. Vectorize NumPy ufuncs, pandas column operations, np.where/np.select, groupby built-ins such as "sum" rather than a Python lambda.
3. Use the right structure Set or dict for membership tests instead of a list; collections.deque for queues; sorted arrays with np.searchsorted.
4. Compile hot loops Numba @njit or Cython for loops that genuinely cannot be vectorized.
5. Parallelize n_jobs=-1 in scikit-learn, joblib.Parallel, process pools, DataLoader workers.
6. Change engine or hardware Polars or DuckDB for large tables, Dask or Spark for multi-machine data, GPU for large matrix work.

import numpy as np
from numba import njit

@njit                                    # compiled to machine code on first call
def count_runs(x):
    runs = 0
    for i in range(1, x.size):          # sequential dependency: hard to vectorize
        if x[i] != x[i - 1]:
            runs += 1
    return runs

from joblib import Parallel, delayed
results = Parallel(n_jobs=-1)(delayed(score_file)(p) for p in paths)

Memory habits

• Use float32 for model inputs and category for repetitive strings; check with memory_usage(deep=True).
• Process in chunks or with generators when data exceeds RAM; write intermediate results to Parquet.
• Delete large temporaries (del df_raw) and call gc.collect() in long notebooks; restart the kernel when in doubt.
• Prefer in-place NumPy operations for huge arrays (np.multiply(a, 2, out=a), a *= 2) to avoid temporary copies.
• On GPU: smaller batches, mixed precision, gradient accumulation, torch.no_grad() for evaluation, and .item() when logging scalars.

Code-quality habits that pay off in ML

Worked example: breast-cancer diagnosis end to end

This project ties every section together on the classic breast-cancer dataset: 569 tumours, 30 numeric features computed from cell-nucleus images, and a binary label. Missing a malignant tumour (a false negative) is far worse than an unnecessary follow-up (a false positive), so the primary metric is recall on the malignant class, with a sanity floor on precision. Every output below comes from running the code with scikit-learn 1.9 and pandas 3.0; small numeric differences across versions are normal.

Step 1: load and relabel

import numpy as np, pandas as pd
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer(as_frame=True)
df = data.frame.copy()
df["target"] = 1 - df["target"]        # raw data: 0 = malignant; now 1 = malignant (positive class)
print(df.shape)                        # (569, 31)
print(df["target"].value_counts())
# target
# 0    357      benign
# 1    212      malignant  (about 37%: mildly imbalanced)
print(df.isna().sum().sum())           # 0  no missing values

Step 2: exploratory analysis

print(df[["mean radius", "mean area", "mean smoothness"]].describe().round(2))
#        mean radius  mean area  mean smoothness
# count       569.00     569.00           569.00
# mean         14.13     654.89             0.10
# std           3.52     351.91             0.01
# min           6.98     143.50             0.05
# ...          (quartile rows omitted)
# max          28.11    2501.00             0.16
# scales differ by 4 orders of magnitude -> scale for linear models, SVM, kNN

print(df.groupby("target")[["mean radius", "worst concave points"]].mean().round(3))
#         mean radius  worst concave points
# target
# 0            12.147                 0.074
# 1            17.463                 0.182       malignant tumours are larger and more irregular

import matplotlib.pyplot as plt, seaborn as sns
fig, axes = plt.subplots(1, 3, figsize=(13, 3.5))
sns.countplot(df, x="target", ax=axes[0])
sns.histplot(df, x="mean radius", hue="target", kde=True, ax=axes[1])
sns.heatmap(df.iloc[:, :10].corr(), cmap="coolwarm", center=0, ax=axes[2])
fig.tight_layout()
# the heatmap shows radius, perimeter and area almost perfectly correlated (they are
# geometrically linked): harmless for trees, handled by regularization in logistic regression

Step 3: split before anything learns from data

from sklearn.model_selection import train_test_split
X, y = df.drop(columns="target"), df["target"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
print(X_train.shape, X_test.shape, round(y_train.mean(), 3), round(y_test.mean(), 3))
# (455, 30) (114, 30) 0.374 0.368      stratification preserved the class ratio

Step 4: pipeline and cross-validated baseline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score

pipe = Pipeline([("scale", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=5000))])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="recall")
print(scores.round(3), round(scores.mean(), 3), round(scores.std(), 3))
# [0.882 0.971 1.    0.941 0.971] 0.953 0.04     fold-to-fold spread matters on small data

Step 5: tune hyperparameters on training data only

from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(pipe,
                    {"clf__C": [0.01, 0.1, 1, 10, 100],
                     "clf__class_weight": [None, "balanced"]},
                    cv=cv, scoring="recall", n_jobs=-1)
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
# {'clf__C': 1, 'clf__class_weight': 'balanced'} 0.959

Step 6: choose the decision threshold from out-of-fold predictions

from sklearn.model_selection import cross_val_predict
from sklearn.metrics import precision_recall_curve
best = grid.best_estimator_
oof = cross_val_predict(best, X_train, y_train, cv=cv, method="predict_proba")[:, 1]
prec, rec, thr = precision_recall_curve(y_train, oof)
ok = np.where(rec[:-1] >= 0.98)[0]           # thresholds meeting the recall target
threshold = thr[ok[-1]]                       # the highest such threshold keeps precision best
print(round(threshold, 3), round(rec[ok[-1]], 3), round(prec[ok[-1]], 3))
# 0.284 0.982 0.923      on training folds: 98% recall at 92% precision

Step 7: open the sealed envelope once

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, average_precision_score
proba = best.predict_proba(X_test)[:, 1]
y_pred = (proba >= threshold).astype(int)
print(confusion_matrix(y_test, y_pred))
# [[70  2]      70 benign correct, 2 false alarms
#  [ 2 40]]      2 missed malignant, 40 caught
print(classification_report(y_test, y_pred, target_names=["benign", "malignant"], digits=3))
#               precision    recall  f1-score   support
#       benign      0.972     0.972     0.972        72
#    malignant      0.952     0.952     0.952        42
#     accuracy                          0.965       114
print(round(roc_auc_score(y_test, proba), 3), round(average_precision_score(y_test, proba), 3))
# 0.995 0.993

Test recall (0.952) is a little below the 0.98 seen on training folds. With only 42 malignant test cases, one tumour moves recall by 2.4 percentage points, so this gap is within noise; the honest report is "about 95 to 98% recall with roughly 95% precision", together with the fold standard deviation. If the clinical requirement is stricter, lower the threshold further and accept more false alarms, deciding on validation data, never by peeking at the test set again.

Step 8: interpret and save

coefs = pd.Series(best.named_steps["clf"].coef_[0], index=X.columns)
print(coefs.sort_values(key=abs, ascending=False).head(5).round(3))
# worst texture          1.466
# radius error           1.309
# worst symmetry         1.077
# mean concave points    0.970
# compactness error     -0.961       coefficients are per standard deviation, because inputs were scaled
import joblib
joblib.dump({"model": best, "threshold": float(threshold), "features": list(X.columns)},
            "breast_cancer_v1.joblib")

Because many features are strongly correlated, individual coefficients can shift sign or size between fits; use permutation importance or a regularized, reduced feature set before drawing medical conclusions. A PyTorch MLP on the same split (see the PyTorch section) reaches the same 0.974 accuracy at the default threshold, which is a useful lesson: on small tabular data, a well-tuned linear model is a very strong baseline.

Quick revision

• Python is the orchestration layer; speed comes from keeping work inside compiled libraries (NumPy, BLAS, cuDNN) rather than Python loops.
• Use one isolated environment per project (venv or conda) with pinned versions; use %pip inside notebooks.
• Restart the kernel and run all cells before sharing a notebook, to expose hidden state.
• Variables are names bound to objects; assignment never copies.
• Lists, dicts and sets are mutable; ints, floats, strings, tuples and frozensets are immutable.
• == compares values, is compares identity; use is only with None and other singletons.
• Never use a mutable default argument; default to None and create the object inside.
• Dict and set lookups are O(1) on average; list membership is O(n). Keys must be hashable.
• *args collects extra positional arguments into a tuple; **kwargs collects extra keyword arguments into a dict.
• A closure remembers variables from its enclosing scope; closures bind late, so loop variables are read at call time.
• A decorator is a function that takes a function and returns a wrapped one; always use functools.wraps.
• Generators (yield) produce values lazily, use constant memory, and can be consumed only once.
• Dunder methods (__len__, __getitem__, __call__, __enter__) plug your classes into Python syntax and into frameworks.
• Type hints are not enforced at runtime; tools such as mypy and libraries such as Pydantic read them.
• with guarantees cleanup via __exit__, even when an exception is raised.
• The GIL allows one thread to run Python bytecode at a time: threads or asyncio for I/O-bound work, processes for CPU-bound Python work.
• A full PyTorch checkpoint (state_dict plus epoch or config) needs torch.load(..., weights_only=False) and must be trusted; tensor-only loads can keep the safer default.
• NumPy arrays are homogeneous, typed and contiguous, which enables vectorized C loops.
• Broadcasting aligns shapes from the right; dimensions must match or be 1.
• Basic slicing returns a view; fancy and boolean indexing return copies.
• axis=0 collapses rows (one result per column); axis=1 collapses columns (one result per row).
• * is element-wise, @ is matrix multiplication.
• Subtract the maximum before exponentiating in softmax for numerical stability.
• NumPy's var/std default to ddof=0; pandas defaults to ddof=1.
• loc selects by label with an inclusive end; iloc selects by position with an exclusive end.
• Combine pandas masks with &, |, ~ and parentheses.
• groupby().agg() returns one row per group; transform() returns a result aligned to the original rows.
• Validate merges (validate=, row counts) to catch duplicate keys that explode joins.
• Vectorized column operations beat apply, which beats iterrows.
• Assign with a single df.loc[mask, col] = value; chained assignment does not modify the original.
• Shrink pandas memory with downcasting, category dtype, usecols, Parquet and chunking.
• EDA checklist: structure, quality, univariate, target, bivariate, multivariate, leakage, decisions.
• scikit-learn contract: fit learns, transform/predict apply; learned attributes end with an underscore.
• Put every data-dependent step inside a Pipeline so cross-validation cannot leak.
• Use CV to choose models; use a paired bootstrap or McNemar on a sealed test set to compare two finished models.
• ColumnTransformer applies different preprocessing to numeric and categorical columns.
• Stratify classification splits; use group splits for repeated entities and time-series splits for temporal data.
• Tune on training data with CV; touch the test set once at the end.
• Choose metrics by error cost: recall when misses are costly, precision when false alarms are costly, PR-AUC for rare positives.
• Pick the decision threshold on validation or out-of-fold predictions, not the test set.
• Scale features for linear models, SVM, kNN, PCA and neural networks; trees do not need it.
• PyTorch loop: zero_grad, forward, loss, backward, step.
• model.eval() changes dropout and BatchNorm behaviour; torch.no_grad() stops gradient tracking; inference needs both.
• CrossEntropyLoss takes raw logits and integer class labels; BCEWithLogitsLoss takes logits and float targets.
• Save state_dicts; weights_only=True is the safe default, but a checkpoint that also stores epoch or config needs weights_only=False from a trusted file.
• Hugging Face: pipeline for quick tasks, AutoTokenizer plus AutoModelFor... for control; tokenizer and model must match.
• Profile before optimising; vectorize first, compile or parallelize second, change engine last.

Glossary

apply

pandas method that runs a Python function per element, row or group; flexible but slow compared with vectorized operations.

asyncio

Python's standard library for single-threaded concurrency using async/await, ideal for many simultaneous network calls.

Autograd

PyTorch's automatic differentiation engine, which records operations and computes gradients with the chain rule on backward().

Axis

A dimension of an array; the axis argument names the dimension an aggregation collapses.

Broadcasting

NumPy and PyTorch rules for combining arrays of different shapes by virtually stretching size-1 dimensions.

Categorical dtype

pandas type storing each distinct value once plus integer codes per row, saving memory for repetitive values.

Closure

A function that remembers variables from the scope in which it was defined.

ColumnTransformer

scikit-learn tool that applies different transformers to different column subsets and concatenates the results.

Context manager

Object used with with that sets something up in __enter__ and guarantees cleanup in __exit__.

Copy-on-Write

pandas behaviour (default from version 3.0) where derived objects act as copies, so modifying one never silently changes another.

Cross-validation

Repeatedly splitting training data into training and validation folds to estimate performance more reliably than a single split.

Data leakage

Information unavailable at prediction time (often from the test set or the target) sneaking into training, inflating reported performance.

Dataclass

Class decorated with @dataclass, which auto-generates __init__, __repr__ and __eq__ from annotated fields.

DataFrame

pandas two-dimensional labelled table made of columns (Series) sharing one index.

DataLoader

PyTorch utility that batches, shuffles and optionally loads data in parallel worker processes from a Dataset.

Decorator

A callable that takes a function and returns a modified or wrapped function, applied with @name.

dtype

The data type of array or column elements, such as float32, int64 or category.

Dunder method

A "double underscore" special method like __len__ that Python calls to implement built-in syntax and functions.

EDA

Exploratory data analysis: systematic inspection of data quality, distributions and relationships before modelling.

Estimator

Any scikit-learn object that learns from data via fit.

Fancy indexing

Indexing an array with lists or arrays of integers; always returns a copy.

Generator

A function using yield that produces values lazily, one at a time, suspending between values.

GIL

Global Interpreter Lock: a CPython mutex allowing only one thread to execute Python bytecode at a time per process.

GridSearchCV

scikit-learn search that cross-validates every combination of a hyperparameter grid and refits the best.

groupby

pandas split-apply-combine operation that groups rows by keys and aggregates, transforms or filters each group.

Hashable

An object with a stable hash and equality, which can therefore be a dict key or set member.

iloc

pandas position-based indexer; slice ends are exclusive.

Iterator

Object implementing __next__ (and __iter__) that yields successive values and raises StopIteration when done.

Jupyter kernel

The live Python process behind a notebook that keeps state between cell executions.

loc

pandas label-based indexer that also accepts boolean masks; slice ends are inclusive.

Logits

Raw, unnormalised model outputs before softmax or sigmoid.

melt

pandas operation that reshapes a wide table into a long one.

merge

pandas SQL-style join of two DataFrames on key columns.

MRO

Method resolution order: the sequence Python searches through base classes to find a method.

Mutable

Able to be changed in place after creation (lists, dicts, sets, arrays).

ndarray

NumPy's n-dimensional, homogeneous, typed array.

nn.Module

Base class for PyTorch layers and models, which registers parameters and submodules and defines forward.

no_grad

PyTorch context that disables gradient tracking, used for inference and manual weight updates.

One-hot encoding

Representing a category as a vector with a single 1 in the column for that category.

Parquet

Compressed, columnar file format that preserves dtypes and allows reading only selected columns.

Pipeline

scikit-learn object chaining transformers and a final estimator so they are fitted and applied together.

pivot_table

pandas reshaping that aggregates values into a grid of index versus column categories.

requires_grad

PyTorch tensor flag indicating that operations on it should be recorded for gradient computation.

Series

pandas one-dimensional labelled array.

state_dict

Dictionary mapping a PyTorch model's or optimizer's parameter names to tensors; the recommended save format.

Stratification

Splitting data so each part keeps the same class proportions as the whole.

Tensor

PyTorch's n-dimensional array, which can live on CPU or GPU and can track gradients.

Tokenizer

Component that converts text into integer token IDs (and back) for a specific language model.

transform (groupby)

pandas group operation that returns a result aligned to the original rows, such as each row's share of its group total.

Transformer (scikit-learn)

An estimator with a transform method, such as a scaler, encoder or imputer.

ufunc

NumPy universal function that operates element-wise on arrays in compiled code.

Vectorization

Expressing computation as whole-array operations instead of explicit Python loops.

View

An array object that shares memory with another array, created for example by basic slicing.

Virtual environment

An isolated folder with its own Python interpreter and packages for one project.

Interview questions

import copy
grid = [[0, 0], [0, 0]]
s = grid.copy(); d = copy.deepcopy(grid)
grid[0][0] = 9
print(s[0][0], d[0][0])   # 9 0

def add(x, acc=[]):
    acc.append(x); return acc
add(1); print(add(2))    # [1, 2]

def add(x, acc=None):    # fix
    if acc is None:
        acc = []
    acc.append(x); return acc

df[(df.salary > 90) & (df.dept == "ml")]
df[df.city.isin(["Pune", "Delhi"]) & ~df.name.str.startswith("A")]
df.query("salary > 90 and dept == 'ml'")

import numpy as np, pandas as pd, statistics
d = [22, 18, 14, 10, 15, 20, 25, 12]
np.mean(d), np.median(d)            # 17.0 16.5
statistics.multimode([1, 2, 2, 3])  # [2]
np.var(d), np.std(d)                # 23.25 4.82  (population, ddof=0)
pd.Series(d).var()                  # 26.57       (sample, ddof=1)
np.percentile(d, [25, 75])          # quartiles

import time, functools
def timer(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        t0 = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            print(f"{func.__name__}: {time.perf_counter() - t0:.3f}s")
    return wrapper

def retry(times=3):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for i in range(times):
                try:
                    return func(*args, **kwargs)
                except ConnectionError:
                    if i == times - 1:
                        raise
        return wrapper
    return decorator

@retry(times=5)
def fetch(): ...

import numpy as np
def minibatches(X, y, batch_size, shuffle=True, seed=0):
    idx = np.arange(len(X))
    if shuffle:
        np.random.default_rng(seed).shuffle(idx)
    for start in range(0, len(X), batch_size):
        b = idx[start:start + batch_size]
        yield X[b], y[b]

for xb, yb in minibatches(np.arange(10).reshape(5, 2), np.arange(5), 2, shuffle=False):
    print(xb.shape)    # (2, 2) (2, 2) (1, 2)

b = np.arange(6); v = b[1:4]; v[0] = 99
print(b)          # [ 0 99  2  3  4  5]

def softmax(z, axis=-1):
    z = z - z.max(axis=axis, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=axis, keepdims=True)
softmax(np.array([1000., 1001., 1002.]))   # [0.09 0.245 0.665]  (naive exp overflows to inf)

Xs = (X - X.mean(axis=0)) / X.std(axis=0)        # broadcasting (n, d) with (d,)

# pairwise distances between A (n, d) and B (m, d)
D = np.sqrt(((A[:, None, :] - B[None, :, :]) ** 2).sum(-1))       # (n, m); memory n*m*d
# memory-lean version using |a-b|^2 = |a|^2 + |b|^2 - 2ab
D2 = (A**2).sum(1)[:, None] + (B**2).sum(1)[None, :] - 2 * A @ B.T
D = np.sqrt(np.maximum(D2, 0))                   # clip tiny negatives from rounding

def cosine_sim(q, M):
    q = q / np.linalg.norm(q)
    M = M / np.linalg.norm(M, axis=1, keepdims=True)
    return M @ q                                 # shape (n,)
top3 = np.argsort(-cosine_sim(q, M))[:3]         # or np.argpartition for large n

df["share"] = df.sales / df.groupby("city").sales.transform("sum")

# top 2 products by revenue within each region
(df.sort_values("revenue", ascending=False)
   .groupby("region")
   .head(2))
# or with ranking, keeping ties explicit
df[df.groupby("region").revenue.rank(method="first", ascending=False) <= 2]
# single best row per group
df.loc[df.groupby("region").revenue.idxmax()]

df = df.sort_values(["user", "date"])
g = df.groupby("user").amount
df["lag_1"] = g.shift(1)                                  # yesterday's value
df["roll_7"] = g.transform(lambda s: s.shift(1).rolling(7).mean())   # past 7 days, excluding today
df["pct_change"] = g.pct_change()

pre = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num_cols),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                          OneHotEncoder(handle_unknown="ignore")), cat_cols),
])
model = make_pipeline(pre, LogisticRegression(max_iter=1000))

from numpy.lib.stride_tricks import sliding_window_view
sliding_window_view(np.arange(6), 3)   # [[0 1 2] [1 2 3] [2 3 4] [3 4 5]], no copy

np.einsum("ij,jk->ik", A, B)          # matrix multiply, same as A @ B
np.einsum("ii->", M)                  # trace
np.einsum("ij->j", A)                 # column sums
np.einsum("bhqd,bhkd->bhqk", Q, K)    # batched attention scores per head

inner = GridSearchCV(pipe, grid, cv=StratifiedKFold(5, shuffle=True, random_state=0))
outer_scores = cross_val_score(inner, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=1))

from sklearn.base import BaseEstimator, TransformerMixin
class Log1p(BaseEstimator, TransformerMixin):
    def __init__(self, cols=None):       # store params unchanged; no logic here
        self.cols = cols
    def fit(self, X, y=None):
        self.n_features_in_ = X.shape[1] # learned state ends with an underscore
        return self
    def transform(self, X):
        X = np.asarray(X, dtype=float).copy()
        idx = self.cols if self.cols is not None else slice(None)
        X[:, idx] = np.log1p(X[:, idx])
        return X

def train_logreg(X, y, lr=0.1, epochs=500, l2=0.0):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(epochs):
        p = 1 / (1 + np.exp(-(X @ w + b)))        # predicted probabilities
        grad_w = X.T @ (p - y) / n + l2 * w       # gradient of mean log loss (+ L2)
        grad_b = (p - y).mean()
        w -= lr * grad_w
        b -= lr * grad_b
    return w, b
# on standardized breast-cancer data: training accuracy about 0.986

def kmeans(X, k, iters=100, seed=0):
    rng = np.random.default_rng(seed)
    C = X[rng.choice(len(X), k, replace=False)]            # init from data points
    for _ in range(iters):
        labels = ((X[:, None, :] - C[None]) ** 2).sum(-1).argmin(1)   # assign
        newC = np.array([X[labels == j].mean(0) if np.any(labels == j) else C[j]
                         for j in range(k)])                          # update
        if np.allclose(newC, C):
            break
        C = newC
    return C, labels

for p in model.backbone.parameters():
    p.requires_grad = False                      # frozen: no gradients computed
opt = torch.optim.AdamW([
    {"params": model.head.parameters(), "lr": 1e-3},
    {"params": [p for p in model.encoder.parameters() if p.requires_grad], "lr": 1e-5},
], weight_decay=0.01)

def attention(Q, K, V, mask=None):
    d_k = Q.shape[-1]
    scores = Q @ K.swapaxes(-1, -2) / np.sqrt(d_k)       # (..., q, k)
    if mask is not None:
        scores = np.where(mask, scores, -1e9)             # block masked positions
    w = softmax(scores, axis=-1)                          # rows sum to 1
    return w @ V, w                                       # (..., q, d_v)