Linux Kernel & BSP

A Board Support Package is the software that makes a generic OS run on a specific SoC and board. It typically contains the bootloader (or its board port), the device tree describing the board's hardware, the kernel configuration (defconfig), and drivers for every peripheral (clocks, pinctrl, regulators, storage, display, sensors, modem interfaces). On Android it also extends into userspace: vendor HALs, init .rc files, ueventd.rc permissions and SELinux vendor policy. BSP work means board bring-up, driver and DT development, and debugging boot, power and peripheral issues.

The kernel is the privileged core that manages hardware and shares it safely among programs. Its main jobs are: process and thread management and CPU scheduling; memory management (virtual memory, allocation, protection); device management through drivers; filesystems and I/O; networking; inter-process communication; and security (permissions, isolation). Programs request these services through system calls rather than touching hardware directly.

Linux is monolithic: scheduler, memory management, filesystems, networking and drivers run together in kernel space and call each other directly, which is fast. The downside is weaker isolation: a bug in any driver can crash the whole system. A microkernel (QNX, seL4, Fuchsia's Zircon) keeps only scheduling, IPC and basic memory in the kernel and runs drivers and filesystems as user processes, giving better fault isolation at the cost of IPC overhead. Linux adds flexibility with loadable modules, so it is often called a modular monolithic kernel.

They are separate privilege levels and address-space regions. User space (EL0 on ARM64, ring 3 on x86) runs applications with no direct hardware access and a private virtual address space per process. Kernel space (EL1 / ring 0) runs the kernel with full access to memory and devices. A bug in user space kills only that process; a bug in kernel space can oops or panic the whole system. User code enters the kernel only through defined entry points: system calls, exceptions (like page faults) and interrupts.

A system call is the controlled way user code asks the kernel for a service (open a file, allocate memory, create a process). The libc wrapper places the syscall number and arguments in registers and executes a trap instruction (svc #0 on ARM64, syscall on x86-64). The CPU switches to kernel mode and jumps to the exception vector; the kernel saves registers, looks up the handler in the syscall table, runs it (copying data with copy_from_user/copy_to_user) and returns the result with eret. Errors come back as negative values that libc turns into -1 plus errno.

A device tree is a data structure (.dts source compiled to a .dtb blob) that describes a board's hardware: buses, register addresses, interrupts, clocks, regulators, GPIOs. It is kept separate from kernel code so one kernel image can support many boards. At boot the kernel creates devices from DT nodes and matches each node's compatible string to a driver's of_match_table, then calls probe(). Before DT, ARM used per-board C files, which did not scale. Overlays (.dtbo) patch the base tree for board variants.

A kernel module (.ko) is code that can be linked into the running kernel on demand, typically a driver. It has an init function (module_init) and exit function (module_exit). Load with insmod file.ko (no dependency handling) or modprobe name (resolves dependencies via modules.dep); unload with rmmod; list with lsmod; inspect with modinfo. In Kconfig, =y builds code into the kernel image and =m builds it as a module. With GKI, all vendor drivers must be modules.

• Character drivers: byte-stream devices accessed via /dev nodes and file_operations (open, read, write, ioctl, mmap), e.g. serial ports, input, sensors, Binder.
• Block drivers: random-access storage in fixed-size blocks, going through the page cache and block layer, e.g. UFS, eMMC, NVMe.
• Network drivers: packet interfaces (net_device) used via sockets, no /dev node, e.g. Wi-Fi, Ethernet.

Orthogonally, drivers sit on buses: platform (on-SoC, from DT), I2C, SPI, USB, PCI.

A platform driver handles a device on the "platform bus", a virtual bus for hardware that cannot be discovered by enumeration, typically IP blocks inside the SoC (UART, I2C controller, GPIO controller, watchdog). The devices are created from device-tree nodes (or ACPI, or board code). The driver registers a struct platform_driver with probe, remove and an of_match_table; when a matching node exists, probe() gets the platform_device, maps its registers (devm_platform_ioremap_resource), gets its IRQ (platform_get_irq) and clocks, and initializes the hardware.

An interrupt is a hardware signal that makes the CPU pause its current work and run a handler, so the device notifies the CPU when something happens. Polling means the CPU repeatedly checks a status register. Interrupts save CPU time and power when events are infrequent; polling can be better at very high event rates (it avoids per-event interrupt overhead) or when latency must be tightly controlled. Linux's network NAPI mixes both: interrupt to start, then poll while traffic is heavy.

The top half is the hard IRQ handler: it runs immediately in interrupt context, cannot sleep, and should do the minimum (acknowledge the hardware, read urgent data, schedule the rest). The bottom half does the remaining work later in a less time-critical context: a softirq (high-frequency, e.g. networking), a tasklet (simple, built on softirq), a workqueue (kernel thread, can sleep), or a threaded IRQ (dedicated kernel thread, can sleep). The split keeps interrupts-off time short so other interrupts are not delayed or lost.

A process is a running program with its own address space and resources (open files, credentials, signal handlers). A thread is an execution flow within a process that shares its address space and resources but has its own stack, registers and scheduling state. Threads communicate cheaply through shared memory but need locking; processes are isolated and need IPC, but a crash in one does not corrupt another. In Linux both are tasks (task_struct) created with clone(); the flags decide what is shared.

• R running or runnable (on a run queue).
• S interruptible sleep: waiting for an event, can be woken by signals.
• D uninterruptible sleep: usually waiting for I/O; ignores signals.
• T stopped (SIGSTOP) or traced by a debugger.
• Z zombie: exited but not yet reaped by its parent.

You can see them in ps output or /proc/<pid>/status.

A zombie has exited, but its parent has not yet called wait()/waitpid(), so the kernel keeps its PID and exit status in the process table. It uses no memory beyond that entry, but many zombies can exhaust PIDs; fix the parent to reap children (or handle SIGCHLD). An orphan is a running process whose parent died; it is re-parented to init (PID 1) or a designated subreaper, which reaps it when it exits.

Virtual memory gives each process its own address space that the MMU translates to physical memory using page tables. Benefits: isolation (processes cannot access each other's memory), protection (read-only code, no-execute data), simpler programming (each process sees a large contiguous space), lazy allocation (pages allocated on first use), sharing (libraries and COW pages mapped into many processes), memory-mapped files, and the ability to swap or compress rarely used pages.

Paging divides virtual and physical memory into fixed-size pages (typically 4 KB). A page table maps each virtual page to a physical frame and stores permission bits (read, write, execute, user/kernel) and status bits (valid, accessed, dirty). To save space, page tables are multi-level (4 levels on ARM64 with 48-bit addresses); the MMU walks them on a TLB miss. The kernel builds and updates the tables; the hardware reads them.

The Translation Lookaside Buffer is a small, fast cache in the MMU holding recent virtual-to-physical translations. A TLB hit translates an address in about a cycle; a miss requires a page-table walk of several memory accesses. When mappings change, the kernel must invalidate affected entries (TLB shootdown on SMP, using broadcast TLBI on ARM). ASIDs tag entries per address space so switching processes does not require flushing the whole TLB. Huge pages increase TLB coverage.

A page fault is an exception raised when the MMU cannot complete an access: the page is not present or the access violates permissions. The kernel handler checks whether the address belongs to a valid VMA. If yes, it resolves the fault: allocate a zero page, load a file page, swap in, or copy a COW page (minor fault if no I/O, major if I/O). If not, the process gets SIGSEGV, or the kernel oopses if it happened in kernel code. Faults are normal and frequent; excessive major faults indicate memory pressure or thrashing.

A context switch is the kernel stopping one task and running another on the same CPU. It saves the current task's registers, stack pointer and program counter, picks the next task, switches the address space if the new task is in a different process (page table base plus ASID), and restores the new task's registers. It is triggered by blocking, preemption (time slice expiry or higher-priority wakeup) or yielding. Direct cost is a few microseconds; the indirect cost of cold caches and TLB often dominates.

A mutex is a lock for mutual exclusion: only one holder at a time, it has an owner, only the owner can unlock it, and waiters sleep. A semaphore is a counter: down() decrements it and sleeps if it would go negative, up() increments it and wakes a waiter. A counting semaphore with N allows N concurrent holders (e.g. a pool of N buffers); a binary semaphore resembles a mutex but has no owner, so any task can signal it. In the Linux kernel, prefer mutex for exclusion and completions for signaling.

A spinlock is a lock where a waiting CPU busy-loops until the lock becomes free, instead of sleeping. It is used for very short critical sections and in contexts that cannot sleep (interrupt handlers, softirqs). In Linux, taking a spinlock disables preemption; the holder must not sleep. Variants like spin_lock_irqsave also disable local interrupts to prevent deadlock with an IRQ handler that takes the same lock. Spinning is only efficient if the hold time is shorter than the cost of sleeping and waking.

A deadlock is when a set of tasks each wait for a resource held by another task in the set, so none can proceed. All four Coffman conditions must hold: mutual exclusion (resources are held exclusively), hold and wait (tasks hold resources while waiting for more), no preemption (resources cannot be forcibly taken), and circular wait (a cycle in the wait-for graph). Breaking any one prevents deadlock; the most practical is a global lock ordering to prevent circular wait.

• Pipes and named pipes (FIFOs): one-way byte streams.
• Signals: asynchronous notifications carrying only a number.
• Shared memory (POSIX shm_open/mmap, System V, memfd): fastest, needs separate synchronization.
• Message queues (POSIX, System V): discrete messages.
• Sockets: Unix domain (local, can pass fds) and TCP/UDP (network).
• Netlink: kernel-userspace messaging.
• eventfd, futex, semaphores for notification and synchronization.

Android adds Binder as its main IPC.

Both are virtual filesystems generated by the kernel on the fly, with no storage behind them. /proc (procfs) exposes process information (/proc/<pid>/status, maps, fd) and kernel state (/proc/meminfo, /proc/interrupts, /proc/cmdline). /sys (sysfs) exposes the device model: devices, drivers, buses and classes, with one value per attribute file, many writable (e.g. cpufreq governor, power controls). debugfs (/sys/kernel/debug) is for developer-only debug data.

dmesg prints the kernel ring buffer: messages written with printk by the kernel and drivers, such as boot progress, driver probe results, errors, warnings, oopses and SELinux denials. Each line has a timestamp since boot and a log level. dmesg -w follows new messages. On Android, logcat -b kernel shows the same messages; logs from the previous boot after a crash are in pstore (/sys/fs/pstore/console-ramoops-0).

kmalloc returns memory that is physically contiguous and in the kernel's linear map. It is fast, suitable for DMA, but large requests can fail when memory is fragmented. vmalloc returns memory that is only virtually contiguous, built from scattered physical pages mapped into a separate kernel virtual range. It suits large buffers the CPU alone uses but is slower (page-table setup, more TLB pressure) and not directly DMA-able. kvmalloc tries kmalloc and falls back to vmalloc. In atomic context use kmalloc(..., GFP_ATOMIC); vmalloc may sleep.

Copy-on-write lets two processes share the same physical pages until one of them writes. After fork(), the kernel copies the page tables and marks shared pages read-only in both. A write triggers a page fault; the kernel then copies just that page, gives the writer a private writable copy, and resumes. This makes fork() fast and memory-efficient, especially when followed by exec(). Android's Zygote relies on it: apps forked from Zygote share its preloaded classes and resources.

fork() creates a new child process that is a copy of the parent (same code, same memory contents via COW, duplicated file descriptors). It returns twice: 0 in the child, the child's PID in the parent. exec() replaces the current process's program image with a new executable, keeping the PID and (unless close-on-exec) open file descriptors. The classic pattern for launching a program is fork then exec in the child, with the parent calling wait().

They are the three providers almost every SoC driver depends on. clk gates and rates clocks (clk_prepare_enable, clk_set_rate); prepare may sleep, enable is the cheap gate. pinctrl muxes SoC balls to a function (UART, I2C, GPIO) and sets pull and drive; GPIO via gpiod is only for lines used as GPIO. regulator turns PMIC rails on and sets voltage. In probe() the usual order is pinctrl default, enable supplies, enable clocks, deassert reset, then talk to the hardware. First dumps: clk_summary, regulator_summary, /sys/kernel/debug/pinctrl/.

It changed twice. Early Android used a userspace FUSE daemon (sdcard) so emulated storage could enforce per-app permissions on top of /data/media. Android 6 replaced that with sdcardfs, an in-kernel wrapfs, for much lower overhead. Android 11 removed sdcardfs and went back to FUSE, this time driven by MediaProvider, to implement scoped storage and richer permission checks. On a current device, /sdcard is FUSE; saying "we use sdcardfs" is an Android 6–10 answer.

Treble (Android 8) is an architecture that separates the Android framework (system partition) from the vendor implementation (vendor partition) through stable, versioned HAL interfaces (HIDL originally, stable AIDL now) over Binder. VINTF manifests and compatibility matrices check that system and vendor are compatible. The benefit is that Google or the OEM can update the framework without the SoC vendor rebuilding its code, which speeds up OS upgrades and is verified by running a Generic System Image with VTS/CTS.

GKI (Generic Kernel Image) is Google's approach to fragmentation in Android kernels. Google builds and signs one core kernel per Android release and kernel version from the Android Common Kernel. SoC and board support is delivered as vendor modules loaded from vendor_boot and vendor_dlkm, and they may use only the stable Kernel Module Interface (KMI). This lets the core kernel receive security fixes without vendors rebasing a fork. GKI 1.0 appeared optionally in Android 11; GKI 2.0 is required for new devices launching with Android 12+ on kernel 5.10+. The OEM still tests and ships the combined image; "independent update" means the core binary and vendor modules can move on different cadences as long as the KMI holds.

Use a spinlock for very short critical sections and whenever the code cannot sleep: interrupt handlers, softirqs, or data shared with them. The holder must not sleep, and preemption is disabled while it is held. Use a mutex for longer sections in process context where waiting by sleeping is cheaper than spinning; the holder may sleep (e.g. do I2C transfers or GFP_KERNEL allocations). Calling a sleeping function while holding a spinlock causes "BUG: scheduling while atomic". If data is shared with a hard IRQ handler, process context must use spin_lock_irqsave so the IRQ cannot interrupt the holder on the same CPU and deadlock.

• Mutex: sleepable, single owner, process context only; the default for protecting driver state touched only from syscalls, workqueues or threaded IRQs.
• Spinlock: non-sleeping, usable in IRQ context (with _irqsave), short sections; use in the top half and for any data shared with it.
• Semaphore: counting, no owner, up() can be called from IRQ; rarely used now for exclusion; completions or wait queues are preferred for signaling, and mutexes for exclusion.

Rule: in a hard IRQ handler use a spinlock; for long-held resources use a mutex in process context.

Softirqs are statically defined and reserved for core high-rate subsystems (network RX/TX, block completion, timers, RCU); drivers do not add new ones. Tasklets are dynamically created on top of softirqs; they cannot sleep and the same tasklet never runs concurrently on two CPUs; they are deprecated in favor of alternatives. Workqueues run in kernel worker threads in process context, so they can sleep, take mutexes and do I2C/SPI transfers. Threaded IRQs give the interrupt its own kernel thread, also able to sleep, and are the modern default for device drivers, especially for devices on slow buses. Choose by whether you need to sleep and how latency-sensitive the work is.

1. The bootloader passes the DTB; the kernel unflattens it into device_node structures.
2. The OF core creates platform_devices for nodes under simple buses with status = "okay"; bus controllers (I2C, SPI) create child devices when they probe.
3. Drivers register with their bus (platform_driver_register), providing an of_match_table.
4. When a device and driver are both present, the bus's match() compares compatible strings; on a match the core calls probe().
5. Probe returns 0 (bound), -EPROBE_DEFER (retried later when dependencies appear), or an error.
6. For modules, udev/ueventd uses the MODALIAS from MODULE_DEVICE_TABLE to autoload the right module.

Drivers often depend on other providers: clocks, regulators, GPIO or pinctrl controllers, PHYs. If a provider has not probed yet, the consumer's probe() gets -EPROBE_DEFER from calls like devm_clk_get() and returns it. The driver core puts the device on a deferred list and retries whenever another driver binds successfully. This avoids fragile hand-tuned init ordering. /sys/kernel/debug/devices_deferred lists devices still waiting, which is a quick way to find a missing dependency. fw_devlink now orders probing automatically from DT phandles, reducing deferral churn.

devm_* functions (e.g. devm_kzalloc, devm_ioremap_resource, devm_request_irq, devm_clk_get) register the resource with the device, and the driver core releases them automatically, in reverse order, when probe fails or the device is unbound. This eliminates most error-path cleanup code and leaks. The caution: release order matters, for example a devm-requested IRQ is freed after remove() runs, so the handler must not use resources remove() already freed.

Implement a struct file_operations with the callbacks you need (open, read, write, unlocked_ioctl, poll, mmap, release), then register it. The simplest way is a miscdevice with a dynamic minor, which creates /dev/name; the fuller way is alloc_chrdev_region + cdev_init/cdev_add + class_create/device_create. In read/write, move data with copy_to_user/copy_from_user, return bytes transferred or a negative errno, protect shared state with a mutex, and support blocking I/O with a wait queue plus poll.

static ssize_t foo_read(struct file *f, char __user *buf, size_t n, loff_t *off)
{
    struct foo *foo = f->private_data;
    size_t len;
    if (wait_event_interruptible(foo->wq, foo->count))
        return -ERESTARTSYS;
    mutex_lock(&foo->lock);
    len = min(n, foo->count);
    if (copy_to_user(buf, foo->data, len)) { mutex_unlock(&foo->lock); return -EFAULT; }
    foo->count -= len;
    mutex_unlock(&foo->lock);
    return len;
}

A wait queue sleeps until a condition becomes true: wait_event_interruptible(wq, foo->ready), with the IRQ or worker setting the flag and calling wake_up. Use it for recurring "data available" or "state changed" waits. Prefer the interruptible or killable variant so signals (including SIGKILL) can break the sleep; a plain wait_event is how tasks get stuck in D state. A completion is a one-shot "this finished" event (wait_for_completion / complete), ideal for "wait until this DMA or probe helper is done". Do not use a binary semaphore for that pattern; completions are the modern API.

The Contiguous Memory Allocator reserves a region of physical memory that can be lent to movable pages when idle and reclaimed for large physically contiguous DMA buffers when a device needs them (camera, display, codecs). Use it when a device cannot use the IOMMU / scatter-gather and needs big contiguous buffers that kmalloc cannot satisfy under fragmentation. Cost: that memory is less flexible for the rest of the system. On modern SoCs with an SMMU, prefer IOMMU + scatter-gather and keep CMA small.

ioctl(fd, cmd, arg) is a system call for device-specific operations that do not fit read/write, such as configuring a mode, starting a DMA, or querying capabilities. Commands are encoded with _IO, _IOR, _IOW, _IOWR macros that include a magic number, command number, direction and argument size. The driver implements unlocked_ioctl (and compat_ioctl for 32-bit apps on a 64-bit kernel), validates the command, and copies arguments with copy_from_user. For simple attributes, sysfs files are often cleaner; for streaming data, read/write or mmap.

Protect it with a spinlock. In the hard IRQ handler use spin_lock() (interrupts for that line are already masked on this CPU). In process context use spin_lock_irqsave(&lock, flags) / spin_unlock_irqrestore(), which disables local interrupts while holding the lock. Otherwise, if the IRQ fires on the same CPU while process context holds the lock, the handler spins forever. For single counters or flags, atomics may suffice; for producer/consumer data, a lock-free kfifo with one reader and one writer works without locking. Keep the locked region short and never sleep inside it.

An edge-triggered interrupt fires on a signal transition (rising or falling). It is raised once per event; if another edge happens while the first is being handled and the controller does not latch it, it can be missed. A level-triggered interrupt stays asserted as long as the line is at the active level; the handler must clear the source in the device, otherwise the interrupt fires again immediately (an interrupt storm). Level triggering is safer for shared lines. The DT interrupts property specifies the type, and getting it wrong is a common bring-up bug.

They are allocation flags. GFP_KERNEL is the normal flag: the allocator may sleep to reclaim memory, write back pages, or compact. Use it in process context when no spinlock is held and preemption is enabled. GFP_ATOMIC never sleeps and may use emergency reserves; use it in interrupt handlers, softirqs, timers, or while holding a spinlock. Atomic allocations fail more easily, so always handle NULL and avoid large atomic allocations. GFP_NOIO/GFP_NOFS prevent recursion into I/O or filesystem code during reclaim.

The buddy allocator manages physical memory as blocks of 2^order contiguous pages per zone. To allocate, it splits a larger free block into two "buddies" until it has the requested size; on free, it merges a block with its free buddy back into a larger block. That keeps external fragmentation manageable; /proc/buddyinfo shows free blocks per order. The slab allocator (SLUB in modern kernels) sits on top: it takes pages from the buddy allocator and carves them into caches of same-sized objects (e.g. task_struct, dentry), giving fast allocation, reduced internal fragmentation and per-CPU caching. kmalloc uses generic size-class slab caches. /proc/slabinfo and slabtop show usage.

With DMA, the driver gives the device a bus address and length; the device reads or writes RAM directly and interrupts when done, freeing the CPU. Problems arise because the CPU caches data: if the bus is not cache-coherent, the device may read stale memory (CPU's writes still in cache) or the CPU may read stale cache lines after the device wrote RAM. The DMA API handles this: streaming mappings (dma_map_single) clean caches before a device read and invalidate before a CPU read; coherent allocations (dma_alloc_coherent) use memory that is uncached or hardware-coherent. The API also translates to device addresses through the IOMMU if present.

An IOMMU is an MMU for devices: it translates device-visible addresses (IOVAs) to physical addresses using per-device page tables. Benefits: devices can use large buffers that are virtually contiguous but physically scattered (no need for big contiguous allocations); devices can be isolated so a buggy or malicious device cannot DMA into arbitrary memory; and 32-bit devices can reach memory above 4 GB. On ARM it is called SMMU. Faults show up as "smmu context fault" messages, which usually indicate a driver using an unmapped or freed buffer.

1. The MMU fails to translate an address (not present) or detects a permission violation and raises an exception with the faulting address (FAR_EL1 on ARM64) and cause.
2. The kernel's fault handler finds the VMA containing the address.
3. If there is no VMA, or the access violates VMA permissions, it sends SIGSEGV to the user process (or oopses if in kernel mode without an exception fixup).
4. If valid, handle_mm_fault() resolves it: allocate a zeroed anonymous page, map a page-cache page (reading from storage if needed = major fault), swap in from ZRAM, or break COW by copying.
5. It updates the PTE, and the faulting instruction is restarted.

A minor fault is resolved without I/O: the page is already in memory (page cache, shared with another process) or can be created (zero page, COW copy); the kernel only updates page tables. A major fault requires reading data from storage or swap, so the task sleeps for I/O and it is much slower (microseconds vs milliseconds). /proc/<pid>/stat fields and /usr/bin/time -v show counts. Many major faults during app start or scrolling indicate memory pressure or cold file cache.

CFS tracks a virtual runtime (vruntime) per task: actual CPU time scaled inversely by the task's weight (derived from its nice value, with each nice level about a 10% share change). Runnable tasks sit in a per-CPU red-black tree ordered by vruntime; the scheduler picks the leftmost (smallest vruntime), i.e. the task that has received the least fair share. A running task is preempted when another task's vruntime is smaller by more than a granularity. Waking tasks get a vruntime close to the minimum so they run soon, which favors interactive tasks. Periodic load balancing moves tasks between CPUs. Since Linux 6.6, EEVDF replaces CFS's pick logic, adding virtual deadlines for better latency control.

• SCHED_DEADLINE: EDF with runtime/period/deadline reservations; highest priority.
• SCHED_FIFO: real-time priority 1-99; runs until it blocks or a higher-priority RT task arrives.
• SCHED_RR: like FIFO with a time slice among tasks of equal priority.
• SCHED_NORMAL/SCHED_OTHER: fair scheduling with nice -20 to +19.
• SCHED_BATCH: CPU-bound, less frequent preemption.
• SCHED_IDLE: runs only when nothing else wants the CPU.

Set with sched_setscheduler() or chrt; RT tasks are throttled to 95% of CPU by default so they cannot hang the system completely.

Both save and restore registers and kernel stack. A process switch additionally changes the address space: it loads a new page table base (TTBR0 on ARM64, CR3 on x86), which may invalidate TLB entries (ASIDs/PCIDs mitigate a full flush) and leaves caches holding the old process's data. Threads of the same process share the address space, so page tables stay the same and the TLB and caches remain warm. Typical costs are a few microseconds for a process switch and noticeably less for a thread switch, plus indirect cache-miss costs afterwards.

fork() creates a child with a copy-on-write copy of the parent's address space and duplicated file descriptors. vfork() creates a child that borrows the parent's address space without copying page tables; the parent is suspended until the child calls exec() or _exit(). It is faster but dangerous if the child modifies memory; COW made it mostly unnecessary (though posix_spawn may use it internally). clone() is the underlying primitive: flags select what is shared (CLONE_VM, CLONE_FILES, CLONE_FS, CLONE_SIGHAND, CLONE_THREAD, namespaces). fork and pthread_create are both implemented with it.

Priority inversion happens when a high-priority task waits for a lock held by a low-priority task, and a medium-priority task preempts the low-priority one, so the high-priority task is effectively blocked by a less important task for an unbounded time. The fix is priority inheritance: the lock holder temporarily inherits the highest waiter's priority until it releases the lock. Linux implements it in rt_mutex and PI futexes (PTHREAD_PRIO_INHERIT); PREEMPT_RT converts most kernel locks to rt_mutexes. Priority ceiling is an alternative. The Mars Pathfinder resets in 1997 are the classic example; Android's Binder also propagates caller priority to the server thread.

Prevention: define and document a global lock order and always acquire in that order; keep lock scopes small; avoid calling unknown code (callbacks) with locks held; use trylock with back-off when ordering is impossible; avoid holding locks across sleeping calls that may need the same lock (e.g. flush_work). Detection: lockdep in debug kernels records lock-dependency chains and reports potential cycles even if the deadlock never actually happened; hung-task and soft-lockup detectors flag real hangs; echo w > /proc/sysrq-trigger dumps blocked tasks; in a ramdump, map lock owners to waiting tasks. In userspace, /data/anr traces and Java lock holders show similar cycles.

Read-Copy-Update is a synchronization mechanism for read-mostly data. Readers enter a read-side critical section (rcu_read_lock()) that costs almost nothing and never blocks; they access data through rcu_dereference(). A writer makes a new copy of the element, updates it, publishes it atomically with rcu_assign_pointer(), then waits for a grace period (synchronize_rcu(), or call_rcu/kfree_rcu asynchronously) until all readers that might see the old version have finished, and only then frees it. It scales extremely well for readers on many CPUs. Use it for lookup tables, lists and configuration that change rarely; writers still need their own lock among themselves.

Both compilers and CPUs reorder memory operations for performance. On a single CPU this is invisible, but on SMP another CPU may observe writes in a different order. ARM is weakly ordered, so this happens in practice. Memory barriers constrain ordering: smp_wmb() orders stores, smp_rmb() orders loads, smp_mb() orders both; acquire/release operations (smp_load_acquire, smp_store_release) give one-way ordering suited to flag-and-data patterns. For device registers, readl/writel include the needed I/O barriers and dma_wmb() orders descriptor writes before ringing a doorbell. Locks and atomic RMW operations with return values already imply barriers.

volatile only tells the compiler not to cache the variable in a register and not to optimize away or reorder accesses to that variable relative to other volatile accesses. It does not make operations atomic (x++ is still read-modify-write), does not prevent the CPU from reordering memory operations, and does not order non-volatile accesses around it. Use C11 atomics, kernel atomics, locks, or READ_ONCE/WRITE_ONCE plus barriers. volatile is appropriate for memory-mapped I/O registers, variables modified by signal handlers (volatile sig_atomic_t) and values changed by a debugger.

Binder is Android's IPC and RPC mechanism, implemented as a kernel driver (/dev/binder). A client's proxy marshals arguments into a Parcel and calls ioctl(BINDER_WRITE_READ); the driver copies the data once into the server's mmap'd buffer and wakes a thread from the server's Binder thread pool, which runs the method and replies. Compared to pipes or sockets, it provides: caller UID/PID filled in by the kernel (unforgeable identity for permission checks), object references with reference counting across processes, death notifications, priority inheritance, fd passing, synchronous and one-way calls, and a name service (servicemanager). It is the backbone of app-framework and framework-HAL communication.

A pipe needs two copies: from the writer's buffer into a kernel buffer, then from the kernel buffer into the reader's buffer. Binder needs one copy: the driver copies from the sender's user memory directly into a buffer that is mmap'd into the receiver, so the receiver reads it without a second copy. Shared memory needs zero copies after setup, since both processes map the same physical pages, but it requires separate synchronization (futex, semaphore, eventfd). For large data such as graphics buffers, Android passes a DMA-BUF or memfd file descriptor over Binder to get zero-copy with Binder's security.

The page cache holds file data in RAM in page-sized units, indexed per file. Reads hit the cache if possible; misses read from storage (with read-ahead for sequential access). write() copies data into the page cache and marks pages dirty, then returns; background writeback threads flush dirty pages to storage later based on age and dirty ratio thresholds. fsync()/fdatasync() force a file's dirty data (and metadata) to stable storage and issue a device cache flush. O_DIRECT bypasses the cache. Clean cache pages are the first thing reclaimed under memory pressure.

An inode is the filesystem structure that holds a file's metadata (type, permissions, owner, size, timestamps, link count) and where its data blocks are, but not its name. Names live in directory entries that map a name to an inode number. A hard link is another directory entry pointing to the same inode; the file exists until the link count drops to zero and no process has it open; hard links cannot span filesystems or (normally) point to directories. A symbolic link is a separate small file containing a path; it can cross filesystems and point to directories, but becomes dangling if the target is removed.

/data gets many small random writes. f2fs (Flash-Friendly File System) is log-structured: it turns random writes into sequential ones, which suits NAND flash and its flash translation layer, improves write performance and reduces wear; it also supports file-based encryption and compression. /system and /vendor are read-only and verified by dm-verity. erofs (Enhanced Read-Only File System) compresses data in fixed-size output blocks, saving significant space while keeping fast random reads, and its immutable layout fits verified boot. ext4 remains in use on many devices and was the historical default for both.

read() copies data from the page cache into a user buffer on each call; it is simple and efficient for sequential streaming. mmap() maps the file's page-cache pages directly into the process's address space; data is loaded lazily by page faults and accessed with normal pointers, avoiding the extra copy and suiting random access to large files and sharing between processes. Costs of mmap: page-fault overhead, TLB pressure, SIGBUS if the file shrinks, and harder error handling. Drivers also implement mmap to expose device memory or DMA buffers to user space.

When suspend is requested (on Android, autosleep once no wakelocks are held), the kernel freezes user tasks and freezable kernel threads, calls every driver's suspend callbacks (suspend, suspend_late, suspend_noirq) in child-before-parent order, takes secondary CPUs offline, and the last CPU enters a deep power state through PSCI firmware while RAM stays in self-refresh. A configured wake interrupt (RTC, button, modem, sensor hub) triggers resume in reverse order. A wakeup source is a kernel object (a kernel wakelock) that, while active, prevents suspend; drivers use pm_stay_awake/pm_relax or __pm_wakeup_event. If any driver's suspend callback fails, the whole suspend is aborted.

Runtime power management lets individual devices power down while the system is running. Drivers call pm_runtime_get_sync() before using the hardware and pm_runtime_put() / pm_runtime_put_autosuspend() when done; the core keeps a usage count and, when it reaches zero (after an optional autosuspend delay), calls the driver's runtime_suspend to gate clocks and regulators, and runtime_resume when needed again. Power domains (genpd) can turn off a whole domain when all its devices are idle. Runtime PM is key to low active and idle current on mobile devices.

When a driver registers a device, the kernel sends a uevent over netlink containing the device's name, major/minor numbers and attributes. On Android, ueventd (part of init) receives it, creates the /dev node, and sets owner, group and mode according to ueventd.rc files (e.g. /dev/foo 0660 system input) and the SELinux label from file_contexts. It also handles firmware loading requests. If a HAL cannot open a node, check the node's permissions, SELinux label, and avc: denied messages.

An oops is the kernel's report of a serious error (such as a bad memory access) in kernel code: it prints registers and a stack trace, kills the offending task, and the kernel may keep running, although it may be in an inconsistent state (for example a lock held forever) and is marked tainted. A panic is a fatal error after which the kernel stops; it halts or reboots after panic_timeout. The kernel default is panic_on_oops=0 (continue). Android typically sets /proc/sys/kernel/panic_on_oops to 1 from init (and ACK builds often also enable CONFIG_PANIC_ON_OOPS), so an oops becomes a panic and a ramdump or reboot rather than a half-dead system.

Process context (syscalls, kernel threads, workqueue items, threaded IRQ handlers) may sleep, as long as no spinlock is held and preemption or interrupts are not disabled. Hard IRQ, softirq and tasklet context, and any code holding a spinlock, must not sleep. Interrupt context borrows whatever task was running; it has no task of its own that the scheduler could put to sleep and wake later, and sleeping would leave the interrupted task and possibly held locks stuck. With a spinlock held, preemption is disabled, so sleeping could deadlock if the next task needs the same lock. might_sleep() annotations with CONFIG_DEBUG_ATOMIC_SLEEP catch violations at runtime.

Modern Linux uses queued spinlocks (qspinlock) on arm64; older kernels used ticket spinlocks. A ticket lock has "next" and "owner" counters: a CPU atomically takes a ticket and spins until owner equals its ticket, which gives FIFO fairness. Qspinlock keeps a 32-bit word for the fast uncontended path and, under contention, queues waiters in per-CPU MCS nodes, so each spins on its own cache line instead of hammering the shared lock line. Atomics use ARMv8.1 LSE instructions (like CAS, LDADD) or LDXR/STXR exclusive loops, and waiters use WFE to reduce power while spinning; acquire/release semantics provide the memory ordering. spin_lock also disables preemption.

PREEMPT_RT (fully merged in Linux 6.12) makes almost all kernel code preemptible to achieve bounded latency. Key changes: most spinlock_t locks become sleeping rt_mutexes with priority inheritance (raw_spinlock_t remains true spinning for the few places that need it); interrupt handlers are forced into threads so they can be prioritized and preempted; softirqs run in thread context; and long non-preemptible sections are broken up. The trade-off is slightly lower throughput for much better worst-case latency. Drivers must use raw_spinlock_t only where truly needed (e.g. inside irqchip code).

EAS adds an energy model (power and capacity per performance state per CPU cluster, from the DT or firmware) to the scheduler. When a task wakes, EAS estimates its utilization with PELT (per-entity load tracking) and evaluates candidate CPUs: which placement meets the task's capacity needs with the lowest total energy, considering the frequency each cluster would have to run at. Small tasks stay on little cores; heavy tasks go to big cores. It works with the schedutil governor, which sets frequency from utilization. It is active only when the system is not overutilized; otherwise normal load balancing takes over. Android steers it with uclamp (min/max utilization clamps per cgroup, e.g. boosting top-app) and cpusets.

Both aim for fair CPU sharing weighted by nice. CFS picks the task with the smallest vruntime and relied on heuristics (wakeup preemption granularity, sleeper credits) to give interactive tasks low latency. EEVDF (Earliest Eligible Virtual Deadline First, default since 6.6) computes a "lag" per task (how much service it is owed); only tasks with non-negative lag are eligible, and among them it picks the one with the earliest virtual deadline, where the deadline is eligible time plus requested slice divided by weight. A task requesting a shorter slice gets an earlier deadline and so runs sooner, without getting more total CPU. This replaces many CFS heuristics with a cleaner model and gives latency-sensitive tasks a proper knob.

Each memory zone has min/low/high watermarks. When free pages drop below low, kswapd wakes and reclaims in the background until high is reached: dropping clean page-cache pages, writing back dirty pages, and swapping anonymous pages (to ZRAM on Android). The LRU lists (active/inactive, or multi-gen LRU in newer kernels) choose victims. If an allocation cannot be satisfied, the allocating task does direct reclaim and compaction, causing latency stalls. If that fails, the OOM killer selects a process by oom_score (memory usage adjusted by oom_score_adj) and kills it. Android avoids reaching that point: LMKD monitors PSI memory stall levels and kills cached/background apps by oom_score_adj set by ActivityManager, keeping the foreground responsive.

Pressure Stall Information (/proc/pressure/cpu, memory, io) reports the percentage of time over 10, 60 and 300 second windows that some tasks ("some") or all non-idle tasks ("full") were stalled waiting for that resource. Unlike free-memory numbers, it measures the actual impact of pressure. Userspace can register triggers such as "notify me when memory some-stall exceeds 70 ms in a 1 s window". Android's LMKD registers PSI triggers; when they fire, it picks a victim with the highest oom_score_adj above a threshold that depends on the pressure level and kills it, replacing the old in-kernel lowmemorykiller that used free-memory thresholds.

ARM64 has two translation table base registers: TTBR0_EL1 for the lower range (user space, per process, tagged with an ASID) and TTBR1_EL1 for the upper range (kernel, shared). With 4 KB pages and 48-bit virtual addresses, translation uses four levels (9 bits each plus a 12-bit offset); block mappings at level 1 or 2 give 1 GB or 2 MB pages. The kernel has a linear map of all physical RAM, a vmalloc area, a fixmap, module space and so on. On a context switch only TTBR0 and the ASID change. Security features include PAN (kernel cannot access user memory without explicit uaccess), PXN/UXN (execute-never), KASLR, and on some systems KPTI (unmapping the kernel while in user space).

copy_from_user/copy_to_user (and get_user/put_user) first check that the pointer range lies in user space (access_ok), then perform the copy with privileged access temporarily enabled (on ARM64, clearing PAN with uaccess_enable). If the user page is not present, a normal page fault occurs and may sleep to bring it in, which is why these functions can sleep. If the address is invalid, the fault handler finds the faulting instruction in the exception table and jumps to a fixup that makes the function return the number of bytes not copied, so the driver returns -EFAULT instead of oopsing. Checks against kernel objects (hardened usercopy) catch overflows.

Implement file_operations.mmap(struct file *f, struct vm_area_struct *vma). For device registers or physically contiguous memory, validate the requested size and offset, set appropriate page protection (e.g. pgprot_noncached or pgprot_writecombine for MMIO), and call remap_pfn_range() or io_remap_pfn_range(). For DMA memory from dma_alloc_coherent, use dma_mmap_coherent(). For scattered pages, use vm_insert_page or a vm_operations_struct with a fault handler that supplies pages on demand. Never allow mapping beyond the buffer, and keep the buffer alive until the VMA is closed (reference counting in vm_ops->open/close). For sharing between devices, exporting a DMA-BUF is usually better.

The Kernel Module Interface is the set of exported kernel symbols plus the layouts of the data types they use that GKI vendor modules may rely on. For each GKI branch (e.g. android14-6.1), Google freezes the KMI after a stabilization period: symbols listed in the vendor symbol lists are guaranteed, and ABI tooling compares every build against the frozen ABI representation to reject incompatible changes. Vendors compile modules against the GKI source and headers; at load time, symbol CRCs (CONFIG_MODVERSIONS) must match. When vendors need a new symbol or a hook into core code, they upstream it to the Android Common Kernel (adding symbols or vendor hooks based on tracepoints) rather than patching the core kernel locally.

With GKI, vendors cannot modify core kernel code, but they sometimes need to change behavior in scheduler, memory or other core paths (e.g. custom task placement or thermal decisions). Vendor hooks are special tracepoints (DECLARE_HOOK / DECLARE_RESTRICTED_HOOK) added to the Android Common Kernel at agreed points; vendor modules register handlers that run there. Restricted hooks can sleep and attach once, normal hooks behave like regular tracepoints. They provide a controlled extension point that keeps the core kernel generic while preserving a stable KMI. They must be proposed and accepted into ACK.

A peripheral asserts a line into the GIC distributor (SPI for shared peripherals, PPI for per-CPU sources like the arch timer, SGI for inter-processor interrupts, LPI for MSI-style message interrupts via the ITS). The distributor routes it to a target CPU's redistributor/CPU interface based on priority and affinity. The CPU takes an IRQ exception, reads the interrupt ID from the acknowledge register, and Linux's generic IRQ layer maps the hardware ID to a Linux IRQ number through an irq domain, then calls the flow handler (e.g. handle_fasteoi_irq) and the driver's handler, and finally signals end of interrupt. irq domains also chain controllers: a GPIO controller or PMIC can be an irqchip whose domain translates its pins into Linux IRQs cascaded under a GIC interrupt, which is what interrupt-parent in the DT expresses.

Symptoms include rare corrupted state, list corruption (list_add corruption warnings), refcount underflows and crashes that move around. Tools: KCSAN detects data races dynamically by watching concurrent unsynchronized accesses; KASAN catches the resulting use-after-free; lockdep verifies locking assumptions (use lockdep_assert_held() in functions that require a lock). Review which contexts touch the data: process, softirq, hard IRQ, other CPUs. Fix by protecting all accesses with the same lock of the right type, converting counters to atomic_t/refcount_t, using RCU for read-mostly structures, or READ_ONCE/WRITE_ONCE for intentionally lockless flags with appropriate barriers.

A futex is a 32-bit integer in user memory plus a kernel wait queue keyed by its address. Acquiring an uncontended pthread_mutex is a single atomic compare-and-swap in user space, with no syscall. Only when the lock is contended does the thread mark it as contended and call futex(FUTEX_WAIT), which sleeps if the value is still as expected (avoiding lost-wakeup races). The unlocking thread, seeing the contended state, calls futex(FUTEX_WAKE). PI futexes (FUTEX_LOCK_PI) store the owner TID so the kernel can apply priority inheritance. Java monitors and ART locks are also built on futexes.

Latency comes from regions where interrupts are disabled (local_irq_save, spin_lock_irqsave), long hard IRQ handlers of other devices, higher-priority interrupts, softirq processing, preemption-disabled sections delaying threaded handlers, CPU idle exit latency from deep C-states, and frequency ramp-up. Measure with the ftrace irqsoff and preemptoff tracers (report the maximum disabled section and its call site), cyclictest for scheduling latency, IRQ and sched events in Perfetto, or a GPIO toggle measured on an oscilloscope for end-to-end latency. Fix by shortening disabled regions, moving work into threaded handlers, adjusting IRQ affinity, raising the priority of the IRQ thread, or limiting deep idle states when latency matters.

With CONFIG_FUNCTION_TRACER, the compiler inserts a call (or NOPs with patchable entries) at the start of every function; at runtime these are NOPs and ftrace patches selected sites to jump to the tracer, so overhead is near zero when disabled. The function_graph tracer also hooks returns to record durations. Tracepoints are static markers placed in code (TRACE_EVENT) that call registered probes via a static key when enabled, recording structured events into per-CPU lockless ring buffers exposed in tracefs. Kprobes dynamically insert a breakpoint at almost any instruction and run a handler; fprobes/fentry and eBPF programs can attach to these points too. Perfetto reads the same ftrace ring buffers plus userspace atrace markers.

Load the dump with the matching vmlinux (and module symbols) into crash or Trace32. First, read the kernel log buffer (log) for the last messages, watchdog reports or lockup warnings. Check each CPU's backtrace (bt -a): is one spinning on a lock, stuck with IRQs disabled, or in a tight loop? List tasks (ps) and focus on those in D state (foreach UN bt) to see what they wait on. For a mutex, inspect its owner field (struct mutex) and follow the owner's stack to find a lock cycle. Examine relevant driver structures with struct, memory usage with kmem -i, and runqueues with runq. The goal is to identify the stuck resource and which task or CPU holds it.

They were three implementations of the kernel's object allocator. SLAB was the original, with complex per-CPU and per-node queues; SLOB was a minimal allocator for tiny systems; SLUB simplified the design with per-CPU slabs and little metadata and became the default. SLOB and SLAB have since been removed, so SLUB is the only one in current kernels. With slub_debug (options like F sanity checks, Z red zones, P poisoning, U user tracking), SLUB detects buffer overruns into red zones, use-after-free via poison patterns, and double frees, and records allocation/free call sites; KASAN and KFENCE provide more precise detection.

ION was an Android-specific allocator (/dev/ion) that allocated buffers from heaps (system, carveout, CMA) and exported them as DMA-BUF file descriptors. Its single ioctl interface with heap IDs and flags was hard to keep ABI-stable and vendors added many incompatible changes. Mainline Linux adopted DMA-BUF heaps instead: each heap is its own character device under /dev/dma_heap/ (e.g. system, linux,cma, vendor heaps as modules), with a simple allocation ioctl returning a DMA-BUF fd. ION was removed from mainline in Linux 5.11 and deprecated in Android's GKI kernels; Android 12+ devices use DMA-BUF heaps through libdmabufheap. Cache maintenance and sharing semantics come from the standard DMA-BUF framework.

The vendor provides a device manifest listing HALs it implements (name, interface, version, instance) plus kernel requirements; the framework provides a compatibility matrix listing HALs and versions it needs and kernel config requirements. The build system checks that the vendor manifest satisfies the framework matrix and vice versa (the device matrix vs framework manifest), and VintfObject checks again at OTA time and boot. Failures happen when the framework requires a newer HAL version than the vendor provides, a required HAL is missing, a HAL is declared but not registered at runtime, or kernel configs/versions do not meet the matrix. Symptoms include OTA rejection, VTS failures, or a service failing to find a HAL.

The logo means the bootloader worked, so the problem is in the kernel or userspace. First get the kernel log: UART console, or /sys/fs/pstore/console-ramoops-0 after a reboot. Look for a panic, a driver hanging in probe, or a root mount failure. If the kernel reached init, check logcat -b all (via adb if available) for a critical service crash-looping (init: Service ... restarting), avc: denied blocking a daemon, a failed mount of /data or /vendor, or system_server/zygote crashing. Then bisect: try the other A/B slot or a known-good build, and diff DT, defconfig, modules and rc changes. For the full stage-by-stage method, see the Android boot page.

1. Read the header: faulting virtual address (a small value like 0x10 means NULL plus a struct field offset), the CPU, the task (Comm), taint flags.
2. Look at pc (e.g. foo_probe+0x48/0x1a0 [foo]) and lr, and the call trace.
3. Symbolize: addr2line -e vmlinux for built-in code, or gdb foo.ko then list *(foo_probe+0x48) for modules; scripts/decode_stacktrace.sh for the whole trace.
4. Identify which pointer was NULL and why: missing error check (e.g. of_get_property returned NULL), a dependency not yet probed, drvdata not set before the IRQ fired, or an early IRQ calling into uninitialized state.
5. Fix and harden: check returns, request the IRQ only after initialization, use -EPROBE_DEFER. Reproduce with KASAN if the pointer might be a freed object.

1. Probe: dmesg | grep for the driver; check /sys/bus/i2c/devices/ for the device and whether it is bound to a driver. No device means the DT node is missing, disabled, or under the wrong bus; device but no driver means a compatible mismatch or module not loaded; deferred means a missing dependency (/sys/kernel/debug/devices_deferred).
2. Bus: i2cdetect -y <bus> to see if the address ACKs. No ACK means wrong address, bus, wiring, or the chip is unpowered or held in reset.
3. Power: regulator_summary to confirm the vdd-supply is enabled at the right voltage; check the reset GPIO polarity.
4. Pins and clocks: pinctrl state for SDA/SCL, controller clock in clk_summary; a logic analyzer or scope on the bus if still unclear.
5. Interrupt: correct GPIO, trigger type and pull; watch /proc/interrupts.

1. Is it suspending? dmesg for "PM: suspend entry/exit"; /sys/kernel/debug/suspend_stats for success and failure counts.
2. Suspend aborted? suspend_stats shows last_failed_dev and the failing step; a driver returning an error from suspend() blocks every attempt.
3. Who holds it awake? /sys/kernel/debug/wakeup_sources: sort by active_since/total_time; dumpsys power for app partial wakelocks.
4. Who wakes it? /proc/interrupts deltas across a sleep period, "wakeup IRQ" messages, Battery Historian or a Perfetto trace. Suspects: chatty sensor or modem IRQ, a GPIO configured as wake source with a floating line, an RTC alarm set too often, an app wakelock.
5. Also check residency of CPU idle states and whether peripherals are runtime-suspended; a rail left on can drain even while suspended.

Code called something that can sleep while in atomic context: holding a spinlock, with preemption or interrupts disabled, or in a hard IRQ, softirq or tasklet. Typical offenders are mutex_lock, kmalloc(GFP_KERNEL), msleep, copy_to_user, wait_event, or a blocking I2C/SPI transfer inside an IRQ handler. The splat's stack trace shows the sleeping call and the preempt count. Fix by moving the work to a threaded IRQ or workqueue, using GFP_ATOMIC, releasing the spinlock before the sleeping call, or using a mutex when all users are in process context. Prevent recurrence by running debug builds with CONFIG_DEBUG_ATOMIC_SLEEP and lockdep.

1. Start from the recorded reboot reason: ro.boot.bootreason, bootloader logs, PMIC reset reason registers, pstore console and panic logs. Never guess.
2. Kernel panic: decode the oops from pstore.
3. Watchdog bite: a CPU stopped petting the watchdog, typically stuck in a spinlock, an IRQs-disabled loop, or a deadlock. Collect a ramdump and inspect per-CPU stacks and lock owners.
4. Thermal: check thermal zone logs and trip points in /sys/class/thermal; a critical trip triggers shutdown.
5. Power: voltage droop or PMIC over-current under load (brown-out); correlate with CPU/GPU frequency peaks and battery state.
6. Reproduce with a stress test and bisect recent kernel, DT or firmware changes.

First decide kernel vs userspace using /proc/meminfo over time. Growing SUnreclaim (slab), KernelStack, VmallocUsed or DMA-BUF totals point to the kernel; growing app PSS (from dumpsys meminfo) points to userspace. For kernel leaks, use slabtop or /proc/slabinfo to find the growing cache, enable kmemleak to list unreferenced allocations with their stacks, and inspect /sys/kernel/debug/dma_buf/bufinfo for leaked buffers (often an fd not closed in userspace). For userspace leaks, use heapprofd in Perfetto, malloc debug, or HWASan, and showmap for mapping growth. Confirm the fix by trending MemAvailable in a long run.

Determine whether any stage prints. If even the bootloader is silent, check the bootloader's UART configuration, pinmux, clock, baud rate, and whether DDR and clocks come up at all (JTAG can tell you where the CPU is). If the bootloader prints but the kernel does not, add earlycon to the kernel command line with the correct UART type and base address, confirm stdout-path in /chosen and the UART DT node (address, clocks, pinctrl), and check that the serial driver is built in, not a module. If the kernel crashes before the console, early printk or reading the log buffer from memory via JTAG helps. As a last resort, toggle a GPIO or LED at known points.

1. Measure each stage: bootloader timestamps, kernel dmesg timestamps up to "Freeing unused kernel memory" and init start, init service start times, sys.boot_completed, and a Perfetto boot trace or bootchart.
2. Kernel: boot with initcall_debug to find slow initcalls; move non-critical drivers to modules loaded later; enable asynchronous probe; remove long msleeps and firmware load waits in probe; trim unused config.
3. Userspace: parallelize services, start non-critical HALs lazily, avoid serial wait_for_prop chains, reduce system_server and Zygote preload work, and check dexopt state.
4. Storage: tune read-ahead, use erofs compression, check that dm-verity and fs-verity are not bottlenecks.
5. Always attack the biggest measured contributor and re-measure after each change.

It is almost certainly in uninterruptible sleep (state D), waiting inside the kernel for something like I/O completion, a mutex, or a driver event that uses wait_event (not the interruptible or killable variant). Signals, including SIGKILL, are handled only when the task returns toward user space, so it cannot die until the wait ends. Check cat /proc/<pid>/stack and /proc/<pid>/wchan to see where it waits, or echo w > /proc/sysrq-trigger to dump all blocked tasks. The root cause is typically a hung storage device, a network filesystem, or a driver that never completes a request. Drivers should use wait_event_killable or timeouts where possible. A zombie (state Z) also cannot be killed, but that is fixed by its parent reaping it.

Capture a Perfetto trace with sched, freq, idle, irq and binder events along with app atrace markers. For the UI thread and RenderThread in the janky frames, check the thread state: Runnable but not Running means CPU contention or poor placement (look at which CPU and what else ran there); Running on a little core at low frequency means utilization estimation, uclamp or cpuset issues; Uninterruptible sleep means I/O or lock waits; Sleeping on Binder means the slowness is in another process. Also look for long IRQ or softirq bursts on the same CPU and thermal throttling (frequency caps). Fixes include correcting cgroup/cpuset assignment, uclamp boosts for top-app, IRQ affinity changes, or removing priority inversions.

The module was built against a kernel whose exported symbol CRCs (from CONFIG_MODVERSIONS) differ from the running kernel, meaning a function signature or a data structure used by that symbol changed, or the module was built against different headers or config. With GKI, this indicates either the module was not rebuilt against the new GKI release, or a KMI break (which the ABI tooling should prevent on a frozen branch). Check modinfo for vermagic, compare with uname -r, rebuild the module against the exact GKI kernel source and config, and ensure the symbols it uses are in the vendor symbol list. Never force-load with mismatched CRCs; that risks memory corruption.

1. Does the node exist? ls -lZ /dev/foo. If not, check that the driver registered the char/misc device and that ueventd processed it.
2. Permissions: owner, group, mode from ueventd.rc; is the HAL's user or group allowed?
3. SELinux: the node's label (from file_contexts) and avc: denied messages in dmesg or logcat for the HAL's domain; add correct type and allow rules in vendor policy (not permissive mode).
4. The driver's open() may itself return an error (e.g. runtime PM resume failing, device busy); check dmesg and the errno the HAL logs.

Suspend may have powered off the device's regulator or power domain, losing its register state, while the driver's resume callback does not restore it. Check whether the driver implements .suspend/.resume (and runtime PM callbacks) and re-initializes the hardware, restores pinctrl state (sleep vs default), re-enables clocks and regulators in the right order, and re-arms interrupts. Use dmesg with pm_debug_messages or initcall_debug to see resume callback order and errors, and compare register dumps before and after. Dependency ordering matters: the device must resume after its parent bus and supplies (device links help). Also verify the firmware or co-processor state if the peripheral runs its own firmware.

• Missing cache maintenance: the CPU reads stale cache lines because the buffer was not invalidated (dma_unmap or dma_sync_single_for_cpu not called) on a non-coherent system.
• The buffer shares cache lines with other data (not cache-line aligned), so CPU writes to neighbors write back stale data over what the device wrote.
• Using a stack or vmalloc buffer for DMA.
• The CPU touches the buffer while the device still owns it (ownership rule violated), or the buffer is freed or reused too early.
• Missing dma_wmb() before ringing a doorbell, so the device reads a descriptor before it is fully written.
• IOMMU mapping errors (check for SMMU faults).

Debug with CONFIG_DMA_API_DEBUG, which warns on API misuse, and add checksums at producer and consumer.

Enable the soft-lockup and hard-lockup detectors and the hung-task detector with panic options so the kernel dumps stacks and panics before the hardware watchdog fires, producing a ramdump. If the hang is a hard lockup (a CPU with interrupts disabled), the arm64 pseudo-NMI (or the SoC's watchdog pre-timeout / FIQ) can capture that CPU's stack. With the ramdump, check all CPU stacks: a CPU spinning in queued_spin_lock_slowpath points to a lock whose owner you then find; a loop in a driver with IRQs off points to a missing timeout while polling hardware. sysrq (l for all CPUs' backtraces) helps if the console is still responsive. Lockdep in a debug build can reveal the ordering problem before it reproduces.

1. Start from the vendor reference board: its bootloader config, DT and defconfig are ground truth; diff the schematics to list what changed.
2. Power and clocks first: PMIC rails, reset sequencing, crystals. Get a UART console working.
3. DDR configuration and training in the bootloader; then boot the kernel with a new board DT derived from the reference.
4. Storage (UFS/eMMC) so the root filesystem mounts; then USB for adb/fastboot.
5. Peripherals one at a time: display, touch, audio, sensors, cameras, modem, Wi-Fi/BT. For each: DT node, pinctrl, supplies, probe, validation via sysfs or test tools.
6. Power and thermal: suspend/resume, idle and active current, thermal zones; then performance tuning.
7. Keep a working ramdump path early so any crash can be analyzed.

Treat the vendor reference board, its BSP and defconfig as the source of truth, and read the SoC technical reference manual for the blocks you will touch (clocks, interrupt controller, the peripherals in question). Set up tooling first: a UART console, ramdump collection, symbol files and a fast build-flash loop, so every experiment produces evidence. Learn the code top-down from the DT: follow a node's compatible to its driver, then to the subsystem it registers with. Use dmesg, deferred-probe lists, ftrace function_graph on the driver, and diffs against the reference DT to understand behavior. Approach every bug the same way: which stage, what evidence, smallest reproducer, fix, regression test.

Trace with Perfetto or ftrace (sched_switch, sched_wakeup, irq, softirq events) around a glitch. Measure wakeup latency: time from wakeup to running. If it is Runnable for long, find what occupies the CPU: a higher-priority RT task, a long IRQ or softirq burst, or an IRQs-off or preemption-off region (use the irqsoff/preemptoff tracers). If it is blocked, look for a lock held by a lower-priority thread (priority inversion; use PI mutexes), a page fault on memory that was not locked (mlock, prefaulting), or a Binder call. Also check CPU frequency ramping and deep idle exit latency on its core, and whether RT throttling kicked in. Fixes include pinning to a quiet core, raising IRQ thread priorities appropriately, avoiding allocations and locks in the audio callback, and PI locking.

cat /sys/kernel/debug/devices_deferred lists the device and often the reason string (missing clock, regulator, GPIO, PHY). Cross-check the DT: is the provider's node status = "okay", is the phandle correct, is the provider itself a module that never loaded? fw_devlink should order probes from DT phandles; if it does not, look for a missing clocks, *-supply or resets property. Also check that the provider driver is in the vendor module list (GKI) and that it did not fail probe earlier. Fix the provider or the DT; do not paper over it with a long msleep in the consumer.

1. Check the upstream or vendor kernel for a driver matching the part and its DT binding documentation (Documentation/devicetree/bindings/).
2. Enable the driver in the vendor defconfig as a module (GKI) and add it to the vendor module list so it is packaged in vendor_dlkm or vendor_boot.
3. Add the DT node under the correct I2C/SPI bus with compatible, reg, interrupt, supplies, reset GPIO and pinctrl as the binding specifies.
4. Verify probe in dmesg, the IIO or input device in sysfs, and raw readings.
5. Add ueventd.rc permissions and SELinux labels for any device nodes, then wire up the sensor HAL.
6. Validate suspend/resume, interrupt wake behavior and power consumption.