Platform Integration & Release

HLOS means High-Level Operating System: the Linux kernel plus Android or Wear OS user space running on the main application processor. The HLOS image is the complete Android software build for a device: the AOSP framework and apps, the vendor BSP (kernel, device tree, drivers), vendor HALs, proprietary libraries and OEM customisation. It is distinct from the non-HLOS firmware that runs on the modem, DSPs and other processors.

Non-HLOS software is firmware or a real-time OS running on processors other than the application processor, or in the secure world. Examples: modem firmware (MPSS), audio and compute DSP firmware (ADSP, CDSP), sensor hub firmware (SLPI), TrustZone and the hypervisor, the XBL boot loaders, the always-on power management processor, and Wi-Fi and Bluetooth firmware. Each is built and versioned by its own team.

A meta build is the integration record that combines one specific, tested version of every software component (the HLOS build and each non-HLOS firmware build) into a single release, along with the partition layout and flashing information. It ensures that the combination flashed onto a device is known to work together. Many hard integration bugs are caused by mismatched component versions, which the meta build is designed to prevent.

It means taking a new release from the upstream source (Google's AOSP, or the Linux kernel) and bringing it into your own code base, which contains your changes on top. You merge or rebase the new upstream code, resolve conflicts with your carried changes, update interfaces, build and test, and promote the result as a new baseline for your products.

Treble, introduced in Android 8, separates the Android framework (system partition) from vendor implementation (vendor partition) behind stable, versioned HAL interfaces. The framework can then be updated without rewriting vendor code, as long as the HAL versions remain compatible. This makes upstream drops and OS upgrades much cheaper and is verified by VTS and by booting a Generic System Image.

VINTF (Vendor Interface object) is a set of manifests and compatibility matrices. The device manifest lists the HALs and versions the vendor provides; the framework compatibility matrix lists what the framework requires, and vice versa. The build and the OTA system check that they match, and the device refuses incompatible updates. A HAL version mismatch is a classic cross-team dependency during an upstream drop.

GKI, the Generic Kernel Image, is a kernel core built by Google from the Android Common Kernel for each supported branch. Vendors put SoC and board-specific code into loadable kernel modules that use only the stable Kernel Module Interface (KMI). This separates kernel updates from vendor changes. Within the same frozen KMI, a core-kernel update does not require those modules to be rebuilt. A rebuild is needed when the KMI changes (new ACK branch) or a module needs a new symbol.

A merge combines two branches by creating a merge commit that has both histories as parents; existing commit IDs are unchanged. A rebase takes your commits and replays them one by one onto a new base, creating new commits with new IDs and a linear history. Merge is safer for shared branches; rebase gives a cleaner history and a tidy patch stack but rewrites history that others may depend on.

A cherry-pick applies the changes of one specific commit onto another branch as a new commit. It is used to port a bug fix from main to a release branch (or the reverse) without bringing other changes. Use git cherry-pick -x to record the original commit ID in the message for traceability, and check for dependent commits that must also be picked.

repo is Google's wrapper around Git that manages the hundreds of Git repositories that make up an Android tree. A manifest XML file lists each project, its path, remote and branch or revision. repo init selects a manifest, repo sync fetches all projects, and repo manifest -r produces a snapshot of exact revisions so a build can be reproduced.

CTS, the Compatibility Test Suite, is Google's automated test suite for the public Android APIs and behaviours required by the Compatibility Definition Document. Passing it shows that apps written against the Android SDK will behave correctly on the device. Passing CTS is required to be Android-compatible and to license Google Mobile Services.

VTS, the Vendor Test Suite, tests the vendor side of the Treble boundary: HAL implementations, VINTF compliance, kernel configuration and GKI requirements, and vendor partition behaviour. It ensures the vendor implementation keeps the contract the framework relies on, so a generic framework can run on the device.

GTS, the GMS Test Suite, checks requirements for devices that ship Google Mobile Services, such as Google Play services and Play Store behaviour, preloaded Google apps and related configuration. It is distributed to GMS licensees, not published in AOSP. A device must pass CTS, VTS and GTS (plus others such as STS) to ship with GMS.

A promotion gate is a set of pass or fail criteria a build must meet to move to the next stage, for example from an integration branch to a baseline delivered to customers. Typical criteria are successful builds on all targets, boot on all SKUs, smoke tests, xTS pass rates, power, stability and performance within thresholds compared to the last good build, and no open P0 or P1 regressions.

Presubmit tests run on a change before it is merged, usually a targeted build and fast tests, so obviously broken changes never land. Postsubmit tests run after merging, on the combined tree, and include full builds on all targets and slower device tests. Presubmit protects the branch; postsubmit catches interactions between changes and anything too slow for presubmit.

Root-cause analysis (RCA) is a structured investigation into why a problem happened, going past the immediate technical fault to the design, process or test gap that allowed it. Its output is the root cause, contributing factors, why it was not detected earlier, and corrective actions with owners and dates. Tools include 5 Whys, fishbone diagrams and timelines.

You state the problem and ask "why did this happen?", then ask "why?" about each answer, typically about five times, until you reach a cause that, if fixed, prevents the problem from recurring (often a process or test gap). Each answer should be supported by evidence, not guesses, and there can be multiple branches. It stops teams from fixing only the surface symptom.

A DRI (Directly Responsible Individual) is the one named person accountable for driving an issue or deliverable to completion. They do not have to do all the work, but they coordinate, track, escalate and make sure it closes. Having a DRI prevents problems from bouncing between teams with nobody driving them.

A code freeze is a milestone after which only approved changes (usually bug fixes for release-blocking issues) may enter the release branch. It stabilises the code so testing results remain valid. Changes after freeze typically need a change control board's approval and may trigger re-running key tests.

Soong is the primary AOSP userspace build. Modules are declared in Android.bp (Blueprint) and compiled via Ninja. Make still composes the product: device.mk, BoardConfig.mk and PRODUCT_PACKAGES decide what is on the image, and some leftover Android.mk modules remain. Google's userspace Bazel migration was halted around 2023; Soong is still primary. Kernel builds use Kleaf (Bazel), which is a different path.

The build target. Recent trees use PRODUCT-RELEASE-VARIANT (for example a Cuttlefish phone, a trunk or named release config, and userdebug). Older trees used two tokens, PRODUCT-VARIANT. The variant is user (ship and certify), userdebug (usual debug image) or eng. Always check the tree you are in rather than memorising one combo.

The Make list of modules installed on that product image. A module can build successfully and still be absent from the device if nobody added it to PRODUCT_PACKAGES (or PRODUCT_PACKAGES_DEBUG for debug-only tools). When a binary is "missing on the image," check the product makefile before rewriting the Android.bp.

GMS is the licensed Google apps and services (Play Store, Play services and related apps). MADA is the commercial agreement under which an OEM may preload them. GTS is the technical test suite for that licence. CTS/VTS prove Android compatibility; they do not grant GMS. Do not quote confidential placement rules; say you would check current licensee docs and the CDD.

A full payload can apply from a wide set of source builds. An incremental (delta) payload is smaller and is built for a specific source fingerprint. update_engine writes the inactive A/B slot (virtual A/B uses snapshots for dynamic partitions). If a device's source build is not in the delta set, it will not be offered that incremental and needs a full payload or another delta.

1. Track the AOSP release tag and read release notes for API, HAL, SELinux, build and kernel changes.
2. Do an early trial merge (for example on preview tags) to estimate conflicts and dependencies.
3. Plan the merge or rebase strategy per repository and sequence the drop across affected SoC baselines.
4. Run dependency analysis: assign each conflict or breakage to its owning domain (framework, BSP, HAL, apps, build) with a date.
5. Update HAL implementations and VINTF manifests for required interface versions.
6. Gate on build health, boot, smoke, VINTF, xTS, and power, stability and performance against last good.
7. Promote the baseline, publish release notes and known issues, and hand off to OEM teams on the agreed schedule.

Throughout, keep one status of record and coordinate multimedia, camera, connectivity, display, power and security stakeholders across sites.

The ingredients are the same: AOSP and Wear OS framework, vendor BSP, vendor HALs, proprietary libraries, sensor hub firmware and OEM customisation, branched per chipset. The differences are the constraints: smaller RAM, flash and display, a dominant power budget, an always-on co-processor handling sensors and ambient display, health sensors as first-class components, a companion and Bluetooth connectivity model (and eSIM or LTE for standalone watches), and tiles, complications and watch faces as the main user surface. The power architecture shapes what is included and how it is tuned to the ship gate.

Choose merge when the branch is shared by many teams and downstream branches, because a merge does not rewrite commit IDs and conflict resolution happens once. Choose rebase when you want to keep the vendor delta as a clean, reviewable patch stack on top of upstream (common for kernel trees or small repositories), which makes it easy to see and upstream your changes. Many organisations mix them: merge for large shared framework repositories, rebase for patch-stack style kernel or HAL repositories.

A semantic conflict is when two changes merge without any textual conflict, but the combined code is wrong: for example upstream changes a default value, renames a behaviour or moves a permission check, and a vendor patch that depended on the old behaviour still applies. Git cannot detect this. You catch it with builds, unit and integration tests, xTS, KPI regression runs, and reviewers who understand why each carried patch exists. Reviewing the upstream diff for areas touched by carried patches helps too.

Do not have one person resolve everything. Pre-scan conflicts early, classify them by domain and type, and assign each group to the team that owns the carried change, with deadlines and a tracking list. Resolve easy mechanical conflicts centrally, and send semantic or interface conflicts to domain experts. Build and test each resolution, reuse earlier resolutions with rerere, and drop vendor patches that upstream has made unnecessary. Report progress daily against the list.

A main development branch that receives upstream drops and new features, preferably with feature flags to keep it close to trunk-based. Release branches cut from main per Android version and chipset baseline at feature freeze, which accept only fixes. OEM or product branches for customer-specific customisation, kept as thin as possible. Fixes land on main and are cherry-picked to supported release branches (or the reverse, but consistently), with automated checks that nothing is missed. Branches are retired on a published schedule.

Require every fix to reference a bug, and track per-branch status on that bug. Use cherry-pick -x or Change-Id matching so tools can compare branches, and run an automated forward-merge or "missing fixes" report that lists changes present on a release branch but not on main. Review the report as part of the release checklist. Without this, the same bug reappears in the next release.

• Build health across all SoC baselines and build variants.
• Boot success on all SKUs, including repeated boot cycles.
• VINTF compatibility and core VTS.
• Smoke or BAT covering calls, data, Wi-Fi, Bluetooth, sensors, display, camera, OTA.
• CTS, VTS and GTS pass rate at or above the previous baseline.
• Power: standby drain, suspend residency, wake lock and wakeup budget versus last good.
• Stability: crash, ANR, panic, watchdog and subsystem restart rates.
• Performance: boot time, launch latency, jank.
• No open P0 or P1 regressions without a signed-off waiver.

First triage: is it a device bug, a test bug, a test environment problem (network, SIM, lab setup) or flakiness? Re-run the single module with Tradefed to confirm, and compare with a previous passing build to find the introducing change. If it is a device bug, assign it to the owning team as a blocker. If it is a genuine test bug, collect evidence and request a waiver through the official process. Either way, record it and add the module to continuous CI so it is caught earlier next time.

CTS-on-GSI means flashing Google's Generic System Image (a pure AOSP system partition) on the vendor's device and running CTS. If the vendor implementation follows Treble correctly, the generic framework should work with the vendor partition. Failures show that the vendor side relies on system partition modifications or breaks the interface contract, which would make future framework updates expensive.

Use Gerrit with presubmit that builds affected targets and runs fast tests and static analysis, with atomic submission for changes that span repositories (topics). Postsubmit builds all targets continuously and runs boot and smoke tests on real devices. Nightly or per-candidate builds run xTS subsets, power and performance benchmarks and stability soaks. Add remote build caching, automated culprit finding, flaky-test quarantine, a revert-first policy for breakages, and store manifest snapshots and artifacts for every build.

Measure flakiness automatically (for example a test that fails and then passes on retry without code changes). Quarantine flaky tests from blocking gates, but keep running them and assign an owner and a deadline to fix or delete them. Track the flaky rate as a metric. Never simply retry until green, because that hides real intermittent bugs such as race conditions, which are often real device defects.

By layer and evidence. Reproduce the bug, capture logs across boot, kernel, HAL and framework on one timeline, and bisect builds to the introducing change. Map the failing component to the team whose code must change. If it is genuinely shared, such as an interface contract between a HAL and firmware, assign one DRI and have the other teams co-own actions rather than letting the bug bounce. The goal is to remove ambiguity quickly so engineers fix instead of argue.

Take ownership as the DRI and acknowledge the customer quickly with a time for the next update. Reproduce and scope the severity and KPI impact. Pull the right cross-domain experts into one triage thread, build a cross-layer log timeline, and bisect to root cause. Weigh fix versus risk versus schedule and agree the plan with the customer. Communicate on a fixed cadence to internal teams and the customer. Close with the verified fix, a regression test or gate, and a post-mortem.

It is blameless and factual: a timeline of when the defect was introduced, when it could have been detected, and when it was detected and fixed. It identifies the root cause and contributing factors (technical and process), the detection gap (which gate or test was missing), and specific corrective actions with owners and dates. It is shared widely, and the actions are tracked to completion rather than forgotten.

Each gate owner reports status against the published release criteria: build, xTS, KPIs, open defects, certification, and customer or carrier sign-offs. Known risks and waivers are reviewed explicitly. The release owner makes the decision (go, no-go, or go with conditions) and records it with reasons. A good meeting is short because the data was prepared in advance; debates about criteria belong before the meeting, not in it.

Release the update to a small percentage of devices first (for example 1 percent), then widen in steps (10, 50, 100 percent) if health metrics are good. Monitor OTA success rate, boot success, crash and ANR rates, battery telemetry and customer-reported issues against the previous build. Define halt criteria in advance, and be ready to pause the rollout and ship a fix. A/B updates with rollback make each step safer.

Work async-first with one source of truth for status, clear DRIs and written decisions. Use follow-the-sun triage with structured handoff notes so critical issues move forward around the clock. Keep a regular status cadence with internal teams and external customers, highlighting what changed, what is blocked and what help is needed. Escalate blockers early with a clear ask, owner and date. Rotate meeting times so the same site is not always inconvenienced.

Battery life and standby drain per hour against the last good build, suspend residency and wakeup counts, wake-to-render latency, crash and ANR rates, kernel panic and watchdog reset rates, boot and OTA success rates, thermal ceiling under sustained load, and connectivity reliability (Bluetooth reconnection, notification delivery). Define regression thresholds in advance and hold promotion if any exceed them.

Map the system and its interfaces first: the image composition, branches, build and test pipelines, and KPI dashboards. Find the two or three people who hold the most context and learn from them. Sit in triage meetings to learn the current problems. Land a small real change early to learn the pipeline end to end. Keep a list of unknowns and burn it down deliberately, and write down what you learn so the next person ramps faster.

Start with the chipset that is most representative and best staffed (often the newest, which will ship the release first) as the lead baseline. Resolve common framework and HAL conflicts there once, then apply the same resolutions to other baselines, handling only their BSP-specific deltas. Older chipsets may stay on an earlier kernel branch or HAL version, so check VINTF and GKI support per chipset. Stagger promotions so teams are not overloaded, and publish the schedule to OEMs.

Before GKI, each vendor carried a large, forked kernel with thousands of patches, and every kernel update meant a painful forward-port. With GKI, the core kernel image comes from Google's Android Common Kernel, and vendor code lives in modules that may only use symbols in the KMI symbol list. Integration work shifts to keeping modules compatible with the frozen KMI, requesting new symbols through the upstream process, and validating with VTS kernel tests. Same-KMI security and bug-fix kernel updates can be taken without rebuilding vendor modules. Modules rebuild when you move to a new ACK/KMI, or when you need a new symbol or a KMI break (which ABI tooling should reject on a frozen branch).

A new Android release may require a newer version of a HAL (or migration from HIDL to AIDL) to support new features, as defined in the framework compatibility matrix. The vendor HAL owner must implement the new version, update the device manifest, and pass VTS for it. As integration lead, identify these requirements from the release notes and compatibility matrix early, create tracked items per HAL with owners and dates, and decide whether the drop can land with the old version (if still allowed) while the new one is completed.

Feature flags let unfinished features merge into main early but remain disabled, so fewer long-lived feature branches are needed and integration happens continuously. AOSP's trunk-stable model uses aconfig flags with release configurations that decide which flags are enabled in each release. The costs are flag management discipline, testing both flag states where it matters, and removing old flags. For release management, flags allow disabling a risky feature late instead of reverting code.

Avoid squashing an upstream drop into one change; keep upstream history so git bisect can walk individual commits. Land the drop in stages where possible (by repository or subsystem) with builds and tests between stages. Store repo manifest -r snapshots for every CI build so any build can be reproduced. When a regression appears, bisect first between CI builds (coarse) and then between commits within the suspect repositories (fine).

Base thresholds on the distribution of results from known-good builds, not on a single run: measure run-to-run variance and set limits outside normal noise (for example the mean plus a margin). Compare against last good on the same hardware and test setup. Require several runs for noisy KPIs such as power and performance. Separate hard blockers (boot, P0 regressions) from soft limits that require a documented waiver. Review thresholds periodically, but never lower them to pass a failing build.

Use dedicated devices with external power monitors or on-device power rails, fixed test profiles (screen off standby, ambient mode, workout, music), controlled radio conditions (shielded boxes or fixed SIM and network), and consistent battery and thermal starting states. Run each profile several times, report mean and variance, and compare with last good. Automatically attach batterystats, wakeup sources and Perfetto traces to failures so they are debuggable. Make standby drain per hour a gate.

First ask whether the patch is still needed: upstream may have fixed the same problem differently, in which case drop the patch and verify the original issue stays fixed. If still needed, re-implement it against the new upstream code with the owning team, adding a test that captures the original intent. Consider upstreaming it so it stops being a carried delta. Record the decision in the commit message.

Make the trade-off explicit and data-driven. Quantify the gate failure (which KPI, by how much, which users), lay out options (slip the date, ship with a documented known issue and a dated fix, disable the feature with a flag, take a targeted fix with focused re-test), and state the risk of each. Decide with stakeholders and record the decision. Some gates are non-negotiable, such as boot success, security, and life-critical paths like emergency calling; others can be waived with sign-off and a follow-up plan.

An escaped defect is a bug found by a customer or in the field that the internal gates should have caught. For each one, do an RCA focused on the detection gap: which test or gate was missing, why the existing tests did not cover it, and what the cheapest reliable way to catch it earlier is. Add that test or gate, and track the escaped defect rate as a process metric. Over time, the gate set becomes a record of lessons learned.

HLOS drivers and HALs talk to firmware through message protocols and shared memory layouts. If a new HLOS build expects a new firmware message or field that an older modem or DSP build does not support (or vice versa), you get failures such as features silently not working, subsystem crashes and restarts, or boot hangs. Prevent them with a meta build that pins tested combinations, versioned interfaces with capability negotiation, compatibility checks at boot, and integration tests that run the exact combination being released.

• Start early: integrate developer previews and betas continuously instead of one big drop.
• Shrink the carried delta by upstreaming patches and removing obsolete ones.
• Automate: trial merges, conflict reports, dependency tracking and CI gates.
• Standardise conflict ownership so work is parallel across domain teams.
• Reuse resolutions (rerere) and make HAL work predictable using the compatibility matrix.
• Measure the lead time per phase to see where time goes.

Increase the reproduction rate first: stress conditions, run many devices in parallel, and automate detection so failures are captured without a person watching. Add targeted instrumentation (always-on ring-buffer tracing, extra logs around the suspected area) and make sure logs survive reboots (pstore, persistent logs). Collect a large sample and look for correlations (build, SKU, temperature, uptime, network). Form hypotheses, test them one at a time, and use the 5 Whys once the technical cause is found.

Security patches follow the monthly Android Security Bulletin and vendor bulletins, with embargo rules before public disclosure. Maintain a dedicated path: patches go into every supported branch on a fixed schedule, verified by STS and a focused regression test set, with the security patch level property updated. Keep this path independent of feature branches so security updates are never delayed by feature integration. Track patch level lag per branch as a metric.

A short set with trends: upstream integration lead time, carried delta size, build green percentage and time to fix breakages, xTS pass rate per baseline, open P0 and P1 counts and defect convergence toward release, customer CR aging and MTTR, escaped defects, and on-time delivery rate. Pair each with a one-line interpretation and the action being taken, and highlight red items and the help needed.

Upstream-first means that changes to shared code (AOSP framework, Linux kernel) are contributed upstream and, ideally, merged there before or instead of being carried privately. Benefits: less carried delta, easier future drops, community review and testing. Trade-offs: upstream review takes time, the change must be generic enough to be accepted, and schedules may require carrying the patch temporarily. Use a clear policy: carry temporarily only with a tracked upstream submission.

Publish clear criteria for what is accepted (for example blocker bugs, security fixes, certification failures). Require each request to include the bug, root cause, risk assessment, test evidence and the branches affected. Meet frequently in short sessions, allow asynchronous approval for low-risk items, and keep the board small with authority to decide. Track approved changes and re-run the relevant tests automatically after they land.

Not as a current fact. Google explored a userspace migration from Soong to Bazel and halted it around 2023; Soong remains the primary userspace build. Kleaf, the Bazel-based kernel build, is still real and is what GKI/kernel teams use. A strong answer splits the two paths and does not treat a 2021-era migration slide as today's architecture.

Vendor API level (ro.vendor.api_level) is the API the vendor partition was built against. A freeze window (often called GRF / Google Requirements Freeze) lets that vendor image pair with newer system images for a documented number of releases, as long as VINTF still matches. The OEM can take a yearly OS upgrade without a full SoC rebase of every HAL. Exact window lengths change; say you would check the current CDD and vendor-API notes. What still moves: new matrix-required HALs, CTS/VTS/GTS for the new release, CDD, and any new KMI.

Do not debug one tombstone at a time. Symbolize stacks for that exact build, cluster by process plus a stable stack signature (not ASLR addresses), rank by volume times user impact, and assign a DRI to each top cluster with one representative report. A cluster is closed when the signature is gone (or below threshold) on the next build and a regression gate exists. Pair crash-free rate with the top-N signatures so a lucky week cannot hide a new system_server cluster.

Reproduce and quantify on a fixed profile, comparing drain per hour with the previous build. Pull a bug report and load batterystats into Battery Historian to spot new wake locks, wakeups, jobs or alarms. Use Perfetto and /sys/kernel/debug/wakeup_sources to find what is keeping the CPU awake, and check suspend residency. Bisect the change set or components to find the culprit. Fix it (remove the wake lock, batch the work, move it to WorkManager, restore sensor batching), re-measure, and add a power KPI gate so it cannot recur silently.

Take ownership as the DRI and move the discussion to one triage thread. Get a reliable repro, then capture logs and traces at every layer on one timeline (logcat, dumpsys, HAL logs, dmesg, Perfetto). Bisect to find the layer where the data first goes wrong and the change that introduced it. Present the evidence and assign the owner whose code must change, with a date. Keep a single status of record, track to closure, and add a regression gate.

Find where it stops. No splash or bootloader output suggests XBL, ABL, AVB or partition layout problems. Splash then reboot loop suggests a kernel panic or init failure, so read the UART console, last_kmsg or pstore. Stuck at the boot animation suggests system_server or a critical service crashing, so read logcat. Compare with the working chipsets to see what differs (BSP, kernel branch, HAL versions, VINTF, SELinux denials). Bisect the drop by repository if needed, and treat it as a P0 blocking promotion for that chipset.

Group the new failures by module to see if they share a cause (one broken service can fail hundreds of tests). Check the lab first: network, SIM, device health and test suite version, because environment problems often cause mass failures. Compare with the previous nightly build and its manifest to list the changes in between, and re-run a sample of failures to confirm. Bisect to the culprit change, revert it if it blocks others, and assign a fix. Add the relevant modules to presubmit if they are cheap enough.

Acknowledge quickly and set an update cadence. Collect data: reboot reasons, kernel panic logs, ramdumps, watchdog and subsystem restart logs, and field telemetry (which builds, SKUs, regions, conditions). Look for patterns and try to reproduce with stress tests under similar conditions. Localise to the layer (kernel panic, modem crash, system_server watchdog), assign the owner and drive root cause. Agree a fix and OTA plan with the customer, roll out staged, confirm the reboot rate drops, then run a post-mortem and add a stability gate.

Quantify impact: which users, how often, how severe, and whether a workaround exists. Check whether a low-risk fix is available and how much re-testing it needs. Lay out options: slip the release, ship with a documented known issue and a dated maintenance fix, disable the feature by flag, or take a targeted fix with focused re-test. Present the options with risks to the release owner and stakeholders, decide transparently, record the decision, and communicate it to customers honestly.

Confirm with repeated measurements on both builds under the same conditions. Capture Perfetto traces of camera launch on both builds and compare the phases: app start, camera service connect, HAL open, sensor configuration, first preview frame. Identify the phase that grew, then look at changes in that area in the drop (framework camera service, HAL interface version, SELinux, scheduling). Bisect commits within the suspect repositories. Assign to the camera or framework owner with the traces, and add launch latency to the performance gate.

Possible reasons: the fix depends on an earlier commit that exists on the other branches but not this one; the code on that branch differs and the conflict was resolved incorrectly; the bug on that branch has a different root cause; the fix is in a component that is built from a different repository or prebuilt on that branch; or the tested build did not actually include the change (check the build's manifest snapshot). Verify the change is in the build, compare the code paths, and re-investigate the root cause for that branch.

Do not refuse in the meeting; understand the business need and the deadline behind it. Assess scope, risk, affected components and test effort with the owning team. Return with options: deliver in the next maintenance release, deliver a limited version behind a flag, or take it now with explicit trade-offs (another item moves, or freeze moves). Make the trade-off visible to program management and the customer, decide together, and document it.

Measure first: which targets break, which kinds of changes cause it, and how long breakages last. Strengthen presubmit to build the affected targets and variants, and enforce atomic submission for multi-repository changes. Adopt a revert-first policy with an on-call build sheriff. Add automated culprit finding to notify authors quickly. Track build green percentage and time to repair as team metrics and review them weekly.

Compare the customer's build against the baseline: OEM apps and services, overlays, configuration, preinstalled watch faces and firmware versions (a different meta build combination). Reproduce with the customer's build on the same power profile and collect batterystats, wakeup sources and Perfetto. Often the cause is an OEM app holding wake locks, a watch face updating too often in ambient mode, or a different sensor or connectivity configuration. Share evidence with the customer, help fix it, and offer them the same power gate you use internally.

In the first 30 days, map the image, branches, teams, KPIs and current red items; sit in triage; do not reorganise yet. By 60 days, introduce one status of record, written promotion gates based on the most common escaped defect types, named DRIs for systemic issues, and work-in-progress limits. By 90 days, run the first gated drop, report gate results and escaped defect trends, set a customer communication cadence, and plan the next set of gate improvements.

Confirm by testing combinations: old modem with new HLOS and new modem with old HLOS, which isolates the component. If the new modem alone breaks it, collect modem logs, IMS and RIL logs and a SIP trace on both firmware versions, and give the modem team the exact failing step. Check whether the new firmware changed an interface (a QMI message or a configuration item) that the HLOS side must adapt to. Pin the old combination in the meta build until a fix is ready, and add a VoLTE call test to the meta build gate.

Move the discussion from opinions to data. Define what matters (failure rate, performance, risk, how many branches or customers are affected), run a quick experiment or stress test on both options, and write a short one-page comparison with a rollback plan. Present it to the architect and agree on the decision criteria before discussing the choice. Once a decision is made, commit fully and help carry it out, even if it was not your preferred option.

Understand why: capacity, unclear requirements, technical blockers or competing priorities. Break the remaining work into smaller tracked pieces with daily visibility. Offer help: pair them with engineers from other teams, clarify the minimum required for promotion, or allow the drop to land with the old HAL version if the compatibility matrix permits. If priorities conflict, escalate to management with a clear ask and the impact of each choice. Communicate the revised plan to stakeholders promptly.

Pause the rollout immediately; A/B rollback protects devices that fail to boot, but the failure still harms users. Collect data from affected devices: models, previous build, storage state, and logs from failed boot attempts. Check whether failures correlate with a specific source build (delta payload problem), storage condition (for example low free space for virtual A/B snapshots), or hardware variant. Fix, test on the affected configurations, then resume the rollout from a small percentage.

List every HAL that must change using the framework compatibility matrix and deprecation notices. For each, identify the owner, effort and dependencies (framework clients, vendor clients, tests). Prioritise HALs that block the release, and check which can remain on HIDL for now. Plan the migration so both versions can coexist during the transition where possible, write or update VTS tests for the AIDL version, and track progress in the integration status. Start early, ideally on preview releases.

Create structure quickly: define the deliverable and the quality bar, list the components and interfaces, and name an owner for each, even if temporary. Set up a single status of record, promotion gates and a regular cadence. Drive cross-team triage on systemic issues and convert unknowns into a tracked list that you burn down. Communicate clearly and early with stakeholders when assumptions change. In an interview, use the STAR format and quantify the result (on-time delivery, defects closed, KPIs held).

First confirm the module exists in the build graph (Android.bp name, vendor: true, no broken required deps). Then check the product makefile: it must be in PRODUCT_PACKAGES (or pulled in by another packaged module). Also check the variant (debug-only packages go in PRODUCT_PACKAGES_DEBUG and will be absent on user), the partition (vendor vs system), and whether a conditional ifeq excluded that SKU. Built is not installed.

The new system must be inside the vendor-API freeze window for that vendor API level, VINTF must still match (no newly required HAL the vendor does not provide), GKI/KMI must still be compatible if the kernel stays, and the device must pass the new release's CTS/VTS/GTS and CDD. Firmware pins in the meta build must still satisfy the new HLOS. If any of those fail, it is not a free upgrade: someone must move vendor code, kernel branch or firmware.

Do not start in the modem C-core. Compare CarrierConfig for that MCC/MNC, IMS and APN overlays, the exact meta-build firmware pins, and whether the lab SIM exercises a different feature flag (VoLTE, VoWiFi, 5G NSA/SA). Reproduce on the same carrier config. If config matches, then collect RIL, IMS and modem logs. PTCRB/GCF failures need the same split: protocol/RF versus HLOS policy. Keep one DRI and one status; name the failed contract, not the company.