# Chrome OS Debugging Instructions Chrome on Chrome OS is tested using a handful of frameworks, each of which you'll find running on Chrome's CQ and waterfalls. If you're investigating failures in these tests, below are some tips for debugging and identifying the cause. *** note This doc outlines tests running in true Chrome OS environments (ie: on virtual machines or real devices). [linux-chromeos] tests, on the other hand, can be debugged like any other linux test. *** ## Tast [Tast] is Chrome OS's integration testing framework. Since Chrome itself is instrumental to the Chrome OS system, it's equally important that we run some of these integration tests on Chrome's waterfalls. If you find one of these tests failing (likely in the `chrome_all_tast_tests` step), you can: - **Inspect the failed test's log snippet**: There should be a log snippet for each failed test in the `Test Results` tab in the build UI. eg: For this [failed build], clicking on the `policy.IncognitoModeAvailability` expands to include stack traces and error messages. - **View browser & system logs**: A common cause of failure on Chrome's builders are browser crashes. When this happens, each test's log snippets will simply contain warnings like "[Chrome probably crashed]". To debug these crashes, expand the list of attached artifacts for the test by clicking the `Artifacts` link under the failed test in the `Test Results` tab. There you'll find an extended log for the test under `log.txt`. Additionally, you can find system logs included in that list. To find a system log for a particular test, match the timestamps printed in the test's log with the timestamps present in the system log filename. For instance, the previous `example.ChromeFixture` failure matches the [chrome/chrome_20210920-051805] browser log, which contains the culprit Chrome crash and backtrace. - **Symbolizing a browser crash dump**: See [below](#symbolizing-a-crash-dump). ### Disabling a test If you are a Chrome Sheriff, please read the sheriff documentation [here](http://go/chrome-sheriff-tast) before disabling any tests. Tast tests are run under both Chrome's builders and CrOS's builders. They can be disabled either completely (in all builders), or in Chrome's builders alone. The latter should be used only for changes which are not expected to occur on CrOS's builders. - **Disabling in all builders**: If you have a full CrOS checkout, you can add the `informational` [attribute] to the test's definition. (You may be able to bypass the need for a full CrOS checkout by using the `Edit code` button in codesearch UI, but this flow is unverified.) This can take time (ie: many hours) to land and propagate onto Chrome's builders. So if you need the test disabled ASAP, consult the next option. - **Disabling in only Chrome's builders**: You can add the test to the list of disabled tests for the step's GN target. For example, to disable a test in the `chrome_all_tast_tests` step, add it to [this list]. **Note**: If the test is failing consistently, and you only disable it here, it will likely start to fail in the next [Chrome uprev] on CrOS's builders, which can lead to further problems down the road. So please make sure you pursue the first option as well in that case. In both cases, please make sure a bug is filed for the test, and route it to the appropriate owners. ### Symbolizing a crash dump If a test fails due to a browser crash, there should be a Minidump crash report present in the test's isolated out under the prefix `crashes/chrome...`. These reports aren't very useful by themselves, but with a few commands you can symbolize the report locally to get insight into what conditions caused Chrome to crash. To do so, first download both the task's input files (this provides the symbols and the symbolizing tools) as well as the task's output results (this provides the crash reports). See the commands listed under the *Reproducing the task locally* section on the task page. For example, to download them for [this task](https://chrome-swarming.appspot.com/task?id=5cc272e0a839b311), `cd` into a tmp directory and run: ``` cipd install "infra/tools/luci/cas/\${platform}" -root bar ./bar/cas login ./bar/cas download -cas-instance projects/chrome-swarming/instances/default_instance -digest 1ad29e201e4ae7e3056a8b17935edbcd62fb54befdfeba221f2e82e54f150c86/812 -dir foo cipd install "infra/tools/luci/swarming/\${platform}" -root bar ./bar/swarming login ./bar/swarming collect -S chrome-swarming.appspot.com -output-dir=foo 5cc272e0a839b311 ``` Once both input and output have been fetched you must then generate the breakpad symbols by pointing the `generate_breakpad_symbols.py` script to the input's build dir: ``` cd foo vpython3 components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir out/Release/ --binary out/Release/chrome ``` That will generate the symbols in the `symbols/` dir. Then to symbolize a Chrome crash report present in the task's output (such as `chrome.20220816.214251.44917.24579.dmp `): ``` ./out/Release/minidump_stackwalk 5cc272e0a839b311/crashes/chrome.20220816.214251.44917.24579.dmp symbols/ ``` ### Running a test locally To run a Tast test the same way it's ran on Chrome's builders: - Decide which Chrome OS device type or VM to test on. - Build Chrome via the [Simple Chrome] workflow for that board. - Deploy your Chrome to the device via the [deploy_chrome.py] tool. - Finally, run the Tast test on the device via the `cros_run_test` tool under `//third_party/chromite/bin/`. eg: `cros_run_test --device $DEVICE_IP --tast login.Chrome`. See [here] for more info on cros_run_test. ## Telemetry >TODO: Add instructions for debugging telemetry failures. ## GTest >TODO: Add instructions for debugging GTest failures. ## Rerunning these tests locally >TODO: Add instructions for rerunning these tests locally. [linux-chromeos]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/chromeos_build_instructions.md [Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md [failed build]: https://ci.chromium.org/ui/p/chromium/builders/ci/chromeos-kevin-rel/37300/test-results [Chrome probably crashed]: https://luci-milo.appspot.com/ui/inv/build-8835572137562508161/test-results?q=example.ChromeFixture [chrome/chrome_20210920-051805]: https://luci-milo.appspot.com/ui/artifact/raw/invocations/task-chromium-swarm.appspot.com-561bed66572a9411/artifacts/chrome%2Fchrome_20210920-051805 [attribute]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md [this list]: https://codesearch.chromium.org/chromium/src/chromeos/tast_control.gni [Chrome uprev]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/chrome_commit_pipeline.md#the-chrome-os-commit-pipeline-for-chrome-changes [Simple Chrome]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md [deploy_chrome.py]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md#Deploying-Chrome-to-the-device [here]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/cros_vm.md#in-simple-chrome