perf_waterfall.md 3.0 KB

chromium.perf Waterfall

Overview

The chrome.perf waterfall continuously builds and runs our performance tests on real Android, Windows, Mac, and Linux hardware; see list of platforms. Results are reported to the Performance Dashboard for analysis. The Perfbot Health Sheriffing Rotation ensures that the benchmarks stay green. The Perf Sheriff Rotation ensures that any regressions detected by those benchmarks are addressed quickly. Together, these rotations maintain Chrome’s Core Principles of speed:

«If you make a change that regresses measured performance, you will be required to fix it or revert».

How It Works

The perf waterfall is split into three stages - builders, testers, and processors.

Builders

For each commit to the Chromium repo, a Builder is invoked. The builder builds Google Chrome, acquires test assets, and bundles everything up to be passed along to Testers. We execute a builder for each platform we support. To ensure we can keep up with the rapid flow of commits, we have a set of builders per platform, each building a different commit.

Testers

For each platform, we have a single tester job running continuously. Each run of a tester covers the set of commits from the end commit of the previous invocation to the latest built commit. Each tester has one or more subdevices (shards). The tester delegates a subset of tests to each shard and aggregates high-level results (pass/fail, runtimes).

Processors

Processors analyze the raw data generated by each Tester and convert it into a form that can be utilized by the Chrome Performance Dashboard. For some jobs, this work is executed by the tester instead.

Contact

  • You can reach the Chromium performance sheriffs at perf-sheriffs@chromium.org.
  • Bugs on waterfall issues should have Component: Speed>Benchmarks>Waterfall.
  • For domain knowledge for a specific benchmark, consider reaching out to benchmark owners listed in benchmark.csv

Links