# Web Tests Tips The recommendations here are intended to help you write new tests that go through code review with a minimal number of round trips, remain useful as Blink evolves, and serve as an asset (rather than a liability) for the team. While reading existing web tests, please keep in mind that they represent snapshots taken over many years of an ever-evolving collective opinion of what good Web pages and solid tests should look like. Thus, it should not come as a surprise that most existing web tests are not consistent with these recommendations, and are not even consistent with each other. *** note This document intentionally uses _should_ a lot more than _must_, as defined in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing web tests is a careful act of balancing many concerns, and this humble document cannot possibly capture the context that rests in the head of an experienced Blink engineer. *** ## General Principles This section contains guidelines adopted from [web-platform-tests documentation](https://web-platform-tests.org/writing-tests/general-guidelines.html) and [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree), with Blink-specific flavoring. ### Concise Tests should be **concise**, without compromising on the principles below. Every element and piece of code on the page should be necessary and relevant to what is being tested. For example, don't build a fully functional signup form if you only need a text field or a button. Content needed to satisfy the principles below is considered necessary. For example, it is acceptable and desirable to add elements that make the test self-describing (see below), and to add code that makes the test more reliable (see below). Content that makes test failures easier to debug is considered necessary (to maintaining a good development speed), and is both acceptable and desirable. *** promo Conciseness is particularly important for reference tests and pixel tests, as the test pages are rendered in an 800x600px viewport. Having content outside the viewport is undesirable because the outside content does not get compared, and because the resulting scrollbars are platform-specific UI widgets, making the test results less reliable. *** ### Fast Tests should be as **fast** as possible, without compromising on the principles below. Blink has several thousand web tests that are run in parallel, and avoiding unnecessary delays is crucial to keeping our Commit Queue in good shape. Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout), as it wastes time on the testing infrastructure. Instead, use specific event handlers, such as [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onload), to decide when to advance to the next step in a test. ### Reliable Tests should be **reliable** and yield consistent results for a given implementation. Flaky tests slow down your fellow developers' debugging efforts and the Commit Queue. `window.setTimeout` is again a primary offender here. Asides from wasting time on a fast system, tests that rely on fixed timeouts can fail when on systems that are slower than expected. When adding or significantly modifying a web test, use the command below to assess its flakiness. While not foolproof, this approach gives you some confidence, and giving up CPU cycles for mental energy is a pretty good trade. ```bash third_party/blink/tools/run_web_tests.py path/to/test.html --repeat-each=100 ``` The [PSA on writing reliable web tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit). also has good guidelines for writing reliable tests. ### Self-Describing Tests should be **self-describing**, so that a project member can recognize whether a test passes or fails without having to read the specification of the feature being tested. `testharness.js` makes a test self-describing when used correctly. Other types of tests, such as reference tests and [tests with manual fallback](./web_tests_with_manual_fallback.md), [must be carefully designed](https://web-platform-tests.org/writing-tests/manual.html#requirements-for-a-manual-test) to be self-describing. ### Minimal Tests should require a **minimal** amount of cognitive effort to read and maintain. Avoid depending on edge case behavior of features that aren't explicitly covered by the test. For example, except where testing parsing, tests should contain valid markup (no parsing errors). Tests should provide as much relevant information as possible when failing. `testharness.js` tests should prefer [rich assert_ functions](https://web-platform-tests.org/writing-tests/testharness-api.html#list-of-assertions) to combining `assert_true()` with a boolean operator. Using appropriate `assert_` functions results in better diagnostic output when the assertion fails. ### Cross-Platform Tests should be as **cross-platform** as reasonably possible. Avoid assumptions about device type, screen resolution, etc. Unavoidable assumptions should be documented. When possible, tests should only use Web platform features, as specified in the relevant standards. When the Web platform's APIs are insufficient, tests should prefer to use WPT extended testing APIs, such as `wpt_automation`, over Blink-specific testing APIs. Test pages should use the HTML5 doctype (``) unless they specifically cover [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode) behavior. Tests should avoid using features that haven't been shipped by the actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When unsure, check [caniuse.com](http://caniuse.com/). By necessity, this recommendation does not apply to the feature targeted by the test. *** note It may be tempting have a test for a bleeding-edge feature X depend on feature Y, which has only shipped in beta / development versions of various browsers. The reasoning would be that all browsers that implement X will have implemented Y. Please keep in mind that Chrome has un-shipped features that made it to the Beta channel in the past. *** *** aside [ES2015](http://benmccormick.org/2015/09/14/es5-es6-es2016-es-next-whats-going-on-with-javascript-versioning/) is shipped by all major browsers under active development (except for modules), so using ES2015 features is acceptable. At the time of this writing, ES2016 is not fully shipped in all major browsers. *** ### Self-Contained Tests must be **self-contained** and not depend on external network resources. Unless used by multiple test files, CSS and JavaScript should be inlined using `