py_testing.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426
  1. U-Boot pytest suite
  2. ===================
  3. Introduction
  4. ------------
  5. This tool aims to test U-Boot by executing U-Boot shell commands using the
  6. console interface. A single top-level script exists to execute or attach to the
  7. U-Boot console, run the entire script of tests against it, and summarize the
  8. results. Advantages of this approach are:
  9. - Testing is performed in the same way a user or script would interact with
  10. U-Boot; there can be no disconnect.
  11. - There is no need to write or embed test-related code into U-Boot itself.
  12. It is asserted that writing test-related code in Python is simpler and more
  13. flexible than writing it all in C.
  14. - It is reasonably simple to interact with U-Boot in this way.
  15. Requirements
  16. ------------
  17. The test suite is implemented using pytest. Interaction with the U-Boot console
  18. involves executing some binary and interacting with its stdin/stdout. You will
  19. need to implement various "hook" scripts that are called by the test suite at
  20. the appropriate time.
  21. In order to run the test suite at a minimum we require that both Python 3 and
  22. pip for Python 3 are installed. All of the required python modules are
  23. described in the requirements.txt file in the /test/py/ directory and can be
  24. installed via the command
  25. .. code-block:: bash
  26. pip install -r requirements.txt
  27. In order to execute certain tests on their supported platforms other tools
  28. will be required. The following is an incomplete list:
  29. * gdisk
  30. * dfu-util
  31. * dtc
  32. * openssl
  33. * sudo OR guestmount
  34. * e2fsprogs
  35. * util-linux
  36. * coreutils
  37. * dosfstools
  38. * efitools
  39. * mount
  40. * mtools
  41. * sbsigntool
  42. * udisks2
  43. Please use the appropriate commands for your distribution to match these tools
  44. up with the package that provides them.
  45. The test script supports either:
  46. - Executing a sandbox port of U-Boot on the local machine as a sub-process,
  47. and interacting with it over stdin/stdout.
  48. - Executing an external "hook" scripts to flash a U-Boot binary onto a
  49. physical board, attach to the board's console stream, and reset the board.
  50. Further details are described later.
  51. Using `virtualenv` to provide requirements
  52. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  53. The recommended way to run the test suite, in order to ensure reproducibility
  54. is to use `virtualenv` to set up the necessary environment. This can be done
  55. via the following commands:
  56. .. code-block:: console
  57. $ cd /path/to/u-boot
  58. $ sudo apt-get install python3 python3-virtualenv
  59. $ virtualenv -p /usr/bin/python3 venv
  60. $ . ./venv/bin/activate
  61. $ pip install -r test/py/requirements.txt
  62. Testing sandbox
  63. ---------------
  64. To run the test suite on the sandbox port (U-Boot built as a native user-space
  65. application), simply execute:
  66. .. code-block:: bash
  67. ./test/py/test.py --bd sandbox --build
  68. The `--bd` option tells the test suite which board type is being tested. This
  69. lets the test suite know which features the board has, and hence exactly what
  70. can be tested.
  71. The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
  72. omit this option and build U-Boot yourself, in whatever way you choose, before
  73. running the test script.
  74. The test script will attach to U-Boot, execute all valid tests for the board,
  75. then print a summary of the test process. A complete log of the test session
  76. will be written to `${build_dir}/test-log.html`. This is best viewed in a web
  77. browser, but may be read directly as plain text, perhaps with the aid of the
  78. `html2text` utility.
  79. Testing under a debugger
  80. ~~~~~~~~~~~~~~~~~~~~~~~~
  81. If you need to run sandbox under a debugger, you may pass the command-line
  82. option `--gdbserver COMM`. This causes two things to happens:
  83. - Instead of running U-Boot directly, it will be run under gdbserver, with
  84. debug communication via the channel `COMM`. You can attach a debugger to the
  85. sandbox process in order to debug it. See `man gdbserver` and the example
  86. below for details of valid values for `COMM`.
  87. - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
  88. time to execute commands. This is useful if U-Boot is stopped at a breakpoint
  89. during debugging.
  90. A usage example is:
  91. Window 1:
  92. .. code-block:: bash
  93. ./test/py/test.py --bd sandbox --gdbserver localhost:1234
  94. Window 2:
  95. .. code-block:: bash
  96. gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
  97. Alternatively, you could leave off the `-ex` option and type the command
  98. manually into gdb once it starts.
  99. You can use any debugger you wish, as long as it speaks the gdb remote
  100. protocol, or any graphical wrapper around gdb.
  101. Some tests deliberately cause the sandbox process to exit, e.g. to test the
  102. reset command, or sandbox's CTRL-C handling. When this happens, you will need
  103. to attach the debugger to the new sandbox instance. If these tests are not
  104. relevant to your debugging session, you can skip them using pytest's -k
  105. command-line option; see the next section.
  106. Command-line options
  107. --------------------
  108. --board-type, --bd, -B
  109. set the type of the board to be tested. For example, `sandbox` or `seaboard`.
  110. --board-identity`, --id
  111. sets the identity of the board to be tested. This allows differentiation
  112. between multiple instances of the same type of physical board that are
  113. attached to the same host machine. This parameter is not interpreted by th
  114. test script in any way, but rather is simply passed to the hook scripts
  115. described below, and may be used in any site-specific way deemed necessary.
  116. --build
  117. indicates that the test script should compile U-Boot itself before running
  118. the tests. If using this option, make sure that any environment variables
  119. required by the build process are already set, such as `$CROSS_COMPILE`.
  120. --buildman
  121. indicates that `--build` should use buildman to build U-Boot. There is no need
  122. to set $CROSS_COMPILE` in this case since buildman handles it.
  123. --build-dir
  124. sets the directory containing the compiled U-Boot binaries. If omitted, this
  125. is `${source_dir}/build-${board_type}`.
  126. --result-dir
  127. sets the directory to write results, such as log files, into.
  128. If omitted, the build directory is used.
  129. --persistent-data-dir
  130. sets the directory used to store persistent test data. This is test data that
  131. may be re-used across test runs, such as file-system images.
  132. `pytest` also implements a number of its own command-line options. Commonly used
  133. options are mentioned below. Please see `pytest` documentation for complete
  134. details. Execute `py.test --version` for a brief summary. Note that U-Boot's
  135. test.py script passes all command-line arguments directly to `pytest` for
  136. processing.
  137. -k
  138. selects which tests to run. The default is to run all known tests. This
  139. option takes a single argument which is used to filter test names. Simple
  140. logical operators are supported. For example:
  141. - `'-k ums'` runs only tests with "ums" in their name.
  142. - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
  143. case, "ut_dm" is a parameter to a test rather than the test name. The full
  144. test name is e.g. "test_ut[ut_dm_leak]".
  145. - `'-k not reset'` runs everything except tests with "reset" in their name.
  146. - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name.
  147. - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in
  148. their name.
  149. -s
  150. prevents pytest from hiding a test's stdout. This allows you to see
  151. U-Boot's console log in real time on pytest's stdout.
  152. Testing real hardware
  153. ---------------------
  154. The tools and techniques used to interact with real hardware will vary
  155. radically between different host and target systems, and the whims of the user.
  156. For this reason, the test suite does not attempt to directly interact with real
  157. hardware in any way. Rather, it executes a standardized set of "hook" scripts
  158. via `$PATH`. These scripts implement certain actions on behalf of the test
  159. suite. This keeps the test suite simple and isolated from system variances
  160. unrelated to U-Boot features.
  161. Hook scripts
  162. ~~~~~~~~~~~~
  163. Environment variables
  164. '''''''''''''''''''''
  165. The following environment variables are set when running hook scripts:
  166. - `UBOOT_BOARD_TYPE` the board type being tested.
  167. - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
  168. specified.
  169. - `UBOOT_SOURCE_DIR` the U-Boot source directory.
  170. - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
  171. - `UBOOT_BUILD_DIR` the U-Boot build directory.
  172. - `UBOOT_RESULT_DIR` the test result directory.
  173. - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
  174. u-boot-test-console
  175. '''''''''''''''''''
  176. This script provides access to the U-Boot console. The script's stdin/stdout
  177. should be connected to the board's console. This process should continue to run
  178. indefinitely, until killed. The test suite will run this script in parallel
  179. with all other hooks.
  180. This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc.
  181. via exec().
  182. If you are able to run U-Boot under a hardware simulator such as QEMU, then
  183. you would likely spawn that simulator from this script. However, note that
  184. `u-boot-test-reset` may be called multiple times per test script run, and must
  185. cause U-Boot to start execution from scratch each time. Hopefully your
  186. simulator includes a virtual reset button! If not, you can launch the
  187. simulator from `u-boot-test-reset` instead, while arranging for this console
  188. process to always communicate with the current simulator instance.
  189. u-boot-test-flash
  190. '''''''''''''''''
  191. Prior to running the test suite against a board, some arrangement must be made
  192. so that the board executes the particular U-Boot binary to be tested. Often
  193. this involves writing the U-Boot binary to the board's flash ROM. The test
  194. suite calls this hook script for that purpose.
  195. This script should perform the entire flashing process synchronously; the
  196. script should only exit once flashing is complete, and a board reset will
  197. cause the newly flashed U-Boot binary to be executed.
  198. It is conceivable that this script will do nothing. This might be useful in
  199. the following cases:
  200. - Some other process has already written the desired U-Boot binary into the
  201. board's flash prior to running the test suite.
  202. - The board allows U-Boot to be downloaded directly into RAM, and executed
  203. from there. Use of this feature will reduce wear on the board's flash, so
  204. may be preferable if available, and if cold boot testing of U-Boot is not
  205. required. If this feature is used, the `u-boot-test-reset` script should
  206. perform this download, since the board could conceivably be reset multiple
  207. times in a single test run.
  208. It is up to the user to determine if those situations exist, and to code this
  209. hook script appropriately.
  210. This script will typically be implemented by calling out to some SoC- or
  211. board-specific vendor flashing utility.
  212. u-boot-test-reset
  213. '''''''''''''''''
  214. Whenever the test suite needs to reset the target board, this script is
  215. executed. This is guaranteed to happen at least once, prior to executing the
  216. first test function. If any test fails, the test infra-structure will execute
  217. this script again to restore U-Boot to an operational state before running the
  218. next test function.
  219. This script will likely be implemented by communicating with some form of
  220. relay or electronic switch attached to the board's reset signal.
  221. The semantics of this script require that when it is executed, U-Boot will
  222. start running from scratch. If the U-Boot binary to be tested has been written
  223. to flash, pulsing the board's reset signal is likely all this script needs to
  224. do. However, in some scenarios, this script may perform other actions. For
  225. example, it may call out to some SoC- or board-specific vendor utility in order
  226. to download the U-Boot binary directly into RAM and execute it. This would
  227. avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
  228. saving wear on the flash chip(s).
  229. Examples
  230. ''''''''
  231. https://source.denx.de/u-boot/u-boot-test-hooks contains some working example hook
  232. scripts, and may be useful as a reference when implementing hook scripts for
  233. your platform. These scripts are not considered part of U-Boot itself.
  234. Board-type-specific configuration
  235. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  236. Each board has a different configuration and behaviour. Many of these
  237. differences can be automatically detected by parsing the `.config` file in the
  238. build directory. However, some differences can't yet be handled automatically.
  239. For each board, an optional Python module `u_boot_board_${board_type}` may exist
  240. to provide board-specific information to the test script. Any global value
  241. defined in these modules is available for use by any test function. The data
  242. contained in these scripts must be purely derived from U-Boot source code.
  243. Hence, these configuration files are part of the U-Boot source tree too.
  244. Execution environment configuration
  245. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  246. Each user's hardware setup may enable testing different subsets of the features
  247. implemented by a particular board's configuration of U-Boot. For example, a
  248. U-Boot configuration may support USB device mode and USB Mass Storage, but this
  249. can only be tested if a USB cable is connected between the board and the host
  250. machine running the test script.
  251. For each board, optional Python modules `u_boot_boardenv_${board_type}` and
  252. `u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
  253. board-specific and board-identity-specific information to the test script. Any
  254. global value defined in these modules is available for use by any test
  255. function. The data contained in these is specific to a particular user's
  256. hardware configuration. Hence, these configuration files are not part of the
  257. U-Boot source tree, and should be installed outside of the source tree. Users
  258. should set `$PYTHONPATH` prior to running the test script to allow these
  259. modules to be loaded.
  260. Board module parameter usage
  261. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  262. The test scripts rely on the following variables being defined by the board
  263. module:
  264. - none at present
  265. U-Boot `.config` feature usage
  266. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  267. The test scripts rely on various U-Boot `.config` features, either directly in
  268. order to test those features, or indirectly in order to query information from
  269. the running U-Boot instance in order to test other features.
  270. One example is that testing of the `md` command requires knowledge of a RAM
  271. address to use for the test. This data is parsed from the output of the
  272. `bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
  273. For a complete list of dependencies, please search the test scripts for
  274. instances of:
  275. - `buildconfig.get(...`
  276. - `@pytest.mark.buildconfigspec(...`
  277. - `@pytest.mark.notbuildconfigspec(...`
  278. Complete invocation example
  279. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  280. Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
  281. any required environment configuration Python modules into $HOME/ubtest/py,
  282. then you would likely invoke the test script as follows:
  283. If U-Boot has already been built:
  284. .. code-block:: bash
  285. PATH=$HOME/ubtest/bin:$PATH \
  286. PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
  287. ./test/py/test.py --bd seaboard
  288. If you want the test script to compile U-Boot for you too, then you likely
  289. need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
  290. follows:
  291. .. code-block:: bash
  292. CROSS_COMPILE=arm-none-eabi- \
  293. PATH=$HOME/ubtest/bin:$PATH \
  294. PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
  295. ./test/py/test.py --bd seaboard --build
  296. or, using buildman to handle it:
  297. .. code-block:: bash
  298. PATH=$HOME/ubtest/bin:$PATH \
  299. PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
  300. ./test/py/test.py --bd seaboard --build --buildman
  301. Writing tests
  302. -------------
  303. Please refer to the pytest documentation for details of writing pytest tests.
  304. Details specific to the U-Boot test suite are described below.
  305. A test fixture named `u_boot_console` should be used by each test function. This
  306. provides the means to interact with the U-Boot console, and retrieve board and
  307. environment configuration information.
  308. The function `u_boot_console.run_command()` executes a shell command on the
  309. U-Boot console, and returns all output from that command. This allows
  310. validation or interpretation of the command output. This function validates
  311. that certain strings are not seen on the U-Boot console. These include shell
  312. error messages and the U-Boot sign-on message (in order to detect unexpected
  313. board resets). See the source of `u_boot_console_base.py` for a complete list of
  314. "bad" strings. Some test scenarios are expected to trigger these strings. Use
  315. `u_boot_console.disable_check()` to temporarily disable checking for specific
  316. strings. See `test_unknown_cmd.py` for an example.
  317. Board- and board-environment configuration values may be accessed as sub-fields
  318. of the `u_boot_console.config` object, for example
  319. `u_boot_console.config.ram_base`.
  320. Build configuration values (from `.config`) may be accessed via the dictionary
  321. `u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
  322. names.