tests_sandbox.rst 6.7 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209
  1. .. SPDX-License-Identifier: GPL-2.0+
  2. Sandbox tests
  3. =============
  4. Test Design
  5. -----------
  6. Most uclasses and many functions of U-Boot have sandbox tests. This allows much
  7. of the code to be checked in an developer-friendly environment.
  8. Sandbox provides a way to write and run unit tests. The traditional approach to
  9. unit tests is to build lots of little executables, one for each test or
  10. category of tests. With sandbox, so far as possible, all the tests share a
  11. small number of executables (e.g. 'u-boot' for sandbox, 'u-boot-spl' and
  12. 'u-boot' for sandbox_spl) and can be run very quickly. The vast majority of
  13. tests can run on the 'sandbox' build,
  14. Available tests
  15. ---------------
  16. Some of the available tests are:
  17. - command_ut: Unit tests for command parsing and handling
  18. - compression: Unit tests for U-Boot's compression algorithms, useful for
  19. security checking. It supports gzip, bzip2, lzma and lzo.
  20. - image: Unit tests for images:
  21. - test/image/test-imagetools.sh - multi-file images
  22. - test/py/tests/test-fit.py - FIT images
  23. - tracing: test/trace/test-trace.sh tests the tracing system (see
  24. README.trace)
  25. - verified boot: test/py/tests/test_vboot.py
  26. If you change or enhance any U-Boot subsystem, you should write or expand a
  27. test and include it with your patch series submission. Test coverage in some
  28. older areas of U-Boot is still somewhat limited and we need to work to improve
  29. it.
  30. Note that many of these tests are implemented as commands which you can
  31. run natively on your board if desired (and enabled).
  32. To run all tests, use 'make check'.
  33. Running sandbox tests directly
  34. ------------------------------
  35. Typically tests are run using the pytest suite. Running pytests on sandbox is
  36. easy and always gets things right. For example some tests require files to be
  37. set up before they can work.
  38. But it is also possible to run some sandbox tests directly. For example, this
  39. runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
  40. $ ./u-boot -T -c "ut dm gpio"
  41. U-Boot 2021.01
  42. Model: sandbox
  43. DRAM: 128 MiB
  44. WDT: Started with servicing (60s timeout)
  45. MMC: mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
  46. In: serial
  47. Out: vidconsole
  48. Err: vidconsole
  49. Model: sandbox
  50. SCSI:
  51. Net: eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: eth@10004000
  52. Test: dm_test_gpio: gpio.c
  53. Test: dm_test_gpio: gpio.c (flat tree)
  54. Failures: 0
  55. The -T option tells the U-Boot sandbox to run with the 'test' devicetree
  56. (test.dts) instead of -D which selects the normal sandbox.dts - this is
  57. necessary because many tests rely on nodes or properties in the test devicetree.
  58. If you try running tests without -T then you may see failures, like::
  59. $ ./u-boot -c "ut dm gpio"
  60. U-Boot 2021.01
  61. DRAM: 128 MiB
  62. WDT: Not found!
  63. MMC:
  64. In: serial
  65. Out: serial
  66. Err: serial
  67. SCSI:
  68. Net: No ethernet found.
  69. Please run with test device tree:
  70. ./u-boot -d arch/sandbox/dts/test.dtb
  71. Test: dm_test_gpio: gpio.c
  72. test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
  73. Test: dm_test_gpio: gpio.c (flat tree)
  74. test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
  75. Failures: 2
  76. The message above should provide a hint if you forget to use the -T flag. Even
  77. running with -D will produce different results.
  78. You can easily use gdb on these tests, without needing --gdbserver::
  79. $ gdb u-boot --args -T -c "ut dm gpio"
  80. ...
  81. (gdb) break dm_test_gpio
  82. Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37.
  83. (gdb) run -T -c "ut dm gpio"
  84. Starting program: u-boot -T -c "ut dm gpio"
  85. Test: dm_test_gpio: gpio.c
  86. Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 <global_dm_test_state>)
  87. at files/test/dm/gpio.c:37
  88. 37 ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio));
  89. (gdb)
  90. You can then single-step and look at variables as needed.
  91. Running sandbox_spl tests directly
  92. ----------------------------------
  93. SPL is the phase before U-Boot proper. It is present in the sandbox_spl build,
  94. so you can run SPL like this::
  95. ./spl/u-boot-spl
  96. SPL tests are special in that they run (only in the SPL phase, of course) if the
  97. -u flag is given::
  98. ./spl/u-boot-spl -u
  99. U-Boot SPL 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700)
  100. Running 5 driver model tests
  101. Test: dm_test_of_plat_base: of_platdata.c (flat tree)
  102. Test: dm_test_of_plat_dev: of_platdata.c (flat tree)
  103. Test: dm_test_of_plat_parent: of_platdata.c (flat tree)
  104. Test: dm_test_of_plat_phandle: of_platdata.c (flat tree)
  105. Test: dm_test_of_plat_props: of_platdata.c (flat tree)
  106. Failures: 0
  107. U-Boot 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700)
  108. DRAM: 128 MiB
  109. ...
  110. It is not possible to run SPL tests in U-Boot proper, firstly because they are
  111. not built into U-Boot proper and secondly because the environment is very
  112. different, e.g. some SPL tests rely on of-platdata which is only available in
  113. SPL.
  114. Note that after running, SPL continues to boot into U-Boot proper. You can add
  115. '-c exit' to make U-Boot quit without doing anything further. It is not
  116. currently possible to run SPL tests and then stop, since the pytests require
  117. that U-Boot produces the expected banner.
  118. You can use the -k flag to select which tests run::
  119. ./spl/u-boot-spl -u -k dm_test_of_plat_parent
  120. Of course you can use gdb with sandbox_spl, just as with sandbox.
  121. Running all tests directly
  122. --------------------------
  123. A fast way to run all sandbox tests is::
  124. ./u-boot -T -c "ut all"
  125. It typically runs single-thread in 6 seconds on 2021 hardware, with 2s of that
  126. to the delays in the time test.
  127. This should not be considered a substitute for 'make check', but can be helpful
  128. for git bisect, etc.
  129. What tests are built in?
  130. ------------------------
  131. Whatever sandbox build is used, which tests are present is determined by which
  132. source files are built. For sandbox_spl, the of_platdata tests are built
  133. because of the build rule in test/dm/Makefile::
  134. ifeq ($(CONFIG_SPL_BUILD),y)
  135. obj-$(CONFIG_SPL_OF_PLATDATA) += of_platdata.o
  136. else
  137. ...other tests for non-spl
  138. endif
  139. You can get a list of tests in a U-Boot ELF file by looking for the
  140. linker_list::
  141. $ nm /tmp/b/sandbox_spl/spl/u-boot-spl |grep 2_dm_test
  142. 000000000001f200 D _u_boot_list_2_dm_test_2_dm_test_of_plat_base
  143. 000000000001f220 D _u_boot_list_2_dm_test_2_dm_test_of_plat_dev
  144. 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent
  145. 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle
  146. 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props
  147. Writing tests
  148. -------------
  149. See :doc:`tests_writing` for how to write new tests.