contribute.txt 29 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685
  1. // -*- mode:doc; -*-
  2. // vim: set syntax=asciidoc:
  3. == Contributing to Buildroot
  4. There are many ways in which you can contribute to Buildroot: analyzing
  5. and fixing bugs, analyzing and fixing package build failures detected by
  6. the autobuilders, testing and reviewing patches sent by other
  7. developers, working on the items in our TODO list and sending your own
  8. improvements to Buildroot or its manual. The following sections give a
  9. little more detail on each of these items.
  10. If you are interested in contributing to Buildroot, the first thing you
  11. should do is to subscribe to the Buildroot mailing list. This list is
  12. the main way of interacting with other Buildroot developers and to send
  13. contributions to. If you aren't subscribed yet, then refer to
  14. xref:community-resources[] for the subscription link.
  15. If you are going to touch the code, it is highly recommended to use a
  16. git repository of Buildroot, rather than starting from an extracted
  17. source code tarball. Git is the easiest way to develop from and directly
  18. send your patches to the mailing list. Refer to xref:getting-buildroot[]
  19. for more information on obtaining a Buildroot git tree.
  20. === Reproducing, analyzing and fixing bugs
  21. A first way of contributing is to have a look at the open bug reports in
  22. the https://bugs.buildroot.org/buglist.cgi?product=buildroot[Buildroot bug
  23. tracker]. As we strive to keep the bug count as small as possible, all
  24. help in reproducing, analyzing and fixing reported bugs is more than
  25. welcome. Don't hesitate to add a comment to bug reports reporting your
  26. findings, even if you don't yet see the full picture.
  27. === Analyzing and fixing autobuild failures
  28. The Buildroot autobuilders are a set of build machines that continuously
  29. run Buildroot builds based on random configurations. This is done for
  30. all architectures supported by Buildroot, with various toolchains, and
  31. with a random selection of packages. With the large commit activity on
  32. Buildroot, these autobuilders are a great help in detecting problems
  33. very early after commit.
  34. All build results are available at http://autobuild.buildroot.org[],
  35. statistics are at http://autobuild.buildroot.org/stats.php[]. Every day,
  36. an overview of all failed packages is sent to the mailing list.
  37. Detecting problems is great, but obviously these problems have to be
  38. fixed as well. Your contribution is very welcome here! There are
  39. basically two things that can be done:
  40. - Analyzing the problems. The daily summary mails do not contain details
  41. about the actual failures: in order to see what's going on you have to
  42. open the build log and check the last output. Having someone doing
  43. this for all packages in the mail is very useful for other developers,
  44. as they can make a quick initial analysis based on this output alone.
  45. - Fixing a problem. When fixing autobuild failures, you should follow
  46. these steps:
  47. . Check if you can reproduce the problem by building with the same
  48. configuration. You can do this manually, or use the
  49. http://git.buildroot.org/buildroot-test/tree/utils/br-reproduce-build[br-reproduce-build]
  50. script that will automatically clone a Buildroot git repository,
  51. checkout the correct revision, download and set the right
  52. configuration, and start the build.
  53. . Analyze the problem and create a fix.
  54. . Verify that the problem is really fixed by starting from a clean
  55. Buildroot tree and only applying your fix.
  56. . Send the fix to the Buildroot mailing list (see
  57. xref:submitting-patches[]). In case you created a patch against the
  58. package sources, you should also send the patch upstream so that the
  59. problem will be fixed in a later release, and the patch in Buildroot
  60. can be removed.
  61. In the commit message of a patch fixing an autobuild failure, add a
  62. reference to the build result directory, as follows:
  63. ---------------------
  64. Fixes: http://autobuild.buildroot.org/results/51000a9d4656afe9e0ea6f07b9f8ed374c2e4069
  65. ---------------------
  66. === Reviewing and testing patches
  67. With the amount of patches sent to the mailing list each day, the
  68. maintainer has a very hard job to judge which patches are ready to apply
  69. and which ones aren't. Contributors can greatly help here by reviewing
  70. and testing these patches.
  71. In the review process, do not hesitate to respond to patch submissions
  72. for remarks, suggestions or anything that will help everyone to
  73. understand the patches and make them better. Please use internet
  74. style replies in plain text emails when responding to patch
  75. submissions.
  76. To indicate approval of a patch, there are three formal tags that keep
  77. track of this approval. To add your tag to a patch, reply to it with the
  78. approval tag below the original author's Signed-off-by line. These tags
  79. will be picked up automatically by patchwork (see
  80. xref:apply-patches-patchwork[]) and will be part of the commit log when
  81. the patch is accepted.
  82. Tested-by:: Indicates that the patch has been tested successfully.
  83. You are encouraged to specify what kind of testing you performed
  84. (compile-test on architecture X and Y, runtime test on target A,
  85. ...). This additional information helps other testers and the
  86. maintainer.
  87. Reviewed-by:: Indicates that you code-reviewed the patch and did your
  88. best in spotting problems, but you are not sufficiently familiar with
  89. the area touched to provide an Acked-by tag. This means that there
  90. may be remaining problems in the patch that would be spotted by
  91. someone with more experience in that area. Should such problems be
  92. detected, your Reviewed-by tag remains appropriate and you cannot
  93. be blamed.
  94. Acked-by:: Indicates that you code-reviewed the patch and you are
  95. familiar enough with the area touched to feel that the patch can be
  96. committed as-is (no additional changes required). In case it later
  97. turns out that something is wrong with the patch, your Acked-by could
  98. be considered inappropriate. The difference between Acked-by and
  99. Reviewed-by is thus mainly that you are prepared to take the blame on
  100. Acked patches, but not on Reviewed ones.
  101. If you reviewed a patch and have comments on it, you should simply reply
  102. to the patch stating these comments, without providing a Reviewed-by or
  103. Acked-by tag. These tags should only be provided if you judge the patch
  104. to be good as it is.
  105. It is important to note that neither Reviewed-by nor Acked-by imply
  106. that testing has been performed. To indicate that you both reviewed and
  107. tested the patch, provide two separate tags (Reviewed/Acked-by and
  108. Tested-by).
  109. Note also that _any developer_ can provide Tested/Reviewed/Acked-by
  110. tags, without exception, and we encourage everyone to do this. Buildroot
  111. does not have a defined group of _core_ developers, it just so happens
  112. that some developers are more active than others. The maintainer will
  113. value tags according to the track record of their submitter. Tags
  114. provided by a regular contributor will naturally be trusted more than
  115. tags provided by a newcomer. As you provide tags more regularly, your
  116. 'trustworthiness' (in the eyes of the maintainer) will go up, but _any_
  117. tag provided is valuable.
  118. Buildroot's Patchwork website can be used to pull in patches for testing
  119. purposes. Please see xref:apply-patches-patchwork[] for more
  120. information on using Buildroot's Patchwork website to apply patches.
  121. [[apply-patches-patchwork]]
  122. ==== Applying Patches from Patchwork
  123. The main use of Buildroot's Patchwork website for a developer is for
  124. pulling in patches into their local git repository for testing
  125. purposes.
  126. When browsing patches in the patchwork management interface, an +mbox+
  127. link is provided at the top of the page. Copy this link address and
  128. run the following commands:
  129. ---------------------
  130. $ git checkout -b <test-branch-name>
  131. $ wget -O - <mbox-url> | git am
  132. ---------------------
  133. Another option for applying patches is to create a bundle. A bundle is
  134. a set of patches that you can group together using the patchwork
  135. interface. Once the bundle is created and the bundle is made public,
  136. you can copy the +mbox+ link for the bundle and apply the bundle
  137. using the above commands.
  138. === Work on items from the TODO list
  139. If you want to contribute to Buildroot but don't know where to start,
  140. and you don't like any of the above topics, you can always work on items
  141. from the http://elinux.org/Buildroot#Todo_list[Buildroot TODO list].
  142. Don't hesitate to discuss an item first on the mailing list or on IRC.
  143. Do edit the wiki to indicate when you start working on an item, so we
  144. avoid duplicate efforts.
  145. [[submitting-patches]]
  146. === Submitting patches
  147. [NOTE]
  148. _Please, do not attach patches to bugs, send them to the mailing list
  149. instead_.
  150. If you made some changes to Buildroot and you would like to contribute
  151. them to the Buildroot project, proceed as follows.
  152. ==== The formatting of a patch
  153. We expect patches to be formatted in a specific way. This is necessary
  154. to make it easy to review patches, to be able to apply them easily to
  155. the git repository, to make it easy to find back in the history how
  156. and why things have changed, and to make it possible to use +git
  157. bisect+ to locate the origin of a problem.
  158. First of all, it is essential that the patch has a good commit
  159. message. The commit message should start with a separate line with a
  160. brief summary of the change, prefixed by the area touched by the
  161. patch. A few examples of good commit titles:
  162. * +package/linuxptp: bump version to 2.0+
  163. * +configs/imx23evk: bump Linux version to 4.19+
  164. * +package/pkg-generic: postpone evaluation of dependency conditions+
  165. * +boot/uboot: needs host-{flex,bison}+
  166. * +support/testing: add python-ubjson tests+
  167. The description that follows the prefix should start with a lower case
  168. letter (i.e "bump", "needs", "postpone", "add" in the above examples).
  169. Second, the body of the commit message should describe _why_ this
  170. change is needed, and if necessary also give details about _how_ it
  171. was done. When writing the commit message, think of how the reviewers
  172. will read it, but also think about how you will read it when you look
  173. at this change again a few years down the line.
  174. Third, the patch itself should do only one change, but do it
  175. completely. Two unrelated or weakly related changes should usually be
  176. done in two separate patches. This usually means that a patch affects
  177. only a single package. If several changes are related, it is often
  178. still possible to split them up in small patches and apply them in a
  179. specific order. Small patches make it easier to review, and often
  180. make it easier to understand afterwards why a change was done.
  181. However, each patch must be complete. It is not allowed that the
  182. build is broken when only the first but not the second patch is
  183. applied. This is necessary to be able to use +git bisect+ afterwards.
  184. Of course, while you're doing your development, you're probably going
  185. back and forth between packages, and certainly not committing things
  186. immediately in a way that is clean enough for submission. So most
  187. developers rewrite the history of commits to produce a clean set of
  188. commits that is appropriate for submission. To do this, you need to
  189. use _interactive rebasing_. You can learn about it
  190. https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History[in the Pro
  191. Git book]. Sometimes, it is even easier to discard you history with
  192. +git reset --soft origin/master+ and select individual changes with
  193. +git add -i+ or +git add -p+.
  194. Finally, the patch should be signed off. This is done by adding
  195. +Signed-off-by: Your Real Name <your@email.address>+ at the end of the
  196. commit message. +git commit -s+ does that for you, if configured
  197. properly. The +Signed-off-by+ tag means that you publish the patch
  198. under the Buildroot license (i.e. GPL-2.0+, except for package patches,
  199. which have the upstream license), and that you are allowed to do so.
  200. See http://developercertificate.org/[the Developer Certificate of
  201. Origin] for details.
  202. When adding new packages, you should submit every package in a
  203. separate patch. This patch should have the update to
  204. +package/Config.in+, the package +Config.in+ file, the +.mk+ file, the
  205. +.hash+ file, any init script, and all package patches. If the package
  206. has many sub-options, these are sometimes better added as separate
  207. follow-up patches. The summary line should be something like
  208. +<packagename>: new package+. The body of the commit message can be
  209. empty for simple packages, or it can contain the description of the
  210. package (like the Config.in help text). If anything special has to be
  211. done to build the package, this should also be explained explicitly in
  212. the commit message body.
  213. When you bump a package to a new version, you should also submit a
  214. separate patch for each package. Don't forget to update the +.hash+
  215. file, or add it if it doesn't exist yet. Also don't forget to check if
  216. the +_LICENSE+ and +_LICENSE_FILES+ are still valid. The summary line
  217. should be something like +<packagename>: bump to version <new
  218. version>+. If the new version only contains security updates compared
  219. to the existing one, the summary should be +<packagename>: security
  220. bump to version <new version>+ and the commit message body should show
  221. the CVE numbers that are fixed. If some package patches can be removed
  222. in the new version, it should be explained explicitly why they can be
  223. removed, preferably with the upstream commit ID. Also any other
  224. required changes should be explained explicitly, like configure
  225. options that no longer exist or are no longer needed.
  226. If you are interested in getting notified of build failures and of
  227. further changes in the packages you added or modified, please add
  228. yourself to the DEVELOPERS file. This should be done in the same patch
  229. creating or modifying the package. See xref:DEVELOPERS[the DEVELOPERS file]
  230. for more information.
  231. Buildroot provides a handy tool to check for common coding style
  232. mistakes on files you created or modified, called +check-package+ (see
  233. xref:check-package[] for more information).
  234. ==== Preparing a patch series
  235. Starting from the changes committed in your local git view, _rebase_
  236. your development branch on top of the upstream tree before generating
  237. a patch set. To do so, run:
  238. ---------------------
  239. $ git fetch --all --tags
  240. $ git rebase origin/master
  241. ---------------------
  242. Now, you are ready to generate then submit your patch set.
  243. To generate it, run:
  244. ---------------------
  245. $ git format-patch -M -n -s -o outgoing origin/master
  246. ---------------------
  247. This will generate patch files in the +outgoing+ subdirectory,
  248. automatically adding the +Signed-off-by+ line.
  249. Once patch files are generated, you can review/edit the commit message
  250. before submitting them, using your favorite text editor.
  251. Buildroot provides a handy tool to know to whom your patches should be
  252. sent, called +get-developers+ (see xref:DEVELOPERS[] for more
  253. information). This tool reads your patches and outputs the appropriate
  254. +git send-email+ command to use:
  255. ---------------------
  256. $ ./utils/get-developers outgoing/*
  257. ---------------------
  258. Use the output of +get-developers+ to send your patches:
  259. ---------------------
  260. $ git send-email --to buildroot@buildroot.org --cc bob --cc alice outgoing/*
  261. ---------------------
  262. Alternatively, +get-developers -e+ can be used directly with the
  263. +--cc-cmd+ argument to +git send-email+ to automatically CC the
  264. affected developers:
  265. ---------------------
  266. $ git send-email --to buildroot@buildroot.org \
  267. --cc-cmd './utils/get-developers -e' origin/master
  268. ---------------------
  269. +git+ can be configured to automatically do this out of the box with:
  270. ---------------------
  271. $ git config sendemail.to buildroot@buildroot.org
  272. $ git config sendemail.ccCmd "$(pwd)/utils/get-developers -e"
  273. ---------------------
  274. And then just do:
  275. ---------------------
  276. $ git send-email origin/master
  277. ---------------------
  278. Note that +git+ should be configured to use your mail account.
  279. To configure +git+, see +man git-send-email+ or google it.
  280. If you do not use +git send-email+, make sure posted *patches are not
  281. line-wrapped*, otherwise they cannot easily be applied. In such a case,
  282. fix your e-mail client, or better yet, learn to use +git send-email+.
  283. ==== Cover letter
  284. If you want to present the whole patch set in a separate mail, add
  285. +--cover-letter+ to the +git format-patch+ command (see +man
  286. git-format-patch+ for further information). This will generate a
  287. template for an introduction e-mail to your patch series.
  288. A 'cover letter' may be useful to introduce the changes you propose
  289. in the following cases:
  290. * large number of commits in the series;
  291. * deep impact of the changes in the rest of the project;
  292. * RFC footnote:[RFC: (Request for comments) change proposal];
  293. * whenever you feel it will help presenting your work, your choices,
  294. the review process, etc.
  295. ==== Patches for maintenance branches
  296. When fixing bugs on a maintenance branch, bugs should be fixed on the
  297. master branch first. The commit log for such a patch may then contain a
  298. post-commit note specifying what branches are affected:
  299. ----
  300. package/foo: fix stuff
  301. Signed-off-by: Your Real Name <your@email.address>
  302. ---
  303. Backport to: 2020.02.x, 2020.05.x
  304. (2020.08.x not affected as the version was bumped)
  305. ----
  306. Those changes will then be backported by a maintainer to the affected
  307. branches.
  308. However, some bugs may apply only to a specific release, for example
  309. because it is using an older version of a package. In that case, patches
  310. should be based off the maintenance branch, and the patch subject prefix
  311. must include the maintenance branch name (for example "[PATCH 2020.02.x]").
  312. This can be done with the +git format-patch+ flag +--subject-prefix+:
  313. ---------------------
  314. $ git format-patch --subject-prefix "PATCH 2020.02.x" \
  315. -M -s -o outgoing origin/2020.02.x
  316. ---------------------
  317. Then send the patches with +git send-email+, as described above.
  318. ==== Patch revision changelog
  319. When improvements are requested, the new revision of each commit
  320. should include a changelog of the modifications between each
  321. submission. Note that when your patch series is introduced by a cover
  322. letter, an overall changelog may be added to the cover letter in
  323. addition to the changelog in the individual commits.
  324. The best thing to rework a patch series is by interactive rebasing:
  325. +git rebase -i origin/master+. Consult the git manual for more
  326. information.
  327. When added to the individual commits, this changelog is added when
  328. editing the commit message. Below the +Signed-off-by+ section, add
  329. +---+ and your changelog.
  330. Although the changelog will be visible for the reviewers in the mail
  331. thread, as well as in http://patchwork.buildroot.org[patchwork], +git+
  332. will automatically ignores lines below +---+ when the patch will be
  333. merged. This is the intended behavior: the changelog is not meant to
  334. be preserved forever in the +git+ history of the project.
  335. Hereafter the recommended layout:
  336. ---------------
  337. Patch title: short explanation, max 72 chars
  338. A paragraph that explains the problem, and how it manifests itself. If
  339. the problem is complex, it is OK to add more paragraphs. All paragraphs
  340. should be wrapped at 72 characters.
  341. A paragraph that explains the root cause of the problem. Again, more
  342. than one paragraph is OK.
  343. Finally, one or more paragraphs that explain how the problem is solved.
  344. Don't hesitate to explain complex solutions in detail.
  345. Signed-off-by: John DOE <john.doe@example.net>
  346. ---
  347. Changes v2 -> v3:
  348. - foo bar (suggested by Jane)
  349. - bar buz
  350. Changes v1 -> v2:
  351. - alpha bravo (suggested by John)
  352. - charly delta
  353. ---------------
  354. Any patch revision should include the version number. The version number
  355. is simply composed of the letter +v+ followed by an +integer+ greater or
  356. equal to two (i.e. "PATCH v2", "PATCH v3" ...).
  357. This can be easily handled with +git format-patch+ by using the option
  358. +--subject-prefix+:
  359. ---------------------
  360. $ git format-patch --subject-prefix "PATCH v4" \
  361. -M -s -o outgoing origin/master
  362. ---------------------
  363. Since git version 1.8.1, you can also use +-v <n>+ (where <n> is the
  364. version number):
  365. ---------------------
  366. $ git format-patch -v4 -M -s -o outgoing origin/master
  367. ---------------------
  368. When you provide a new version of a patch, please mark the old one as
  369. superseded in http://patchwork.buildroot.org[patchwork]. You need to
  370. create an account on http://patchwork.buildroot.org[patchwork] to be
  371. able to modify the status of your patches. Note that you can only change
  372. the status of patches you submitted yourself, which means the email
  373. address you register in http://patchwork.buildroot.org[patchwork] should
  374. match the one you use for sending patches to the mailing list.
  375. You can also add the +--in-reply-to <message-id>+ option when
  376. submitting a patch to the mailing list. The id of the mail to reply to
  377. can be found under the "Message Id" tag on
  378. http://patchwork.buildroot.org[patchwork]. The advantage of
  379. *in-reply-to* is that patchwork will automatically mark the previous
  380. version of the patch as superseded.
  381. [[reporting-bugs]]
  382. === Reporting issues/bugs or getting help
  383. Before reporting any issue, please check in
  384. xref:community-resources[the mailing list archive] whether someone has
  385. already reported and/or fixed a similar problem.
  386. However you choose to report bugs or get help, either by
  387. opening a bug in the xref:community-resources[bug tracker] or by
  388. xref:community-resources[sending a mail to the mailing list], there are
  389. a number of details to provide in order to help people reproduce and
  390. find a solution to the issue.
  391. Try to think as if you were trying to help someone else; in
  392. that case, what would you need?
  393. Here is a short list of details to provide in such case:
  394. * host machine (OS/release)
  395. * version of Buildroot
  396. * target for which the build fails
  397. * package(s) for which the build fails
  398. * the command that fails and its output
  399. * any information you think that may be relevant
  400. Additionally, you should add the +.config+ file (or if you know how, a
  401. +defconfig+; see xref:customize-store-buildroot-config[]).
  402. If some of these details are too large, do not hesitate to use a
  403. pastebin service. Note that not all available pastebin services will
  404. preserve Unix-style line terminators when downloading raw pastes.
  405. Following pastebin services are known to work correctly:
  406. - https://gist.github.com/
  407. - http://code.bulix.org/
  408. === Using the run-tests framework
  409. Buildroot includes a run-time testing framework called run-tests built
  410. upon Python scripting and QEMU runtime execution. There are two types of
  411. test cases within the framework, one for build time tests and another for
  412. run-time tests that have a QEMU dependency. The goals of the framework are
  413. the following:
  414. * build a well defined configuration
  415. * optionally, verify some properties of the build output
  416. * if it is a run-time test:
  417. ** boot it under QEMU
  418. ** run some test condition to verify that a given feature is working
  419. The run-tests tool has a series of options documented in the tool's help '-h'
  420. description. Some common options include setting the download folder, the
  421. output folder, keeping build output, and for multiple test cases, you can set
  422. the JLEVEL for each.
  423. Here is an example walk through of running a test case.
  424. * For a first step, let us see what all the test case options are. The test
  425. cases can be listed by executing +support/testing/run-tests -l+. These tests
  426. can all be run individually during test development from the console. Both
  427. one at a time and selectively as a group of a subset of tests.
  428. ---------------------
  429. $ support/testing/run-tests -l
  430. List of tests
  431. test_run (tests.utils.test_check_package.TestCheckPackage)
  432. Test the various ways the script can be called in a simple top to ... ok
  433. test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootMusl) ... ok
  434. test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc) ... ok
  435. test_run (tests.toolchain.test_external.TestExternalToolchainCCache) ... ok
  436. test_run (tests.toolchain.test_external.TestExternalToolchainCtngMusl) ... ok
  437. test_run (tests.toolchain.test_external.TestExternalToolchainLinaroArm) ... ok
  438. test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv4) ... ok
  439. test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv5) ... ok
  440. test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv7) ... ok
  441. [snip]
  442. test_run (tests.init.test_systemd.TestInitSystemSystemdRoFull) ... ok
  443. test_run (tests.init.test_systemd.TestInitSystemSystemdRoIfupdown) ... ok
  444. test_run (tests.init.test_systemd.TestInitSystemSystemdRoNetworkd) ... ok
  445. test_run (tests.init.test_systemd.TestInitSystemSystemdRwFull) ... ok
  446. test_run (tests.init.test_systemd.TestInitSystemSystemdRwIfupdown) ... ok
  447. test_run (tests.init.test_systemd.TestInitSystemSystemdRwNetworkd) ... ok
  448. test_run (tests.init.test_busybox.TestInitSystemBusyboxRo) ... ok
  449. test_run (tests.init.test_busybox.TestInitSystemBusyboxRoNet) ... ok
  450. test_run (tests.init.test_busybox.TestInitSystemBusyboxRw) ... ok
  451. test_run (tests.init.test_busybox.TestInitSystemBusyboxRwNet) ... ok
  452. Ran 157 tests in 0.021s
  453. OK
  454. ---------------------
  455. Those runtime tests are regularly executed by Buildroot Gitlab CI
  456. infrastructure, see .gitlab.yml and https://gitlab.com/buildroot.org/buildroot/-/jobs.
  457. ==== Creating a test case
  458. The best way to get familiar with how to create a test case is to look at a
  459. few of the basic file system +support/testing/tests/fs/+ and init
  460. +support/testing/tests/init/+ test scripts. Those tests give good examples
  461. of a basic build and build with run type of tests. There are other more
  462. advanced cases that use things like nested +br2-external+ folders to provide
  463. skeletons and additional packages.
  464. The test cases by default use a br-arm-full-* uClibc-ng toolchain and the
  465. prebuild kernel for a armv5/7 cpu. It is recommended to use the default
  466. defconfig test configuration except when Glibc/musl or a newer kernel are
  467. necessary. By using the default it saves build time and the test would
  468. automatically inherit a kernel/std library upgrade when the default is
  469. updated.
  470. The basic test case definition involves
  471. * Creation of a new test file
  472. * Defining a unique test class
  473. * Determining if the default defconfig plus test options can be used
  474. * Implementing a +def test_run(self):+ function to optionally startup the
  475. emulator and provide test case conditions.
  476. After creating the test script, add yourself to the +DEVELOPERS+ file to
  477. be the maintainer of that test case.
  478. ==== Debugging a test case
  479. Within the Buildroot repository, the testing framework is organized at the
  480. top level in +support/testing/+ by folders of +conf+, +infra+ and +tests+.
  481. All the test cases live under the +test+ folder and are organized in various
  482. folders representing the catagory of test.
  483. Lets walk through an example.
  484. * Using the Busybox Init system test case with a read/write rootfs
  485. +tests.init.test_busybox.TestInitSystemBusyboxRw+
  486. * A minimal set of command line arguments when debugging a test case would
  487. include '-d' which points to your dl folder, '-o' to an output folder, and
  488. '-k' to keep any output on both pass/fail. With those options, the test will
  489. retain logging and build artifacts providing status of the build and
  490. execution of the test case.
  491. ---------------------
  492. $ support/testing/run-tests -d dl -o output_folder -k tests.init.test_busybox.TestInitSystemBusyboxRw
  493. 15:03:26 TestInitSystemBusyboxRw Starting
  494. 15:03:28 TestInitSystemBusyboxRw Building
  495. 15:08:18 TestInitSystemBusyboxRw Building done
  496. 15:08:27 TestInitSystemBusyboxRw Cleaning up
  497. .
  498. Ran 1 test in 301.140s
  499. OK
  500. ---------------------
  501. * For the case of a successful build, the +output_folder+ would contain a
  502. <test name> folder with the Buildroot build, build log and run-time log. If
  503. the build failed, the console output would show the stage at which it failed
  504. (setup / build / run). Depending on the failure stage, the build/run logs
  505. and/or Buildroot build artifacts can be inspected and instrumented. If the
  506. QEMU instance needs to be launched for additional testing, the first few
  507. lines of the run-time log capture it and it would allow some incremental
  508. testing without re-running +support/testing/run-tests+.
  509. * You can also make modifications to the current sources inside the
  510. +output_folder+ (e.g. for debug purposes) and rerun the standard
  511. Buildroot make targets (in order to regenerate the complete image with
  512. the new modifications) and then rerun the test. Modifying the sources
  513. directly can speed up debugging compared to adding patch files, wiping the
  514. output directoy, and starting the test again.
  515. ---------------------
  516. $ ls output_folder/
  517. TestInitSystemBusyboxRw/
  518. TestInitSystemBusyboxRw-build.log
  519. TestInitSystemBusyboxRw-run.log
  520. ---------------------
  521. * The source file used to implement this example test is found under
  522. +support/testing/tests/init/test_busybox.py+. This file outlines the
  523. minimal defconfig that creates the build, QEMU configuration to launch
  524. the built images and the test case assertions.
  525. To test an existing or new test case within Gitlab CI, there is a method of
  526. invoking a specific test by creating a Buildroot fork in Gitlab under your
  527. account. This can be handy when adding/changing a run-time test or fixing a
  528. bug on a use case tested by a run-time test case.
  529. In the examples below, the <name> component of the branch name is a unique
  530. string you choose to identify this specific job being created.
  531. * to trigger all run-test test case jobs:
  532. ---------------------
  533. $ git push gitlab HEAD:<name>-runtime-tests
  534. ---------------------
  535. * to trigger one test case job, a specific branch naming string is used that
  536. includes the full test case name.
  537. ---------------------
  538. $ git push gitlab HEAD:<name>-<test case name>
  539. ---------------------