bitbake-user-manual-execution.rst 31 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733
  1. .. SPDX-License-Identifier: CC-BY-2.5
  2. =========
  3. Execution
  4. =========
  5. |
  6. The primary purpose for running BitBake is to produce some kind of
  7. output such as a single installable package, a kernel, a software
  8. development kit, or even a full, board-specific bootable Linux image,
  9. complete with bootloader, kernel, and root filesystem. Of course, you
  10. can execute the ``bitbake`` command with options that cause it to
  11. execute single tasks, compile single recipe files, capture or clear
  12. data, or simply return information about the execution environment.
  13. This chapter describes BitBake's execution process from start to finish
  14. when you use it to create an image. The execution process is launched
  15. using the following command form: ::
  16. $ bitbake target
  17. For information on
  18. the BitBake command and its options, see ":ref:`The BitBake Command
  19. <bitbake-user-manual-command>`" section.
  20. .. note::
  21. Prior to executing BitBake, you should take advantage of available
  22. parallel thread execution on your build host by setting the
  23. :term:`BB_NUMBER_THREADS` variable in
  24. your project's ``local.conf`` configuration file.
  25. A common method to determine this value for your build host is to run
  26. the following: ::
  27. $ grep processor /proc/cpuinfo
  28. This command returns
  29. the number of processors, which takes into account hyper-threading.
  30. Thus, a quad-core build host with hyper-threading most likely shows
  31. eight processors, which is the value you would then assign to
  32. ``BB_NUMBER_THREADS``.
  33. A possibly simpler solution is that some Linux distributions (e.g.
  34. Debian and Ubuntu) provide the ``ncpus`` command.
  35. Parsing the Base Configuration Metadata
  36. =======================================
  37. The first thing BitBake does is parse base configuration metadata. Base
  38. configuration metadata consists of your project's ``bblayers.conf`` file
  39. to determine what layers BitBake needs to recognize, all necessary
  40. ``layer.conf`` files (one from each layer), and ``bitbake.conf``. The
  41. data itself is of various types:
  42. - **Recipes:** Details about particular pieces of software.
  43. - **Class Data:** An abstraction of common build information (e.g. how to
  44. build a Linux kernel).
  45. - **Configuration Data:** Machine-specific settings, policy decisions,
  46. and so forth. Configuration data acts as the glue to bind everything
  47. together.
  48. The ``layer.conf`` files are used to construct key variables such as
  49. :term:`BBPATH` and :term:`BBFILES`.
  50. ``BBPATH`` is used to search for configuration and class files under the
  51. ``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used
  52. to locate both recipe and recipe append files (``.bb`` and
  53. ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
  54. user has set the ``BBPATH`` and ``BBFILES`` directly in the environment.
  55. Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable
  56. that was just constructed. The ``bitbake.conf`` file may also include
  57. other configuration files using the ``include`` or ``require``
  58. directives.
  59. Prior to parsing configuration files, BitBake looks at certain
  60. variables, including:
  61. - :term:`BB_ENV_WHITELIST`
  62. - :term:`BB_ENV_EXTRAWHITE`
  63. - :term:`BB_PRESERVE_ENV`
  64. - :term:`BB_ORIGENV`
  65. - :term:`BITBAKE_UI`
  66. The first four variables in this list relate to how BitBake treats shell
  67. environment variables during task execution. By default, BitBake cleans
  68. the environment variables and provides tight control over the shell
  69. execution environment. However, through the use of these first four
  70. variables, you can apply your control regarding the environment
  71. variables allowed to be used by BitBake in the shell during execution of
  72. tasks. See the
  73. ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`"
  74. section and the information about these variables in the variable
  75. glossary for more information on how they work and on how to use them.
  76. The base configuration metadata is global and therefore affects all
  77. recipes and tasks that are executed.
  78. BitBake first searches the current working directory for an optional
  79. ``conf/bblayers.conf`` configuration file. This file is expected to
  80. contain a :term:`BBLAYERS` variable that is a
  81. space-delimited list of 'layer' directories. Recall that if BitBake
  82. cannot find a ``bblayers.conf`` file, then it is assumed the user has
  83. set the ``BBPATH`` and ``BBFILES`` variables directly in the
  84. environment.
  85. For each directory (layer) in this list, a ``conf/layer.conf`` file is
  86. located and parsed with the :term:`LAYERDIR` variable
  87. being set to the directory where the layer was found. The idea is these
  88. files automatically set up :term:`BBPATH` and other
  89. variables correctly for a given build directory.
  90. BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in
  91. the user-specified ``BBPATH``. That configuration file generally has
  92. include directives to pull in any other metadata such as files specific
  93. to the architecture, the machine, the local environment, and so forth.
  94. Only variable definitions and include directives are allowed in BitBake
  95. ``.conf`` files. Some variables directly influence BitBake's behavior.
  96. These variables might have been set from the environment depending on
  97. the environment variables previously mentioned or set in the
  98. configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`"
  99. chapter presents a full list of
  100. variables.
  101. After parsing configuration files, BitBake uses its rudimentary
  102. inheritance mechanism, which is through class files, to inherit some
  103. standard classes. BitBake parses a class when the inherit directive
  104. responsible for getting that class is encountered.
  105. The ``base.bbclass`` file is always included. Other classes that are
  106. specified in the configuration using the
  107. :term:`INHERIT` variable are also included. BitBake
  108. searches for class files in a ``classes`` subdirectory under the paths
  109. in ``BBPATH`` in the same way as configuration files.
  110. A good way to get an idea of the configuration files and the class files
  111. used in your execution environment is to run the following BitBake
  112. command: ::
  113. $ bitbake -e > mybb.log
  114. Examining the top of the ``mybb.log``
  115. shows you the many configuration files and class files used in your
  116. execution environment.
  117. .. note::
  118. You need to be aware of how BitBake parses curly braces. If a recipe
  119. uses a closing curly brace within the function and the character has
  120. no leading spaces, BitBake produces a parsing error. If you use a
  121. pair of curly braces in a shell function, the closing curly brace
  122. must not be located at the start of the line without leading spaces.
  123. Here is an example that causes BitBake to produce a parsing error: ::
  124. fakeroot create_shar() {
  125. cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
  126. usage()
  127. {
  128. echo "test"
  129. ###### The following "}" at the start of the line causes a parsing error ######
  130. }
  131. EOF
  132. }
  133. Writing the recipe this way avoids the error:
  134. fakeroot create_shar() {
  135. cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
  136. usage()
  137. {
  138. echo "test"
  139. ###### The following "}" with a leading space at the start of the line avoids the error ######
  140. }
  141. EOF
  142. }
  143. Locating and Parsing Recipes
  144. ============================
  145. During the configuration phase, BitBake will have set
  146. :term:`BBFILES`. BitBake now uses it to construct a
  147. list of recipes to parse, along with any append files (``.bbappend``) to
  148. apply. ``BBFILES`` is a space-separated list of available files and
  149. supports wildcards. An example would be: ::
  150. BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
  151. BitBake parses each
  152. recipe and append file located with ``BBFILES`` and stores the values of
  153. various variables into the datastore.
  154. .. note::
  155. Append files are applied in the order they are encountered in BBFILES.
  156. For each file, a fresh copy of the base configuration is made, then the
  157. recipe is parsed line by line. Any inherit statements cause BitBake to
  158. find and then parse class files (``.bbclass``) using
  159. :term:`BBPATH` as the search path. Finally, BitBake
  160. parses in order any append files found in ``BBFILES``.
  161. One common convention is to use the recipe filename to define pieces of
  162. metadata. For example, in ``bitbake.conf`` the recipe name and version
  163. are used to set the variables :term:`PN` and
  164. :term:`PV`: ::
  165. PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
  166. PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
  167. In this example, a recipe called "something_1.2.3.bb" would set
  168. ``PN`` to "something" and ``PV`` to "1.2.3".
  169. By the time parsing is complete for a recipe, BitBake has a list of
  170. tasks that the recipe defines and a set of data consisting of keys and
  171. values as well as dependency information about the tasks.
  172. BitBake does not need all of this information. It only needs a small
  173. subset of the information to make decisions about the recipe.
  174. Consequently, BitBake caches the values in which it is interested and
  175. does not store the rest of the information. Experience has shown it is
  176. faster to re-parse the metadata than to try and write it out to the disk
  177. and then reload it.
  178. Where possible, subsequent BitBake commands reuse this cache of recipe
  179. information. The validity of this cache is determined by first computing
  180. a checksum of the base configuration data (see
  181. :term:`BB_HASHCONFIG_WHITELIST`) and
  182. then checking if the checksum matches. If that checksum matches what is
  183. in the cache and the recipe and class files have not changed, BitBake is
  184. able to use the cache. BitBake then reloads the cached information about
  185. the recipe instead of reparsing it from scratch.
  186. Recipe file collections exist to allow the user to have multiple
  187. repositories of ``.bb`` files that contain the same exact package. For
  188. example, one could easily use them to make one's own local copy of an
  189. upstream repository, but with custom modifications that one does not
  190. want upstream. Here is an example: ::
  191. BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
  192. BBFILE_COLLECTIONS = "upstream local"
  193. BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
  194. BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
  195. BBFILE_PRIORITY_upstream = "5" BBFILE_PRIORITY_local = "10"
  196. .. note::
  197. The layers mechanism is now the preferred method of collecting code.
  198. While the collections code remains, its main use is to set layer
  199. priorities and to deal with overlap (conflicts) between layers.
  200. .. _bb-bitbake-providers:
  201. Providers
  202. =========
  203. Assuming BitBake has been instructed to execute a target and that all
  204. the recipe files have been parsed, BitBake starts to figure out how to
  205. build the target. BitBake looks through the ``PROVIDES`` list for each
  206. of the recipes. A ``PROVIDES`` list is the list of names by which the
  207. recipe can be known. Each recipe's ``PROVIDES`` list is created
  208. implicitly through the recipe's :term:`PN` variable and
  209. explicitly through the recipe's :term:`PROVIDES`
  210. variable, which is optional.
  211. When a recipe uses ``PROVIDES``, that recipe's functionality can be
  212. found under an alternative name or names other than the implicit ``PN``
  213. name. As an example, suppose a recipe named ``keyboard_1.0.bb``
  214. contained the following: ::
  215. PROVIDES += "fullkeyboard"
  216. The ``PROVIDES``
  217. list for this recipe becomes "keyboard", which is implicit, and
  218. "fullkeyboard", which is explicit. Consequently, the functionality found
  219. in ``keyboard_1.0.bb`` can be found under two different names.
  220. .. _bb-bitbake-preferences:
  221. Preferences
  222. ===========
  223. The ``PROVIDES`` list is only part of the solution for figuring out a
  224. target's recipes. Because targets might have multiple providers, BitBake
  225. needs to prioritize providers by determining provider preferences.
  226. A common example in which a target has multiple providers is
  227. "virtual/kernel", which is on the ``PROVIDES`` list for each kernel
  228. recipe. Each machine often selects the best kernel provider by using a
  229. line similar to the following in the machine configuration file: ::
  230. PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
  231. The default :term:`PREFERRED_PROVIDER` is the provider
  232. with the same name as the target. BitBake iterates through each target
  233. it needs to build and resolves them and their dependencies using this
  234. process.
  235. Understanding how providers are chosen is made complicated by the fact
  236. that multiple versions might exist for a given provider. BitBake
  237. defaults to the highest version of a provider. Version comparisons are
  238. made using the same method as Debian. You can use the
  239. :term:`PREFERRED_VERSION` variable to
  240. specify a particular version. You can influence the order by using the
  241. :term:`DEFAULT_PREFERENCE` variable.
  242. By default, files have a preference of "0". Setting
  243. ``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used
  244. unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to
  245. "1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides
  246. any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used
  247. to mark newer and more experimental recipe versions until they have
  248. undergone sufficient testing to be considered stable.
  249. When there are multiple "versions" of a given recipe, BitBake defaults
  250. to selecting the most recent version, unless otherwise specified. If the
  251. recipe in question has a
  252. :term:`DEFAULT_PREFERENCE` set lower than
  253. the other recipes (default is 0), then it will not be selected. This
  254. allows the person or persons maintaining the repository of recipe files
  255. to specify their preference for the default selected version.
  256. Additionally, the user can specify their preferred version.
  257. If the first recipe is named ``a_1.1.bb``, then the
  258. :term:`PN` variable will be set to "a", and the
  259. :term:`PV` variable will be set to 1.1.
  260. Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by
  261. default. However, if you define the following variable in a ``.conf``
  262. file that BitBake parses, you can change that preference: ::
  263. PREFERRED_VERSION_a = "1.1"
  264. .. note::
  265. It is common for a recipe to provide two versions -- a stable,
  266. numbered (and preferred) version, and a version that is automatically
  267. checked out from a source code repository that is considered more
  268. "bleeding edge" but can be selected only explicitly.
  269. For example, in the OpenEmbedded codebase, there is a standard,
  270. versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
  271. is also a Git-based version, ``busybox_git.bb``, which explicitly
  272. contains the line ::
  273. DEFAULT_PREFERENCE = "-1"
  274. to ensure that the
  275. numbered, stable version is always preferred unless the developer
  276. selects otherwise.
  277. .. _bb-bitbake-dependencies:
  278. Dependencies
  279. ============
  280. Each target BitBake builds consists of multiple tasks such as ``fetch``,
  281. ``unpack``, ``patch``, ``configure``, and ``compile``. For best
  282. performance on multi-core systems, BitBake considers each task as an
  283. independent entity with its own set of dependencies.
  284. Dependencies are defined through several variables. You can find
  285. information about variables BitBake uses in the
  286. :doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a
  287. basic level, it is sufficient to know that BitBake uses the
  288. :term:`DEPENDS` and
  289. :term:`RDEPENDS` variables when calculating
  290. dependencies.
  291. For more information on how BitBake handles dependencies, see the
  292. :ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies`
  293. section.
  294. .. _ref-bitbake-tasklist:
  295. The Task List
  296. =============
  297. Based on the generated list of providers and the dependency information,
  298. BitBake can now calculate exactly what tasks it needs to run and in what
  299. order it needs to run them. The
  300. :ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks`
  301. section has more information on how BitBake chooses which task to
  302. execute next.
  303. The build now starts with BitBake forking off threads up to the limit
  304. set in the :term:`BB_NUMBER_THREADS`
  305. variable. BitBake continues to fork threads as long as there are tasks
  306. ready to run, those tasks have all their dependencies met, and the
  307. thread threshold has not been exceeded.
  308. It is worth noting that you can greatly speed up the build time by
  309. properly setting the ``BB_NUMBER_THREADS`` variable.
  310. As each task completes, a timestamp is written to the directory
  311. specified by the :term:`STAMP` variable. On subsequent
  312. runs, BitBake looks in the build directory within ``tmp/stamps`` and
  313. does not rerun tasks that are already completed unless a timestamp is
  314. found to be invalid. Currently, invalid timestamps are only considered
  315. on a per recipe file basis. So, for example, if the configure stamp has
  316. a timestamp greater than the compile timestamp for a given target, then
  317. the compile task would rerun. Running the compile task again, however,
  318. has no effect on other providers that depend on that target.
  319. The exact format of the stamps is partly configurable. In modern
  320. versions of BitBake, a hash is appended to the stamp so that if the
  321. configuration changes, the stamp becomes invalid and the task is
  322. automatically rerun. This hash, or signature used, is governed by the
  323. signature policy that is configured (see the
  324. :ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`
  325. section for information). It is also
  326. possible to append extra metadata to the stamp using the
  327. ``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this
  328. flag to make some tasks machine-specific.
  329. .. note::
  330. Some tasks are marked as "nostamp" tasks. No timestamp file is
  331. created when these tasks are run. Consequently, "nostamp" tasks are
  332. always rerun.
  333. For more information on tasks, see the
  334. :ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section.
  335. Executing Tasks
  336. ===============
  337. Tasks can be either a shell task or a Python task. For shell tasks,
  338. BitBake writes a shell script to
  339. ``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then
  340. executes the script. The generated shell script contains all the
  341. exported variables, and the shell functions with all variables expanded.
  342. Output from the shell script goes to the file
  343. ``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in
  344. the run file and the output in the log files is a useful debugging
  345. technique.
  346. For Python tasks, BitBake executes the task internally and logs
  347. information to the controlling terminal. Future versions of BitBake will
  348. write the functions to files similar to the way shell tasks are handled.
  349. Logging will be handled in a way similar to shell tasks as well.
  350. The order in which BitBake runs the tasks is controlled by its task
  351. scheduler. It is possible to configure the scheduler and define custom
  352. implementations for specific use cases. For more information, see these
  353. variables that control the behavior:
  354. - :term:`BB_SCHEDULER`
  355. - :term:`BB_SCHEDULERS`
  356. It is possible to have functions run before and after a task's main
  357. function. This is done using the ``[prefuncs]`` and ``[postfuncs]``
  358. flags of the task that lists the functions to run.
  359. .. _checksums:
  360. Checksums (Signatures)
  361. ======================
  362. A checksum is a unique signature of a task's inputs. The signature of a
  363. task can be used to determine if a task needs to be run. Because it is a
  364. change in a task's inputs that triggers running the task, BitBake needs
  365. to detect all the inputs to a given task. For shell tasks, this turns
  366. out to be fairly easy because BitBake generates a "run" shell script for
  367. each task and it is possible to create a checksum that gives you a good
  368. idea of when the task's data changes.
  369. To complicate the problem, some things should not be included in the
  370. checksum. First, there is the actual specific build path of a given task
  371. - the working directory. It does not matter if the working directory
  372. changes because it should not affect the output for target packages. The
  373. simplistic approach for excluding the working directory is to set it to
  374. some fixed value and create the checksum for the "run" script. BitBake
  375. goes one step better and uses the
  376. :term:`BB_HASHBASE_WHITELIST` variable
  377. to define a list of variables that should never be included when
  378. generating the signatures.
  379. Another problem results from the "run" scripts containing functions that
  380. might or might not get called. The incremental build solution contains
  381. code that figures out dependencies between shell functions. This code is
  382. used to prune the "run" scripts down to the minimum set, thereby
  383. alleviating this problem and making the "run" scripts much more readable
  384. as a bonus.
  385. So far we have solutions for shell scripts. What about Python tasks? The
  386. same approach applies even though these tasks are more difficult. The
  387. process needs to figure out what variables a Python function accesses
  388. and what functions it calls. Again, the incremental build solution
  389. contains code that first figures out the variable and function
  390. dependencies, and then creates a checksum for the data used as the input
  391. to the task.
  392. Like the working directory case, situations exist where dependencies
  393. should be ignored. For these cases, you can instruct the build process
  394. to ignore a dependency by using a line like the following: ::
  395. PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
  396. This example ensures that the
  397. ``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``,
  398. even if it does reference it.
  399. Equally, there are cases where we need to add dependencies BitBake is
  400. not able to find. You can accomplish this by using a line like the
  401. following: ::
  402. PACKAGE_ARCHS[vardeps] = "MACHINE"
  403. This example explicitly
  404. adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
  405. Consider a case with in-line Python, for example, where BitBake is not
  406. able to figure out dependencies. When running in debug mode (i.e. using
  407. ``-DDD``), BitBake produces output when it discovers something for which
  408. it cannot figure out dependencies.
  409. Thus far, this section has limited discussion to the direct inputs into
  410. a task. Information based on direct inputs is referred to as the
  411. "basehash" in the code. However, there is still the question of a task's
  412. indirect inputs - the things that were already built and present in the
  413. build directory. The checksum (or signature) for a particular task needs
  414. to add the hashes of all the tasks on which the particular task depends.
  415. Choosing which dependencies to add is a policy decision. However, the
  416. effect is to generate a master checksum that combines the basehash and
  417. the hashes of the task's dependencies.
  418. At the code level, there are a variety of ways both the basehash and the
  419. dependent task hashes can be influenced. Within the BitBake
  420. configuration file, we can give BitBake some extra information to help
  421. it construct the basehash. The following statement effectively results
  422. in a list of global variable dependency excludes - variables never
  423. included in any checksum. This example uses variables from OpenEmbedded
  424. to help illustrate the concept: ::
  425. BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
  426. SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
  427. USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
  428. PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
  429. CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
  430. The previous example excludes the work directory, which is part of
  431. ``TMPDIR``.
  432. The rules for deciding which hashes of dependent tasks to include
  433. through dependency chains are more complex and are generally
  434. accomplished with a Python function. The code in
  435. ``meta/lib/oe/sstatesig.py`` shows two examples of this and also
  436. illustrates how you can insert your own policy into the system if so
  437. desired. This file defines the two basic signature generators
  438. OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there
  439. is a dummy "noop" signature handler enabled in BitBake. This means that
  440. behavior is unchanged from previous versions. ``OE-Core`` uses the
  441. "OEBasicHash" signature handler by default through this setting in the
  442. ``bitbake.conf`` file: ::
  443. BB_SIGNATURE_HANDLER ?= "OEBasicHash"
  444. The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic"
  445. version but adds the task hash to the stamp files. This results in any
  446. metadata change that changes the task hash, automatically causing the
  447. task to be run again. This removes the need to bump
  448. :term:`PR` values, and changes to metadata automatically
  449. ripple across the build.
  450. It is also worth noting that the end result of these signature
  451. generators is to make some dependency and hash information available to
  452. the build. This information includes:
  453. - ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the
  454. recipe.
  455. - ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each
  456. dependent task.
  457. - ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for
  458. each task.
  459. - ``BB_TASKHASH``: The hash of the currently running task.
  460. It is worth noting that BitBake's "-S" option lets you debug BitBake's
  461. processing of signatures. The options passed to -S allow different
  462. debugging modes to be used, either using BitBake's own debug functions
  463. or possibly those defined in the metadata/signature handler itself. The
  464. simplest parameter to pass is "none", which causes a set of signature
  465. information to be written out into ``STAMPS_DIR`` corresponding to the
  466. targets specified. The other currently available parameter is
  467. "printdiff", which causes BitBake to try to establish the closest
  468. signature match it can (e.g. in the sstate cache) and then run
  469. ``bitbake-diffsigs`` over the matches to determine the stamps and delta
  470. where these two stamp trees diverge.
  471. .. note::
  472. It is likely that future versions of BitBake will provide other
  473. signature handlers triggered through additional "-S" parameters.
  474. You can find more information on checksum metadata in the
  475. :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
  476. section.
  477. Setscene
  478. ========
  479. The setscene process enables BitBake to handle "pre-built" artifacts.
  480. The ability to handle and reuse these artifacts allows BitBake the
  481. luxury of not having to build something from scratch every time.
  482. Instead, BitBake can use, when possible, existing build artifacts.
  483. BitBake needs to have reliable data indicating whether or not an
  484. artifact is compatible. Signatures, described in the previous section,
  485. provide an ideal way of representing whether an artifact is compatible.
  486. If a signature is the same, an object can be reused.
  487. If an object can be reused, the problem then becomes how to replace a
  488. given task or set of tasks with the pre-built artifact. BitBake solves
  489. the problem with the "setscene" process.
  490. When BitBake is asked to build a given target, before building anything,
  491. it first asks whether cached information is available for any of the
  492. targets it's building, or any of the intermediate targets. If cached
  493. information is available, BitBake uses this information instead of
  494. running the main tasks.
  495. BitBake first calls the function defined by the
  496. :term:`BB_HASHCHECK_FUNCTION` variable
  497. with a list of tasks and corresponding hashes it wants to build. This
  498. function is designed to be fast and returns a list of the tasks for
  499. which it believes in can obtain artifacts.
  500. Next, for each of the tasks that were returned as possibilities, BitBake
  501. executes a setscene version of the task that the possible artifact
  502. covers. Setscene versions of a task have the string "_setscene" appended
  503. to the task name. So, for example, the task with the name ``xxx`` has a
  504. setscene task named ``xxx_setscene``. The setscene version of the task
  505. executes and provides the necessary artifacts returning either success
  506. or failure.
  507. As previously mentioned, an artifact can cover more than one task. For
  508. example, it is pointless to obtain a compiler if you already have the
  509. compiled binary. To handle this, BitBake calls the
  510. :term:`BB_SETSCENE_DEPVALID` function for
  511. each successful setscene task to know whether or not it needs to obtain
  512. the dependencies of that task.
  513. Finally, after all the setscene tasks have executed, BitBake calls the
  514. function listed in
  515. :term:`BB_SETSCENE_VERIFY_FUNCTION2`
  516. with the list of tasks BitBake thinks has been "covered". The metadata
  517. can then ensure that this list is correct and can inform BitBake that it
  518. wants specific tasks to be run regardless of the setscene result.
  519. You can find more information on setscene metadata in the
  520. :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
  521. section.
  522. Logging
  523. =======
  524. In addition to the standard command line option to control how verbose
  525. builds are when execute, bitbake also supports user defined
  526. configuration of the `Python
  527. logging <https://docs.python.org/3/library/logging.html>`__ facilities
  528. through the :term:`BB_LOGCONFIG` variable. This
  529. variable defines a json or yaml `logging
  530. configuration <https://docs.python.org/3/library/logging.config.html>`__
  531. that will be intelligently merged into the default configuration. The
  532. logging configuration is merged using the following rules:
  533. - The user defined configuration will completely replace the default
  534. configuration if top level key ``bitbake_merge`` is set to the value
  535. ``False``. In this case, all other rules are ignored.
  536. - The user configuration must have a top level ``version`` which must
  537. match the value of the default configuration.
  538. - Any keys defined in the ``handlers``, ``formatters``, or ``filters``,
  539. will be merged into the same section in the default configuration,
  540. with the user specified keys taking replacing a default one if there
  541. is a conflict. In practice, this means that if both the default
  542. configuration and user configuration specify a handler named
  543. ``myhandler``, the user defined one will replace the default. To
  544. prevent the user from inadvertently replacing a default handler,
  545. formatter, or filter, all of the default ones are named with a prefix
  546. of "``BitBake.``"
  547. - If a logger is defined by the user with the key ``bitbake_merge`` set
  548. to ``False``, that logger will be completely replaced by user
  549. configuration. In this case, no other rules will apply to that
  550. logger.
  551. - All user defined ``filter`` and ``handlers`` properties for a given
  552. logger will be merged with corresponding properties from the default
  553. logger. For example, if the user configuration adds a filter called
  554. ``myFilter`` to the ``BitBake.SigGen``, and the default configuration
  555. adds a filter called ``BitBake.defaultFilter``, both filters will be
  556. applied to the logger
  557. As an example, consider the following user logging configuration file
  558. which logs all Hash Equivalence related messages of VERBOSE or higher to
  559. a file called ``hashequiv.log`` ::
  560. {
  561. "version": 1,
  562. "handlers": {
  563. "autobuilderlog": {
  564. "class": "logging.FileHandler",
  565. "formatter": "logfileFormatter",
  566. "level": "DEBUG",
  567. "filename": "hashequiv.log",
  568. "mode": "w"
  569. }
  570. },
  571. "formatters": {
  572. "logfileFormatter": {
  573. "format": "%(name)s: %(levelname)s: %(message)s"
  574. }
  575. },
  576. "loggers": {
  577. "BitBake.SigGen.HashEquiv": {
  578. "level": "VERBOSE",
  579. "handlers": ["autobuilderlog"]
  580. },
  581. "BitBake.RunQueue.HashEquiv": {
  582. "level": "VERBOSE",
  583. "handlers": ["autobuilderlog"]
  584. }
  585. }
  586. }