123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651 |
- .. SPDX-License-Identifier: CC-BY-2.5
- ========
- Overview
- ========
- |
- Welcome to the BitBake User Manual. This manual provides information on
- the BitBake tool. The information attempts to be as independent as
- possible regarding systems that use BitBake, such as OpenEmbedded and
- the Yocto Project. In some cases, scenarios or examples within the
- context of a build system are used in the manual to help with
- understanding. For these cases, the manual clearly states the context.
- .. _intro:
- Introduction
- ============
- Fundamentally, BitBake is a generic task execution engine that allows
- shell and Python tasks to be run efficiently and in parallel while
- working within complex inter-task dependency constraints. One of
- BitBake's main users, OpenEmbedded, takes this core and builds embedded
- Linux software stacks using a task-oriented approach.
- Conceptually, BitBake is similar to GNU Make in some regards but has
- significant differences:
- - BitBake executes tasks according to provided metadata that builds up
- the tasks. Metadata is stored in recipe (``.bb``) and related recipe
- "append" (``.bbappend``) files, configuration (``.conf``) and
- underlying include (``.inc``) files, and in class (``.bbclass``)
- files. The metadata provides BitBake with instructions on what tasks
- to run and the dependencies between those tasks.
- - BitBake includes a fetcher library for obtaining source code from
- various places such as local files, source control systems, or
- websites.
- - The instructions for each unit to be built (e.g. a piece of software)
- are known as "recipe" files and contain all the information about the
- unit (dependencies, source file locations, checksums, description and
- so on).
- - BitBake includes a client/server abstraction and can be used from a
- command line or used as a service over XML-RPC and has several
- different user interfaces.
- History and Goals
- =================
- BitBake was originally a part of the OpenEmbedded project. It was
- inspired by the Portage package management system used by the Gentoo
- Linux distribution. On December 7, 2004, OpenEmbedded project team
- member Chris Larson split the project into two distinct pieces:
- - BitBake, a generic task executor
- - OpenEmbedded, a metadata set utilized by BitBake
- Today, BitBake is the primary basis of the
- `OpenEmbedded <http://www.openembedded.org/>`__ project, which is being
- used to build and maintain Linux distributions such as the `Angstrom
- Distribution <http://www.angstrom-distribution.org/>`__, and which is
- also being used as the build tool for Linux projects such as the `Yocto
- Project <http://www.yoctoproject.org>`__.
- Prior to BitBake, no other build tool adequately met the needs of an
- aspiring embedded Linux distribution. All of the build systems used by
- traditional desktop Linux distributions lacked important functionality,
- and none of the ad hoc Buildroot-based systems, prevalent in the
- embedded space, were scalable or maintainable.
- Some important original goals for BitBake were:
- - Handle cross-compilation.
- - Handle inter-package dependencies (build time on target architecture,
- build time on native architecture, and runtime).
- - Support running any number of tasks within a given package,
- including, but not limited to, fetching upstream sources, unpacking
- them, patching them, configuring them, and so forth.
- - Be Linux distribution agnostic for both build and target systems.
- - Be architecture agnostic.
- - Support multiple build and target operating systems (e.g. Cygwin, the
- BSDs, and so forth).
- - Be self-contained, rather than tightly integrated into the build
- machine's root filesystem.
- - Handle conditional metadata on the target architecture, operating
- system, distribution, and machine.
- - Be easy to use the tools to supply local metadata and packages
- against which to operate.
- - Be easy to use BitBake to collaborate between multiple projects for
- their builds.
- - Provide an inheritance mechanism to share common metadata between
- many packages.
- Over time it became apparent that some further requirements were
- necessary:
- - Handle variants of a base recipe (e.g. native, sdk, and multilib).
- - Split metadata into layers and allow layers to enhance or override
- other layers.
- - Allow representation of a given set of input variables to a task as a
- checksum. Based on that checksum, allow acceleration of builds with
- prebuilt components.
- BitBake satisfies all the original requirements and many more with
- extensions being made to the basic functionality to reflect the
- additional requirements. Flexibility and power have always been the
- priorities. BitBake is highly extensible and supports embedded Python
- code and execution of any arbitrary tasks.
- .. _Concepts:
- Concepts
- ========
- BitBake is a program written in the Python language. At the highest
- level, BitBake interprets metadata, decides what tasks are required to
- run, and executes those tasks. Similar to GNU Make, BitBake controls how
- software is built. GNU Make achieves its control through "makefiles",
- while BitBake uses "recipes".
- BitBake extends the capabilities of a simple tool like GNU Make by
- allowing for the definition of much more complex tasks, such as
- assembling entire embedded Linux distributions.
- The remainder of this section introduces several concepts that should be
- understood in order to better leverage the power of BitBake.
- Recipes
- -------
- BitBake Recipes, which are denoted by the file extension ``.bb``, are
- the most basic metadata files. These recipe files provide BitBake with
- the following:
- - Descriptive information about the package (author, homepage, license,
- and so on)
- - The version of the recipe
- - Existing dependencies (both build and runtime dependencies)
- - Where the source code resides and how to fetch it
- - Whether the source code requires any patches, where to find them, and
- how to apply them
- - How to configure and compile the source code
- - How to assemble the generated artifacts into one or more installable
- packages
- - Where on the target machine to install the package or packages
- created
- Within the context of BitBake, or any project utilizing BitBake as its
- build system, files with the ``.bb`` extension are referred to as
- recipes.
- .. note::
- The term "package" is also commonly used to describe recipes.
- However, since the same word is used to describe packaged output from
- a project, it is best to maintain a single descriptive term -
- "recipes". Put another way, a single "recipe" file is quite capable
- of generating a number of related but separately installable
- "packages". In fact, that ability is fairly common.
- Configuration Files
- -------------------
- Configuration files, which are denoted by the ``.conf`` extension,
- define various configuration variables that govern the project's build
- process. These files fall into several areas that define machine
- configuration, distribution configuration, possible compiler tuning,
- general common configuration, and user configuration. The main
- configuration file is the sample ``bitbake.conf`` file, which is located
- within the BitBake source tree ``conf`` directory.
- Classes
- -------
- Class files, which are denoted by the ``.bbclass`` extension, contain
- information that is useful to share between metadata files. The BitBake
- source tree currently comes with one class metadata file called
- ``base.bbclass``. You can find this file in the ``classes`` directory.
- The ``base.bbclass`` class files is special since it is always included
- automatically for all recipes and classes. This class contains
- definitions for standard basic tasks such as fetching, unpacking,
- configuring (empty by default), compiling (runs any Makefile present),
- installing (empty by default) and packaging (empty by default). These
- tasks are often overridden or extended by other classes added during the
- project development process.
- Layers
- ------
- Layers allow you to isolate different types of customizations from each
- other. While you might find it tempting to keep everything in one layer
- when working on a single project, the more modular your metadata, the
- easier it is to cope with future changes.
- To illustrate how you can use layers to keep things modular, consider
- customizations you might make to support a specific target machine.
- These types of customizations typically reside in a special layer,
- rather than a general layer, called a Board Support Package (BSP) layer.
- Furthermore, the machine customizations should be isolated from recipes
- and metadata that support a new GUI environment, for example. This
- situation gives you a couple of layers: one for the machine
- configurations and one for the GUI environment. It is important to
- understand, however, that the BSP layer can still make machine-specific
- additions to recipes within the GUI environment layer without polluting
- the GUI layer itself with those machine-specific changes. You can
- accomplish this through a recipe that is a BitBake append
- (``.bbappend``) file.
- .. _append-bbappend-files:
- Append Files
- ------------
- Append files, which are files that have the ``.bbappend`` file
- extension, extend or override information in an existing recipe file.
- BitBake expects every append file to have a corresponding recipe file.
- Furthermore, the append file and corresponding recipe file must use the
- same root filename. The filenames can differ only in the file type
- suffix used (e.g. ``formfactor_0.0.bb`` and
- ``formfactor_0.0.bbappend``).
- Information in append files extends or overrides the information in the
- underlying, similarly-named recipe files.
- When you name an append file, you can use the "``%``" wildcard character
- to allow for matching recipe names. For example, suppose you have an
- append file named as follows: ::
- busybox_1.21.%.bbappend
- That append file
- would match any ``busybox_1.21.``\ x\ ``.bb`` version of the recipe. So,
- the append file would match the following recipe names: ::
- busybox_1.21.1.bb
- busybox_1.21.2.bb
- busybox_1.21.3.bb
- .. note::
- The use of the " % " character is limited in that it only works directly in
- front of the .bbappend portion of the append file's name. You cannot use the
- wildcard character in any other location of the name.
- If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the
- append name would not match. However, if you named the append file
- ``busybox_1.%.bbappend``, then you would have a match.
- In the most general case, you could name the append file something as
- simple as ``busybox_%.bbappend`` to be entirely version independent.
- Obtaining BitBake
- =================
- You can obtain BitBake several different ways:
- - **Cloning BitBake:** Using Git to clone the BitBake source code
- repository is the recommended method for obtaining BitBake. Cloning
- the repository makes it easy to get bug fixes and have access to
- stable branches and the master branch. Once you have cloned BitBake,
- you should use the latest stable branch for development since the
- master branch is for BitBake development and might contain less
- stable changes.
- You usually need a version of BitBake that matches the metadata you
- are using. The metadata is generally backwards compatible but not
- forward compatible.
- Here is an example that clones the BitBake repository: ::
- $ git clone git://git.openembedded.org/bitbake
- This command clones the BitBake
- Git repository into a directory called ``bitbake``. Alternatively,
- you can designate a directory after the ``git clone`` command if you
- want to call the new directory something other than ``bitbake``. Here
- is an example that names the directory ``bbdev``: ::
- $ git clone git://git.openembedded.org/bitbake bbdev
- - **Installation using your Distribution Package Management System:**
- This method is not recommended because the BitBake version that is
- provided by your distribution, in most cases, is several releases
- behind a snapshot of the BitBake repository.
- - **Taking a snapshot of BitBake:** Downloading a snapshot of BitBake
- from the source code repository gives you access to a known branch or
- release of BitBake.
- .. note::
- Cloning the Git repository, as described earlier, is the preferred
- method for getting BitBake. Cloning the repository makes it easier
- to update as patches are added to the stable branches.
- The following example downloads a snapshot of BitBake version 1.17.0: ::
- $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz
- $ tar zxpvf bitbake-1.17.0.tar.gz
- After extraction of the tarball using
- the tar utility, you have a directory entitled ``bitbake-1.17.0``.
- - **Using the BitBake that Comes With Your Build Checkout:** A final
- possibility for getting a copy of BitBake is that it already comes
- with your checkout of a larger BitBake-based build system, such as
- Poky. Rather than manually checking out individual layers and gluing
- them together yourself, you can check out an entire build system. The
- checkout will already include a version of BitBake that has been
- thoroughly tested for compatibility with the other components. For
- information on how to check out a particular BitBake-based build
- system, consult that build system's supporting documentation.
- .. _bitbake-user-manual-command:
- The BitBake Command
- ===================
- The ``bitbake`` command is the primary interface to the BitBake tool.
- This section presents the BitBake command syntax and provides several
- execution examples.
- Usage and syntax
- ----------------
- Following is the usage and syntax for BitBake: ::
- $ bitbake -h
- Usage: bitbake [options] [recipename/target recipe:do_task ...]
- Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
- It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
- will provide the layer, BBFILES and other configuration information.
- Options:
- --version show program's version number and exit
- -h, --help show this help message and exit
- -b BUILDFILE, --buildfile=BUILDFILE
- Execute tasks from a specific .bb recipe directly.
- WARNING: Does not handle any dependencies from other
- recipes.
- -k, --continue Continue as much as possible after an error. While the
- target that failed and anything depending on it cannot
- be built, as much as possible will be built before
- stopping.
- -f, --force Force the specified targets/task to run (invalidating
- any existing stamp file).
- -c CMD, --cmd=CMD Specify the task to execute. The exact options
- available depend on the metadata. Some examples might
- be 'compile' or 'populate_sysroot' or 'listtasks' may
- give a list of the tasks available.
- -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
- Invalidate the stamp for the specified task such as
- 'compile' and then run the default task for the
- specified target(s).
- -r PREFILE, --read=PREFILE
- Read the specified file before bitbake.conf.
- -R POSTFILE, --postread=POSTFILE
- Read the specified file after bitbake.conf.
- -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also
- print bb.note(...) messages to stdout (in addition to
- writing them to ${T}/log.do_<task>).
- -D, --debug Increase the debug level. You can specify this more
- than once. -D sets the debug level to 1, where only
- bb.debug(1, ...) messages are printed to stdout; -DD
- sets the debug level to 2, where both bb.debug(1, ...)
- and bb.debug(2, ...) messages are printed; etc.
- Without -D, no debug messages are printed. Note that
- -D only affects output to stdout. All debug messages
- are written to ${T}/log.do_taskname, regardless of the
- debug level.
- -q, --quiet Output less log message data to the terminal. You can
- specify this more than once.
- -n, --dry-run Don't execute, just go through the motions.
- -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
- Dump out the signature construction information, with
- no task execution. The SIGNATURE_HANDLER parameter is
- passed to the handler. Two common values are none and
- printdiff but the handler may define more/less. none
- means only dump the signature, printdiff means compare
- the dumped signature with the cached one.
- -p, --parse-only Quit after parsing the BB recipes.
- -s, --show-versions Show current and preferred versions of all recipes.
- -e, --environment Show the global or per-recipe environment complete
- with information about where variables were
- set/changed.
- -g, --graphviz Save dependency tree information for the specified
- targets in the dot syntax.
- -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
- Assume these dependencies don't exist and are already
- provided (equivalent to ASSUME_PROVIDED). Useful to
- make dependency graphs more appealing
- -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
- Show debug logging for the specified logging domains
- -P, --profile Profile the command and save reports.
- -u UI, --ui=UI The user interface to use (knotty, ncurses or taskexp
- - default knotty).
- --token=XMLRPCTOKEN Specify the connection token to be used when
- connecting to a remote server.
- --revisions-changed Set the exit code depending on whether upstream
- floating revisions have changed or not.
- --server-only Run bitbake without a UI, only starting a server
- (cooker) process.
- -B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind
- to.
- -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT
- Set timeout to unload bitbake server due to
- inactivity, set to -1 means no unload, default:
- Environment variable BB_SERVER_TIMEOUT.
- --no-setscene Do not run any setscene tasks. sstate will be ignored
- and everything needed, built.
- --setscene-only Only run setscene tasks, don't run any real tasks.
- --remote-server=REMOTE_SERVER
- Connect to the specified server.
- -m, --kill-server Terminate any running bitbake server.
- --observe-only Connect to a server as an observing-only client.
- --status-only Check the status of the remote bitbake server.
- -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
- Writes the event log of the build to a bitbake event
- json file. Use '' (empty string) to assign the name
- automatically.
- --runall=RUNALL Run the specified task for any recipe in the taskgraph
- of the specified target (even if it wouldn't otherwise
- have run).
- --runonly=RUNONLY Run only the specified task within the taskgraph of
- the specified targets (and any task dependencies those
- tasks may have).
- .. _bitbake-examples:
- Examples
- --------
- This section presents some examples showing how to use BitBake.
- .. _example-executing-a-task-against-a-single-recipe:
- Executing a Task Against a Single Recipe
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Executing tasks for a single recipe file is relatively simple. You
- specify the file in question, and BitBake parses it and executes the
- specified task. If you do not specify a task, BitBake executes the
- default task, which is "build". BitBake obeys inter-task dependencies
- when doing so.
- The following command runs the build task, which is the default task, on
- the ``foo_1.0.bb`` recipe file: ::
- $ bitbake -b foo_1.0.bb
- The following command runs the clean task on the ``foo.bb`` recipe file: ::
- $ bitbake -b foo.bb -c clean
- .. note::
- The "-b" option explicitly does not handle recipe dependencies. Other
- than for debugging purposes, it is instead recommended that you use
- the syntax presented in the next section.
- Executing Tasks Against a Set of Recipe Files
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- There are a number of additional complexities introduced when one wants
- to manage multiple ``.bb`` files. Clearly there needs to be a way to
- tell BitBake what files are available and, of those, which you want to
- execute. There also needs to be a way for each recipe to express its
- dependencies, both for build-time and runtime. There must be a way for
- you to express recipe preferences when multiple recipes provide the same
- functionality, or when there are multiple versions of a recipe.
- The ``bitbake`` command, when not using "--buildfile" or "-b" only
- accepts a "PROVIDES". You cannot provide anything else. By default, a
- recipe file generally "PROVIDES" its "packagename" as shown in the
- following example: ::
- $ bitbake foo
- This next example "PROVIDES" the
- package name and also uses the "-c" option to tell BitBake to just
- execute the ``do_clean`` task: ::
- $ bitbake -c clean foo
- Executing a List of Task and Recipe Combinations
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The BitBake command line supports specifying different tasks for
- individual targets when you specify multiple targets. For example,
- suppose you had two targets (or recipes) ``myfirstrecipe`` and
- ``mysecondrecipe`` and you needed BitBake to run ``taskA`` for the first
- recipe and ``taskB`` for the second recipe: ::
- $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB
- Generating Dependency Graphs
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- BitBake is able to generate dependency graphs using the ``dot`` syntax.
- You can convert these graphs into images using the ``dot`` tool from
- `Graphviz <http://www.graphviz.org>`__.
- When you generate a dependency graph, BitBake writes two files to the
- current working directory:
- - ``task-depends.dot``: Shows dependencies between tasks. These
- dependencies match BitBake's internal task execution list.
- - ``pn-buildlist``: Shows a simple list of targets that are to be
- built.
- To stop depending on common depends, use the "-I" depend option and
- BitBake omits them from the graph. Leaving this information out can
- produce more readable graphs. This way, you can remove from the graph
- ``DEPENDS`` from inherited classes such as ``base.bbclass``.
- Here are two examples that create dependency graphs. The second example
- omits depends common in OpenEmbedded from the graph: ::
- $ bitbake -g foo
- $ bitbake -g -I virtual/kernel -I eglibc foo
- Executing a Multiple Configuration Build
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- BitBake is able to build multiple images or packages using a single
- command where the different targets require different configurations
- (multiple configuration builds). Each target, in this scenario, is
- referred to as a "multiconfig".
- To accomplish a multiple configuration build, you must define each
- target's configuration separately using a parallel configuration file in
- the build directory. The location for these multiconfig configuration
- files is specific. They must reside in the current build directory in a
- sub-directory of ``conf`` named ``multiconfig``. Following is an example
- for two separate targets:
- .. image:: figures/bb_multiconfig_files.png
- :align: center
- The reason for this required file hierarchy is because the ``BBPATH``
- variable is not constructed until the layers are parsed. Consequently,
- using the configuration file as a pre-configuration file is not possible
- unless it is located in the current working directory.
- Minimally, each configuration file must define the machine and the
- temporary directory BitBake uses for the build. Suggested practice
- dictates that you do not overlap the temporary directories used during
- the builds.
- Aside from separate configuration files for each target, you must also
- enable BitBake to perform multiple configuration builds. Enabling is
- accomplished by setting the
- :term:`BBMULTICONFIG` variable in the
- ``local.conf`` configuration file. As an example, suppose you had
- configuration files for ``target1`` and ``target2`` defined in the build
- directory. The following statement in the ``local.conf`` file both
- enables BitBake to perform multiple configuration builds and specifies
- the two extra multiconfigs: ::
- BBMULTICONFIG = "target1 target2"
- Once the target configuration files are in place and BitBake has been
- enabled to perform multiple configuration builds, use the following
- command form to start the builds: ::
- $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
- Here is an example for two extra multiconfigs: ``target1`` and ``target2``: ::
- $ bitbake mc::target mc:target1:target mc:target2:target
- .. _bb-enabling-multiple-configuration-build-dependencies:
- Enabling Multiple Configuration Build Dependencies
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Sometimes dependencies can exist between targets (multiconfigs) in a
- multiple configuration build. For example, suppose that in order to
- build an image for a particular architecture, the root filesystem of
- another build for a different architecture needs to exist. In other
- words, the image for the first multiconfig depends on the root
- filesystem of the second multiconfig. This dependency is essentially
- that the task in the recipe that builds one multiconfig is dependent on
- the completion of the task in the recipe that builds another
- multiconfig.
- To enable dependencies in a multiple configuration build, you must
- declare the dependencies in the recipe using the following statement
- form: ::
- task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
- To better show how to use this statement, consider an example with two
- multiconfigs: ``target1`` and ``target2``: ::
- image_task[mcdepends] = "mc:target1:target2:image2:rootfs_task"
- In this example, the
- ``from_multiconfig`` is "target1" and the ``to_multiconfig`` is "target2". The
- task on which the image whose recipe contains image_task depends on the
- completion of the rootfs_task used to build out image2, which is
- associated with the "target2" multiconfig.
- Once you set up this dependency, you can build the "target1" multiconfig
- using a BitBake command as follows: ::
- $ bitbake mc:target1:image1
- This command executes all the tasks needed to create ``image1`` for the "target1"
- multiconfig. Because of the dependency, BitBake also executes through
- the ``rootfs_task`` for the "target2" multiconfig build.
- Having a recipe depend on the root filesystem of another build might not
- seem that useful. Consider this change to the statement in the image1
- recipe: ::
- image_task[mcdepends] = "mc:target1:target2:image2:image_task"
- In this case, BitBake must create ``image2`` for the "target2" build since
- the "target1" build depends on it.
- Because "target1" and "target2" are enabled for multiple configuration
- builds and have separate configuration files, BitBake places the
- artifacts for each build in the respective temporary build directories.
|