adding-packages-python.txt 10 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260
  1. // -*- mode:doc; -*-
  2. // vim: set syntax=asciidoc:
  3. === Infrastructure for Python packages
  4. This infrastructure applies to Python packages that use the standard
  5. Python setuptools mechanism as their build system, generally
  6. recognizable by the usage of a +setup.py+ script.
  7. [[python-package-tutorial]]
  8. ==== +python-package+ tutorial
  9. First, let's see how to write a +.mk+ file for a Python package,
  10. with an example :
  11. ------------------------
  12. 01: ################################################################################
  13. 02: #
  14. 03: # python-foo
  15. 04: #
  16. 05: ################################################################################
  17. 06:
  18. 07: PYTHON_FOO_VERSION = 1.0
  19. 08: PYTHON_FOO_SOURCE = python-foo-$(PYTHON_FOO_VERSION).tar.xz
  20. 09: PYTHON_FOO_SITE = http://www.foosoftware.org/download
  21. 10: PYTHON_FOO_LICENSE = BSD-3-Clause
  22. 11: PYTHON_FOO_LICENSE_FILES = LICENSE
  23. 12: PYTHON_FOO_ENV = SOME_VAR=1
  24. 13: PYTHON_FOO_DEPENDENCIES = libmad
  25. 14: PYTHON_FOO_SETUP_TYPE = distutils
  26. 15:
  27. 16: $(eval $(python-package))
  28. ------------------------
  29. On line 7, we declare the version of the package.
  30. On line 8 and 9, we declare the name of the tarball (xz-ed tarball
  31. recommended) and the location of the tarball on the Web. Buildroot
  32. will automatically download the tarball from this location.
  33. On line 10 and 11, we give licensing details about the package (its
  34. license on line 10, and the file containing the license text on line
  35. 11).
  36. On line 12, we tell Buildroot to pass custom options to the Python
  37. +setup.py+ script when it is configuring the package.
  38. On line 13, we declare our dependencies, so that they are built
  39. before the build process of our package starts.
  40. On line 14, we declare the specific Python build system being used. In
  41. this case the +distutils+ Python build system is used. The two
  42. supported ones are +distutils+ and +setuptools+.
  43. Finally, on line 16, we invoke the +python-package+ macro that
  44. generates all the Makefile rules that actually allow the package to be
  45. built.
  46. [[python-package-reference]]
  47. ==== +python-package+ reference
  48. As a policy, packages that merely provide Python modules should all be
  49. named +python-<something>+ in Buildroot. Other packages that use the
  50. Python build system, but are not Python modules, can freely choose
  51. their name (existing examples in Buildroot are +scons+ and
  52. +supervisor+).
  53. Packages that are only compatible with one version of Python (as in:
  54. Python 2 or Python 3) should depend on that version explicitely in
  55. their +Config.in+ file (+BR2_PACKAGE_PYTHON+ for Python 2,
  56. +BR2_PACKAGE_PYTHON3+ for Python 3). Packages that are compatible
  57. with both versions should not explicitely depend on them in their
  58. +Config.in+ file, since that condition is already expressed for the
  59. whole "External python modules" menu.
  60. The main macro of the Python package infrastructure is
  61. +python-package+. It is similar to the +generic-package+ macro. It is
  62. also possible to create Python host packages with the
  63. +host-python-package+ macro.
  64. Just like the generic infrastructure, the Python infrastructure works
  65. by defining a number of variables before calling the +python-package+
  66. or +host-python-package+ macros.
  67. All the package metadata information variables that exist in the
  68. xref:generic-package-reference[generic package infrastructure] also
  69. exist in the Python infrastructure: +PYTHON_FOO_VERSION+,
  70. +PYTHON_FOO_SOURCE+, +PYTHON_FOO_PATCH+, +PYTHON_FOO_SITE+,
  71. +PYTHON_FOO_SUBDIR+, +PYTHON_FOO_DEPENDENCIES+, +PYTHON_FOO_LICENSE+,
  72. +PYTHON_FOO_LICENSE_FILES+, +PYTHON_FOO_INSTALL_STAGING+, etc.
  73. Note that:
  74. * It is not necessary to add +python+ or +host-python+ in the
  75. +PYTHON_FOO_DEPENDENCIES+ variable of a package, since these basic
  76. dependencies are automatically added as needed by the Python
  77. package infrastructure.
  78. * Similarly, it is not needed to add +host-setuptools+ to
  79. +PYTHON_FOO_DEPENDENCIES+ for setuptools-based packages, since it's
  80. automatically added by the Python infrastructure as needed.
  81. One variable specific to the Python infrastructure is mandatory:
  82. * +PYTHON_FOO_SETUP_TYPE+, to define which Python build system is used
  83. by the package. The two supported values are +distutils+ and
  84. +setuptools+. If you don't know which one is used in your package,
  85. look at the +setup.py+ file in your package source code, and see
  86. whether it imports things from the +distutils+ module or the
  87. +setuptools+ module.
  88. A few additional variables, specific to the Python infrastructure, can
  89. optionally be defined, depending on the package's needs. Many of them
  90. are only useful in very specific cases, typical packages will
  91. therefore only use a few of them, or none.
  92. * +PYTHON_FOO_SUBDIR+ may contain the name of a subdirectory inside the
  93. package that contains the main +setup.py+ file. This is useful,
  94. if for example, the main +setup.py+ file is not at the root of
  95. the tree extracted by the tarball. If +HOST_PYTHON_FOO_SUBDIR+ is not
  96. specified, it defaults to +PYTHON_FOO_SUBDIR+.
  97. * +PYTHON_FOO_ENV+, to specify additional environment variables to
  98. pass to the Python +setup.py+ script (for both the build and install
  99. steps). Note that the infrastructure is automatically passing
  100. several standard variables, defined in +PKG_PYTHON_DISTUTILS_ENV+
  101. (for distutils target packages), +HOST_PKG_PYTHON_DISTUTILS_ENV+
  102. (for distutils host packages), +PKG_PYTHON_SETUPTOOLS_ENV+ (for
  103. setuptools target packages) and +HOST_PKG_PYTHON_SETUPTOOLS_ENV+
  104. (for setuptools host packages).
  105. * +PYTHON_FOO_BUILD_OPTS+, to specify additional options to pass to the
  106. Python +setup.py+ script during the build step. For target distutils
  107. packages, the +PKG_PYTHON_DISTUTILS_BUILD_OPTS+ options are already
  108. passed automatically by the infrastructure.
  109. * +PYTHON_FOO_INSTALL_TARGET_OPTS+, +PYTHON_FOO_INSTALL_STAGING_OPTS+,
  110. +HOST_PYTHON_FOO_INSTALL_OPTS+ to specify additional options to pass
  111. to the Python +setup.py+ script during the target installation step,
  112. the staging installation step or the host installation,
  113. respectively. Note that the infrastructure is automatically passing
  114. some options, defined in +PKG_PYTHON_DISTUTILS_INSTALL_TARGET_OPTS+
  115. or +PKG_PYTHON_DISTUTILS_INSTALL_STAGING_OPTS+ (for target distutils
  116. packages), +HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS+ (for host
  117. distutils packages), +PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS+ or
  118. +PKG_PYTHON_SETUPTOOLS_INSTALL_STAGING_OPTS+ (for target setuptools
  119. packages) and +HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS+ (for host
  120. setuptools packages).
  121. * +HOST_PYTHON_FOO_NEEDS_HOST_PYTHON+, to define the host python
  122. interpreter. The usage of this variable is limited to host
  123. packages. The two supported value are +python2+ and +python3+. It
  124. will ensure the right host python package is available and will
  125. invoke it for the build. If some build steps are overloaded, the
  126. right python interpreter must be explicitly called in the commands.
  127. With the Python infrastructure, all the steps required to build and
  128. install the packages are already defined, and they generally work well
  129. for most Python-based packages. However, when required, it is still
  130. possible to customize what is done in any particular step:
  131. * By adding a post-operation hook (after extract, patch, configure,
  132. build or install). See xref:hooks[] for details.
  133. * By overriding one of the steps. For example, even if the Python
  134. infrastructure is used, if the package +.mk+ file defines its own
  135. +PYTHON_FOO_BUILD_CMDS+ variable, it will be used instead of the
  136. default Python one. However, using this method should be restricted
  137. to very specific cases. Do not use it in the general case.
  138. [[scanpypi]]
  139. ==== Generating a +python-package+ from a PyPI repository
  140. If the Python package for which you would like to create a Buildroot
  141. package is available on PyPI, you may want to use the +scanpypi+ tool
  142. located in +utils/+ to automate the process.
  143. You can find the list of existing PyPI packages
  144. https://pypi.python.org[here].
  145. +scanpypi+ requires Python's +setuptools+ package to be installed on
  146. your host.
  147. When at the root of your buildroot directory just do :
  148. -----------------------
  149. utils/scanpypi foo bar -o package
  150. -----------------------
  151. This will generate packages +python-foo+ and +python-bar+ in the package
  152. folder if they exist on https://pypi.python.org.
  153. Find the +external python modules+ menu and insert your package inside.
  154. Keep in mind that the items inside a menu should be in alphabetical order.
  155. Please keep in mind that you'll most likely have to manually check the
  156. package for any mistakes as there are things that cannot be guessed by
  157. the generator (e.g. dependencies on any of the python core modules
  158. such as BR2_PACKAGE_PYTHON_ZLIB). Also, please take note that the
  159. license and license files are guessed and must be checked. You also
  160. need to manually add the package to the +package/Config.in+ file.
  161. If your Buildroot package is not in the official Buildroot tree but in
  162. a br2-external tree, use the -o flag as follows:
  163. -----------------------
  164. utils/scanpypi foo bar -o other_package_dir
  165. -----------------------
  166. This will generate packages +python-foo+ and +python-bar+ in the
  167. +other_package_directory+ instead of +package+.
  168. Option +-h+ will list the available options:
  169. -----------------------
  170. utils/scanpypi -h
  171. -----------------------
  172. [[python-package-cffi-backend]]
  173. ==== +python-package+ CFFI backend
  174. C Foreign Function Interface for Python (CFFI) provides a convenient
  175. and reliable way to call compiled C code from Python using interface
  176. declarations written in C. Python packages relying on this backend can
  177. be identified by the appearance of a +cffi+ dependency in the
  178. +install_requires+ field of their +setup.py+ file.
  179. Such a package should:
  180. * add +python-cffi+ as a runtime dependency in order to install the
  181. compiled C library wrapper on the target. This is achieved by adding
  182. +select BR2_PACKAGE_PYTHON_CFFI+ to the package +Config.in+.
  183. ------------------------
  184. config BR2_PACKAGE_PYTHON_FOO
  185. bool "python-foo"
  186. select BR2_PACKAGE_PYTHON_CFFI # runtime
  187. ------------------------
  188. * add +host-python-cffi+ as a build-time dependency in order to
  189. cross-compile the C wrapper. This is achieved by adding
  190. +host-python-cffi+ to the +PYTHON_FOO_DEPENDENCIES+ variable.
  191. ------------------------
  192. ################################################################################
  193. #
  194. # python-foo
  195. #
  196. ################################################################################
  197. ...
  198. PYTHON_FOO_DEPENDENCIES = host-python-cffi
  199. $(eval $(python-package))
  200. ------------------------