xz.rst 5.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127
  1. ============================
  2. XZ data compression in Linux
  3. ============================
  4. Introduction
  5. ============
  6. XZ is a general purpose data compression format with high compression
  7. ratio and relatively fast decompression. The primary compression
  8. algorithm (filter) is LZMA2. Additional filters can be used to improve
  9. compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
  10. improve compression ratio of executable data.
  11. The XZ decompressor in Linux is called XZ Embedded. It supports
  12. the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
  13. for integrity checking. The home page of XZ Embedded is at
  14. <https://tukaani.org/xz/embedded.html>, where you can find the
  15. latest version and also information about using the code outside
  16. the Linux kernel.
  17. For userspace, XZ Utils provide a zlib-like compression library
  18. and a gzip-like command line tool. XZ Utils can be downloaded from
  19. <https://tukaani.org/xz/>.
  20. XZ related components in the kernel
  21. ===================================
  22. The xz_dec module provides XZ decompressor with single-call (buffer
  23. to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
  24. module is documented in include/linux/xz.h.
  25. The xz_dec_test module is for testing xz_dec. xz_dec_test is not
  26. useful unless you are hacking the XZ decompressor. xz_dec_test
  27. allocates a char device major dynamically to which one can write
  28. .xz files from userspace. The decompressed output is thrown away.
  29. Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
  30. See the xz_dec_test source code for the details.
  31. For decompressing the kernel image, initramfs, and initrd, there
  32. is a wrapper function in lib/decompress_unxz.c. Its API is the
  33. same as in other decompress_*.c files, which is defined in
  34. include/linux/decompress/generic.h.
  35. scripts/xz_wrap.sh is a wrapper for the xz command line tool found
  36. from XZ Utils. The wrapper sets compression options to values suitable
  37. for compressing the kernel image.
  38. For kernel makefiles, two commands are provided for use with
  39. $(call if_needed). The kernel image should be compressed with
  40. $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
  41. dictionary. It will also append a four-byte trailer containing the
  42. uncompressed size of the file, which is needed by the boot code.
  43. Other things should be compressed with $(call if_needed,xzmisc)
  44. which will use no BCJ filter and 1 MiB LZMA2 dictionary.
  45. Notes on compression options
  46. ============================
  47. Since the XZ Embedded supports only streams with no integrity check or
  48. CRC32, make sure that you don't use some other integrity check type
  49. when encoding files that are supposed to be decoded by the kernel. With
  50. liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
  51. when encoding. With the xz command line tool, use --check=none or
  52. --check=crc32.
  53. Using CRC32 is strongly recommended unless there is some other layer
  54. which will verify the integrity of the uncompressed data anyway.
  55. Double checking the integrity would probably be waste of CPU cycles.
  56. Note that the headers will always have a CRC32 which will be validated
  57. by the decoder; you can only change the integrity check type (or
  58. disable it) for the actual uncompressed data.
  59. In userspace, LZMA2 is typically used with dictionary sizes of several
  60. megabytes. The decoder needs to have the dictionary in RAM, thus big
  61. dictionaries cannot be used for files that are intended to be decoded
  62. by the kernel. 1 MiB is probably the maximum reasonable dictionary
  63. size for in-kernel use (maybe more is OK for initramfs). The presets
  64. in XZ Utils may not be optimal when creating files for the kernel,
  65. so don't hesitate to use custom settings. Example::
  66. xz --check=crc32 --lzma2=dict=512KiB inputfile
  67. An exception to above dictionary size limitation is when the decoder
  68. is used in single-call mode. Decompressing the kernel itself is an
  69. example of this situation. In single-call mode, the memory usage
  70. doesn't depend on the dictionary size, and it is perfectly fine to
  71. use a big dictionary: for maximum compression, the dictionary should
  72. be at least as big as the uncompressed data itself.
  73. Future plans
  74. ============
  75. Creating a limited XZ encoder may be considered if people think it is
  76. useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
  77. the fastest settings, so it isn't clear if LZMA2 encoder is wanted
  78. into the kernel.
  79. Support for limited random-access reading is planned for the
  80. decompression code. I don't know if it could have any use in the
  81. kernel, but I know that it would be useful in some embedded projects
  82. outside the Linux kernel.
  83. Conformance to the .xz file format specification
  84. ================================================
  85. There are a couple of corner cases where things have been simplified
  86. at expense of detecting errors as early as possible. These should not
  87. matter in practice all, since they don't cause security issues. But
  88. it is good to know this if testing the code e.g. with the test files
  89. from XZ Utils.
  90. Reporting bugs
  91. ==============
  92. Before reporting a bug, please check that it's not fixed already
  93. at upstream. See <https://tukaani.org/xz/embedded.html> to get the
  94. latest code.
  95. Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
  96. Freenode and talk to Larhzu. I don't actively read LKML or other
  97. kernel-related mailing lists, so if there's something I should know,
  98. you should email to me personally or use IRC.
  99. Don't bother Igor Pavlov with questions about the XZ implementation
  100. in the kernel or about XZ Utils. While these two implementations
  101. include essential code that is directly based on Igor Pavlov's code,
  102. these implementations aren't maintained nor supported by him.