2.Process.rst 24 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503
  1. .. _development_process:
  2. How the development process works
  3. =================================
  4. Linux kernel development in the early 1990's was a pretty loose affair,
  5. with relatively small numbers of users and developers involved. With a
  6. user base in the millions and with some 2,000 developers involved over the
  7. course of one year, the kernel has since had to evolve a number of
  8. processes to keep development happening smoothly. A solid understanding of
  9. how the process works is required in order to be an effective part of it.
  10. The big picture
  11. ---------------
  12. The kernel developers use a loosely time-based release process, with a new
  13. major kernel release happening every two or three months. The recent
  14. release history looks like this:
  15. ====== =================
  16. 5.0 March 3, 2019
  17. 5.1 May 5, 2019
  18. 5.2 July 7, 2019
  19. 5.3 September 15, 2019
  20. 5.4 November 24, 2019
  21. 5.5 January 6, 2020
  22. ====== =================
  23. Every 5.x release is a major kernel release with new features, internal
  24. API changes, and more. A typical release can contain about 13,000
  25. changesets with changes to several hundred thousand lines of code. 5.x is
  26. the leading edge of Linux kernel development; the kernel uses a
  27. rolling development model which is continually integrating major changes.
  28. A relatively straightforward discipline is followed with regard to the
  29. merging of patches for each release. At the beginning of each development
  30. cycle, the "merge window" is said to be open. At that time, code which is
  31. deemed to be sufficiently stable (and which is accepted by the development
  32. community) is merged into the mainline kernel. The bulk of changes for a
  33. new development cycle (and all of the major changes) will be merged during
  34. this time, at a rate approaching 1,000 changes ("patches," or "changesets")
  35. per day.
  36. (As an aside, it is worth noting that the changes integrated during the
  37. merge window do not come out of thin air; they have been collected, tested,
  38. and staged ahead of time. How that process works will be described in
  39. detail later on).
  40. The merge window lasts for approximately two weeks. At the end of this
  41. time, Linus Torvalds will declare that the window is closed and release the
  42. first of the "rc" kernels. For the kernel which is destined to be 5.6,
  43. for example, the release which happens at the end of the merge window will
  44. be called 5.6-rc1. The -rc1 release is the signal that the time to
  45. merge new features has passed, and that the time to stabilize the next
  46. kernel has begun.
  47. Over the next six to ten weeks, only patches which fix problems should be
  48. submitted to the mainline. On occasion a more significant change will be
  49. allowed, but such occasions are rare; developers who try to merge new
  50. features outside of the merge window tend to get an unfriendly reception.
  51. As a general rule, if you miss the merge window for a given feature, the
  52. best thing to do is to wait for the next development cycle. (An occasional
  53. exception is made for drivers for previously-unsupported hardware; if they
  54. touch no in-tree code, they cannot cause regressions and should be safe to
  55. add at any time).
  56. As fixes make their way into the mainline, the patch rate will slow over
  57. time. Linus releases new -rc kernels about once a week; a normal series
  58. will get up to somewhere between -rc6 and -rc9 before the kernel is
  59. considered to be sufficiently stable and the final release is made.
  60. At that point the whole process starts over again.
  61. As an example, here is how the 5.4 development cycle went (all dates in
  62. 2019):
  63. ============== ===============================
  64. September 15 5.3 stable release
  65. September 30 5.4-rc1, merge window closes
  66. October 6 5.4-rc2
  67. October 13 5.4-rc3
  68. October 20 5.4-rc4
  69. October 27 5.4-rc5
  70. November 3 5.4-rc6
  71. November 10 5.4-rc7
  72. November 17 5.4-rc8
  73. November 24 5.4 stable release
  74. ============== ===============================
  75. How do the developers decide when to close the development cycle and create
  76. the stable release? The most significant metric used is the list of
  77. regressions from previous releases. No bugs are welcome, but those which
  78. break systems which worked in the past are considered to be especially
  79. serious. For this reason, patches which cause regressions are looked upon
  80. unfavorably and are quite likely to be reverted during the stabilization
  81. period.
  82. The developers' goal is to fix all known regressions before the stable
  83. release is made. In the real world, this kind of perfection is hard to
  84. achieve; there are just too many variables in a project of this size.
  85. There comes a point where delaying the final release just makes the problem
  86. worse; the pile of changes waiting for the next merge window will grow
  87. larger, creating even more regressions the next time around. So most 5.x
  88. kernels go out with a handful of known regressions though, hopefully, none
  89. of them are serious.
  90. Once a stable release is made, its ongoing maintenance is passed off to the
  91. "stable team," currently Greg Kroah-Hartman. The stable team will release
  92. occasional updates to the stable release using the 5.x.y numbering scheme.
  93. To be considered for an update release, a patch must (1) fix a significant
  94. bug, and (2) already be merged into the mainline for the next development
  95. kernel. Kernels will typically receive stable updates for a little more
  96. than one development cycle past their initial release. So, for example, the
  97. 5.2 kernel's history looked like this (all dates in 2019):
  98. ============== ===============================
  99. July 7 5.2 stable release
  100. July 14 5.2.1
  101. July 21 5.2.2
  102. July 26 5.2.3
  103. July 28 5.2.4
  104. July 31 5.2.5
  105. ... ...
  106. October 11 5.2.21
  107. ============== ===============================
  108. 5.2.21 was the final stable update of the 5.2 release.
  109. Some kernels are designated "long term" kernels; they will receive support
  110. for a longer period. As of this writing, the current long term kernels
  111. and their maintainers are:
  112. ====== ================================ =======================
  113. 3.16 Ben Hutchings (very long-term kernel)
  114. 4.4 Greg Kroah-Hartman & Sasha Levin (very long-term kernel)
  115. 4.9 Greg Kroah-Hartman & Sasha Levin
  116. 4.14 Greg Kroah-Hartman & Sasha Levin
  117. 4.19 Greg Kroah-Hartman & Sasha Levin
  118. 5.4 Greg Kroah-Hartman & Sasha Levin
  119. ====== ================================ =======================
  120. The selection of a kernel for long-term support is purely a matter of a
  121. maintainer having the need and the time to maintain that release. There
  122. are no known plans for long-term support for any specific upcoming
  123. release.
  124. The lifecycle of a patch
  125. ------------------------
  126. Patches do not go directly from the developer's keyboard into the mainline
  127. kernel. There is, instead, a somewhat involved (if somewhat informal)
  128. process designed to ensure that each patch is reviewed for quality and that
  129. each patch implements a change which is desirable to have in the mainline.
  130. This process can happen quickly for minor fixes, or, in the case of large
  131. and controversial changes, go on for years. Much developer frustration
  132. comes from a lack of understanding of this process or from attempts to
  133. circumvent it.
  134. In the hopes of reducing that frustration, this document will describe how
  135. a patch gets into the kernel. What follows below is an introduction which
  136. describes the process in a somewhat idealized way. A much more detailed
  137. treatment will come in later sections.
  138. The stages that a patch goes through are, generally:
  139. - Design. This is where the real requirements for the patch - and the way
  140. those requirements will be met - are laid out. Design work is often
  141. done without involving the community, but it is better to do this work
  142. in the open if at all possible; it can save a lot of time redesigning
  143. things later.
  144. - Early review. Patches are posted to the relevant mailing list, and
  145. developers on that list reply with any comments they may have. This
  146. process should turn up any major problems with a patch if all goes
  147. well.
  148. - Wider review. When the patch is getting close to ready for mainline
  149. inclusion, it should be accepted by a relevant subsystem maintainer -
  150. though this acceptance is not a guarantee that the patch will make it
  151. all the way to the mainline. The patch will show up in the maintainer's
  152. subsystem tree and into the -next trees (described below). When the
  153. process works, this step leads to more extensive review of the patch and
  154. the discovery of any problems resulting from the integration of this
  155. patch with work being done by others.
  156. - Please note that most maintainers also have day jobs, so merging
  157. your patch may not be their highest priority. If your patch is
  158. getting feedback about changes that are needed, you should either
  159. make those changes or justify why they should not be made. If your
  160. patch has no review complaints but is not being merged by its
  161. appropriate subsystem or driver maintainer, you should be persistent
  162. in updating the patch to the current kernel so that it applies cleanly
  163. and keep sending it for review and merging.
  164. - Merging into the mainline. Eventually, a successful patch will be
  165. merged into the mainline repository managed by Linus Torvalds. More
  166. comments and/or problems may surface at this time; it is important that
  167. the developer be responsive to these and fix any issues which arise.
  168. - Stable release. The number of users potentially affected by the patch
  169. is now large, so, once again, new problems may arise.
  170. - Long-term maintenance. While it is certainly possible for a developer
  171. to forget about code after merging it, that sort of behavior tends to
  172. leave a poor impression in the development community. Merging code
  173. eliminates some of the maintenance burden, in that others will fix
  174. problems caused by API changes. But the original developer should
  175. continue to take responsibility for the code if it is to remain useful
  176. in the longer term.
  177. One of the largest mistakes made by kernel developers (or their employers)
  178. is to try to cut the process down to a single "merging into the mainline"
  179. step. This approach invariably leads to frustration for everybody
  180. involved.
  181. How patches get into the Kernel
  182. -------------------------------
  183. There is exactly one person who can merge patches into the mainline kernel
  184. repository: Linus Torvalds. But, for example, of the over 9,500 patches
  185. which went into the 2.6.38 kernel, only 112 (around 1.3%) were directly
  186. chosen by Linus himself. The kernel project has long since grown to a size
  187. where no single developer could possibly inspect and select every patch
  188. unassisted. The way the kernel developers have addressed this growth is
  189. through the use of a lieutenant system built around a chain of trust.
  190. The kernel code base is logically broken down into a set of subsystems:
  191. networking, specific architecture support, memory management, video
  192. devices, etc. Most subsystems have a designated maintainer, a developer
  193. who has overall responsibility for the code within that subsystem. These
  194. subsystem maintainers are the gatekeepers (in a loose way) for the portion
  195. of the kernel they manage; they are the ones who will (usually) accept a
  196. patch for inclusion into the mainline kernel.
  197. Subsystem maintainers each manage their own version of the kernel source
  198. tree, usually (but certainly not always) using the git source management
  199. tool. Tools like git (and related tools like quilt or mercurial) allow
  200. maintainers to track a list of patches, including authorship information
  201. and other metadata. At any given time, the maintainer can identify which
  202. patches in his or her repository are not found in the mainline.
  203. When the merge window opens, top-level maintainers will ask Linus to "pull"
  204. the patches they have selected for merging from their repositories. If
  205. Linus agrees, the stream of patches will flow up into his repository,
  206. becoming part of the mainline kernel. The amount of attention that Linus
  207. pays to specific patches received in a pull operation varies. It is clear
  208. that, sometimes, he looks quite closely. But, as a general rule, Linus
  209. trusts the subsystem maintainers to not send bad patches upstream.
  210. Subsystem maintainers, in turn, can pull patches from other maintainers.
  211. For example, the networking tree is built from patches which accumulated
  212. first in trees dedicated to network device drivers, wireless networking,
  213. etc. This chain of repositories can be arbitrarily long, though it rarely
  214. exceeds two or three links. Since each maintainer in the chain trusts
  215. those managing lower-level trees, this process is known as the "chain of
  216. trust."
  217. Clearly, in a system like this, getting patches into the kernel depends on
  218. finding the right maintainer. Sending patches directly to Linus is not
  219. normally the right way to go.
  220. Next trees
  221. ----------
  222. The chain of subsystem trees guides the flow of patches into the kernel,
  223. but it also raises an interesting question: what if somebody wants to look
  224. at all of the patches which are being prepared for the next merge window?
  225. Developers will be interested in what other changes are pending to see
  226. whether there are any conflicts to worry about; a patch which changes a
  227. core kernel function prototype, for example, will conflict with any other
  228. patches which use the older form of that function. Reviewers and testers
  229. want access to the changes in their integrated form before all of those
  230. changes land in the mainline kernel. One could pull changes from all of
  231. the interesting subsystem trees, but that would be a big and error-prone
  232. job.
  233. The answer comes in the form of -next trees, where subsystem trees are
  234. collected for testing and review. The older of these trees, maintained by
  235. Andrew Morton, is called "-mm" (for memory management, which is how it got
  236. started). The -mm tree integrates patches from a long list of subsystem
  237. trees; it also has some patches aimed at helping with debugging.
  238. Beyond that, -mm contains a significant collection of patches which have
  239. been selected by Andrew directly. These patches may have been posted on a
  240. mailing list, or they may apply to a part of the kernel for which there is
  241. no designated subsystem tree. As a result, -mm operates as a sort of
  242. subsystem tree of last resort; if there is no other obvious path for a
  243. patch into the mainline, it is likely to end up in -mm. Miscellaneous
  244. patches which accumulate in -mm will eventually either be forwarded on to
  245. an appropriate subsystem tree or be sent directly to Linus. In a typical
  246. development cycle, approximately 5-10% of the patches going into the
  247. mainline get there via -mm.
  248. The current -mm patch is available in the "mmotm" (-mm of the moment)
  249. directory at:
  250. https://www.ozlabs.org/~akpm/mmotm/
  251. Use of the MMOTM tree is likely to be a frustrating experience, though;
  252. there is a definite chance that it will not even compile.
  253. The primary tree for next-cycle patch merging is linux-next, maintained by
  254. Stephen Rothwell. The linux-next tree is, by design, a snapshot of what
  255. the mainline is expected to look like after the next merge window closes.
  256. Linux-next trees are announced on the linux-kernel and linux-next mailing
  257. lists when they are assembled; they can be downloaded from:
  258. https://www.kernel.org/pub/linux/kernel/next/
  259. Linux-next has become an integral part of the kernel development process;
  260. all patches merged during a given merge window should really have found
  261. their way into linux-next some time before the merge window opens.
  262. Staging trees
  263. -------------
  264. The kernel source tree contains the drivers/staging/ directory, where
  265. many sub-directories for drivers or filesystems that are on their way to
  266. being added to the kernel tree live. They remain in drivers/staging while
  267. they still need more work; once complete, they can be moved into the
  268. kernel proper. This is a way to keep track of drivers that aren't
  269. up to Linux kernel coding or quality standards, but people may want to use
  270. them and track development.
  271. Greg Kroah-Hartman currently maintains the staging tree. Drivers that
  272. still need work are sent to him, with each driver having its own
  273. subdirectory in drivers/staging/. Along with the driver source files, a
  274. TODO file should be present in the directory as well. The TODO file lists
  275. the pending work that the driver needs for acceptance into the kernel
  276. proper, as well as a list of people that should be Cc'd for any patches to
  277. the driver. Current rules require that drivers contributed to staging
  278. must, at a minimum, compile properly.
  279. Staging can be a relatively easy way to get new drivers into the mainline
  280. where, with luck, they will come to the attention of other developers and
  281. improve quickly. Entry into staging is not the end of the story, though;
  282. code in staging which is not seeing regular progress will eventually be
  283. removed. Distributors also tend to be relatively reluctant to enable
  284. staging drivers. So staging is, at best, a stop on the way toward becoming
  285. a proper mainline driver.
  286. Tools
  287. -----
  288. As can be seen from the above text, the kernel development process depends
  289. heavily on the ability to herd collections of patches in various
  290. directions. The whole thing would not work anywhere near as well as it
  291. does without suitably powerful tools. Tutorials on how to use these tools
  292. are well beyond the scope of this document, but there is space for a few
  293. pointers.
  294. By far the dominant source code management system used by the kernel
  295. community is git. Git is one of a number of distributed version control
  296. systems being developed in the free software community. It is well tuned
  297. for kernel development, in that it performs quite well when dealing with
  298. large repositories and large numbers of patches. It also has a reputation
  299. for being difficult to learn and use, though it has gotten better over
  300. time. Some sort of familiarity with git is almost a requirement for kernel
  301. developers; even if they do not use it for their own work, they'll need git
  302. to keep up with what other developers (and the mainline) are doing.
  303. Git is now packaged by almost all Linux distributions. There is a home
  304. page at:
  305. https://git-scm.com/
  306. That page has pointers to documentation and tutorials.
  307. Among the kernel developers who do not use git, the most popular choice is
  308. almost certainly Mercurial:
  309. https://www.selenic.com/mercurial/
  310. Mercurial shares many features with git, but it provides an interface which
  311. many find easier to use.
  312. The other tool worth knowing about is Quilt:
  313. https://savannah.nongnu.org/projects/quilt/
  314. Quilt is a patch management system, rather than a source code management
  315. system. It does not track history over time; it is, instead, oriented
  316. toward tracking a specific set of changes against an evolving code base.
  317. Some major subsystem maintainers use quilt to manage patches intended to go
  318. upstream. For the management of certain kinds of trees (-mm, for example),
  319. quilt is the best tool for the job.
  320. Mailing lists
  321. -------------
  322. A great deal of Linux kernel development work is done by way of mailing
  323. lists. It is hard to be a fully-functioning member of the community
  324. without joining at least one list somewhere. But Linux mailing lists also
  325. represent a potential hazard to developers, who risk getting buried under a
  326. load of electronic mail, running afoul of the conventions used on the Linux
  327. lists, or both.
  328. Most kernel mailing lists are run on vger.kernel.org; the master list can
  329. be found at:
  330. http://vger.kernel.org/vger-lists.html
  331. There are lists hosted elsewhere, though; a number of them are at
  332. redhat.com/mailman/listinfo.
  333. The core mailing list for kernel development is, of course, linux-kernel.
  334. This list is an intimidating place to be; volume can reach 500 messages per
  335. day, the amount of noise is high, the conversation can be severely
  336. technical, and participants are not always concerned with showing a high
  337. degree of politeness. But there is no other place where the kernel
  338. development community comes together as a whole; developers who avoid this
  339. list will miss important information.
  340. There are a few hints which can help with linux-kernel survival:
  341. - Have the list delivered to a separate folder, rather than your main
  342. mailbox. One must be able to ignore the stream for sustained periods of
  343. time.
  344. - Do not try to follow every conversation - nobody else does. It is
  345. important to filter on both the topic of interest (though note that
  346. long-running conversations can drift away from the original subject
  347. without changing the email subject line) and the people who are
  348. participating.
  349. - Do not feed the trolls. If somebody is trying to stir up an angry
  350. response, ignore them.
  351. - When responding to linux-kernel email (or that on other lists) preserve
  352. the Cc: header for all involved. In the absence of a strong reason (such
  353. as an explicit request), you should never remove recipients. Always make
  354. sure that the person you are responding to is in the Cc: list. This
  355. convention also makes it unnecessary to explicitly ask to be copied on
  356. replies to your postings.
  357. - Search the list archives (and the net as a whole) before asking
  358. questions. Some developers can get impatient with people who clearly
  359. have not done their homework.
  360. - Avoid top-posting (the practice of putting your answer above the quoted
  361. text you are responding to). It makes your response harder to read and
  362. makes a poor impression.
  363. - Ask on the correct mailing list. Linux-kernel may be the general meeting
  364. point, but it is not the best place to find developers from all
  365. subsystems.
  366. The last point - finding the correct mailing list - is a common place for
  367. beginning developers to go wrong. Somebody who asks a networking-related
  368. question on linux-kernel will almost certainly receive a polite suggestion
  369. to ask on the netdev list instead, as that is the list frequented by most
  370. networking developers. Other lists exist for the SCSI, video4linux, IDE,
  371. filesystem, etc. subsystems. The best place to look for mailing lists is
  372. in the MAINTAINERS file packaged with the kernel source.
  373. Getting started with Kernel development
  374. ---------------------------------------
  375. Questions about how to get started with the kernel development process are
  376. common - from both individuals and companies. Equally common are missteps
  377. which make the beginning of the relationship harder than it has to be.
  378. Companies often look to hire well-known developers to get a development
  379. group started. This can, in fact, be an effective technique. But it also
  380. tends to be expensive and does not do much to grow the pool of experienced
  381. kernel developers. It is possible to bring in-house developers up to speed
  382. on Linux kernel development, given the investment of a bit of time. Taking
  383. this time can endow an employer with a group of developers who understand
  384. the kernel and the company both, and who can help to train others as well.
  385. Over the medium term, this is often the more profitable approach.
  386. Individual developers are often, understandably, at a loss for a place to
  387. start. Beginning with a large project can be intimidating; one often wants
  388. to test the waters with something smaller first. This is the point where
  389. some developers jump into the creation of patches fixing spelling errors or
  390. minor coding style issues. Unfortunately, such patches create a level of
  391. noise which is distracting for the development community as a whole, so,
  392. increasingly, they are looked down upon. New developers wishing to
  393. introduce themselves to the community will not get the sort of reception
  394. they wish for by these means.
  395. Andrew Morton gives this advice for aspiring kernel developers
  396. ::
  397. The #1 project for all kernel beginners should surely be "make sure
  398. that the kernel runs perfectly at all times on all machines which
  399. you can lay your hands on". Usually the way to do this is to work
  400. with others on getting things fixed up (this can require
  401. persistence!) but that's fine - it's a part of kernel development.
  402. (https://lwn.net/Articles/283982/).
  403. In the absence of obvious problems to fix, developers are advised to look
  404. at the current lists of regressions and open bugs in general. There is
  405. never any shortage of issues in need of fixing; by addressing these issues,
  406. developers will gain experience with the process while, at the same time,
  407. building respect with the rest of the development community.