460 lines
22 KiB
Text
460 lines
22 KiB
Text
![]() |
2: HOW THE DEVELOPMENT PROCESS WORKS
|
||
|
|
||
|
Linux kernel development in the early 1990's was a pretty loose affair,
|
||
|
with relatively small numbers of users and developers involved. With a
|
||
|
user base in the millions and with some 2,000 developers involved over the
|
||
|
course of one year, the kernel has since had to evolve a number of
|
||
|
processes to keep development happening smoothly. A solid understanding of
|
||
|
how the process works is required in order to be an effective part of it.
|
||
|
|
||
|
|
||
|
2.1: THE BIG PICTURE
|
||
|
|
||
|
The kernel developers use a loosely time-based release process, with a new
|
||
|
major kernel release happening every two or three months. The recent
|
||
|
release history looks like this:
|
||
|
|
||
|
2.6.26 July 13, 2008
|
||
|
2.6.25 April 16, 2008
|
||
|
2.6.24 January 24, 2008
|
||
|
2.6.23 October 9, 2007
|
||
|
2.6.22 July 8, 2007
|
||
|
2.6.21 April 25, 2007
|
||
|
2.6.20 February 4, 2007
|
||
|
|
||
|
Every 2.6.x release is a major kernel release with new features, internal
|
||
|
API changes, and more. A typical 2.6 release can contain over 10,000
|
||
|
changesets with changes to several hundred thousand lines of code. 2.6 is
|
||
|
thus the leading edge of Linux kernel development; the kernel uses a
|
||
|
rolling development model which is continually integrating major changes.
|
||
|
|
||
|
A relatively straightforward discipline is followed with regard to the
|
||
|
merging of patches for each release. At the beginning of each development
|
||
|
cycle, the "merge window" is said to be open. At that time, code which is
|
||
|
deemed to be sufficiently stable (and which is accepted by the development
|
||
|
community) is merged into the mainline kernel. The bulk of changes for a
|
||
|
new development cycle (and all of the major changes) will be merged during
|
||
|
this time, at a rate approaching 1,000 changes ("patches," or "changesets")
|
||
|
per day.
|
||
|
|
||
|
(As an aside, it is worth noting that the changes integrated during the
|
||
|
merge window do not come out of thin air; they have been collected, tested,
|
||
|
and staged ahead of time. How that process works will be described in
|
||
|
detail later on).
|
||
|
|
||
|
The merge window lasts for two weeks. At the end of this time, Linus
|
||
|
Torvalds will declare that the window is closed and release the first of
|
||
|
the "rc" kernels. For the kernel which is destined to be 2.6.26, for
|
||
|
example, the release which happens at the end of the merge window will be
|
||
|
called 2.6.26-rc1. The -rc1 release is the signal that the time to merge
|
||
|
new features has passed, and that the time to stabilize the next kernel has
|
||
|
begun.
|
||
|
|
||
|
Over the next six to ten weeks, only patches which fix problems should be
|
||
|
submitted to the mainline. On occasion a more significant change will be
|
||
|
allowed, but such occasions are rare; developers who try to merge new
|
||
|
features outside of the merge window tend to get an unfriendly reception.
|
||
|
As a general rule, if you miss the merge window for a given feature, the
|
||
|
best thing to do is to wait for the next development cycle. (An occasional
|
||
|
exception is made for drivers for previously-unsupported hardware; if they
|
||
|
touch no in-tree code, they cannot cause regressions and should be safe to
|
||
|
add at any time).
|
||
|
|
||
|
As fixes make their way into the mainline, the patch rate will slow over
|
||
|
time. Linus releases new -rc kernels about once a week; a normal series
|
||
|
will get up to somewhere between -rc6 and -rc9 before the kernel is
|
||
|
considered to be sufficiently stable and the final 2.6.x release is made.
|
||
|
At that point the whole process starts over again.
|
||
|
|
||
|
As an example, here is how the 2.6.25 development cycle went (all dates in
|
||
|
2008):
|
||
|
|
||
|
January 24 2.6.24 stable release
|
||
|
February 10 2.6.25-rc1, merge window closes
|
||
|
February 15 2.6.25-rc2
|
||
|
February 24 2.6.25-rc3
|
||
|
March 4 2.6.25-rc4
|
||
|
March 9 2.6.25-rc5
|
||
|
March 16 2.6.25-rc6
|
||
|
March 25 2.6.25-rc7
|
||
|
April 1 2.6.25-rc8
|
||
|
April 11 2.6.25-rc9
|
||
|
April 16 2.6.25 stable release
|
||
|
|
||
|
How do the developers decide when to close the development cycle and create
|
||
|
the stable release? The most significant metric used is the list of
|
||
|
regressions from previous releases. No bugs are welcome, but those which
|
||
|
break systems which worked in the past are considered to be especially
|
||
|
serious. For this reason, patches which cause regressions are looked upon
|
||
|
unfavorably and are quite likely to be reverted during the stabilization
|
||
|
period.
|
||
|
|
||
|
The developers' goal is to fix all known regressions before the stable
|
||
|
release is made. In the real world, this kind of perfection is hard to
|
||
|
achieve; there are just too many variables in a project of this size.
|
||
|
There comes a point where delaying the final release just makes the problem
|
||
|
worse; the pile of changes waiting for the next merge window will grow
|
||
|
larger, creating even more regressions the next time around. So most 2.6.x
|
||
|
kernels go out with a handful of known regressions though, hopefully, none
|
||
|
of them are serious.
|
||
|
|
||
|
Once a stable release is made, its ongoing maintenance is passed off to the
|
||
|
"stable team," currently comprised of Greg Kroah-Hartman and Chris Wright.
|
||
|
The stable team will release occasional updates to the stable release using
|
||
|
the 2.6.x.y numbering scheme. To be considered for an update release, a
|
||
|
patch must (1) fix a significant bug, and (2) already be merged into the
|
||
|
mainline for the next development kernel. Continuing our 2.6.25 example,
|
||
|
the history (as of this writing) is:
|
||
|
|
||
|
May 1 2.6.25.1
|
||
|
May 6 2.6.25.2
|
||
|
May 9 2.6.25.3
|
||
|
May 15 2.6.25.4
|
||
|
June 7 2.6.25.5
|
||
|
June 9 2.6.25.6
|
||
|
June 16 2.6.25.7
|
||
|
June 21 2.6.25.8
|
||
|
June 24 2.6.25.9
|
||
|
|
||
|
Stable updates for a given kernel are made for approximately six months;
|
||
|
after that, the maintenance of stable releases is solely the responsibility
|
||
|
of the distributors which have shipped that particular kernel.
|
||
|
|
||
|
|
||
|
2.2: THE LIFECYCLE OF A PATCH
|
||
|
|
||
|
Patches do not go directly from the developer's keyboard into the mainline
|
||
|
kernel. There is, instead, a somewhat involved (if somewhat informal)
|
||
|
process designed to ensure that each patch is reviewed for quality and that
|
||
|
each patch implements a change which is desirable to have in the mainline.
|
||
|
This process can happen quickly for minor fixes, or, in the case of large
|
||
|
and controversial changes, go on for years. Much developer frustration
|
||
|
comes from a lack of understanding of this process or from attempts to
|
||
|
circumvent it.
|
||
|
|
||
|
In the hopes of reducing that frustration, this document will describe how
|
||
|
a patch gets into the kernel. What follows below is an introduction which
|
||
|
describes the process in a somewhat idealized way. A much more detailed
|
||
|
treatment will come in later sections.
|
||
|
|
||
|
The stages that a patch goes through are, generally:
|
||
|
|
||
|
- Design. This is where the real requirements for the patch - and the way
|
||
|
those requirements will be met - are laid out. Design work is often
|
||
|
done without involving the community, but it is better to do this work
|
||
|
in the open if at all possible; it can save a lot of time redesigning
|
||
|
things later.
|
||
|
|
||
|
- Early review. Patches are posted to the relevant mailing list, and
|
||
|
developers on that list reply with any comments they may have. This
|
||
|
process should turn up any major problems with a patch if all goes
|
||
|
well.
|
||
|
|
||
|
- Wider review. When the patch is getting close to ready for mainline
|
||
|
inclusion, it will be accepted by a relevant subsystem maintainer -
|
||
|
though this acceptance is not a guarantee that the patch will make it
|
||
|
all the way to the mainline. The patch will show up in the maintainer's
|
||
|
subsystem tree and into the staging trees (described below). When the
|
||
|
process works, this step leads to more extensive review of the patch and
|
||
|
the discovery of any problems resulting from the integration of this
|
||
|
patch with work being done by others.
|
||
|
|
||
|
- Merging into the mainline. Eventually, a successful patch will be
|
||
|
merged into the mainline repository managed by Linus Torvalds. More
|
||
|
comments and/or problems may surface at this time; it is important that
|
||
|
the developer be responsive to these and fix any issues which arise.
|
||
|
|
||
|
- Stable release. The number of users potentially affected by the patch
|
||
|
is now large, so, once again, new problems may arise.
|
||
|
|
||
|
- Long-term maintenance. While it is certainly possible for a developer
|
||
|
to forget about code after merging it, that sort of behavior tends to
|
||
|
leave a poor impression in the development community. Merging code
|
||
|
eliminates some of the maintenance burden, in that others will fix
|
||
|
problems caused by API changes. But the original developer should
|
||
|
continue to take responsibility for the code if it is to remain useful
|
||
|
in the longer term.
|
||
|
|
||
|
One of the largest mistakes made by kernel developers (or their employers)
|
||
|
is to try to cut the process down to a single "merging into the mainline"
|
||
|
step. This approach invariably leads to frustration for everybody
|
||
|
involved.
|
||
|
|
||
|
|
||
|
2.3: HOW PATCHES GET INTO THE KERNEL
|
||
|
|
||
|
There is exactly one person who can merge patches into the mainline kernel
|
||
|
repository: Linus Torvalds. But, of the over 12,000 patches which went
|
||
|
into the 2.6.25 kernel, only 250 (around 2%) were directly chosen by Linus
|
||
|
himself. The kernel project has long since grown to a size where no single
|
||
|
developer could possibly inspect and select every patch unassisted. The
|
||
|
way the kernel developers have addressed this growth is through the use of
|
||
|
a lieutenant system built around a chain of trust.
|
||
|
|
||
|
The kernel code base is logically broken down into a set of subsystems:
|
||
|
networking, specific architecture support, memory management, video
|
||
|
devices, etc. Most subsystems have a designated maintainer, a developer
|
||
|
who has overall responsibility for the code within that subsystem. These
|
||
|
subsystem maintainers are the gatekeepers (in a loose way) for the portion
|
||
|
of the kernel they manage; they are the ones who will (usually) accept a
|
||
|
patch for inclusion into the mainline kernel.
|
||
|
|
||
|
Subsystem maintainers each manage their own version of the kernel source
|
||
|
tree, usually (but certainly not always) using the git source management
|
||
|
tool. Tools like git (and related tools like quilt or mercurial) allow
|
||
|
maintainers to track a list of patches, including authorship information
|
||
|
and other metadata. At any given time, the maintainer can identify which
|
||
|
patches in his or her repository are not found in the mainline.
|
||
|
|
||
|
When the merge window opens, top-level maintainers will ask Linus to "pull"
|
||
|
the patches they have selected for merging from their repositories. If
|
||
|
Linus agrees, the stream of patches will flow up into his repository,
|
||
|
becoming part of the mainline kernel. The amount of attention that Linus
|
||
|
pays to specific patches received in a pull operation varies. It is clear
|
||
|
that, sometimes, he looks quite closely. But, as a general rule, Linus
|
||
|
trusts the subsystem maintainers to not send bad patches upstream.
|
||
|
|
||
|
Subsystem maintainers, in turn, can pull patches from other maintainers.
|
||
|
For example, the networking tree is built from patches which accumulated
|
||
|
first in trees dedicated to network device drivers, wireless networking,
|
||
|
etc. This chain of repositories can be arbitrarily long, though it rarely
|
||
|
exceeds two or three links. Since each maintainer in the chain trusts
|
||
|
those managing lower-level trees, this process is known as the "chain of
|
||
|
trust."
|
||
|
|
||
|
Clearly, in a system like this, getting patches into the kernel depends on
|
||
|
finding the right maintainer. Sending patches directly to Linus is not
|
||
|
normally the right way to go.
|
||
|
|
||
|
|
||
|
2.4: STAGING TREES
|
||
|
|
||
|
The chain of subsystem trees guides the flow of patches into the kernel,
|
||
|
but it also raises an interesting question: what if somebody wants to look
|
||
|
at all of the patches which are being prepared for the next merge window?
|
||
|
Developers will be interested in what other changes are pending to see
|
||
|
whether there are any conflicts to worry about; a patch which changes a
|
||
|
core kernel function prototype, for example, will conflict with any other
|
||
|
patches which use the older form of that function. Reviewers and testers
|
||
|
want access to the changes in their integrated form before all of those
|
||
|
changes land in the mainline kernel. One could pull changes from all of
|
||
|
the interesting subsystem trees, but that would be a big and error-prone
|
||
|
job.
|
||
|
|
||
|
The answer comes in the form of staging trees, where subsystem trees are
|
||
|
collected for testing and review. The older of these trees, maintained by
|
||
|
Andrew Morton, is called "-mm" (for memory management, which is how it got
|
||
|
started). The -mm tree integrates patches from a long list of subsystem
|
||
|
trees; it also has some patches aimed at helping with debugging.
|
||
|
|
||
|
Beyond that, -mm contains a significant collection of patches which have
|
||
|
been selected by Andrew directly. These patches may have been posted on a
|
||
|
mailing list, or they may apply to a part of the kernel for which there is
|
||
|
no designated subsystem tree. As a result, -mm operates as a sort of
|
||
|
subsystem tree of last resort; if there is no other obvious path for a
|
||
|
patch into the mainline, it is likely to end up in -mm. Miscellaneous
|
||
|
patches which accumulate in -mm will eventually either be forwarded on to
|
||
|
an appropriate subsystem tree or be sent directly to Linus. In a typical
|
||
|
development cycle, approximately 10% of the patches going into the mainline
|
||
|
get there via -mm.
|
||
|
|
||
|
The current -mm patch can always be found from the front page of
|
||
|
|
||
|
http://kernel.org/
|
||
|
|
||
|
Those who want to see the current state of -mm can get the "-mm of the
|
||
|
moment" tree, found at:
|
||
|
|
||
|
http://userweb.kernel.org/~akpm/mmotm/
|
||
|
|
||
|
Use of the MMOTM tree is likely to be a frustrating experience, though;
|
||
|
there is a definite chance that it will not even compile.
|
||
|
|
||
|
The other staging tree, started more recently, is linux-next, maintained by
|
||
|
Stephen Rothwell. The linux-next tree is, by design, a snapshot of what
|
||
|
the mainline is expected to look like after the next merge window closes.
|
||
|
Linux-next trees are announced on the linux-kernel and linux-next mailing
|
||
|
lists when they are assembled; they can be downloaded from:
|
||
|
|
||
|
http://www.kernel.org/pub/linux/kernel/people/sfr/linux-next/
|
||
|
|
||
|
Some information about linux-next has been gathered at:
|
||
|
|
||
|
http://linux.f-seidel.de/linux-next/pmwiki/
|
||
|
|
||
|
How the linux-next tree will fit into the development process is still
|
||
|
changing. As of this writing, the first full development cycle involving
|
||
|
linux-next (2.6.26) is coming to an end; thus far, it has proved to be a
|
||
|
valuable resource for finding and fixing integration problems before the
|
||
|
beginning of the merge window. See http://lwn.net/Articles/287155/ for
|
||
|
more information on how linux-next has worked to set up the 2.6.27 merge
|
||
|
window.
|
||
|
|
||
|
Some developers have begun to suggest that linux-next should be used as the
|
||
|
target for future development as well. The linux-next tree does tend to be
|
||
|
far ahead of the mainline and is more representative of the tree into which
|
||
|
any new work will be merged. The downside to this idea is that the
|
||
|
volatility of linux-next tends to make it a difficult development target.
|
||
|
See http://lwn.net/Articles/289013/ for more information on this topic, and
|
||
|
stay tuned; much is still in flux where linux-next is involved.
|
||
|
|
||
|
|
||
|
2.5: TOOLS
|
||
|
|
||
|
As can be seen from the above text, the kernel development process depends
|
||
|
heavily on the ability to herd collections of patches in various
|
||
|
directions. The whole thing would not work anywhere near as well as it
|
||
|
does without suitably powerful tools. Tutorials on how to use these tools
|
||
|
are well beyond the scope of this document, but there is space for a few
|
||
|
pointers.
|
||
|
|
||
|
By far the dominant source code management system used by the kernel
|
||
|
community is git. Git is one of a number of distributed version control
|
||
|
systems being developed in the free software community. It is well tuned
|
||
|
for kernel development, in that it performs quite well when dealing with
|
||
|
large repositories and large numbers of patches. It also has a reputation
|
||
|
for being difficult to learn and use, though it has gotten better over
|
||
|
time. Some sort of familiarity with git is almost a requirement for kernel
|
||
|
developers; even if they do not use it for their own work, they'll need git
|
||
|
to keep up with what other developers (and the mainline) are doing.
|
||
|
|
||
|
Git is now packaged by almost all Linux distributions. There is a home
|
||
|
page at
|
||
|
|
||
|
http://git.or.cz/
|
||
|
|
||
|
That page has pointers to documentation and tutorials. One should be
|
||
|
aware, in particular, of the Kernel Hacker's Guide to git, which has
|
||
|
information specific to kernel development:
|
||
|
|
||
|
http://linux.yyz.us/git-howto.html
|
||
|
|
||
|
Among the kernel developers who do not use git, the most popular choice is
|
||
|
almost certainly Mercurial:
|
||
|
|
||
|
http://www.selenic.com/mercurial/
|
||
|
|
||
|
Mercurial shares many features with git, but it provides an interface which
|
||
|
many find easier to use.
|
||
|
|
||
|
The other tool worth knowing about is Quilt:
|
||
|
|
||
|
http://savannah.nongnu.org/projects/quilt/
|
||
|
|
||
|
Quilt is a patch management system, rather than a source code management
|
||
|
system. It does not track history over time; it is, instead, oriented
|
||
|
toward tracking a specific set of changes against an evolving code base.
|
||
|
Some major subsystem maintainers use quilt to manage patches intended to go
|
||
|
upstream. For the management of certain kinds of trees (-mm, for example),
|
||
|
quilt is the best tool for the job.
|
||
|
|
||
|
|
||
|
2.6: MAILING LISTS
|
||
|
|
||
|
A great deal of Linux kernel development work is done by way of mailing
|
||
|
lists. It is hard to be a fully-functioning member of the community
|
||
|
without joining at least one list somewhere. But Linux mailing lists also
|
||
|
represent a potential hazard to developers, who risk getting buried under a
|
||
|
load of electronic mail, running afoul of the conventions used on the Linux
|
||
|
lists, or both.
|
||
|
|
||
|
Most kernel mailing lists are run on vger.kernel.org; the master list can
|
||
|
be found at:
|
||
|
|
||
|
http://vger.kernel.org/vger-lists.html
|
||
|
|
||
|
There are lists hosted elsewhere, though; a number of them are at
|
||
|
lists.redhat.com.
|
||
|
|
||
|
The core mailing list for kernel development is, of course, linux-kernel.
|
||
|
This list is an intimidating place to be; volume can reach 500 messages per
|
||
|
day, the amount of noise is high, the conversation can be severely
|
||
|
technical, and participants are not always concerned with showing a high
|
||
|
degree of politeness. But there is no other place where the kernel
|
||
|
development community comes together as a whole; developers who avoid this
|
||
|
list will miss important information.
|
||
|
|
||
|
There are a few hints which can help with linux-kernel survival:
|
||
|
|
||
|
- Have the list delivered to a separate folder, rather than your main
|
||
|
mailbox. One must be able to ignore the stream for sustained periods of
|
||
|
time.
|
||
|
|
||
|
- Do not try to follow every conversation - nobody else does. It is
|
||
|
important to filter on both the topic of interest (though note that
|
||
|
long-running conversations can drift away from the original subject
|
||
|
without changing the email subject line) and the people who are
|
||
|
participating.
|
||
|
|
||
|
- Do not feed the trolls. If somebody is trying to stir up an angry
|
||
|
response, ignore them.
|
||
|
|
||
|
- When responding to linux-kernel email (or that on other lists) preserve
|
||
|
the Cc: header for all involved. In the absence of a strong reason (such
|
||
|
as an explicit request), you should never remove recipients. Always make
|
||
|
sure that the person you are responding to is in the Cc: list. This
|
||
|
convention also makes it unnecessary to explicitly ask to be copied on
|
||
|
replies to your postings.
|
||
|
|
||
|
- Search the list archives (and the net as a whole) before asking
|
||
|
questions. Some developers can get impatient with people who clearly
|
||
|
have not done their homework.
|
||
|
|
||
|
- Avoid top-posting (the practice of putting your answer above the quoted
|
||
|
text you are responding to). It makes your response harder to read and
|
||
|
makes a poor impression.
|
||
|
|
||
|
- Ask on the correct mailing list. Linux-kernel may be the general meeting
|
||
|
point, but it is not the best place to find developers from all
|
||
|
subsystems.
|
||
|
|
||
|
The last point - finding the correct mailing list - is a common place for
|
||
|
beginning developers to go wrong. Somebody who asks a networking-related
|
||
|
question on linux-kernel will almost certainly receive a polite suggestion
|
||
|
to ask on the netdev list instead, as that is the list frequented by most
|
||
|
networking developers. Other lists exist for the SCSI, video4linux, IDE,
|
||
|
filesystem, etc. subsystems. The best place to look for mailing lists is
|
||
|
in the MAINTAINERS file packaged with the kernel source.
|
||
|
|
||
|
|
||
|
2.7: GETTING STARTED WITH KERNEL DEVELOPMENT
|
||
|
|
||
|
Questions about how to get started with the kernel development process are
|
||
|
common - from both individuals and companies. Equally common are missteps
|
||
|
which make the beginning of the relationship harder than it has to be.
|
||
|
|
||
|
Companies often look to hire well-known developers to get a development
|
||
|
group started. This can, in fact, be an effective technique. But it also
|
||
|
tends to be expensive and does not do much to grow the pool of experienced
|
||
|
kernel developers. It is possible to bring in-house developers up to speed
|
||
|
on Linux kernel development, given the investment of a bit of time. Taking
|
||
|
this time can endow an employer with a group of developers who understand
|
||
|
the kernel and the company both, and who can help to train others as well.
|
||
|
Over the medium term, this is often the more profitable approach.
|
||
|
|
||
|
Individual developers are often, understandably, at a loss for a place to
|
||
|
start. Beginning with a large project can be intimidating; one often wants
|
||
|
to test the waters with something smaller first. This is the point where
|
||
|
some developers jump into the creation of patches fixing spelling errors or
|
||
|
minor coding style issues. Unfortunately, such patches create a level of
|
||
|
noise which is distracting for the development community as a whole, so,
|
||
|
increasingly, they are looked down upon. New developers wishing to
|
||
|
introduce themselves to the community will not get the sort of reception
|
||
|
they wish for by these means.
|
||
|
|
||
|
Andrew Morton gives this advice for aspiring kernel developers
|
||
|
|
||
|
The #1 project for all kernel beginners should surely be "make sure
|
||
|
that the kernel runs perfectly at all times on all machines which
|
||
|
you can lay your hands on". Usually the way to do this is to work
|
||
|
with others on getting things fixed up (this can require
|
||
|
persistence!) but that's fine - it's a part of kernel development.
|
||
|
|
||
|
(http://lwn.net/Articles/283982/).
|
||
|
|
||
|
In the absence of obvious problems to fix, developers are advised to look
|
||
|
at the current lists of regressions and open bugs in general. There is
|
||
|
never any shortage of issues in need of fixing; by addressing these issues,
|
||
|
developers will gain experience with the process while, at the same time,
|
||
|
building respect with the rest of the development community.
|