appa.xml

<appendix id="LKN-appendix-making-changes">

<title>
Helpful Utilities
</title>

<para>
Retrieving, building, updating, and maintaining a Linux kernel
source tree involves a lot of different steps, as this book has shown.
Developers being naturally lazy creatures, they have created some programs
to help with the various routine tasks.  Here we describe a few of these
useful tools and the basics on how to use them.
</para>

<para>
Linux kernel development differs in many ways from traditional software
development. Some of the special demands on kernel programmers include:
</para>

<itemizedlist>

<listitem>
<para>
<!--
  AO: The following is probably worse for the Linux kernel than other
  projects.

  gkh: yes it is...
-->
Constantly applying your changes to the moving target of a fast-based
kernel development release schedule.
</para>
</listitem>

<listitem>
<para>
<!--

  AO: The following seems pretty universal.

  gkh: no, usually you work in your own area, and no one touches code
  there.  Merging is very rare in most commercial work because of this.  Or
  at least that was my experience in all my years doing it...

-->
Resolving any merge conflicts between changes you have made and changes
made by other people.
</para>
</listitem>

<listitem>
<para>
Exporting your changes in a format that lets others incorporate and work
with it easily.
</para>
</listitem>

</itemizedlist>

<sect1>
<title>patch and diff
<footnote>
<!--
  AO: I think it's fair and proper to add the following. By the way,
  do you have the right to reuse the article? Forgive me if you told
  me already.

  gkh: Yes, I have the right to reuse it, I retain the original copyright on
  any article published by them, after a few months after publication happens.
-->
<para>
This section is based on an article originally published in Linux Journal.
<!-- The article can be found at http://www.linuxjournal.com/article/6183 -->
</para>
</footnote>
</title>

<para>
One of the most common methods of doing kernel work is to use the
<command>patch</command> and <command>diff</command> programs.
<!--

  AO: I removed a sentence here that didn't seem necessary given the
  introduction that lays out the appendix.

-->
To use these tools, two different directory trees: a "clean" one and a
"working" one must be used. The clean tree is a released kernel version,
while the working one is based on the same version but contains your
modifications.  Then you can use <command>patch</command> and
<command>diff</command> to extract your changes and port them forward to a
new kernel release.
</para>

<!--

  AO: The following step-by-step, slow-moving description of how to
  set up source code was probably useful for the Linux Journal, but
  more detailed than necessary for readers of this book. I would tell
  people how to do the download, and then just say they can create two
  directories of different names with the same contents.

-->
<para>
For an example, create two directories containing the latest kernel version
as described in <xref linkend="LKN-chapter-retrieve-source" />:
<screen>
$ <userinput>tar -zxf linux-2.6.19.tar.gz</userinput>
$ <userinput>mv linux-2.6.19 linux-2.6.19-dirty</userinput>
$ <userinput>tar -zxf linux-2.6.19.tar.gz</userinput>
$ <userinput>ls</userinput>
linux-2.6.19/
linux-2.6.19-dirty/
</screen>
</para>

<para>
Now make all of the different changes you wish to do in the
<filename>-dirty</filename> directory and leave the clean, original kernel
directory alone.  After finishing making changes, you should create a patch
to send it to other people:
<screen>
$ <userinput>diff -Naur -X linux-2.6.19/Documentation/dontdiff linux-2.6.19/ linux-2.6.19-dirty/ &gt; my_patch</userinput>
</screen>
</para>

<para>
This will create a file called <filename>my_patch</filename> that contains
the difference between your work and a clean 2.6.19 kernel tree. This patch
then can be sent to other people via e-mail.
</para>

<sect2>
<title>New Kernel Versions</title>

<para>
If a new kernel version is released, and you wish to port your changes to
the new version, you need to try to apply your generated patch onto a clean
kernel version. This can be done in the following steps:
</para>

<orderedlist>
<listitem>
<para>
Generate your original patch, as in the previous example.
</para>
</listitem>

<listitem>
<para>
Using the official patch from
<emphasis>kernel.org</emphasis>, move the old kernel version
forward one release:
</para>

<screen>
$ <userinput>cd linux-2.6.19</userinput>
$ <userinput>patch -p1 &lt; ../patch-2.6.20</userinput>
$ <userinput>cd ..</userinput>
$ <userinput>mv linux-2.6.19 linux-2.6.20</userinput>
</screen>
</listitem>

<listitem>
<para>
Move your working directory forward one release by removing your patch,
then apply the new update:
<screen>
$ <userinput>cd linux-2.6.19-dirty</userinput>
$ <userinput>patch -p1 -R &lt; ../my_patch</userinput>
$ <userinput>patch -p1 &lt; ../patch-2.6.20</userinput>
$ <userinput>cd ..</userinput>
$ <userinput>mv linux-2.4.19-dirty linux-2.6.20-dirty</userinput>
</screen>
</para>
</listitem>

<listitem>
<para>
Try to apply your patch on top of the new update:
<screen>
$ <userinput>cd linux-2.6.20-dirty</userinput>
$ <userinput>patch -p1 &lt; ../my_patch</userinput>
</screen>
</para>

<para>
If your patch does not apply cleanly, resolve all of the conflicts that are
created (the <command>patch</command> command will tell you about these,
leaving behind <filename>.rej</filename> and <filename>.orig</filename>
files for you to compare and fix up manually using your favorite
editor).  This merge process can be the most difficult part if you have
made changes to portions of the source tree that have been changed by other
people.
</para>
</listitem>

</orderedlist>

<para>
If you use this development process, I highly recommend getting the
excellent <literal>patchutils</literal> set of programs (found at
<systemitem role="url">http://cyberelk.net/tim/patchutils</systemitem>). These
programs enable you to manipulate text patches easily in all sorts of
useful ways, and have saved kernel developers many hours of tedious
work.
</para>

</sect2>
</sect1>

<sect1>
<title>Managing Your Patches With quilt</title>

<para>
Kernel development using <command>patch</command> and
<command>diff</command> generally works quite well. But after a while, most
people grow tired of it and look for a different way to work that does not
involve so much tedious patching and merging.  Luckily, a few kernel
developers came up with a program called <command>quilt</command> that
handles the process of manipulating a number of patches made against an
external source tree much easier.
</para>

<para>
The idea for <command>quilt</command> came from a set of scripts written by
Andrew Morton that he used to first maintain the memory management
subsystem, and then later the entire development kernel tree.  His scripts
were tied very tightly to his workflow, but the ideas behind them were very
powerful.  Andreas Gruenbacher took those ideas and created the
<command>quilt</command> tool.
</para>

<para>
The basic idea behind <command>quilt</command> is that you work with a
pristine source tree, and add a bunch of patches on top of it.  You can
push and pop different patches off of the source tree, and maintain this
list of patches in a simple manner.
</para>

<para>
To get started, create a kernel source tree like always:
<screen>
$ <userinput>tar -zxf linux-2.6.19.tar.gz</userinput>
$ <userinput>ls</userinput>
linux-2.6.19/
</screen>
And go into that directory:
<screen>
$ <userinput>cd linux-2.6.19</userinput>
</screen>
To get started, create a directory called <filename>patches</filename> that
will hold all of our kernel patches:
<screen>
$ <userinput>mkdir patches</userinput>
</screen>
Then tell <command>quilt</command> to create a new patch called
<literal>patch1</literal>:
<screen>
$ <userinput>quilt new patch1</userinput>
Patch patches/patch1 is now on top
</screen>
<command>quilt</command> needs to be told about all of the different files
that will be modifed by this new patch.  To do this, use the
<command>add</command> command:
<screen>
$ <userinput>quilt add Makefile</userinput>
File Makefile added to patch patches/patch1
</screen>
Edit the file <filename>Makefile</filename>, modify the
<literal>EXTRAVERSION</literal> line, and save the change.
After you finish, tell <command>quilt</command> to refresh the patch:
<screen>
$ <userinput>quilt refresh</userinput>
Refreshed patch patches/patch1
</screen>
The file <filename>patches/patch1</filename> will contain
a patch with the changes that you have just made:
<screen>
$ <userinput>cat patches/patch1</userinput>
Index: linux-2.6.19/Makefile
===================================================================
--- linux-2.6.19.orig/Makefile
+++ linux-2.6.19/Makefile
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 19
-EXTRAVERSION =
+EXTRAVERSION = -dirty
 NAME=Crazed Snow-Weasel
 
 # *DOCUMENTATION*
</screen>

</para>

<para>
You can continue on, working with this single patch, or create a new one to
go on top of this patch.  As an example, if three different patches had
been created, <filename>patch1</filename>, <filename>patch2</filename>, and
<filename>patch3</filename>, they will be applied one on top of one another.
</para>

<para>
To see the list of patches that are currently applied:
<screen>
$ <userinput>quilt series -v</userinput>
+ patches/patch1
+ patches/patch2
= patches/patch3
</screen>
This output shows that all three patches are applied, and that the current
one is <filename>patch3</filename>.
</para>

<para>
If a new kernel version is released, and you wish to port your changes to
the new version, quilt can handle this easily with the following steps:
</para>

<orderedlist>
<listitem>
<para>
Pop off all of the patches that are currently on the tree:
<screen>
$ <userinput>quilt pop -a</userinput>
Removing patch patches/patch3
Restoring drivers/usb/Makefile

Removing patch patches/patch2
Restoring drivers/Makefile

Removing patch patches/patch1
Restoring Makefile

No patches applied
</screen>
</para>
</listitem>

<listitem>
<para>
Using the official patch from
<systemitem role="url">kernel.org</systemitem>, move the old kernel version
forward one release:
</para>

<screen>
$ <userinput>patch -p1 &lt; ../patch-2.6.20</userinput>
$ <userinput>cd ..</userinput>
$ <userinput>mv linux-2.6.19 linux-2.6.20</userinput>
</screen>
</listitem>

<listitem>
<para>
Now have quilt push all of the patches back on top of the new tree:
<screen>
$ <userinput>quilt push</userinput>
Applying patch patches/patch1
patching file Makefile
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- rejects in file Makefile
Patch patches/patch1 does not apply (enforce with -f)
</screen>
</para>
</listitem>

<listitem>
<para>
As the first patch doesn't apply cleanly, force the patch to be applied and
then fix it up:
<screen>
$ <userinput>quilt push -f</userinput>
Applying patch patches/patch1
patching file Makefile
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file Makefile.rej
Applied patch patches/patch1 (forced; needs refresh)
$ <userinput>vim Makefile.rej Makefile</userinput>
</screen>
</para>
</listitem>

<listitem>
<para>
After the patch is applied by hand, refresh the patch:
<screen>
$ <userinput>quilt refresh</userinput>
Refreshed patch patches/patch1
</screen>
</para>
</listitem>

<listitem>
<para>
And continue on pushing the other patches:
<screen>
$ <userinput>quilt push</userinput>
Applying patch patches/patch2
patching file drivers/Makefile

Now at patch patches/patch2
$ <userinput>quilt push</userinput>
Applying patch patches/patch3
patching file drivers/usb/Makefile

Now at patch patches/patch3
</screen>
</para>
</listitem>
</orderedlist>

<para>
<command>quilt</command> also has options that will automatically email out
all of the patches in the series to a group of people or a mailing list,
delete specific patches in the middle of the series, go up or down the
series of patches until a specific patch is found, and many more powerful
options.
</para>

<para>
If you want to do any kind of kernel development,
<command>quilt</command> is strongly recommended, even for tracking a
few patches, instead of using the more difficult <command>diff</command>
and <command>patch</command> method.  It is much simpler and will save you
much time and effort.
</para>

<para>
On a personal note, I cannot recommend this tool enough, as I
use it every day to manage hundreds of patches in different development
trees.  It is also used by numerous Linux distributions to maintain their
kernel packages, and has an involved and responsive development community.
</para>

</sect1>


<sect1>
<title>git</title>

<para>
<command>git</command> is a source code control tool that was originally
written by Linus Torvalds when the Linux kernel was looking for a new
source code control system.  It is a distributed system, which differs from
traditional source code control systems such as <command>cvs</command> in that
it is not required to be connected to a server in order to make a commit to
the repository.
</para>

<para>
<command>git</command> is one of the most powerful, flexible, and fast source code control
systems currently available, and has an active development team working
behind it.  The main web page for <command>git</command> can be found at
<systemitem role="url">http://git.or.cz/</systemitem>.  It is recommended
that any new user of <command>git</command> go through the published
tutorials in order to become familiar with how <command>git</command>
works, and how to use it properly.
</para>

<para>
The Linux kernel is developed using <command>git</command>, and the latest
<command>git</command> kernel tree can be found at
<systemitem role="url">http://www.kernel.org/git/</systemitem>, along with
a large list of other kernel developer's <command>git</command> repositories.
</para>

<para>
It is not necessary to use <command>git</command> in order to do Linux
kernel development, but it is very handy in helping to track down kernel
bugs.  If you report a bug to the Linux kernel developers, they might ask
you to use <command>git bisect</command> in order to find the exact change
that caused the bug to happen.  If so, follow the directions in the
<command>git</command> documentation for how to use this.
</para>

</sect1>

<sect1 id="lkn_ketchup">
<title>ketchup</title>

<para>
<command>ketchup</command> is a very handy tool used to update or
switch between different versions of the Linux kernel source tree.  It has
the ability to:
<itemizedlist>
<listitem>
<para>
Find the latest version of the kernel, download it, and uncompress it.
</para>
</listitem>
<listitem>
<para>
Update a currently installed version of the kernel source tree to any other
version, by patching the tree to the proper version.
</para>
</listitem>
<listitem>
<para>
Handle the different development and stable branches of the kernel
tree, including the <literal>-mm</literal> and <literal>-stable</literal> trees.
</para>
</listitem>
<listitem>
<para>
Download any patches or tarballs needed to do the update, if they
are not present on the machine already.
</para>
</listitem>
<listitem>
<para>
Check the GPG signatures of the tarball and patches to verify that it
has downloaded a correct file.
</para>
</listitem>
</itemizedlist>
</para>

<para>
<command>ketchup</command> can be found at
<systemitem role="url">http://www.selenic.com/ketchup/</systemitem>
and has lots of additional documentation in the wiki at
<systemitem role="url">http://www.selenic.com/ketchup/wiki/</systemitem>.
.
</para>

<para>
Here is a set of steps that show how simple it is to use
<command>ketchup</command> to download a specific kernel version, and then
have it switch the directory to another kernel version with only a minimal
number of commands.
</para>

<para>
To have <command>ketchup</command> download the 2.6.16.24 version of the
kernel source tree into a directory, and rename the directory to be the
same as the kernel version, enter:
<screen>
$ <userinput>mkdir foo</userinput>
$ <userinput>cd foo</userinput>
$ <userinput>ketchup -r 2.6.16.24</userinput>
None -> 2.6.16.24
Unpacking linux-2.6.17.tar.bz2
Applying patch-2.6.17.bz2 -R
Applying patch-2.6.16.24.bz2
Current directory renamed to /home/gregkh/linux/linux-2.6.16.24
</screen>
</para>

<para>
Now, to upgrade this kernel to contain the latest stable kernel version, just
enter:
<screen>
$ <userinput>ketchup -r 2.6</userinput>
2.6.16.24 -> 2.6.17.11
Applying patch-2.6.16.24.bz2 -R
Applying patch-2.6.17.bz2
Downloading patch-2.6.17.11.bz2
--22:21:14--  http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.17.11.bz2
           => `/home/greg/.ketchup/patch-2.6.17.11.bz2.partial'
Resolving www.kernel.org... 204.152.191.37, 204.152.191.5
Connecting to www.kernel.org|204.152.191.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36,809 (36K) [application/x-bzip2]

100%[====================================>] 36,809        93.32K/s

22:21:14 (92.87 KB/s) - `/home/greg/.ketchup/patch-2.6.17.11.bz2.partial' saved [36809/36809]

Downloading patch-2.6.17.11.bz2.sign
--22:21:14--  http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.17.11.bz2.sign
           => `/home/greg/.ketchup/patch-2.6.17.11.bz2.sign.partial'
Resolving www.kernel.org... 204.152.191.37, 204.152.191.5
Connecting to www.kernel.org|204.152.191.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 248 [application/pgp-signature]

100%[====================================>] 248           --.--K/s

22:21:14 (21.50 MB/s) - `/home/greg/.ketchup/patch-2.6.17.11.bz2.sign.partial' saved [248/248]

Verifying signature...
gpg: Signature made Wed Aug 23 15:01:04 2006 PDT using DSA key ID 517D0F0E
gpg: Good signature from "Linux Kernel Archives Verification Key &gt;ftpadmin@kernel.org&lt;"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C75D C40A 11D7 AF88 9981  ED5B C86B A06A 517D 0F0E
Applying patch-2.6.17.11.bz2
Current directory renamed to /home/greg/linux/tmp/x/linux-2.6.17.11
</screen>
This shows that <command>ketchup</command> automatically determined that
the newest stable version was 2.6.17.11, and downloaded the needed patch
files in order to get to that version.
</para>

<para>
It is highly recommended that you use <command>ketchup</command> if you
want to download any Linux kernel source trees.  It takes all of the work
in finding where on the server the correct patch file is, and automatically
will apply the patch in the properly format, after checking that the
downloaded file is properly signed.  Combine <command>ketchup</command>
with <command>quilt</command> and you have a very powerful setup that
contains everything that you need in order to deal effectively with
kernel sources as a Linux kernel
developer.
</para>

</sect1>


</appendix>

<!-- vim: set ai tw=72 : -->