DebianBuildd
This documentation is written to explain how to setup a build daemon to recompile the whole Debian unstable archive.
Note: this howto has been written for etch - things may have changed since.
How it works
Buildd setup works with 3 packages:
wanna-build is used to store informations about packages in a
database. sbuild is used to actually build packages into a chroot.
buildd is used to fetch packages to build via wanna-build and run sbuild
to build them.
Setting up the box
Install Debian
Install debian. Version installed here is etch.
Install packages
The following packages are needed:
build-essential devscripts bzip2 fakeroot patch subversion
Fetch the code and install packages
Currently, everything is not packaged. That's a shame.
sbuild is in the archive, but wanna-build and buildd are not. You fetch the code via Subversion from
Here is how I build sbuild and wanna-build:
$ svn co
$ cd wanna-build
$ vi debian/control
--> Remove versionned dependency on sbuild, because we can use the version from unstable
$ debuild -uc -us
Then, install packages:
# aptitude install sbuild debootstrap cron-apt libmldbm-perl dupload quinn-diff
# dpkg -i wanna-build_2006.01.08_sparc.deb buildd_2006.01.08_sparc.deb
You can use the sbuild version from sid or from the svn repository. Currently I use both.
Create system environment
You can use /usr/share/doc/buildd/examples/buildd-setup-system.gz directly (zcat | sh), but I prefer to do everything by myself.
# as root
adduser --system --shell /bin/sh --uid 60000 --gecos 'Build Daemon' --group --disabled-password buildd
mkdir -p /var/lib/debbuild/srcdep-lock
chown -R buildd:buildd /var/lib/debbuild
chmod -R 2775 /var/lib/debbuild
cd ~buildd
zcat /usr/share/doc/buildd/examples/buildd.conf.gz > buildd.conf
zcat /usr/share/doc/sbuild/examples/sbuildrc.gz > .sbuildrc
chown buildd:buildd buildd.conf .sbuildrc
mkdir -p logs build old-logs upload bin stats/graphs mqueue
echo "|/usr/bin/buildd-mail-wrapper" > .forward
chown buildd:buildd logs build old-logs upload bin stats .forward mqueue
Setup chroot
Now it's time to create our chroot! The buildd-make-chroot command
is not available in the sbuild package from sid, so use the one you
grabbed from svn, be smart!
# cd ~buildd
# ~/wanna-build/buildd-make-chroot buildd etch build/chroot-etch
Note: actually, this command fails because of dependencies, with the following error:
dpkg: dependency problems prevent removal of makedev:
gnupg depends on makedev (>= 2.3.1-13) | devfsd | hurd; however:
Package makedev is to be removed.
Package devfsd is not installed.
Package hurd is not installed.
The solution is to keep makedev (which is removed by buildd-make-chroot). Apply the following patch_:
--- buildd-make-chroot.orig 2007-05-11 12:31:44.000000000 +0000
+++ buildd-make-chroot 2007-05-11 12:31:57.000000000 +0000
@@ -74,7 +74,7 @@
chmod -R 02775 $TARGET/var/debbuild
echo include /etc/ld.so.conf.d/*.conf >> $TARGET/etc/ld.so.conf
(cd $TARGET/dev ; ./MAKEDEV fd)
-sudo chroot $TARGET dpkg -P debconf-i18n debconf liblocale-gettext-perl libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl procps makedev
+sudo chroot $TARGET dpkg -P debconf-i18n debconf liblocale-gettext-perl libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl procps
echo "Successfully setup chroot for a buildd"
echo Possible commands to append to fstab:
echo echo $SUITE-proc $TARGET/proc proc defaults 0 0 \>\> /etc/fstab
You can finish the install:
# as root
mkdir -p ~buildd/build/chroot-etch/var/lib/sbuild/srcdep-lock
chown -R buildd.buildd ~buildd/build/chroot-etch/var/lib/sbuild/
mkdir ~buildd/build/chroot-etch/home/buildd
chown buildd.buildd ~buildd/build/chroot-etch/home/buildd
cd ~buildd/build && ln -s chroot-etch chroot-stable
After, edit the /etc/apt/sources.list in your chroot:
# as root
vi ~buildd/build/chroot-etch/etc/apt/sources.list
chroot build/chroot-etch /bin/sh
apt-get update
exit
You should mount proc into your chroot otherwise some build will failed. Add the following line to your /etc/fstab:
etch-proc /home/buildd/build/chroot-etch/proc proc defaults 0 0
Give root to your buildd
sbuild should be able to use schroot, but, in fact, that's terribly b0rked right now. So we will use the old sudo method.
# visudo
Add:
Defaults:buildd env_keep+="APT_CONFIG DEBIAN_FRONTEND SHELL"
buildd ALL= NOPASSWD: ALL
Setup sbuild
Change your user to buildd (sudo -H -s -u buildd), and configure sbuild.
# vi .sbuildrc
# egrep -v '^(#|$)' .sbuildrc
$build_dir="$HOME/build";
$log_dir = "$HOME/logs";
$mailto = "pollux\@inl.fr";
$maintainer_name='Pierre Chifflier
';
$purge_build_directory="always";
$stalled_pkg_timeout = 90;
1;
Setup buildd
Change your user to buildd and edit ~buildd/buildd.conf:
$ egrep -v '^(#|$)' buildd.conf
require "/etc/buildd.conf.global";
@take_from_dists = qw(stable testing unstable);
@no_auto_build = qw(boot-floppies);
@weak_no_auto_build = qw();
$autoclean_interval = 86400;
$admin_mail = "buildd admin ";
$build_log_keep = 1;
$statistics_mail = 'pollux\@inl.fr';
$dupload_to = "anonymous-inl";
$dupload_to_security = "anonymous-inl";
$log_queued_messages = 0;
$wanna_build_user = "buildd_arch-hostname";
$wanna_build_dbbase = "build-db";
$sshcmd = "";
$sshsocket = "";
1;
Configure dupload
The buildd user should have a working dupload installation (.dupload.conf):
package config;
$default_host = 'anonymous-inl';
$cfg{'anonymous-inl'} = {
fqdn => "xxx.xxx.xxx.xxx",
method => "scpb",
login => "pollux",
incoming => "/var/www/debian/mini-dinstall/incoming/",
dinstall_runs => 1,
};
Add cron jobs
Some commands should be added to the crontab, to ensure that buildd is running, and to upload packages:
$ crontab -l
# m h dom mon dow command
MAILTO=pollux@inl.fr
30 0,6,12,18 * * * /usr/bin/buildd-uploader
20 0,6,12,18 * * * /usr/bin/buildd-watcher
Setup wanna-build
Copy wanna-build local configuration:
# zcat /usr/share/doc/wanna-build/examples/wanna-build.conf.local.gz > /etc/wanna-build.conf.local
# chown buildd /etc/wanna-build.conf.local
Then you can create an empty wanna-build database:
# # as root:
# install -m 755 -d -obuildd -gbuildd /var/debbuild
# # as user buildd:
$ wanna-build -d stable --create-db --clean-db
Now you will need to fill up the database with all packages. That's easy to do with a little script (thanks to Julien Louis).
#!/bin/sh
cd /tmp
wget -q -O Sources.main.gz
wget -q -O Sources.contrib.gz
if [ -e Sources.main.gz ] && [ -e Sources.contrib.gz ]; then
zcat Sources.main.gz Sources.contrib.gz > Sources
fi
touch Packages
quinn-diff -A @dpkg-architecture -qDEB_HOST_ARCH@ 2>/dev/null | wanna-build --merge-quinn -d stable
rm -f Sources Packages
You can run this script once a day to update the database.
Building arch=all packages
The default configuration of buildd is not to build
architecture-independant packages. To force buildd to build arch=all
packages, apply the following patch to '/usr/bin/buildd':
--- buildd.OLD 2007-05-14 11:59:20.000000000 +0000
+++ buildd 2007-05-14 11:59:47.000000000 +0000
@@ -466,7 +466,8 @@
my @sbuild_args = ( 'nice', '-n', "$conf::nice_level", 'sbuild',
'--batch', "--stats-dir=$main::HOME/stats",
- "--dist=$dist" );
+ "--dist=$dist",
+ "--arch-all" );
my $sbuild_gb = '--auto-give-back';
if ($conf::sshcmd) {
$sbuild_gb .= "=";
Useful commands
List packages which needs to be build
$ /usr/bin/wb-short/list-needs-build -d stable
unknown/glpi_0.68.3-3 [unknown:uncompiled]
Total 1 package(s)
Give back a package for build (aka retry)
$ /usr/bin/wb-short/give-back-build -d stable -U buildd_arch-hostname glpi_0.68.3-3
Remove package state from database
$ /usr/bin/wb-short/forget-build -d stable -U buildd_arch ocsinventory-server_1.01-2
binNMU
$ cd ~buildd/build
$ sbuild --make-binNMU="Rebuild for erase unwelcomed dependency" --binNMU=1 -d unstable -v hello_2.1.1-4
- "--make-binNMU" defines message to add changelog.
- "--binNMU" defines minor revision. If you set 1, hello_2.1.1-4.0.1 will be built.
Mutt script for buildd auto-reply
When a build is complete (and successful), a mail is sent (see ~buildd/.sbuildrc).
To complete the build and upload the package, the .changes file must be
extracted from the build log, must be signed and sent back to the
buildd. As this is quite boring, here is some scripts to make this part
easier.
Folder-hook
First, we will apply the script only to mails stored in the =buildd folder. So we use a folder-hook to associate this folder with a specific script:
folder-hook =buildd source ~/.mutt/defaults.buildd
Mutt preferences and send-hook
This is the big part: we set some preferences for how to display the
folder, and we set a send-hook to make an almost automatic reply,
passing the build log to a shell script to extract the .changes file,
and then signing it. Practically, all what you have to do is to press
'r' (reply) on a mail, then 'y' to send it (you will have to enter the
passphrase if not in memory).
# -*- muttrc -*-
# Use the global defaults first.
source ~/.mutt/defaults.global
# This is the index format for mailing lists.
set index_format="%4C %Z %{%b %d} %-15.15n (%4l) %s"
# Sort on threads
set sort=threads
# Give mailing lists a more "slrn" like feel.
set pager_index_lines=9
# do not ask for 'to' and 'subject' on reply
set fast_reply=yes
# do not save a copy of the reply
set record=""
send-hook '~t buildd ~s success' "set signature=_ pgp_autosign=yes indent_string=_ edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=chifflier@inl.fr"
send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes"
'Note': please note that I am no mutt expert, so the line can be
wrong, or there may be an easier way to do that. In both cases, I'm
interested if you have any suggestion.
Extracting changes file from build log
We use a shell script buildd-script.sh, containing a bit of sed magic to suppress all lines before and after the .changes file:
#!/bin/bash +e
cat $1 > ~/.buildd/orig
# 11 is the number of lines of your headers in default reply
egrep -v '^Fcc:' $1 | sed -e '11,/\.changes\:$/d' -e '/\/.*\.deb\:/,$d' > ~/.buildd/changes
cat ~/.buildd/changes > $1
Related Links
