Chinaunix首页 | 论坛 | 博客
  • 博客访问: 1155087
  • 博文数量: 290
  • 博客积分: 3468
  • 博客等级: 中校
  • 技术积分: 3461
  • 用 户 组: 普通用户
  • 注册时间: 2010-12-28 22:21
文章分类

全部博文(290)

文章存档

2016年(13)

2015年(3)

2014年(42)

2013年(67)

2012年(90)

2011年(75)

分类: LINUX

2012-06-25 22:05:47

DebianBuildd

This documentation is written to explain how to setup a build daemon to recompile the whole Debian unstable archive.

Note: this howto has been written for etch - things may have changed since.

How it works

Buildd setup works with 3 packages:

  • wanna-build
  • sbuild
  • buildd

wanna-build is used to store informations about packages in a database. sbuild is used to actually build packages into a chroot. buildd is used to fetch packages to build via wanna-build and run sbuild to build them.

Setting up the box Install Debian

Install debian. Version installed here is etch.

Install packages

The following packages are needed:

build-essential devscripts bzip2 fakeroot patch subversion Fetch the code and install packages

Currently, everything is not packaged. That's a shame.

sbuild is in the archive, but wanna-build and buildd are not. You fetch the code via Subversion from http://svn.cyberhqz.com/svn/wanna-build

Here is how I build sbuild and wanna-build:

$ svn co http://svn.cyberhqz.com/svn/wanna-build $ cd wanna-build $ vi debian/control --> Remove versionned dependency on sbuild, because we can use the version from unstable $ debuild -uc -us

Then, install packages:

# aptitude install sbuild debootstrap cron-apt libmldbm-perl dupload quinn-diff # dpkg -i wanna-build_2006.01.08_sparc.deb buildd_2006.01.08_sparc.deb

You can use the sbuild version from sid or from the svn repository. Currently I use both.

Create system environment

You can use /usr/share/doc/buildd/examples/buildd-setup-system.gz directly (zcat | sh), but I prefer to do everything by myself.

# as root adduser --system --shell /bin/sh --uid 60000 --gecos 'Build Daemon' --group --disabled-password buildd mkdir -p /var/lib/debbuild/srcdep-lock chown -R buildd:buildd /var/lib/debbuild chmod -R 2775 /var/lib/debbuild cd ~buildd zcat /usr/share/doc/buildd/examples/buildd.conf.gz > buildd.conf zcat /usr/share/doc/sbuild/examples/sbuildrc.gz > .sbuildrc chown buildd:buildd buildd.conf .sbuildrc mkdir -p logs build old-logs upload bin stats/graphs mqueue echo "|/usr/bin/buildd-mail-wrapper" > .forward chown buildd:buildd logs build old-logs upload bin stats .forward mqueue Setup chroot

Now it's time to create our chroot! The buildd-make-chroot command is not available in the sbuild package from sid, so use the one you grabbed from svn, be smart!

# cd ~buildd # ~/wanna-build/buildd-make-chroot buildd etch build/chroot-etch http://ftp2.fr.debian.org/debian

Note: actually, this command fails because of dependencies, with the following error:

dpkg: dependency problems prevent removal of makedev: gnupg depends on makedev (>= 2.3.1-13) | devfsd | hurd; however: Package makedev is to be removed. Package devfsd is not installed. Package hurd is not installed.

The solution is to keep makedev (which is removed by buildd-make-chroot). Apply the following patch_:

--- buildd-make-chroot.orig 2007-05-11 12:31:44.000000000 +0000 +++ buildd-make-chroot 2007-05-11 12:31:57.000000000 +0000 @@ -74,7 +74,7 @@ chmod -R 02775 $TARGET/var/debbuild echo include /etc/ld.so.conf.d/*.conf >> $TARGET/etc/ld.so.conf (cd $TARGET/dev ; ./MAKEDEV fd) -sudo chroot $TARGET dpkg -P debconf-i18n debconf liblocale-gettext-perl libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl procps makedev +sudo chroot $TARGET dpkg -P debconf-i18n debconf liblocale-gettext-perl libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl procps echo "Successfully setup chroot for a buildd" echo Possible commands to append to fstab: echo echo $SUITE-proc $TARGET/proc proc defaults 0 0 \>\> /etc/fstab

You can finish the install:

# as root mkdir -p ~buildd/build/chroot-etch/var/lib/sbuild/srcdep-lock chown -R buildd.buildd ~buildd/build/chroot-etch/var/lib/sbuild/ mkdir ~buildd/build/chroot-etch/home/buildd chown buildd.buildd ~buildd/build/chroot-etch/home/buildd cd ~buildd/build && ln -s chroot-etch chroot-stable

After, edit the /etc/apt/sources.list in your chroot:

# as root vi ~buildd/build/chroot-etch/etc/apt/sources.list chroot build/chroot-etch /bin/sh apt-get update exit

You should mount proc into your chroot otherwise some build will failed. Add the following line to your /etc/fstab:

etch-proc /home/buildd/build/chroot-etch/proc proc defaults 0 0 Give root to your buildd

sbuild should be able to use schroot, but, in fact, that's terribly b0rked right now. So we will use the old sudo method.

# visudo Add: Defaults:buildd env_keep+="APT_CONFIG DEBIAN_FRONTEND SHELL" buildd ALL= NOPASSWD: ALL Setup sbuild

Change your user to buildd (sudo -H -s -u buildd), and configure sbuild.

# vi .sbuildrc # egrep -v '^(#|$)' .sbuildrc $build_dir="$HOME/build"; $log_dir = "$HOME/logs"; $mailto = "pollux\@inl.fr"; $maintainer_name='Pierre Chifflier '; $purge_build_directory="always"; $stalled_pkg_timeout = 90; 1; Setup buildd

Change your user to buildd and edit ~buildd/buildd.conf:

$ egrep -v '^(#|$)' buildd.conf require "/etc/buildd.conf.global"; @take_from_dists = qw(stable testing unstable); @no_auto_build = qw(boot-floppies); @weak_no_auto_build = qw(); $autoclean_interval = 86400; $admin_mail = "buildd admin "; $build_log_keep = 1; $statistics_mail = 'pollux\@inl.fr'; $dupload_to = "anonymous-inl"; $dupload_to_security = "anonymous-inl"; $log_queued_messages = 0; $wanna_build_user = "buildd_arch-hostname"; $wanna_build_dbbase = "build-db"; $sshcmd = ""; $sshsocket = ""; 1; Configure dupload

The buildd user should have a working dupload installation (.dupload.conf):

package config; $default_host = 'anonymous-inl'; $cfg{'anonymous-inl'} = { fqdn => "xxx.xxx.xxx.xxx", method => "scpb", login => "pollux", incoming => "/var/www/debian/mini-dinstall/incoming/", dinstall_runs => 1, }; Add cron jobs

Some commands should be added to the crontab, to ensure that buildd is running, and to upload packages:

$ crontab -l # m h dom mon dow command MAILTO=pollux@inl.fr 30 0,6,12,18 * * * /usr/bin/buildd-uploader 20 0,6,12,18 * * * /usr/bin/buildd-watcher Setup wanna-build

Copy wanna-build local configuration:

# zcat /usr/share/doc/wanna-build/examples/wanna-build.conf.local.gz > /etc/wanna-build.conf.local # chown buildd /etc/wanna-build.conf.local
Then you can create an empty wanna-build database:
# # as root: # install -m 755 -d -obuildd -gbuildd /var/debbuild # # as user buildd: $ wanna-build -d stable --create-db --clean-db

Now you will need to fill up the database with all packages. That's easy to do with a little script (thanks to Julien Louis).

#!/bin/sh cd /tmp wget -q http://ftp.fr.debian.org/debian/dists/unstable/main/source/Sources.gz -O Sources.main.gz wget -q http://ftp.fr.debian.org/debian/dists/unstable/contrib/source/Sources.gz -O Sources.contrib.gz if [ -e Sources.main.gz ] && [ -e Sources.contrib.gz ]; then zcat Sources.main.gz Sources.contrib.gz > Sources fi touch Packages quinn-diff -A @dpkg-architecture -qDEB_HOST_ARCH@ 2>/dev/null | wanna-build --merge-quinn -d stable rm -f Sources Packages
You can run this script once a day to update the database. Building arch=all packages

The default configuration of buildd is not to build architecture-independant packages. To force buildd to build arch=all packages, apply the following patch to '/usr/bin/buildd':

--- buildd.OLD 2007-05-14 11:59:20.000000000 +0000 +++ buildd 2007-05-14 11:59:47.000000000 +0000 @@ -466,7 +466,8 @@ my @sbuild_args = ( 'nice', '-n', "$conf::nice_level", 'sbuild', '--batch', "--stats-dir=$main::HOME/stats", - "--dist=$dist" ); + "--dist=$dist", + "--arch-all" ); my $sbuild_gb = '--auto-give-back'; if ($conf::sshcmd) { $sbuild_gb .= "="; Useful commands List packages which needs to be build $ /usr/bin/wb-short/list-needs-build -d stable unknown/glpi_0.68.3-3 [unknown:uncompiled] Total 1 package(s) Give back a package for build (aka retry) $ /usr/bin/wb-short/give-back-build -d stable -U buildd_arch-hostname glpi_0.68.3-3 Remove package state from database $ /usr/bin/wb-short/forget-build -d stable -U buildd_arch ocsinventory-server_1.01-2 binNMU $ cd ~buildd/build $ sbuild --make-binNMU="Rebuild for erase unwelcomed dependency" --binNMU=1 -d unstable -v hello_2.1.1-4
  • "--make-binNMU" defines message to add changelog.
  • "--binNMU" defines minor revision. If you set 1, hello_2.1.1-4.0.1 will be built.
Mutt script for buildd auto-reply

When a build is complete (and successful), a mail is sent (see ~buildd/.sbuildrc). To complete the build and upload the package, the .changes file must be extracted from the build log, must be signed and sent back to the buildd. As this is quite boring, here is some scripts to make this part easier.

Folder-hook

First, we will apply the script only to mails stored in the =buildd folder. So we use a folder-hook to associate this folder with a specific script:

folder-hook =buildd source ~/.mutt/defaults.buildd Mutt preferences and send-hook

This is the big part: we set some preferences for how to display the folder, and we set a send-hook to make an almost automatic reply, passing the build log to a shell script to extract the .changes file, and then signing it. Practically, all what you have to do is to press 'r' (reply) on a mail, then 'y' to send it (you will have to enter the passphrase if not in memory).

# -*- muttrc -*- # Use the global defaults first. source ~/.mutt/defaults.global # This is the index format for mailing lists. set index_format="%4C %Z %{%b %d} %-15.15n (%4l) %s" # Sort on threads set sort=threads # Give mailing lists a more "slrn" like feel. set pager_index_lines=9 # do not ask for 'to' and 'subject' on reply set fast_reply=yes # do not save a copy of the reply set record="" send-hook '~t buildd ~s success' "set signature=_ pgp_autosign=yes indent_string=_ edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=chifflier@inl.fr" send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes"

'Note': please note that I am no mutt expert, so the line can be wrong, or there may be an easier way to do that. In both cases, I'm interested if you have any suggestion.

Extracting changes file from build log

We use a shell script buildd-script.sh, containing a bit of sed magic to suppress all lines before and after the .changes file:

#!/bin/bash +e cat $1 > ~/.buildd/orig # 11 is the number of lines of your headers in default reply egrep -v '^Fcc:' $1 | sed -e '11,/\.changes\:$/d' -e '/\/.*\.deb\:/,$d' > ~/.buildd/changes cat ~/.buildd/changes > $1 Related Links



个人记录:
# apt-get install wanna-build  buildd   sbuild  quinn-diff
# mkdir -p /root/upload-security
# mkdir -p /root/upload
# mkdir -p /root/build
# touch build.sh

点击(此处)折叠或打开

  1. #!/bin/sh
  2. user=$LOGNAME
  3. dist=unstable
  4. # default to host architecture. you can also use i386 on amd64 or vice versa
  5. arch=$(dpkg --print-architecture)
  6. mirror=http://cdn.debian.net/debian

  7. sbuild-adduser $user
  8. sbuild-update --keygen

  9. # optional, but recommended if not using a local mirror:
  10. echo /var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0 >>/etc/schroot/sbuild/fstab

  11. mkdir -p /srv/chroot
  12. sbuild-createchroot --keyring=/etc/apt/trusted.gpg --arch=$arch $dist /srv/chroot/$dist-$arch-sbuild $mirror
# sh -x build.sh

I: Configuring dpkg-dev...
I: Configuring build-essential...
I: Base system installed successfully.
I: Configured /etc/hosts:
   ┌────────────────────────────────────────────────────────────────────────
   │127.0.0.1 debian-loongson localhost
   └────────────────────────────────────────────────────────────────────────
I: Configured /usr/sbin/policy-rc.d:
   ┌─────────────────────────────────────────────────────────────────────   ┌────────────────────────────────────────────────────────────────────────
   │[unstable-i386-sbuild]
   │type=directory
   │description=Debian unstable/i386 autobuilder
   │directory=/srv/chroot/unstable-i386-sbuild
   │groups=root,sbuild
   │root-groups=root,sbuild
   └────────────────────────────────────────────────────────────────────────
I: Please rename and modify this file as required.
mkdir /etc/sbuild/chroot
I: sudo chroot configuration linked as /etc/sbuild/chroot/unstable-i386-sbuild.
I: Setting reference package list.
I: Updating chroot.
命中 http://cdn.debian.net unstable Release.gpg
获取:1 http://cdn.debian.net/debian/ unstable/main Translation-en [4,008 kB]
获取:2 http://cdn.debian.net/debian/ unstable/main Translation-zh [1,528 B]  
获取:3 http://cdn.debian.net/debian/ unstable/main Translation-zh_CN [134 kB]
命中 http://cdn.debian.net unstable Release                                   
获取:4 http://cdn.debian.net unstable/main Sources [6,333 kB]                
获取:5 http://cdn.debian.net unstable/main i386 Packages/DiffIndex [7,876 B] 
下载 10.5 MB,耗时 2分 20秒 (74.8 kB/s)                                       
正在读取软件包列表... 完成
正在读取软件包列表... 完成
正在分析软件包的依赖关系树... 完成
正在对升级进行计算... 完成
升级了 0 个软件包,新安装了 0 个软件包,要卸载 0 个软件包,有 0 个软件包未被升级。
I: Successfully set up unstable chroot.
I: Run "sbuild-adduser" to add new sbuild users.


参考链接:
http://www.debian.org/devel/buildd/
https://buildd.debian.org/docs/buildd-setup.txt
http://kmuto.jp/open.cgi?buildd
https://www.wzdftpd.net/redmine/projects/general/wiki/DebianBuildd
http://blog.openrays.org/blog.php?do=showone&tid=64
http://wiki.debian.org/sbuild


附原文:
Buildd是自动打包整套系统的一个通用名字,具体的包括下面几个软件组成:

* wanna-build (http://svn.cyberhqz.com/svn/wanna-build/)
通过一个记录软件包列表及状态的数据库来帮助对软件包的打包过程。每一个架构有一个数据库,用来存储软件包的状态,版本和一些其他信息。

* buildd (http://svn.cyberhqz.com/svn/wanna-build/)
一个周期性的检查由wanna-build所维护数据库 的后台进程,并且调用sbuild来进行打包。buildd记录打包的成功或者失败的情况,当管理员对打包日志确认之后,它将上传打好的包。

* sbuild (http://packages.debian.org/sbuild)
sbuild负责在独立的chroot环境中进行实际的打包工作。它主要使用一些标准的Debian工具来完成这个工作,而且它也会关注源的依赖关系和一些其他小问题。

* quinn-diff (http://packages.debian.org/quinn-diff)
它向wanna-build的数据库提供新的软件包。它会对比可用的软件包不同平台的版本,并输出差别。(对Source和Package文件进行对比)。更多的关于quinn-diff的信息在(http://buildd.debian.org/quinn-diff/)。

* andrea
自动生成一些源依赖关系,并且将些数据和手动添加的依赖关系数据进行合并。


上面所有这些部分通力合作使得自动打包系统正常运转。

阅读(2775) | 评论(1) | 转发(1) |
给主人留下些什么吧!~~

fbwfbi2013-11-06 10:17:46

谢谢楼主,正研究这个方面,不知道楼主能再给点中文资料不?