Homemade Embedded BSD Systems(自制的嵌入式BSD系统)

News flash: computers are expensive, film at 11.

Many people use a dedicated computer as a firewall and NAT device. It's quite true that a high-end 486 running an open source UNIX-like operating system is perfectly capable of handling network connectivity in a SOHO environment, but computers have costs beyond the base hardware. These systems are bulky, and old hard drives frequently stagger on the verge of death. Plus, if you're in an enterprise environment, your manager does not want to hear statements like, "You know that desktop that was so abominably slow that you refused to let the receptionist endure it any longer? I put it back into service as the corporate firewall." Of course, you can always use your IT budget to purchase a nice new machine with a full warranty, but we all have things we'd rather buy than a computer that will sit 99% idle. All of this adds up to "expensive."

It's enough to make even the most rabid open source advocate consider purchasing an inexpensive consumer firewall appliance, losing a bit of flexibility in exchange for low-power silence.

Or, you can build a small firewall box yourself. While there are several alternatives for small, quiet systems, my favorites are the  models. Soekris boxes are roughly the size of a paperback book, fanless, low-power, and designed specifically for open source operating systems. They're sold either as a plain board or mounted in a case.

For this article, I used the Soekris NET4801 -- a high-end miniature computer with a 266MHz processor and 128MB RAM, case-mounted and with a DC power supply, all purchased direct from Soekris.

While the NET4801 can use a laptop hard drive, that's a bit much for a basic network gateway. I chose to use a compact flash card for the operating system, as firewalls generally don't have to write to the local hard drive.

To install OpenBSD on a compact flash card, you need a CF drive and an OpenBSD system with a CF reader. The good news is that the OpenBSD system doesn't have to be a server-grade system; it's perfectly acceptable to take that secretary's obsolete desktop and use it as your bootstrap platform. I used a very recent OpenBSD-current on my bootstrap station. Using this method installs the same version of OpenBSD on the Soekris as on the workstation, so your choice of OpenBSD version is important. (You can install other releases on the Soekris, but that's beyond the scope of this article.)

Hardware Information

Take the cover off of your Soekris, insert the CF card, and then hook a null modem cable to the console port and a serial port on your bootstrap station. On your bootstrap station, open up a tip(1) window at 19200 baud.

# tip -19200 tty00 connected

Plug in the Soekris, and you'll see the following in your tip(1) window.

comBIOS ver. 1.20 20030721 Copyright (C) 2000-2003 Soekris Engineering. net4801 0128 Mbyte Memory CPU Geode 266 Mhz Pri Mas Hitachi XXM2.3.0 LBA 993-16-63 500 Mbyte PXE-M00: BootManage UNDI, PXE-2.0 (build 082)

The LBA space gives the disk geometry that the Soekris sees. The Soekris BIOS believes this CF card has 993 cylinders, 16 tracks per cylinder, and 63 sectors per track. Take note of those numbers; you'll probably need them. These numbers are for my 512MB flash card, which turned out to be vastly too large. My complete install, with applications, used about 20MB, or roughly 4% of that disk. A 32MB or 64MB card would have been more than sufficient and far less expensive.

flashdist

Now get Chris Cappuccio's . flashdist is a set of tools to install a stripped-down OpenBSD on an attached hard drive device. In addition to a shell script that does most of the work, it includes some modified /etc scripts designed for embedded systems, kernel configurations tuned for Soekris hardware, and a text file listing every component to install to provide basic OpenBSD networking features.

I started by building a Soekris kernel using the NET4801 configuration in theflashdist distribution.

# cp NET4801 /sys/arch/i386/conf # config NET4801 # cd ../compile/NET4801/ # make depend && make

Once this finishes churning, copy the bsd file somewhere safe. I recommend renaming it to something like NET4801-bsd and moving the copy to your flashdistdirectory, just so you don't get confused and install it on your bootstrap station. (The Soekris kernel configurations don't bother with unnecessary hardware such as PC keyboards or video cards, so your bootstrap station would probably behave badly with that kernel.)

Now remove the flash card from the Soekris and put it in the bootstrap station's CF reader. Take note of which device the CF card shows up as, and then runflashdist.sh from the directory in which you extracted it.

# ./flashdist.sh -d sd0 flashsmall.txt ./NET4801-bsd /

Use the -d flag to tell flashdist which local device to install onto. In this example, the CF card is /dev/sd0, so we use -d sd0.

We also need a list of files to install on the system, provided in flashdist asflashsmall.txt. The third argument is the name of the kernel file to be installed on the disk, and finally we have the directory path from which flashdist should copy the files. In this case, we're copying the system files from the installed OpenBSD system. You could extract the distribution files elsewhere and use that path here, to install a different version of OpenBSD than is on your bootstrap station.

The flashdist.sh script will walk through an install, making its best guess at the disk geometry and prompting you for items such as the IP address of a and a root password for the complete system. Once complete, remove the CF card from your workstation, insert the CF in the Soekris, and power the Soekris back on. You should see a standard OpenBSD bootup in your tip(1) window, ending with a command prompt.

Congratulations! You have a working embedded OpenBSD system. It doesn't do much yet, but at least it boots. Now to make it useful.

flashdist Variables

You can set several variables in flashdist.sh to make the script stop asking you questions. You will probably be repeatedly re-installing on the CF card, so making the install as silent as possible is a good thing. Once you know the install works the way you want, you can even comment out the "hit enter to continue" sections and make the script run perfectly silently.

The cylinders, trackscylinder, and sectorstrack variables come straight from the Soekris BIOS screen. (While several other disk variables appear in the script,flashdist.sh can calculate them on the fly.) You can also define variables for the syslog server IP address, or a password in either encrypted or plain text format. Theetccopyfiles variable lists files to copy from the flashdist directory to the target disk's/etc directory. We'll make copious use of this later.

Also edit the rc file included in flashdist. Unlike the standard BSD-style rc system with lots of shell tricks and nifty variables, this rc script is just a basic shell script of commands to be run upon boot. There is no rc.conf or equivalent. The most obvious things to change are the ifconfig statements, to assign the appropriate IP addresses to each interface. Add any local startup scripts you have to this file.

Run flashdist.sh again once you have set all of the variables and configured the rcfile, and confirm that the bare OpenBSD install still works. This is a fairly complicated process, and I recommend you reinstall until you have a customized installation process that works perfectly every time.

Customizing the Installation

While flashdist includes a very minimal OpenBSD install, you can probably strip it down further. Edit the flashsmall.txt file to remove things you don't need. This particular system will be a firewall, so it doesn't need /usr/sbin/dhcpd, /sbin/dhclient, or /sbin/dhclient-script. There is no wireless card, so wicontrol(8) can go away. Do you need telnet, mg, or tcpdump? If it's not needed in your environment, remove it. Once you've gutted flashsmall.txt, reinstall again to be sure the system still works; if you trim too enthusiastically and without sufficient comprehension, you can easily create a non-working system. Be sure your base system works before adding programs!

Adding Programs

This particular system needs two additional programs, Snort with MySQL and libnet, and snmpd. The basic process is to build the program on the bootstrap server, identify the files absolutely necessary for the program to run, and add them to flashsmall.txt. Let's start with Snort.

Build and install Snort with the appropriate flavors on the bootstrap server, then get it running on your bootstrap system exactly as you want it to be run on your firewall. This will tell you which configuration files Snort requires. In my case, I installed the configuration under /etc/snort.

Add the list of required configuration files to flashsmall.txt. While this will handle the instructions the binary needs, what about the associated programs and shared libraries the program binary needs?

The port places a list of all its installed files in the +CONTENTS file under /var/db/pkg. This file also contains port instructions and comments, all of which are prefaced with the "at" symbol (@). Get rid of those first.

# grep -v '@' /var/db/pkg/snort-2.0.0p1-mysql-flexresp/+CONTENTS > \ $HOME/stripped-snort-contents

This creates a list of all files installed by the port. We won't be reading man pages on the firewall, so get rid of them. We won't install sample rules, so zap them. At the end, only one program file remains: /usr/local/bin/snort.

Even though the Snort port didn't install additional binaries, that doesn't mean that the compiled Snort binary doesn't expect a certain shared library infrastructure to be available. If you ask nicely with ldd(1), the program binary will tell you which libraries it expects.

# ldd /usr/local/bin/snort /usr/local/bin/snort: Start End Type Ref Name 00000000 00000000 exe 1 /usr/local/bin/snort 001d1000 201d5000 rlib 1 /usr/local/lib/libnet.so.0.0 088ca000 288d0000 rlib 2 /usr/lib/libz.so.2.0 0c47a000 2c487000 rlib 1 /usr/lib/libpcap.so.2.0 01a30000 21a37000 rlib 2 /usr/lib/libm.so.1.0 09fdf000 29fe8000 rlib 1 /usr/local/lib/libmysqlclient.so.10.0 055d5000 2560e000 rlib 1 /usr/lib/libc.so.30.1 08f9a000 08f9a000 rtld 1 /usr/libexec/ld.so

This really isn't surprising. Snort requires four standard libraries from /usr/lib, all of which are already appear in flashsmall.txt. We have to add in the shared libraries from /usr/local/lib, in this case libmysqlclient.so.10.0 and libnet.so.0.0. Add these files to flashsmall.txt.

By default, OpenBSD only checks for shared libraries under /usr/lib. You must tell the Soekris' /etc/rc to check for additional shared libraries under /usr/local/lib. Add the following lines to the flashdist rc file. I put them just after the ifconfig statements.

ldconfig ldconfig -m /usr/local/lib

With these additions to flashsmall.txt, re-run the installation and boot the Soekris.

In this particular case, I found that snort would not run. It complained about not being able to find libmysqlclient.so.10.0, even though ldconfig -r showed that the shared library was available!

After losing several clenched fists' worth of hair, I learned about libtool. By design, shared libraries are not supposed to call other shared libraries. libtool allows a shared library to break this restriction. The MySQL libraries use libtool to glue them together. We have the following files under /usr/local/lib/mysql:

libmysqlclient.so.10.0 libmysqlclient.a libmysqlclient.la

The .so.10.0 is a standard binary shared library. The .la file is a text file that configures libtool support for this library. Buried in this file we find the following:

# The name of the static archive. old_library='libmysqlclient.a' # Libraries that this one depends upon. dependency_libs=' -lz -lm'

So we need the libmysqlclient.a file. This library also depends on two others, libzand libm. Add these to flashsmall.txt, if they're not there. We also need the libtoolbinary and whatever libraries that requires. Some work with ldd(1) gives the following extra files:

./usr/local/bin/libtool ./usr/local/lib/libltdl.a ./usr/local/lib/libltdl.la ./usr/local/lib/libltdl.so.1.2

Also add /usr/local/lib/mysql to the ldconfig(8) paths in the Soekris' /etc/rc file.

ldconfig ldconfig -m /usr/local/lib ldconfig -m /usr/local/lib/mysql

Finally, add the snort startup to the very tail end of /etc/rc.

echo snort: starting daemon... /usr/local/bin/snort -c /etc/snort/snort.conf -Dz &

After all of this, following the same process for snmpd was trivial.

Conclusion

You will almost certainly have to do some troubleshooting on every application you install. I've found applications with files that look important but are irrelevant, while the developers have apparently disguised vital files as Frozen Bubble levels.

After all this, however, you will have a small silent machine that takes up a fraction of the power of a standard PC, with sufficient processing power for small network tasks. You can roll out hundreds of small systems while only having to back up the bootstrap station. By alternating flash cards, you can easily revert when an upgrade goes bad. These small systems make it possible to deploy complicated systems with a minimum of fuss.

Once you have hundreds of homemade embedded boxes, though, shuffling CF cards turns into an annoyance. Wouldn't it be easier to just skip the flash card and have the Soekris boxes create their own minds? We'll do that next time.