install razor&pyzor&dcc

Razor helps SpamAssassin detect spam messages. This is from its web site:

What is Vipul's Razor?

Vipul's Razor is a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content. User input is validated through reputation assignments based on consensus on report and revoke assertions which in turn is used for computing confidence values associated with individual signatures.

We start by installing the SDK, which loads some required Perl modules that Razor will use.

Change to the directory where Razor’s install is in:

cd /usr/local/src/

Untar Razor’s SDK:

tar xjvf razor-agents-sdk-#.##.tar.gz

After a few seconds, Razor’s SDK will be extracted.

Change to the directory it was extracted to:

cd razor-agents-sdk-#.##

Now, let’s compile and install the SDK:

perl Makefile.PL

make

make test

make install

After a minute or two the SDK will be installed.

Now let’s install Razor itself. The procedure is similar to the installation of the SDK:

We go one directory level up: 

cd ..

Untar Razor:

tar xjvf razor-agents-#.##.tar.gz

After a few seconds, Razor will be extracted.

Change to the directory it was extracted to:

cd razor-agents-#.##

Now, let’s compile and install Razor:

perl Makefile.PL

make

make test

make install

First, make sure that you allow TCP port 2703 outgoing from your server to the Internet:

razor-admin -d -create

If the command completed successfully, you should see:

razor-admin finished successfully.

This connected you to the Razor servers, and configured your server to use them.

Next, run this command:

razor-admin -register

This command creates a razor identity on your server. You should see:

Register successful.

or

razor-admin finished successfully.

If it gives you an error, run it again. Note that you may need to open some ports on your firewall for this command to complete successfully (TCP 2703).

We will now change some settings in the Razor configuration file:

vi /root/.razor/razor-agent.conf

Find the line that says:

debuglevel             = 3

And change the 3 to a 0.

In addition, add this line:

razorhome = /var/amavis/.razor

This tells Razor that its files are now going to be under /var/amavis/.

Since the amavis user is going to need to have access to Razor’s files, we tell Razor in its configuration file that its new home is under the /var/amavis/.razor/ directory. We will then move all of Razor’s files into their new home.

First, save and close the file you were editing above..

Now let’s move Razor to the /var/amavis/ directory:

cp -r /root/.razor/ /var/amavis/

Answer “y” to any overwrite question if you see one.

We now tell Razor to reconfigure itself to the new location:

razor-admin -d -create -home=/var/amavis/.razor/

If the command completed successfully, you should see:

razor-admin finished successfully.

Due to a change in the license agreement for Razor and DCC, SpamAssassin v3.1.x will not use both applications by default.

If the situation described in the license agreement allows you to run Razor, do the following to re-enable it:

Edit the file v310.pre:

vi /etc/mail/spamassassin/v310.pre

Find the line that looks like this:

#loadplugin Mail::SpamAssassin::Plugin::Razor2

And remove the #:

loadplugin Mail::SpamAssassin::Plugin::Razor2

Save and close the file.

That’s it!

Pyzor is a system of distributed servers that is very similar to Razor. We use it here in order to fine tune our server’s spam-fighting abilities. It uses UDP Port 24441 to communicate with its server(s). Pyzor requires Python v2.2.1 and above. You can issue the following command to see if you have it installed:

yum list python

If you don’t have it installed, install it with yum:

yum install python

Start the installation by extracting the archive:

cd /usr/local/src/

tar xjvf pyzor-#.#.#.tar.bz2

cd pyzor-#.#.#

Once in the directory, issue the commands:

python setup.py build

python setup.py install

The install may mess up some of the permissions. We can fix it by issuing this command:

chmod -R a+rX /usr/share/doc/pyzor /usr/lib/python#.#/site-packages/pyzor /usr/bin/pyzor /usr/bin/pyzord

The gdbm module is required for Pyzor’s operation as well. gdbm is a GNU implementation of the standard Unix dbm library. You can check if it’s installed by running:

python -c 'import gdbm' && echo 'gdbm found'

If you get a “gdbm found”, you’re all set.

If not, run:

yum install gdbm

We’ll now tell Pyzor to find the Pyzor server(s):

pyzor discover

(If you get some errors running it for the first time, run it again)

This creates a file named servers in the /root/.pyzor/ directory. The directory will be created if not already there. However, since Amavis will have to have access to this file as well, we’ll copy it to the .pyzor directory that we created under our amavis user directory tree:

cp /root/.pyzor/servers /var/amavis/.pyzor/

To be absolutely sure that the /var/amavis/ directory is really owned by the amavis user and group, it’s always good to issue this command:

chown -R amavis:amavis /var/amavis/

By running this command, you switch temporarily to the amavis user, and then ping the Pyzor server:

su amavis -c 'pyzor ping'

You should see the server’s IP and port, followed by (200, 'OK').

If you don’t get this, make sure that port 24441 UDP is open from your server to the world.

Much like Razor and Pyzor, DCC is a collaborative network of spam signatures. It can greatly enhance our server’s detection rate. DCC requires that port 6277 UDP will be open out. We start by switching to the source directory:

cd /usr/local/src/

Then we untar the package that we downloaded earlier, and change to the extracted directory:

tar xzvf dcc.tar.Z

cd dcc-#.#.#/

Now we’ll configure and install it:

./configure --with-uid=amavis

make

make install

This gets DCC installed. Now we need to tell it not to search any local DCC server, since we’re not going to run any servers on our machine. We will use the public DCC servers available on the Internet. If you have a huge site, and receive more than 100,000 messages per day, consider having a local DCC server. For now, run these commands:

cdcc "delete 127.0.0.1"
cdcc "delete 127.0.0.1 Greylist"

Note that you can add more DCC servers to your list. Visit DCC’s webpage for a current list.

There’s one last thing that we need to do. DCC comes with a script that prunes some of DCC’s log file. The installation instructions for DCC tell us to make run this script daily, or even more frequently. You will soon see that we’re going to have several scripts that are running at specific times. In Linux, you use the cron daemon to schedule jobs to run at a specific time. We will first create a symbolic link from /usr/bin/cron-dccd to the real script, located at /var/dcc/libexec/cron-dccd:

ln -s /var/dcc/libexec/cron-dccd /usr/bin/cron-dccd

Now, we’ll tell cron to run it every morning, at 5:00 am:

crontab -e

Add this line:

0 5 * * * /usr/bin/cron-dccd

Save and close the file. You should see:

crontab: installing new crontab

Don’t worry about the meaning of the numbers and asterisks. We’ll go over all of this later, when we set-up some more cron jobs.

This concludes the configuration of DCC as a stand-alone client. SpamAssassin will run DCC every time a message arrives, and this can be pretty time-consuming. If you have a server with a lot of RAM (512MB+), a better way will be to run DCC as a daemon. This way it’ll be constantly running, and the interaction between it and SpamAssassin will be more efficient. Understand that this is not a required step. SpamAssassin will run just fine without a “daemonized” DCC. The daemon version of DCC is named dccifd.

We start by editing the dcc_conf file:

vi /var/dcc/dcc_conf

Find the line that reads:

DCCIFD_ENABLE=off

(note: not the DCCD_ENABLE=OFF line)

and change it to:

DCCIFD_ENABLE=on

Save and close the file.

We’ll now make this daemon run every time the system starts:

cp /var/dcc/libexec/rcDCC /etc/init.d/DCC

chkconfig --add DCC

Let’s make sure that the amavis user and group have full access to the DCC directory:

chown -R amavis:amavis /var/dcc

And finally, we’ll edit SpamAssassin’s configuration file, and tell it where to find the path to DCC:

vi /etc/mail/spamassassin/local.cf

Add this to the end of the file:

dcc_home /var/dcc

Save and close the file.

Due to a change in the license agreement for Razor and DCC, SpamAssassin v3.1.x will not use both applications by default.

If the situation described in the license agreement allows you to run DCC, do the following to re-enable it:

Edit the file v310.pre:

vi /etc/mail/spamassassin/v310.pre

Find the line that looks like this:

#loadplugin Mail::SpamAssassin::Plugin::DCC

And remove the #:

loadplugin Mail::SpamAssassin::Plugin::DCC

Save and close the file.

Next, we’ll copy some files from /root/.spamassassin/ to /var/amavis/.spamassassin/.

What we’re trying to accomplish is to make the amavis and root users “see” that SpamAssassin’s configuration is under /var/amavis/.spamassassin/ (root will assume by default that it’s under /root/.spamassassin/).

Run:

spamassassin --lint -D

This will show you some debug information, and since you’re running it as root, it’ll create the .spamassassin directory under /root/ (it creates such a directory for every user running spamassassin). The --lint command checks the rule base for errors. We’ll now copy the settings that were created for root, into the home directory of amavis:

When running the next command, please answer “n” to the possible “overwrite?” question:

cp -ir /root/.spamassassin /var/amavis

We’ll now make the root and amavis users read from the same preferences file, by creating a symbolic link:

cd /root/.spamassassin/

rm -f user_prefs 

ln -s /var/amavis/.spamassassin/user_prefs user_prefs

What we did in the above steps was basically creating a link, so that every time root is looking for the user_prefs file under /root/.spamassassin, it’ll actually be the file /var/amavis/.spamassassin/user_prefs. This creates an interesting situation: if you find yourself working as root, and you run commands such as spamassassin --lint -D, or others, such as sa-learn, root may reclaim ownership of them. You may find yourself with a file named user_prefs that’s under the /var/amavis/.spamassassin/ directory, but Amavis won’t be able to access it (it’s owned by root).

To solve this, get used to this command (you don’t have to run it now, we’ll run it soon, below):

chown -R amavis:amavis /var/amavis

In English: give the amavis user and group full access to the directory and sub directories under /var/amavis. Feel free to run this whenever!

Next, we’ll edit SpamAssassin’s configuration file, and modify some files’ locations:

vi /etc/mail/spamassassin/local.cf

Add the following lines to the file:

razor_config /var/amavis/.razor/razor-agent.conf
bayes_path /var/amavis/.spamassassin/bayes
auto_whitelist_path /var/amavis/.spamassassin/auto-whitelist

lock_method flock

Lines 1 and 3 above refer to already-existing files that are in these directories from previous steps that we did. Line 4 tells SpamAssassin that the Bayes database files are located here, on this machine’s hard drive, and not somewhere else (such as an NFS server). 

If you want to get a better report in the mail header, about the tests that SpamAssassin did along with the actual scores that they each contribute to the total score, ad this line:

add_header all Status _YESNO_, score=_SCORE_ required=_REQD_ tests=_TESTSSCORES(,)_ _DCCR_ _PYZOR_ _RBL_ autolearn=_AUTOLEARN_ version=_VERSION_

Save and exit the file.

We now need to create the Bayes database, referenced in the second line that we added to local.cf:

sa-learn --sync

This will initialize the Bayes database.

The last step that we have to do is give Amavisd full ownership of all the files under /var/amavis/. Issue this command:

chown -R amavis:amavis /var/amavis/

This is a good time to stop and test Postfix, Amavis, SpamAssassin, Pyzor, Razor, and DCC. We’re about to start the installation of our Anti Virus software, so before we do, let’s test that everything is OK thus far!

Reboot your new server:

reboot

After rebooting, Postfix and Amavisd should be starting automatically. If they’re not, you probably made some mistakes above. Go to the appropriate sections and fix the problems.

You can check the process list (ps aux) to see that Postfix and Amavisd are running, but you can also do the following:

amavisd reload

If amavisd is not running, you’ll get an error:

No PID file /var/amavis/amavisd.pid, can't reload the process

Make sure that you correctly did the steps for making it load automatically at boot-up.

If it was running, it will kill itself, and then reload itself:

daemon terminated, waiting for the dust to settle...

becoming a new daemon...

To start Amavisd, issue this command:

amavisd start

Note that Amavisd only reports errors. If everything is well, it’ll simply show you the command prompt after it loads.

With Postfix the situation is similar. First, issue a reload:

postfix reload

If Postfix is not running, you’ll get this error:

postfix/postfix-script: fatal: the Postfix mail system is not running

Make sure that you correctly did the steps for making it load automatically at boot-up.

If Postfix was running, it’ll reload itself really fast:

postfix/postfix-script: refreshing the Postfix mail system

To start Postfix, issue the following command:

postfix start

You should see:

postfix/postfix-script: starting the Postfix mail system

You may see the following when starting Postfix:

postfix/postfix-script: warning: /var/spool/postfix/etc/passwd and /etc/passwd differ

If you do, issue this command:

LINUX2

Stopping Postfix is done by issuing postfix stop. Stopping Amavisd is done by issuing amavisd stop. Amavisd should always be started before Postfix, since Postfix depends on Amavisd.

Now we’ll actually test Amavisd with SpamAssassin, and see that everything is OK. Type these commands:

amavisd stop

amavisd debug-sa

This will throw tons of text to the screen, but it should then stop, and give no errors.

(By the way: if you see nothing on the screen, you may have some permissions issues. Run chown -R amavis:amavis /var/amavis/)

Scroll up and down (if you use an ssh client), and look for lines like:

debug: using "/var/amavis/.spamassassin/user_prefs" for user prefs file

This line tells you that the user_prefs file was read from /var/amavis/.spamassassin/-exactly what we wanted!

Also look for this:

dbg: bayes: xxxxx tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks

dbg: bayes: xxxxx tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen

This tells us that the Bayes database is exactly where we wanted it.

A good Razor configuration should show you this:

dbg: razor2: razor2 is available, version 2.77

A good Pyzor configuration should show you:

dbg: plugin: loading Mail::SpamAssassin::Plugin::Pyzor from @INC

dbg: pyzor: network tests on, attempting Pyzor

And finally, a good DCC configuration should show you:

dbg: plugin: loading Mail::SpamAssassin::Plugin::DCC from @INC

dbg: dcc: network tests on, registering DCC

You can end the debug session with CTRL+C.

Lets start Amavisd again:

amavisd start

Once both Postfix and Amavis are up, the easiest way to test is to do the firewall changes that I suggested, and test from Gmail. You can also use Outlook Express as discussed before. To simulate a spam message, you can send the GTUBE message (Generic Test for Unsolicited Bulk Email). Spam scanners such as SpamAssassin are designed to see this message as spam, and report it as such.

Simply e-mail yourself the following in the body of the message from Gmail:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X 

You can read more about it, and get a text file with GTUBE in it here:

Your message should be delivered to the e-mail address that you sent it to.

It should have the spam tags (you will need to view the message source to see them), and it should say “[Possible Spam] “ (unless you chose something else) in the subject line, before your original subject.

If it didn’t work, then you know that the problem is probably with Amavisd, since we know that Postfix works. Look in the log (less /var/log/maillog), and try to determine where the problem is. Also check that you did all the steps in the configuration and setup of Amavisd.

Another problem that you may have is that Amavisd doesn’t have access rights to all the files and directories under /var/amavis. Issue the following commands to fix it:

chown -R amavis:amavis /var/amavis/
chmod -R 750 /var/amavis/