View source for QMT Failover replication Setup

==QMT Failover replication Setup==
Craig Smith - 26th October 2006 - craig@doc-net.com

Thanks to Jake for taking the time to review this for me before posting.  It always helps to have a sounding board and Jake was kind enough to be that board for me. 


This page gives you a procedure to configure a backup qmt server that will be available for failover in the event of primary server failure.  The backup server will only ever be 10 minute out from the primary.(depending on cronjob timing)

Please note initial replication (the first run) will take some time, so schedule this for off peak hours.  Once the first run has finished and unison has a db of what it is working with subsequent runs are pretty quick. So enable the cron job settings at a time that you can manage the traffic for initial replication.

Also this setup is based on 2 servers where the port used is internal and not visible publically.  If you cannot do this on a private network, then read up on using ssh for replication as this is not a secure transport and should not be used on open networks.

This was setup and tested on Fedora core 5 on both servers, and it works without any hiccups.

The details are pretty much cut and paste.  

I would test this installation on a similar setup first as I only have my system to compare with, but given the ease of installation and use of unison, I don't forsee any major problems.

This setup assumes that QMT is installed and configured on both servers.  The backup servers version is to match the primary's.

I have also setup a script that will test your server availability by pinging it.  This is posted at the bottom of this page, but still needs work.  The mail commands are purely for testing. If the primary server really were off, the emails wouldn't go out until you switch to the backup which doesn't help.  I think replacing these mail commands to sms text messages would certainly help.  A quick and easy monitoring system, which can be added to cron to run every x minutes.

This is a first draft and also my fist proper linux/unix scripts so please forgive any errors and let me know if you have any problems. If the primary server fails and the switch is made to backup in a short time, clients will probably not even notice the downtime.

*Unison is a great replication utility.  http://www.cis.upenn.edu/~bcpierce/unison/

*We use the command line version available from http://www.cis.upenn.edu/~bcpierce/unison/
*This is based on unison-2.13.16-linux-text.bz2 which is the current stable release as of 10/10/2006

*To modify files/folders either replicated or ignored edit the unison profile (qmail.prf).

*This setup is for servers that are in a lan environment with the unison port not visible to the public world. Therefore the security overhead of ssh is not needed.  However unison can be run with ssh if needed.

*To configure unison and ssh see http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html#remote.  The procedure stays largely the same, but the socket server is not needed.

== Comments and Notes ==

Please feel free to leave comments about your experience with this procedure.

Craig Smith - 26th October 2006
On our setup of 2 x fedora core 5 boxes, whenever I make a change on primary I check on backup to make sure it has replicated, and so far everytime it works a charm. I've not run into anything strange.

Also I'm not sure how people prefer the logs.  I had it initially set to log each run, but that was changed to keep logging for 20MB which is roughly 24 hours worth of logging.  Times are logged so finding a specific run should be fairly easy.  However it's quite easy for me to change back to log each run in a seperate file.  I will go with preferance really, please let me know.

;'''* Note about non FC versions'''.

One of the variables in the script obtains file size by cutting the relevant field from an ls listing.  This field it turns out is not the same as on Fedora, so if your script runs into errors, the problem lies here more than likely.  To fix it run this (ls -l  $LOG|cut -d ' ' -f 5)portion in the command line and change the f no. until it correctly displays the size field and then change the number in the script and this should fix any script errors.

It is the following variable.

 size=`ls -l  $LOG|cut -d ' ' -f 5`

I believe for Centos it is f6 so that variable would be changed to 
 size=`ls -l  $LOG|cut -d ' ' -f 6`

== Primary Server (Client) setup ==

*Primary server is the main server being replicated. This has the unison program and the client side script to replicate data based on the profile. 

*All scripts assume a default path of /unison.

*The script calls unison to run as follows
 /unison/unison -force / -batch qmail
*-force / is very important, as it specifies this root as the primary and will default all conflicts to this root. NB IF THIS IS NOT INCLUDED ANY CHANGES MADE ON THE BACKUP SERVER WILL REPLICATE TO PRIMARY. 
*-batch tells unison to run without promts and qmail refers to the qmail.prf file. (see below)

;'''To configure unison on the client side take the following steps as root/su

 mkdir /unison
 cd /unison
 wget https://svn.cis.upenn.edu/svnroot/unison-contributed-binaries/linux/unison-2.13.16-linux-text.bz2
 bzip2 -d unison-2.13.16-linux-text.bz2
 mv unison-2.13.16-linux-text unison
 chmod 755 unison
 ./unison                   *This will create the initial /root/.unison database folder
 vi qmail-replicatec        *Once in the editor, copy and paste the qmail-replicatec script from below
 :wq
 vi /root/.unison/qmail.prf *Once in the editor, copy and paste the qmail.prf details from below
 :wq''

'''Client script gets added to cron to run every 10 minutes.
 crontab -e
 */10 * * * * /unison/qmail-replicatec

'''*Below is the full script for the client side, script name qmail-replicatec

 #!/bin/sh
 
 #Version 1.0 - Oct 11th 2006
 #Script created by Craig Smith - craig@doc-net.com
 
 #This script is the client (Primary  mail server) side script to replicate based on the
 #qmail.prf file located under /root/.unison.  The Mysql variables and dump were taken from 
 #qmailbackup script written initially by Nate Davis. 
 
 #To add to or change files paths edit /root/.unison/qmail.prf accordingly.
 
 #This script assumes unison and scripts are placed in /unison
 
 #To keep logs from building too excessively as this script will run fairly regularly
 #they will be moved to the /unison/oldlogs folder after 10.  The oldlogs folder will
 #be emptied after each group of 10.  You can increase this if needed.  See comment below.
 #With default settings the last 20 runs are available. Add email address below for notifications of log rotation.
 
 #starting with mysql so the dumped file can replicate
 
 #Checking for mysql.dump and oldlogs folder
 
 folder1=/unison/mysql.dump
 folder2=/unison/oldlogs
 lock=/unison/replicate.lock
 email=(email address)

 if [ ! -d $folder1 ]; then
 mkdir -p $folder1
 fi
 
 if [ ! -d $folder2 ];then
 mkdir -p $folder2 
 fi
 
 if [ ! -f $lock ]; then
 touch $lock
 elif [ -f $lock ]; then
 echo "Lock file still in place, investigate."|mail -s "unison script problems  please investiagate server" $email
 exit 0
 fi

 # MYSQL variables
 mysqlfile="/home/vpopmail/etc/vpopmail.mysql";
 mysqlhost=`cut -d\| -f1 < $mysqlfile`;
 mysqlport=`cut -d\| -f2 < $mysqlfile`;
 mysqluser=`cut -d\| -f3 < $mysqlfile`;
 mysqlpswd=`cut -d\| -f4 < $mysqlfile`;
 mysqldb=`cut -d\| -f5 < $mysqlfile`;
 if [ $mysqlport == "0" ]; then
 mysqlport=""
 else
 mysqlport="-P$mysqlport"
 fi
 
 #echo "Backing up MYSQL Data"
 mysqldump -u$mysqluser -p$mysqlpswd -h$mysqlhost $mysqlport $mysqldb > /unison/mysql.dump/vpopmail
 
 #script for log clearup
 #Only last 10 log files are kept in /unison/oldlogs. 
 #The full log will be replaced every 20MB and moved to /unison/oldlogs.
 
 LOG=/unison/unisonlog.full
 LOGCOUNT=`ls $folder2 |wc -l`
 size=`ls -l  $LOG|cut -d ' ' -f 5`
 LOGSAVE=/unison/oldlogs/unisonlog.`date +%Y%m%d%H%M`
 
 #echo $size 
 
 if [ $size -gt 20000000 ];then
 echo "this is bigger than 20MB, moving">>$LOG
 mv $LOG $LOGSAVE
 echo "" >$LOG
 fi
 
 #LOG COMMENT : if you want to increase the saved logs change the 10 below.
 
 if [ $LOGCOUNT -gt 15 ];
 then
 echo "more than 15 logs exist, moving to folder /unison/oldlogs/previous"
   #keep previous batch in previous
   rm -f /unison/oldlogs/previous/*
     mv /unison/oldlogs/* /unison/oldlogs/previous
    clear
     echo "`date` : log files have been moved to oldlogs" >>$LOG
 echo "Unison Log files moved, please check folders on server"|mail -s"Unison Logile Rotation : `date`" $email
 fi
 
 echo "" >>$LOG
 echo "`date` ***STARTING REPLICATION RUN**** " >>$LOG
 
  
 #Begin unison replication using /root/.unison/qmail.prf file for folder locations
 #Log file location and format
 
 /unison/unison -force / -batch qmail >> $LOG  2>&1
 echo "Deleting lock file" >> $LOG  2>&1
 rm -f $lock
 echo "Done `date`"  >> $LOG  2>&1



''';*Below is the contents of the qmail.prf file.  This file points to the backup server ip and port and specifies the folders to be replicated.  It will only replicate what is specified by Path.  If you don't include path, then it will replicate the whole root.  If you want to add details e.g. squirrellmail prefs etc, then add path = path to folder.  note no trailing / add the correct ip address and port no when you paste this into the qmail.prf file. eg.
root = socket://10.0.0.1:1234//

 #Root and path setup for qmail backup
 root = /
 root = socket://xxx.xxx.xxx.xxx:port no.//
 
 path = home/vpopmail
 path = var/qmail/control
 path = var/qmail/users
 path = etc/mail/spamassassin
 path = etc/tcprules.d
 path = unison/mysql.dump
 
 ignore = Name *.lock

==Backup Server (Server) Setup==

*The backup server runs the unison program with the -sockets option.  This creates a listening socket that the client will try and connect to.  There is also a script that will deal with qmail cleanup and configuration as unison does not replicate ownerships.  

'''*To configure Unison on the Server side take the following steps.  The same unison application as above is used.

 mkdir /unison
 cd /unison
 wget https://svn.cis.upenn.edu/svnroot/unison-contributed-binaries/linux/unison-2.13.16-linux-text.bz2
 bzip2 -d unison-2.13.16-linux-text.bz2
 mv unison-2.13.16-linux-text unison
 chmod 755 unison
 ./unison                   *This will create the initial /root/.unison database folder
 vi qmail-replicateb        *Once in the editor, copy and paste the script from below
 :wq
 chmod 755 qmail-replicateb
 vi unison-run              *Once in the editor, copy and paste the unison-run script from below
 :wq
 chmod 755 unison-run
 vi qmail-switch            *Once in the editor, copy and paste the qmail-switch script from below.
 :wq
 chmod 755 qmail-switch
 cp unison-run /etc/init.d
 ln -s /etc/init.d/unison-run /etc/rc3.d/S50unison-run  *This will configure unison to start on boot up.

*To manually start or stop the socket do the following
 /unison/unison-run start 
or 
 /unison/unison-run stop

*The server side script uses a qmail queue repair tool.  This needs to be configured before running the script
 cd /unison
 mkdir queue-repair
 wget http://pyropus.ca/software/queue-repair/queue-repair-0.9.0.tar.gz
 tar -xzf queue-repair-0.9.0.tar.gz
 mv queue-repair-0.9.0 queue-repair

*Run the client script if you need to switch to the backup server

''';Below is the script for the server called qmail-replicateb
 #!/bin/sh
 
 #Version 1.0 - Oct 11th 2006
 #Script created by Craig Smith - craig@doc-net.com
 
 #This script is the server (backup mail server) side script to monitor 
 #changes and import the changes and fix qmail accordingly.
 #It also sets the correct ownership on the vpopmail folders as unison doesn't
 #replicate ownership.
 
 #This script assumes unison and scripts are placed in /unison
 
 #The mysql commands,qmail-newu  and queue repair were taken from Jake Vickers' qmail 
 #restore scripts.
 #Put this script in cron job to finalize changes based on the main server
 #Check the qmail-replicate.log file for each run to see if changes were made.
 #Log file will be adjusted to similarly match client side script.
 
 #Currently qmail is left off to prevent unecessary direct mailings to the
 #backup server.  This can be uncommented to leave it running.
 
 #Please add your mysql root password below
 mysqlrootpass="mysql password"
 
 
 #This portion compares the vpopmail files between replication for changes and 
 #acts accordingly.  If you want this to run with every cron job, comment out 
 #the next 4 script lines as well as the last 3 script lines.
 
 #Checking for mysql.dump and oldlogs folder
 
 folder1=/unison/mysql.dump
 folder2=/unison/oldlogs
 
 if [ ! -d $folder1 ]; then
   mkdir -p $folder1
 fi
  
 if [ ! -d $folder2 ]; then
    mkdir -p $folder2
 fi
 
 #script for log clearup
 #Only last 10 run log files are kept in /unison. The rest are moved to
 #/unison/oldlogs and cleared out each time 10 are exceeded to prevent excessive
 #buildup. 
 
 #Log file location and format
 
 LOG=/unison/unisonlog.`date +%Y%m%d%H%M` 2>&1
 
 LOGCOUNT=`ls /unison/unisonlog.* |wc -l`
 
 #mysql file to be imported
 FILE=/unison/mysql.dump/vpopmail
 
 #LOG COMMENT : if you want to increase the saved logs change the 10 below.
 
 if [ $LOGCOUNT -gt 10 ]
 then
 # echo "more than 10 logs exist, moving to folder /unison/oldlogs" 
   rm -f /unison/oldlogs/*
     mv /unison/unisonlog.* /unison/oldlogs
    clear
     echo "`date` : log files have been moved to oldlogs" >$LOG
   elif [ $LOGCOUNT -le 10 ]
 
 then
 
 echo "`date` : Log count is fine, nothing to do" >>$LOG
 
 fi
 
 mysqladmin -f -uroot -p$mysqlrootpass drop vpopmail >>$LOG 
 mysqladmin -uroot -p$mysqlrootpass flush-tables >>$LOG
 mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG
 mysqladmin -uroot -p$mysqlrootpass reload >>$LOG
 
 mysqladmin create vpopmail -uroot -p$mysqlrootpass >>$LOG
 
 mysql -uroot -p$mysqlrootpass vpopmail < $FILE
 
 #Set the file permission for vpopmail.vchkpw on vpopmail folder
 cd /home
 chown -R vpopmail:vchkpw vpopmail
 #cd /home/vpopmail/
 #chown -R vpopmail:vchkpw *
 
 #Run queue repair
 
 echo "Running qmail-newu for any other loose ends....">>$LOG
 /var/qmail/bin/qmail-newu >>$LOG
 
 #Reload mysql data and restart httpd
 
 echo "Reloading (and refreshing) MySQL and Apache...." >>$LOG
 mysqladmin -uroot -p$mysqlrootpass reload >>$LOG
 mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG
 /sbin/service httpd reload >>$LOG
 
 #Rebuild qmail cdb's to accomadate any replicated changes
 echo "Rebuilding QMail's CDB and starting the mail server...." >>$LOG
 echo "If qmail is not running the svc errors are normal and can be ignored." >>$LOG
 qmailctl stop >>$LOG 2>&1
 sleep 3
 qmailctl cdb >>$LOG
 #Leaving qmail stopped to prevent unecessary direct mailing etc.
 #qmailctl start
 echo "`date` The database has been imported into MYSQL. Verify user details." >>$LOG


''';Below is the script for the unison control file called unison-run

 !/bin/sh
 
 #Script created by Craig Smith - craig@doc-net.com
 #Script for Unison Start up shutdown
 #Set the socket number in the /unison/unison -socket # line.
 
 
 case "$1" in
  start)
    # have a silent kill in case someone tries to start the service when it
    # is already running
    /unison/unison-run stop >/dev/null 2>&1
    echo -n "Starting Unison Socket Server"
    /unison/unison -socket xxxx  &
    echo $! > /var/run/unison.pid
    echo "."
    ;;
  stop)
    echo -n "Stopping Unison Socket Server"
    kill `cat /var/run/unison.pid`
    rm -rf /var/run/unison.pid
    ;;
  help)
 HELP
    ;;
  *)
    echo "Usage: unison-run{start|stop}"
    exit 1
    ;;
 
  esac

Below is the script to use, when you need to switch to your backup server. Change the backup server's ip to match the primary and then run this.  By changing the ip you don't have to worry about dns updates by changing the host record.  This script will do a final clearup and then start qmail.
/unison/qmail-switch to run this file.

'''Below is the qmail-switch script

 #!/bin/sh 
 
 #Version 1.0 - Oct 11th 2006
 #Script created by Craig Smith - craig@doc-net.com
 
 #Please add your mysql root password below
 mysqlrootpass="mysql rootpass"
 LOG=/unison/qmailswitch.log
 FILE=/unison/mysql.dump/vpopmail
 
 mysqladmin -f -uroot -p$mysqlrootpass drop vpopmail >>$LOG
 mysqladmin -uroot -p$mysqlrootpass flush-tables >>$LOG
 mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG
 mysqladmin -uroot -p$mysqlrootpass reload >>$LOG
 
 mysqladmin create vpopmail -uroot -p$mysqlrootpass >>$LOG
 
 mysql -uroot -p$mysqlrootpass vpopmail < $FILE
 
 #Set the file permission for vpopmail.vchkpw on vpopmail folder
 cd /home
 chown -R vpopmail:vchkpw vpopmail
 
 #Run queue repair
 
 echo "Running Charles Cazabon's queue_repair python utility to fix any loose  ends...." >>$LOG
 sleep 3
 cd /unison/queue-repair
 chmod 777 queue_repair.py
 ./queue_repair.py -r >>$LOG
 
 echo
 echo "Running qmail-newu for any other loose ends....">>$LOG
 /var/qmail/bin/qmail-newu >>$LOG
 
 #Reload mysql data and restart httpd
 
 echo "Reloading (and refreshing) MySQL and Apache...." >>$LOG
 mysqladmin -uroot -p$mysqlrootpass reload >>$LOG
 mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG
 /sbin/service httpd reload >>$LOG
 
 #Rebuild qmail cdb's to accomadate any replicated changes
 echo "Rebuilding QMail's CDB and starting the mail server...." >>$LOG
 echo "If qmail is not running the svc errors are normal and can be ignored." >>$LOG
 qmailctl stop >>$LOG 2>&1
 sleep 3
 qmailctl cdb >>$LOG
 qmailctl start
 echo "`date` The database has been imported into MYSQL. Verify user details." >>$LOG
 echo "QMAIL has been fixed and started. It should now be live and workings as  primary mail. please verify." >>$LOG

==Base State Check script==
This script will ping your host for uptime, and will create a fail based on failure. To use, vi filename e.g. statecheck and paste the contents from below. Then chmod 755 filename.  At this time it's not really that helpful without a method of notifying you.  if you have an email address that you monitor regularly that is not on your primary server, then uncomment all email options below and you will be notified on the various levels of failure.  



 #!/bin/sh
 
 #Created by Craig Smith 16/10/2006
 
 #Script file to check for up time on hosts.  Will ping at x min intervals, and email after each consequtive failure.
 #To increase failure numbers, add more variables i.e. fouth, fifth etc. Add to cron to specify check rate.
 #There are probably other ways of doing this, but for now, this is what came about.
 
 HOST=host to be pinged
 #EMAIL=mail for testing
 FOLDER=/unison/uptime
 first=/unison/uptime/fail1
 second=/unison/uptime/fail2
 third=/unison/uptime/fail3
 fine=/unison/uptime/fine
 
 if [ ! -d $FOLDER ];then
 mkdir /unison/uptime
 fi
 
 #If increasing error count, extend these counts.
 
 if [ -f $first ];then
  file=$second
 elif [ -f $second ];then
  file=$third
 elif [ ! -f $first ];then
  file=$first
 fi
 
 rm -f $FOLDER/*
 ping -c 4 $HOST >/dev/null
 if [ $? -eq 0 ]; then
   echo "$HOST is up and visible" > $fine
 elif [ $? -eq 1 ];then
   echo "failure `date`" >$file
 fi
 
 #used for testing. Should be replaced with sms based commands.
 #if [ -f $first ];then
 #echo "$HOST has failed to respond for the 1st time on `date`"|  mail - s "Please perform first time $HOST state check" $EMAIL
 #elif [ -f $second ];then
 #echo "$HOST has failed to respond for the 2nd time on `date`"|  mail -s "Please perform second time $HOST state check!!" $EMAIL
 #elif [ -f $third ];then
 #  echo "$HOST has failed to respond for the 3rd time on `date`"|  mail -s "$HOST state!!! OFFLINE 3RD FAILURE" $EMAIL
 #elif [ -f $fine ]; then
 #echo "$HOST is alive and well"| mail -s "No problems were detected `date`." $EMAIL;
 #fi


==[[User Tips & Tricks]]==