FreeBSD network settings for highly utilized squid proxy server by dial-up users

Port config knobs:
cat /var/db/ports/squid26/options
# This file is auto-generated by 'make config'.
# No user-servicable parts inside!
# Options for squid-2.6.9
_OPTIONS_READ=squid-2.6.9
WITHOUT_SQUID_LDAP_AUTH=true
WITHOUT_SQUID_SASL_AUTH=true
WITHOUT_SQUID_DELAY_POOLS=true
WITH_SQUID_SNMP=true
WITHOUT_SQUID_CARP=true
WITHOUT_SQUID_SSL=true
WITHOUT_SQUID_PINGER=true
WITHOUT_SQUID_DNS_HELPER=true
WITH_SQUID_HTCP=true
WITHOUT_SQUID_VIA_DB=true
WITH_SQUID_CACHE_DIGESTS=true
WITH_SQUID_WCCP=true
WITHOUT_SQUID_WCCPV2=true
WITH_SQUID_STRICT_HTTP=true
WITHOUT_SQUID_IDENT=true
WITHOUT_SQUID_REFERER_LOG=true
WITHOUT_SQUID_USERAGENT_LOG=true
WITHOUT_SQUID_ARP_ACL=true
WITH_SQUID_PF=true
WITHOUT_SQUID_IPFILTER=true
WITH_SQUID_FOLLOW_XFF=true
WITHOUT_SQUID_ICAP=true
WITH_SQUID_AUFS=true
WITH_SQUID_COSS=true
WITH_SQUID_KQUEUE=true
WITH_SQUID_LARGEFILE=true
WITH_SQUID_STACKTRACES=true

Boot network settings:

cat /boot/loader.conf
accf_http_load="YES"
accf_data_load="YES"

#kern.hz=1000
kern.maxproc=6164
kern.maxdsiz="1536M"
kern.dfldsiz="1536M"
kern.maxssiz="512M"

kern.ipc.msgseg=768
kern.ipc.msgssz=128
kern.ipc.msgtql=3072
kern.ipc.msgmnb=12288
kern.ipc.msgmni=60

kern.ipc.shmall=6144
kern.ipc.shmseg=24
kern.ipc.shmmni=48
kern.ipc.shmmax=51457280

kern.ipc.shm_use_phys=1
kern.ipc.nmbclusters=131072
kern.ipc.maxsockbuf=524288

net.inet.tcp.tcbhashsize=16384

Run-time settings:

cat /etc/sysctl.conf
security.bsd.see_other_uids=0

kern.maxfiles=40960
kern.maxfilesperproc=22190
kern.timecounter.hardware=TSC
kern.ipc.somaxconn=4096

To repartition disk drive /duplicate file system in FreeBSD :
1. create new slice using bsdlabel
2. newfs newly created slice
3. mount it to /mnt (or wherever)
4. cd /mnt (or wherever)
5. try to minimize amount of the open files on source filesystem by killing all unnecessary daemons
6. dump -0ab 128 -C 32 -f - /dev/slicename | restore -rb 128 -f -
7. ensure backup is adequate
8. modify /etc/fstab and point new filesystem to desired mount point
9. check is everything ok throughly.
10 umount old mount point && mount new one or reboot if the live fs like e.g. /usr

Organize files backup and upload them to ftp server

This script performs a daily files backup. The goal is to get possibility to restore any file within seven days. To economically waste disk space, the full backup is performed only once within a week on Sunday morning. On all other days only differential backup is performed (backup files changes from last FULL backup only). This technique allows to significantly speedup the restore. Scheme is simple, one full backup followed by six differential ones.

The /usr/local/etc/backup.conf file stores directories to include/exclude to/from backup, syntax is simple: the first "+" means to backup, and the "-" means to exclude from the backup.

cat /usr/local/etc/backup.conf
+/etc
+/usr/local/etc
+/usr/local/opt
+/home
-/home/www/logs
+/var/mail
+/var/qmail/control
+/var/qmail/alias
+/var/cron/tabs
-/var/mail/spam

So, here we go:
cat /usr/local/opt/files_backup.sh
#!/bin/sh
to_backup=/usr/local/etc/backup.conf
backupdir=/path/to/store/backups
ftp_to=ftp://host.name.to.put.backups.to/`hostname -s`/files
# Days to make diffs
diff_days=6
# Days to keep backup
restore_days=7

### Do not edit below
umask 037
PATH=/bin:/usr/bin

date=`date "+%Y-%m-%d"`

# Find last full backup in ${diff_days} days if exists
last_full=`find ${backupdir} -name "*-full.tbz2" -type f -mtime -${diff_days} | sort -r | head -1`

# Enumerate directories to include in backup
for dir in `cat ${to_backup}` ; do
case `expr -- "${dir}" : '\(^.\)'` in
+) include="${include} `expr -- "${dir}" : '+\(.*\)'`" ;;
-) exclude="${exclude} --exclude `expr -- "${dir}" : '-\(.*\)'`" ;;
esac
done

if [ ${last_full} ]; then
if [ "`find ${to_backup} -newer ${last_full}`" ]; then
filesuff="full.tbz2"
else
filesuff="diff.tbz2"
newer="-W newer-mtime-than=${last_full}"
fi
else
filesuff="full.tbz2"
fi
filename=${date}-${filesuff}

# Backup files and put to ftp server
tar jPpcf ${backupdir}/${filename} ${newer} ${exclude} ${include} && \
ftp -Vu ${ftp_to}/${filename} ${backupdir}/${filename}

# Find Last Full Backup to Keep ( LFB2K ) in ${restore_days} days
full2keep=`find ${backupdir} -name '*-full.tbz2' -type f -mtime +${restore_days} | sort -r | head -1`

if [ ${full2keep} ]; then

full2keep_name=`expr "//${full2keep}" : '.*/\(.*\)'`

# Delete files older than LFB2K
find ${backupdir} -type f ! -newer ${full2keep} ! -name ${full2keep_name} -name '*.tbz2' -delete

# Delete unnecessary diff files belongs to LFB2K
find ${backupdir} -type f -newerBm ${full2keep} -Btime +${restore_days} -name '*.tbz2' -delete
fi

This script can be downloaded from here.

There were some fixes and modifications.

Organize MySQL backups and put to ftp server

So we have very important MySQL databases hosted on FreeBSD OS we do not want to lose.
First step is to create backup procedures to make backups to local disk.
But this will not help us much if server will crash.
As the second step we will put local backups to remote ftp server configured to allow only upload with unique filenames. The download and even delete or overwrite is prohibited.
FTP server configuration is the subject of another post.

I prefer to put my scripts in /usr/local/opt/. Let's name the script "mysql_backup.sh"

# cat > /usr/local/opt/mysql_backup.sh
#!/bin/sh
username=USERNAME
password=PASSWORD
backupdir=/path/to/backup
ftp_to=ftp://host.name.to.put.backups.to/`hostname -s`/db
days=7

### Do not edit below
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
umask 077
now=`date "+%Y-%m-%dT%H:%M:%S"`

databases=`mysql -u ${username} -p${password} -N -e "show databases"`

for db in ${databases} ; do
mysqldump -u ${username} -p${password} --opt -F -l ${db} > ${backupdir}/${db}-${now} && \
bzip2 -9 ${backupdir}/${db}-${now} && \
ftp -Vu ${ftp_to}/${db}-${now}.bz2 ${backupdir}/${db}-${now}.bz2
done

find ${backupdir} -name '*-*.bz2' -a -type f -mtime +${days} -delete

^D

Script enumerates all databases and then dumps each in backup folder using DBNAME-YEAR-MONTH-DAY_HOUR:MINUTE scheme as the name. After, script archives the dump using bzip2 and upload archive to the ftp server. Last part of the script removes files in backup folder older than, in my case, 7 days.

So, everything is ready, and let's modify /etc/crontab to run script every night at 05:30

# echo "30 5 * * * root /usr/local/opt/mysql_backup.sh" >> /etc/crontab

That's it.

P.S.
Will be wise to run script manually before putting to crontab.

Script to flush logs from replicant host to master

#!/bin/sh
# Format host:username:password
hosts="host1:username1:password1 host2:username2:password2"

###
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
for system in $hosts ; do
host=`echo $system | cut -d : -f 1`
user=`echo $system | cut -d : -f 2`
password=`echo $system | cut -d : -f 3`

log=`mysql -S /tmp/$host-mysql.sock -B --skip-column-names mysql -e "show slave status" | cut -f 6`

if [ ${log} ]; then
mysql -h $host -u $user -p$password -e "PURGE MASTER LOGS TO '$log';"
fi
pid="/home/$host/`hostname`.pid"
kill -HUP `cat $pid`
done