Notes about creating FreeBSD jails

Something like that:

/bin/sh
mkdir /var/jails && cd /var/jails

for D in `ls`; do
mkdir -p $D/root $D/bin $D/dev $D/etc $D/lib $D/libexec $D/tmp $D/usr $D/var $D/home/$D
chmod o+rwxt $D/tmp
mtree -deU -f /etc/mtree/BSD.var.dist -p $D/var
done

cd /usr/src
for D in `ls /var/jails/`; do
make distribution DESTDIR=/var/jails/$D
done


cat >> /etc/rc.conf
# Jails
jail_enable="YES"
jail_mount_enable="YES"
jail_sysvipc_allow="YES"
jail_list="jail1 jail2 jail3"

## jail1
jail_jail1_name="jail1"
jail_jail1_hostname="jail1.com"
jail_jail1_ip="127.0.0.10/8"
jail_jail1_interface="lo0"
jail_jail1_rootdir="/var/jails/jail1"
jail_jail1_devfs_enable="YES"
jail_jail1_devfs_ruleset="devfsrules_jail"

## jail2
jail_jail2_name="jail2"
jail_jail2_hostname="jail2.com"
jail_jail2_ip="127.0.0.20/8"
jail_jail2_interface="lo0"
jail_jail2_rootdir="/var/jails/jail2"
jail_jail2_devfs_enable="YES"
jail_jail2_devfs_ruleset="devfsrules_jail"


## jail3
jail_jail3_name="jail3"
jail_jail3_hostname="jail3.com"
jail_jail3_ip="127.0.0.30/8"
jail_jail3_interface="lo0"
jail_jail3_rootdir="/var/jails/jail3"
jail_jail3_devfs_enable="YES"
jail_jail3_devfs_ruleset="devfsrules_jail"

^D


cat /etc/fstab.jail1
/var/jails/ruby/bin /var/jails/jail1/bin nullfs ro 0 0
/var/jails/ruby/lib /var/jails/jail1/lib nullfs ro 0 0
/var/jails/ruby/libexec /var/jails/jail1/libexec nullfs ro 0 0
/var/jails/ruby/usr /var/jails/jail1/usr nullfs ro 0 0
tmpfs /var/jails/jail1/tmp tmpfs rw,size=1G 0 0

Simple script to monitor replication in PostgreSQL 9

Here is the simple script to monitor PostgreSQL replication on FreeBSD:

========= Cut here =========
#!/bin/sh

# Timeout in seconds
readonly timeout=600

last_chkp=`pg_controldata /var/db/pgsql/ | grep "Time of latest checkpoint" | sed "s/.*checkpoint: *//"`
last_chkp=`date -j -f "%a %b %d %T %Z %Y" "$last_chkp" "+%s"`

now=`date`
now=`date -j -f "%a %b %d %T %Z %Y" "$now" "+%s"`

if [ `expr $now - $last_chkp` -gt $timeout ]; then
echo Replication kaput!
fi

Tarsnap cleanup on FreeBSD

The simple script to preform tarsnap cleanup on FreeBSD:


#!/bin/sh

keyfile="/path/to/rw_keyfile"

# man(1) date will help (keep files for last 7 days in this example)
# we prefer to keep files in this format 2009-10-18+06:15-database as mentioned in earlier post
rd=`date -v-7d -v0H -v0M -v0S`
pd=`date -j -f "%a %b %d %T %Z %Y" "$rd" "+%s"`

for file in `tarsnap --keyfile $keyfile --list-archives | sort`; do
fd=`echo -n $file | cut -d'+' -f 1`
fd=`date -j -f "%Y-%m-%d %T" "$fd 00:00:00" "+%s"`
if [ $fd -lt $pd ]; then
echo Deleting the $file archive
tarsnap --keyfile $keyfile -d -f $file
fi
done

Script to perform tarsnapping of data on FreeBSD

Here is the script to perform tarsnapping on FreeBSD:


#!/bin/sh

# Directories to backup
DIRS=/usr/local/etc/tarsnap-backup.conf

#EXTRA_FLAGS=--dry-run

BACKUP=`date "+%Y-%m-%d+%H:%M"`
TARSNAP="/usr/local/bin/tarsnap"

# Do backups
IFS='
'
for dirs in `grep -v -e '^#.' $DIRS`; do
# Needs to set IFS to space to allow passing arguments to tarsnap
IFS=' '
echo ${dirs}
$TARSNAP $EXTRA_FLAGS -c -f $BACKUP-${dirs}
done


The conf file should look like:
archive_name1 /list /of /directories1
archive_name2 /list /of /directories2
misc /root /etc /boot/loader.conf /usr/local/etc /usr/local/opt /var/cron/tabs /var/services /var/mail /var/qmail/control /var/qmail/alias

Get NetFlow entries from a flowd via logsock to put to PostgreSQL in Perl

Here is a small piece of code in Perl to get NetFlow entries from a flowd via logging socket (logsock) and put to a PostgreSQL database.
Beware, no sanity checking at all!!!

=== cut here ===

#!/usr/bin/perl
use IO::Socket;
use Socket;
use Flowd;
use DBI;

# Database settings
my $DBI_DRIVER = "Pg"; # or one of "Pg" "mysql" "mysqlPP"
my $DB = "netflow";
my $HOST = "localhost";
my $TABLE = "flow";
my $USER = "netflow";
my $PASS = "password";

$sock_addr="/var/run/flowd/flowd.sock";
unlink($sock_addr);

$sock = IO::Socket::UNIX->new( Local => $sock_addr, Type => SOCK_DGRAM)
or die "Can't bind to Unix Socket: $!\n";
$sock->setsockopt(SOL_SOCKET, SO_RCVBUF, 65440);

my $db = DBI->connect("dbi:$DBI_DRIVER:host=$HOST;database=$DB", $USER, $PASS)
or die "DBI->connect error: " . $DBI::errstr;

print "Started.\n";
while ($bytes = $sock->recv($input,1024)) {
$flowfields = Flowd::deserialise($input);

$recv_time = sprintf "%s.%03d",$flowfields->{recv_sec}, $flowfields->{recv_usec};
$flow_start = $recv_time + ($flowfields->{flow_start} - $flowfields->{sys_uptime_ms})/1000;
$flow_finish = $recv_time + ($flowfields->{flow_finish} - $flowfields->{sys_uptime_ms})/1000;

$sql = sprintf("INSERT INTO flows (recv_time, agent_addr, protocol_id, src_addr, src_port, dst_addr, dst_port, packets, octets, flow_start, flow_finish) VALUES (to_timestamp('%s'), '%s', '%u', '%s', '%u', '%s', '%u', '%s', '%s', to_timestamp('%s'), to_timestamp('%s'))",
$recv_time,
$flowfields->{agent_addr},
$flowfields->{protocol},
$flowfields->{src_addr},
$flowfields->{src_port},
$flowfields->{dst_addr},
$flowfields->{dst_port},
$flowfields->{flow_packets},
$flowfields->{flow_octets},
$flow_start,
$flow_finish
);
$db->do($sql) or die "db->do failed: " . $DBI::errstr;
}
1;

=== cut here ===

The SQL schema:
CREATE TABLE flows (
id serial NOT NULL,
recv_time timestamp with time zone DEFAULT now() NOT NULL,
agent_addr inet NOT NULL,
protocol_id integer NOT NULL,
src_addr inet NOT NULL,
src_port integer NOT NULL,
dst_addr inet NOT NULL,
dst_port integer NOT NULL,
packets bigint DEFAULT 0 NOT NULL,
octets bigint DEFAULT 0 NOT NULL,
flow_start timestamp with time zone NOT NULL,
flow_finish timestamp with time zone NOT NULL
);

Patching Squid to support gzip/deflate encoding

Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator.
Unfortunately, Squid is HTTP/1.0 due to the lack of certain features and the main disadvantage, from my point of view, is a lack of gzip/deflate content encoding.
There is a patch created by Swell Technology and committed to the HEAD. But it was not tested and little bit outdated.
So I've spent around six late night/early morning hours to merge the patch to Squid 3.0-Stable1. Right now patched squid it running on the old AMD Athlon64 3000+ machine serving around 500 simultaneously connected clients through the dial-up lines. The main concern was the CPU utilization, but FreeBSD team made an excellent work in regards of the performance in FreeBSD 7.
So here is the snapshot from top:
56897 squid 1 4 0 1085M 1044M kqread 9:28 5.18% squid

and uptime is:
1:51PM up 7 days, 10:45, 2 users, load averages: 0.01, 0.06, 0.15


The FreeBSD port patch is here.

Enjoy.
UPDATE:

Link to FreeBSD PR

UPDATE 1:
FreeBSD maintainer refused to integrate patch to ports tree:

From:	Thomas-Martin Seck
Date:	Sat, 8 Mar 2008 15:59:46 +0100
I am sorry, but I am not going to integrate third-party patches into the Squid ports any more (this includes patchsets that are available from devel.squid-cache.org). Please work with the Squid developers on integrating this feature into mainline Squid. Short rationale: Third party patches are a headache to maintain, especially when they are no longer maintained (cf ICAP for Squid-2) and they can be a major source of trouble when they contain bugs that are then wrongly attributed to bugs in Squid itself. I would therefore like to keep the port as close to the mainline source as possible to make it easy for users to get support for it from the Squid developers in case of problems.

Small script to check MySQL replication

Small script to check MySQL replication on FreeBSD. Path to mysql.sock as a parameter:

cat /usr/local/opt/check_relication.sh
#!/bin/sh
treshold=1200

### Do not edit below
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin

if [ -z $1 ] ; then
echo "Usage: $0 /path/to/mysql.sock"
exit 255
fi

seconds=`mysql -S $1 -N -e "show slave status\G" | grep Seconds_Behind_Master | cut -d: -f 2`

if [ -z $seconds ] ; then
exit 255
fi

if [ $seconds = "NULL" ]; then
echo Replication kaput!!!
exit 255
fi

if [ $seconds -gt $treshold ]; then
echo Slave behind the master more than $treshold seconds!
fi


Just put to crontab something like this:
*/30 * * * * /usr/local/opt/check_relication.sh /tmp/mysql.sock