brick oven pizza franchise estate sales services in new jersey movers nyc dallas roofing company

Analyze snapshot of pf state table

September 30th, 2008

Ubuntu 8.04 LTS Desktop

August 31st, 2008

So I’ve been using the Ubuntu 8.04 LTS Desktop Edition on my laptop since its release and I must say that its a huge improvement since their last LTS release which I was running prior. I was previously on a dual boot configuration but I have been so satisfied with the current state of Ubuntu that I have not felt the need to boot into Windows at all. Anything I need to do on Windows has been far and few between lately and terminal services is just a click away, not to mention wine 1.0, wow has that come along way. Sure this version of ubuntu has its bugs but what doesn’t, and its not necessarily the bugs that bug me, but its the annoyances that get under my skin, sometimes caused by bugs or just design issues or lack of features. The previous Ubuntu LTS was filled with so many annoyances that it drove me crazy sometimes, such as WPA support out the box, buggy suspend/hibernation modes,ipod recognition, audio/video/codec annoyances and numerous other minor issues that just left me with a bad taste in my mouth. Their has been very little that has annoyed me with Ubuntu 8.04 LTS and I have never been a huge fan of Ubuntu, but it was the lesser of three evils. I still haven’t gotten to the point where I feel I need to migrate all my debian installations to ubuntu, but this desktop lts version might persuade me to at least give the ubuntu server version another shot. I had a horrendous experience with the last ubuntu server version and I havent looked back since. I dont remember what the exact issues were but debian was just so much less of a headache. And with the lenny branch frozen with a pending release in september this may be a good time to start evaluating my options that will be runnin on my servers for the next couple of years. So thanks to the Ubuntu/Debian teams for releasing a great product. Its been a pleasure.

The Dan Kaminsky Microsoft DNS Patch Sideeffect

July 30th, 2008

So its been a few weeks since most of us have patched our vulnerable dns servers, but I hadn’t noticed this little bonus until today which actually made me laugh. You see a few years ago I had noticed an annoying little behavior with the way Microsoft’s DNS Server handles outgoing client connection for domains/servers that are listed under the Forwarders tab. We use this Forwarders tab to list frequently queried domains in which we host a copy of the zone file in rbldnsd so as to not go to the internet to find the answer to. This gives us the benefit of returning an answer to a dns query much faster and saves us the extra bandwidth. This is highly beneficial to our mail systems which process on average 100 million messages per month, mostly spam of course. So back when we had implemented the rbldnsd system, we had placed Linux Virtual Server in front of the rbldnsd to load balance the traffic accross 8 or so machines. After pointing the Forwarded domains to the LVS VIP, I had expected hundreds, even thousands of connections to get sprayed accross the rbldnsd farm, but uh-uh, nope. There were only two connections listed to two of the backend servers, however all the queries were getting answered.

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 1
-> server2.domain.com:domain Route 2 0 0
-> server3.domain.com:domain Route 2 0 0
-> server4.domain.com:domain Route 2 0 1
-> server5.domain.com:domain Route 2 0 0
-> server6.domain.com:domain Route 2 0 0
-> server7.domain.com:domain Route 2 0 0
-> server8.domain.com:domain Route 2 0 0

This had me scratching my head at first and then after a few packet captures later, I realized that Microsoft was opening 1 socket connection and pushing all the forwarded queries through it. Gee Wiz Microsoft! Why would you do such a thing? I figured that opening and closing socket connections carries an overhead and could also potentially exhaust all available udp ports in a very short amount of time, I can understand why Microsoft would implement it in this way. However this is exactly the insufficient socket entropy that is described in Dan’s advisory as flawed, and from my perspective I hated it as I couldn’t load balance all the forwarded dns queries across each machine that had rbldnsd running on them. Luckily rbldnsd wasn’t the primary service on those machines that we were load balancing so I had decided to just let it be after a spending a few minutes of looking for a workaround then banging my head on my desk out of frustration. Availability was still guaranteed and rbldnsd being as fast and memory efficient as it was, was performing fine in this configuration, so I let it be. I had bigger fish to fry at the time. Fast forward a few years later and a Dan Kaminsky patched Microsoft DNS Server, and wallah, this is what I noticed today…

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 264
-> server2.domain.com:domain Route 2 0 258
-> server3.domain.com:domain Route 2 0 256
-> server4.domain.com:domain Route 2 0 252
-> server5.domain.com:domain Route 2 0 250
-> server6.domain.com:domain Route 2 0 252
-> server7.domain.com:domain Route 2 0 252
-> server8.domain.com:domain Route 2 0 252

and this is with a modified udp timeout of 10 seconds…

me@director:~$ sudo ipvsadm -L --timeout
Timeout (tcp tcpfin udp): 60 10 10

Awesome, entropy, security, and load balancing :). Thanks Dan!

Gentoo - PHP - Fatal error: Call to undefined function ctype_alnum()

July 9th, 2008

If you need access to any of the ctype functions such as ctype_alnum() in any of your php applications and you are running gentoo, make sure you add the ctype use flag or you will get this lovely error :).

cfengine, can’t stat in copy and reverse dns

July 9th, 2008

Well I’ve been using cfengine for a number of years now and thought I had paid my dues already when I initially took on its steep learning curve… Well today I had a little run in with cfengine that made me feel as frustrated as when I was a newbie to this software, but I guess it was a newbie mistake that Im sure I learned years ago that I just happened to forget over the years when adding a cluster of new hosts to the mix - reverse dns.
The issue came about when I was configuring a new group of servers. I was on the final one when I simply installed cfengine on the host, scp’ed over cfagent.conf, cfservd.conf, and update.conf from a host that I had just been successful with. But after running “cfagent -v” i ran into the familiar “Can’t stat /var/lib/cfengine… in copy” which struck me as odd because it had just worked on all the other hosts. After checking the usual suspects such as the grant: function in the cfservd.conf to make sure permissions were explicitly granted on the server side, the hostname and domain name configured on the client, typos, cfkeys, cfservd started ?, etc, nothing seemed to work and adding the debug options -d seemed to frustrate me even more. As a last resort I took a packet capture to see what was going on between the client-> server for both the system that was failing and one that was working. I didnt think it would help much but sure enough after crawling through the capture packet by packet I seen the issue in one of the packets data field that looked something like this…

CAUTH IP IP user - non-working host
CAUTH IP hostname user - working host

This is when the little cfl lightbulb went off in my head and I decided to have a look at reverse dns. Sure enough all the hosts had reverse dns configured but this last one.

Although other functions such as directories,files,editfiles seemed to authenticate and run fine without reverse dns it seemed the copy function was failing because authentication under cfservd and the grant directive is based on the domain *.domain.com and not the IP… sheesh… it seems the parameter SkipVerify can be applied globally here and workaround hosts not having reverse dns, however I decided not to use this option since we control the reverse dns and it really should of been configured, not sure why it was not…

as soon as I added the reverse dns for the host cfagent ran without a hitch…

Ldirectord missing dependency in Debian

June 23rd, 2008

So i came across this the other day while trying to configure ldirectord to load balance pop3 services.

Can’t locate Mail/POP3Client.pm

It seems there is a missing dependency that is specific to debian etch I believe. I was a little disappointed as I’ve had little other issues with LVS and ldirectord but the fix was easy enough and I was able to find bug #421415 in Debian’s bug tracking system so Im sure I was not the first or the last to run into this. If you run into this just run apt-get install libmail-pop3client-perl and you should be good to go.

Slackware 12.1

May 4th, 2008

Slackware has been a staple in the linux community since the beginning and its great to see another release as this distribution evolves thanks to the hard work by Patrick J. Volkerding. Slackware 12.1 was released yesterday and boasts a 2.6.24.5 kernel, and of course the simplicity, stability and security we’ve always expected and recieved from this distro.

OpenBSD 4.3 released

May 4th, 2008

OpenBSD 4.3 was publicly released on schedule last week with astonishing amount of improvements, new features, and bug fixes. Hats off to the OpenBSD Developers that are putting out some great work to make a stable and reliable product.

However, I must say, I did notice some bug fixes in the changelogs that I’ve actually came across and been caught up on that were fixed in this release. These issues had been referenced in the openbsd mailing lists by other users but were never acknowledged by the developers leaving me frustrated and at a dead end without getting into the code. Its good to see these issues finally acknowledged in the changelog.

xen filling up /var/ partition

May 4th, 2008

So the other day I noticed my /var partition filled up in on my Dom0. One of the symtoms was not allowing me to start up any more DomU’s. So I tracked the disk usage to /var/lib/xen/save. There were files of each of my running dom u’s here which were the culprit.

Apparently, on a Debian system, upon a shutdown of the Dom0, Xen attempts to take a snapshot of the running DomU instead of issuing a shutdown command across all the DomU’s. This snapshot is stored in - you guessed it - /var/lib/xen/save - and is controlled by the XENDOMAINS_SAVE parameter in /etc/default/xendomains. In order to disable this future and always execute a shutdown of DomU’s during a shutdown of the Dom0, you can set XENDOMAINS_SAVE=”". The corresponding restore command is controlled by XENDOMAINS_RESTORE parameter. It is also safe to delete these files provided you have a newer instance of the DomU.

After realizing this I changed the path of where the running domains are stored to in case of a shutdown to a partition with sufficient space to hold the running instances. I also set the XENDOMAINS_RESTORE=true.

Thanks to the helpful individuals on the xen-users mailing lists for the info.

rsync bug

April 17th, 2008

The rsync folks just recently released rsync 3.0 last month with a bug-fix release and a security release earlier this month. Unfortunately, after upgrading one of my critical systems that feeds a cluster of about 10 machines, I ran into an ugly little bugger that prevented my cluster’d nodes from successfully pulling there data from the central rsync machine resulting in stale files on the clustered nodes. Heres the error I seen when running my rsync manually:

$ rsync -t 10.9.8.7::module/* /dest
rsync: link_stat “/*” (in module) failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(1515) [receiver=3.0.2]

Instead of using the wildcard I tried one file specifically and that seemed to work just fine so I knew something was up with the wildcard thrown in there…After a little searching I confirmed my suspicions…

https://bugzilla.samba.org/show_bug.cgi?id=5388

Unfortunately this required a manual patch as the current version remains unpatched at the time of this writing and unavailable via package managers such as apt-get, portage, ports, yum, etc. Luckily this was easy enough as the patching and compilation was very smooth as I would expect…

#cd /usr/local/src/
#wget http://samba.org/ftp/rsync/src/rsync-3.0.2.tar.gz
#wget http://samba.org/ftp/rsync/src/rsync-patches-3.0.2.tar.gz
#tar -zxvf rsync-3.0.2.tar.gz
#tar -zxvf rsync-patches-3.0.2.tar.gz
#cd rsync-3.0.2
#patch util.c patches/util.c
#./configure
#make
#make install
#cp /usr/local/bin/rsync /usr/bin/rsync
#/etc/init.d/rsync restart

And once again my cluster’d nodes are happy again :).