Parallelizing rsync

Last week I had a massive hardware failure on one of the GlusterFS storage nodes in the ILRI, Kenya Research Computing cluster; two drives failed simultaneously on the underlying RAID5. As RAID5 can only withstand one drive failure, the entire 31TB array was toast. FML. After replacing the failed disks, rebuilding the array, and formatting […]

Hacking on the Eudyptula Challenge

Last weekend a few of us met up at a coffee shop in Nairobi to hack on the Eudyptula Challenge. From their website, the Eudyptula Challenge is: … a series of programming exercises for the Linux kernel, that start from a very basic “Hello world” kernel module, moving on up in complexity to getting patches […]

New Linodes are fast!

In April, 2014 Linode rolled out a major infrastructure upgrade for their VPSes. When I saw that their update included SSDs (among other things) on all new instances, my mind instantly went to this dog-slow Tomcat application I currently have running in an Amazon EC2 m1.medium instance. I bought a shiny new Linode 2048 and […]

Tweaking Monit to avoid false positives

I use Monit to do lightweight monitoring and alerting for my services. Nothing advanced, just one host running monit which probes other servers via TCP, UDP, ICMP, etc to see if services are accessible. Monit can do much more if you install it on each machine itself, but I’m not interested in that right now […]

GlusterFS mounts fail at boot on CentOS

After nearly a year of running GlusterFS on my compute cluster, I’m surprised it took me so long to run into this bug. I rebooted one of my compute nodes and found that the machine hung at Mounting network filesystems during the boot sequence. From the volume log — /var/log/glusterfs/<volname>.log — on the client, it […]