Nickebo.net
I am a man of science
Partition alignment, how important is it?
0To start with, what is alignment? I’ll illustrate it below.
Misaligned partition
This is the most common problem that usually occurs on older OS:es like Windows XP. The partition gets an offset from the physical sectors because of 63 hidden sectors (512 bytes) used by the boot mechanism. This is solved by instead of let’s say 63 sectors (512 bytes) you use 1 MB in the beginning of the disk. This is more than enough space for any boot info and the disk now gets aligned since it’ll use a number of blocks that can be divided by 4k, not just a part of the first one in the beginning.
There’s still a problem though, the disk uses 4k blocks and in this case it reports it’s using 512b blocks to the OS. This effectively means that if you want to modify a 512b block you have to read 4k, modify 512b and write down 4k again. Quite a waste and it becomes an irritating issue with ZFS.
What about ZFS?
ZFS uses variable block size. This means that it’ll use down to 512b blocks if that’s what’s reported by the disk. This can take quite a toll in performance, but there’s a way to force ZFS to use a minimum of 4k blocks. I’m using a program called gnop in FreeBSD. It’s used like this and forces ZFS to use a block size minimum of 4k.
# gnop create -S 4096 /dev/ada0
# gnop create -S 4096 /dev/ada1
# gnop create -S 4096 /dev/ada2
# gnop create -S 4096 /dev/ada3
# zpool create pool raidz2 /dev/ada0.nop /dev/ada1.nop /dev/ada2.nop /dev/ada3.nop
What’s the impact in performance? I actually did a test with two striped SSDs. They’re a couple of Corsair F60, about two years old and uses SATA 2.0 (3 Gb/s).
bonnie++ with misaligned ZFS on mirrored SSDs
bonnie++ with aligned ZFS on mirrored SSDs
As expected the read performance was the same, but the write performance was considerably higher with alignment. It went from 261 MB/s to 331 MB/s, quite a bump by just doing things the right way!
The firmware bug that made me crazy
0
The evil disk of death?
My file server has been a pretty frequent topic on this blog and it probably won’t be a surprise when I’m bringing it up again. This time there’s been an issue with firmware, although I didn’t realize it until a just a few days ago.
I’ve always had some minor issues with my file server, most prominently in performance. It used to stall now and then, but I read and read and ZFS seemed to have some problems with “bursty” writes etc. so I didn’t pay that much attention to it. Not until about a week ago. I had to replace one of the disks in the RAID, I had a WD Green of the same model as the others laying as a cold spare so I used it and started resilvering. I got about 2-5 MB/s in resilvering speed, something was definitely wrong. But what as it?
My first though was “this is probably an alignment problem, I knew the disks were 4k (Advanced Format) drives with 4k sectors reporting 512k sectors to the OS. This can cause a bit of a problem, so I decided to backup all my important data from the RAIDZ store and align the disks, like this, using the gnop utility in FreeBSD. However this didn’t make a difference. The RAID was still AWFULLY slow.
I started reading blog after blog, but couldn’t figure out what the problem was. Then it hit me, I’d heard about the problem with WD Green disks a couple of years ago and some of my disks were from around that time. I decided to try them out one by one and check if there was a difference in performance between the separate disks. My two oldest 1,5 TB disks seemed to be the problem. Now and then they stalled for about 5-20 seconds, no data could be read or written. This seemed to indicate I had disks with the Intellipark bug, extremely aggressive power savings that made the disks pretty much useless. I tried running a configuration tool from WD that would sort out the problem but with no luck. It seemed the RAID was kind of OK with one of the disks with buggy firmware but when I used two of them it just went bananas and stopped working at all, pretty much.
What to do? I replaced the drives, plain and simple. Two fresh WD Green 2 TB disks. OK, the other disks are 1,5 TB so I’m ending up loosing 1 TB but the RAID works! What about the performance? I used to get stalls and about really bad performance with bonnie++ (25 MB/s write and 100 MB/s read from the RAID in average).
Screen shot from a round of bonnie++
If you have a look at the screen shot above I now get 179 MB/s write and 237 MB/s read with bonnie++ on my RAIDZ2. By no means extremely fast, but it’s not bad either.
Conclusion
If you have disks from the WD Green series performing strangely, make sure it isn’t bad firmware that’s the problem. In my case I had problems with WD15EARS-00Z5B1 using firmware 80.00A80. It seems that’s the firmware causing the most problems.
Virtual machine storage performance, NFS vs. iSCSI
0I’m running VMware ESXi to virtualize my computers and FreeBSD 9.0 with ZFS as backend storage. I’ve done some tests a couple of years ago telling me to always choose NFS over iSCSI for virtual machine storage. Now, is this the best solution for performance? I had to test this, totally unscientific but still worth reading I hope.
I did two different tests, one with bonnie++ and one with dd. Each test was performed inside a virtual machine (Ubuntu 12.04 LTS 64 bit) running off iSCSI storage and one stored on NFS. They were both stored on the same storage pool. At last the test was performed directly on the FreeBSD backend server to see the potential maximum. This isn’t true though as I was limited by my 1 Gb/s Ethernet connecting the ESXi machine and the FreeBSD server.
I just ran bonnie++ with standard settings, no tweaking at all. Here are the results.
Bonnie++ results
The second test was done with dd. I used dd if=/dev/zero of=./test bs=`expr 1024 \* 1024` count=4096 to create a 4 GB file just measuring the write performance.
dd results
Conclusion
As you can see there was a slight advantage for NFS, although not a huge one. In my opinion, the fact that the virtual machines are stored as files on the FreeBSD servers filesystem in NFS, is a giant reason for NFS as the first choice. You can take backup of your machines with ease and won’t have to use VMFS at all.
Using op5 graphs without logging in
0I use the graphs generated in op5 to show the temperatures on my temperature page. Every time op5 (or pnp4nagios to be more precise) is updated the config file gets updated. This adds a security feature saying that you have to be logged in to download the graphs. This is easy to turn off if you want to be able to download the graphs via for example wget (which I do).
Edit /opt/monitor/etc/pnp/config.php and change auth_enable to FALSE:
$conf['auth_enabled'] = FALSE;
You can now download the graphs without authorization.
Problems during update to op5 5.6.0
2Yesterday I updated to op5 5.6.0 and it broke. Big time! Every time I tried to view Tactical overview, Services, basically anything using Ninja I just got a stack trace. I tried reinstalling Ninja using yum, but with no luck. However it told me to run a script called /opt/monitor/op5/ninja/install_scripts/ninja_db_init.sh, but this script gave me the following errors:
[root@charon marcus]# /opt/monitor/op5/ninja/install_scripts/ninja_db_init.sh Installing database tables for Ninja GUI /opt/monitor/op5/ninja/install_scripts/ninja_db_init.sh: line 42: [: : integer expression expected Installing database tables for SLA report configuration /opt/monitor/op5/ninja/op5-upgradescripts/merlin-reports-db-upgrade.sh: line 52: [: : integer expression expected Installing database tables for AVAIL report configuration /opt/monitor/op5/ninja/op5-upgradescripts/merlin-reports-db-upgrade.sh: line 90: [: : integer expression expected Installing database tables for scheduled reports configuration /opt/monitor/op5/ninja/op5-upgradescripts/merlin-reports-db-upgrade.sh: line 136: [: : integer expression expected Database upgrade complete.
After some research I concluded that the script couldn’t log into the database, the root account in MySQL had a password set all of a sudden. (op5 uses a blank root password by default).
I used the instructions at http://dev.mysql.com/doc/refman/5.0/en/resetting-permissions.html#resetting-permissions-unix to reset the root password and reinstalled all op5 packages. This didn’t work, so I ran the /opt/monitor/op5/ninja/install_scripts/ninja_db_init.sh script manually. This did the trick!
[root@charon ~]# /opt/monitor/op5/ninja/install_scripts/ninja_db_init.sh Upgrading ninja db from v4 to v5 Upgrading AVAIL tables from v8 to v9 ... done. Upgrading scheduled reports tables from v7 to v8.sql ... done. Importing old scheduled reports Schedules seems to be already imported. Database upgrade complete.
As you can see all the errors are gone and the upgrade seemed to be successful, and it was successful!
Upgrade to 5.6.0 finally successful
Now, I’m not sure if it was something I did or a bug in op5. I’m just very pleased that I fixed the problem.
Backup of virtual machines with ZFS snapshots and rsync
2I use VMware ESXi to run virtual machines, they’re all stored via NFS on my file server which uses ZFS. Even though I use RAID-Z it would be a good idea to back them up to a remote site. I do this by snapshoting the ZFS volume and rsyncing the snapshotted machines to my Xserve at work.
#!/usr/local/bin/bash #Create snapshot backupdate=$(date "+%Y-%m-%d") zfs snapshot storage/vmware@$backupdate #rsync virtual machines rsync -avz /storage/vmware/.zfs/snapshot/$backupdate/Callisto marcus@[remote host]:/Volumes/Storage/backup/VMs/ rsync -avz /storage/vmware/.zfs/snapshot/$backupdate/Ganymede marcus@[remote host]:/Volumes/Storage/backup/VMs/ rsync -avz /storage/vmware/.zfs/snapshot/$backupdate/Mercury_1 marcus@[remote host]:/Volumes/Storage/backup/VMs/
This is by no means a perfect solution. I’m backing up live machines, this has a potential of going wrong. But since I’ve snapshotted them I hope the problem is negligible. As you can see in the script I’ve changed the remote host to [remote host], replace with host of choice plus modify the rest of the paths. I think you get it.
Right now I’ve added a cron job to do a backup once a week.
Rescuing my iPhoto Library from a Time Machine backup
2I recently reinstalled my Macbook Air, of course I backed everything up and even copied my music to a share on my server. The reason for the reinstallation was that I had installed OS X Mountain Lion DP 2, this didn’t work well with for example VMware Fusion. I thought I’d just get my old installation back with Time Machine, but it just refused to work. What to do?
I reinstalled Lion, copied my music back and remembered that I hadn’t copied my iPhoto Library! Crap, all my pictures. I had to figure out how to get them back from my Time Machine backup somehow. The Time Machine backup was done to my Xserve at work.
I started by downloading BackupLoupe, thought it looked like a good program to extract the iPhoto Library. So I installed it on the Xserve, mounted the TM image and started messing around.
BackupLoupe in action
BackupLoupe didn’t do what I wanted. At all. When I tried to extract the library it just failed, no idea why. Then I googled some more and figured out that I could actually extract the library with the ordinary TM tool in OS X, just right click on the TM icon and choose to browse a TM backup from another computer. I managed to extract the library!
The next problem is that the library is really just a directory with a huge number of files, pictures and movies. It would take forever to transfer them one and one to my computer, so I just tarballed them.
I then copied the tarball to my computer.
Copying in progress
The last step was to unrar the file and import the library. Everything works now, puh. I did NOT want to lose all my pictures.
What did I learn from this? The most important thing about backups is to make sure you can restore the files!
Adding DTrace support to FreeBSD
0I’ve been interested in learning about DTrace for quite a while now, to start with I thought I’d get support for it on my file server running FreeBSD. The generic FreeBSD kernel that comes with the standard installation doesn’t have DTrace support enabled. Time to get that sorted.
1. To start with, make sure your system has the kernel source installed under /usr/src/sys.
2. Then, because I run a 64-bit system, I’m changing to /usr/src/sys/amd64/conf and copy GENERIC to a new conf file called DTRACE. At the bottom of the DTRACE file I add the following.
# Dtrace support
options KDTRACE_HOOKS # all architectures - enable general DTrace hooks
options DDB_CTF # all architectures - kernel ELF linker loads CTF data
options KDTRACE_FRAME # amd64 - ensure frames are compiled in
makeoptions DEBUG="-g" # amd64? - build kernel with gdb(1) debug symbols
makeoptions WITH_CTF=1
This adds the DTrace support to the new kernel I’m compiling. I also removed some crap (like WiFi support) since I’ll never need that in a file server. Now, time to recompile the kernel.
- Run config /usr/src/sys/amd64/conf/DTRACE
- Go to /usr/src/sys/amd64/compile/DTRACE and run make cleandepend; make depend; make; make install; reboot
There we go, last run kldload dtraceall and dtrace -l hopefully gives you a list of all the DTrace probes avaliable.
Loads o' probes
DuckDuckGo as default search engine in Firefox
0For some time I’ve been trying to get rid of all connections to Google, mainly because I don’t want my life cataloged and sold to the highest bidder. Anyway, today I switched my main search engine in Firefox from Google to DuckDuckGo. I installed a simple addon to make it my search engine and it did pop up in the search field. But whenever typed something in the address bar it still used Google. This was frustrating, but you can change that behavior easily.
Open about:config in Firefox and make sure these fields are set to “DuckDuckGo”.
Preferences set in about:config
Hopefully this will make my web browsing even better since I feel that DuckDuckGo is a bit more like the old school web. Just search, nothing more. No Gmail, YouTube or Google+.
Installing Ubuntu on a Zotac Zbox Nano VD01 part II
0So, how did I do? I gave up, that’s how. The graphical installer just won’t work without drivers and I can’t install drivers before I’ve installed the OS. Plus the drivers seems to be very outdated. Don’t buy this computer unless you want to get really frustrated before you get X up and running (if you ever get it running). I’d buy something based on the ION platform or AMD Fusion instead.
VD01
So, what to use it for? Ubuntu Server works nicely on it, so I guess as a small and power efficient server maybe?