Btrfs Experiment

After watching this video and reading this blog post, I decided to have a play with the new btrfs (ButterFS) filesystem.

So I downloaded OEL6u2 and made a minimal install in VirtualBox (which doesn’t even include wget or scp!)

I decided not to use LVM, but to use LUKS encryption for my 8Gb ext4 /, a 500Mb ext4 /boot with no encryption, and no swap partition.

Upon first boot I installed the Base Yum repo from here which is essentially just what’s on the DVD, and also the beta repo from here. I then installed the Unbreakable2 kernel and btrfs:

yum -y update kernel-uek
yum -y install btrfs-progs openssh wget
reboot

I then booted from the Fedora 16 LiveCD (not the install DVD) to do the ext4-to-btrfs conversion, as the OEL installer hasn’t yet been updated to include btrfs, although eventually it will allow direct btrfs root installation.

Next I had to decrypt the LUKS partition and do the conversion of the underlying ext4 filesystem:

cryptsetup luksOpen /dev/sda2 luks_root
btrfs-convert /dev/mapper/luks_root

As you need to edit the fstab, make a temporary directory to mount your root filesystem inside root’s $HOME directory:

cd /root
mkdir root
mount /dev/mapper/luks_root root/
vi root/etc/fstab

Simply replace “ext4” with “btrfs” on the “/” line and :wq

For some reason when I tried to boot at this stage I got all sorts of permissions issues and it wouldn’t boot. The fix is to disable SELinux:

vi root/etc/selinux/config
SELINUX=disabled
reboot

Prove we’ve booted into our btrfs root filesystem:

mount | grep btrfs
/dev/mapper/luks-f3b07133-ac92-4d75-a782-d0556e7655dc on / type btrfs (rw)

Remove the old ext4 snapshot and defragment:

btrfs subvolume list /
ID 256 top level 5 path ext2_saved

btrfs subvolume delete /ext2_saved
Delete subvolume '//ext2_saved'

btrfs subvolume list /
btrfs filesystem defragment /

Next I thought I’d try out the the yum plugin that takes a snapshot every time you run yum, so you can rollback (Debian equivalent is apt-btrfs-snapshot):

yum install yum-plugin-fs-snapshot.noarch

Then install Apache to try out the plugin:

yum install httpd
Loaded plugins: fs-snapshot
Setting up Install Process
....
Running Transaction
fs-snapshot: snapshotting /: /yum_20120204121740
....
Complete!

Check for the snapshot, note its ID and reboot:

btrfs subvol list /
ID 256 top level 5 path yum_20120204121740

reboot

To boot into the snapshot made before you installed Apache, press “e” at grub prompt to edit the parameters and insert the following into the kernel line:

rootflags=subvolid=256

Then press “b” to boot. Once booted prove Apache isn’t installed anymore, and reboot back into the default snapshot:

rpm -qa |grep httpd
reboot

Check apache is installed again:

rpm -qa |grep httpd
httpd-tools-2.2.15-15.0.1.el6.x86_64
httpd-2.2.15-15.0.1.el6.x86_64

Magic! So no we’ve converted an ext4 filesystem to btrfs on top of a LUKS partition, and have proven yum snapshots work and can boot into a snapshot without needing any backup/restore system.

The snapshot is just regular files, so could be tarred up and moved to an external drive for backup I guess:

ls /yum_20120204121740/
bin   dev  home  lib64       media  opt   root  selinux  sys  usr
boot  etc  lib   lost+found  mnt    proc  sbin  srv      tmp  var

I tried to simulate a catastrophic drive failure – i.e. if you’ve got a tarball of a snapshot, can you restore that to a blank disk with empty ext4 /boot and btrfs / partitions? Well the answer is no if you’re using LUKS. By using dd to backup the MBR and /boot partition I got somewhere, however when I restored the snapshot (untarred it into an empty btrfs filesystem) it booted without prompting for the LUKS key, and then died.

So it still seems a bit like btrfs is for rolling back to a previous point, or to be used with one or more drives in a RAID array, but no use when a single drive fails.

Update: I found that because btrfs stores the BLKID of the disk in every data block, you could never actually move a btrfs filesystem to another disk! The only way to do it (as I have) is to dd the entire disk to another disk, and then they’re BLKID’s will be the same, also it solves the problem of trying to piece together the /boot sector and grub in the MBR.

The problem however, is that as you can’t mount two devices with the same BLKID’s at the same time, you couldn’t use rsync to sync the two disks. So for backup of btrfs, you’re stuck with dd’ing the entire disk, or don’t use LUKS and use hardware RAID (not btrfs RAID or software RAID) instead, which is pretty crap.

Update

Been a while since my last post.

Today I tried out my new mountain bike. I cycled to M&D’s and back, it wasn’t too bad despite the hills, only took 10mins. I do seem to be suffering now though – inhaling all the cold air has screwed with my lungs it seems.

Dad and I have been collecting free firewood from a neighbour, which is nice (although very hard work lugging around) but now we’re itching a bit, so hopefully don’t have poison oak or something. On the way there today we decided to take a shortcut and I ended up getting stuck in a waterlogged ditch, luckily right next to a tractor which pulled us out. I’ll never moan about slow tractors on the road again!

M&D got a new DivX player, its got a USB port so I’ve ordered a couple of 8Gb thumbdrives to try it out with. Its also got a SD card slot, so should be interesting to see if it will play video’s from Mum’s digital camera (as well as obviously the photo’s).

I switched my fileserver to NFSv4, which seems to have improved performance and stability as I’m not getting the “server not responding” errors and Nautilus dying as I was with v3. Its a very different setup though – it has the concept of a root share, and then individual shares beneath it.

Strangely enough MacOSX 10.5 is not supposed to have an NFSv4 client and NFSv4 is not supposed to be backwards compatible, however I can still mount the root share albeit not the individual shares beneath it.

I’m still getting strange speeds though – like NFSv3 took 8mins to transfer 2.7Gb, NFSv4 took 3m30s, NFSv3 on the Mac took 1m30s, SCP took 3m30s and FTP took just 70secs!

I also tried out NIC bonding on my Fedora7 box, you basically setup two GigE cards to round-robin between themselves, appearing as one IP address externally (bit like IPMP on Solaris). It increased my bandwidth from 580-690Mbit/sec to 910-940Mbit/sec – or 112MBytes/sec, and also can cope with you unplugging one of the ethernet cables. I might set it up on my fileserver with a second PCI card.

I tried out Oracle Unbreakable Linux 5.3, which is a shameless ripoff of RHEL 5u3, but not in the free vein of CentOS, it still requires commercial support (a RHN/up2date ripoff) instead of free updates via YUM, and it has a few Oracle-specific patches which either haven’t been accepted upstream or are closed source.

Virtual Oracle

Over the last few days I’ve been installing a few different instances of Oracle into VirtualBox machines.

To ease the OS install headache I made a CentOS 5.2 guest with all the Oracle kernel optimisations and filesystem layouts as well as the required packages (such as the legacy C++ libraries). I then cloned that twice – once for Oracle10gR2 and once for Oracle11iR1; using the instructions here; which are essentiallty: clone the VDI file, register a new VM, change the UUID and mac address and attach the new disk.

It works well actually, although in typical VirtualBox fashion, requires a fair bit of manual hacking around instead of just copying a directory and clicking a button like with VMWare.

Oracle install instructions were from here for 11g and here for 10g

So then I tried to install Oracle 9iR2, well that was a different matter altogether. I had to go back to CentOS 4.7 and fudge a lot of settings and scripts to even get it to install – including turning off NumLock to fix the “font.properties not found” error! I used a combination of instructions here for Fedora Core 2 and here for RHEL 4ASu4.

I found a fix for the 100% CPU load on CentOS 4.7 – pass the “divider=10” kernel parameter to grub; also turn off acpid and cpuspeed in the service menu. Still no fix for XP64, and strangely enough the issue doesn’t occur on CentOS 5.2

After all that installing, I found that the VDI’s were around 14Gb, as the ~4Gb from the deleted cpio’s hadn’t been free’d! So I had a hunt around the forums and found a utility called zerofree that zeroes any unused sectors. I found it best to run it from a rescue CD (in my case RHEL 5.0), remounting the root partition as readonly first:

mount -n -o remount,ro -t ext3 /dev/sda3 /
zerofree -v /dev/sda3

Then shutdown the VM, and on the host run:

VBoxManage modifyhd centos47.vdi compact

Which after a few minutes of disk chugging saved between 2 and 5Gb per VDI!

Last night I watched Death Race, which I actually thought was a lot better than the original Death Race 2000.

The gas has been installed and the plumber has been to measure up, so installation of radiators etc. should start tomorrow for the next two weeks.

Update – NOT

Its been way too long since my last post, and now I’m back in France I have plenty to write about in the Personal category and a few for the Movies category, so much that I’m going to leave it for another day or just skip it altogether!

So I thought I’d bung in some quick techie snippets instead:

I’ve been playing with Oracle 11gR1 on Linux and 10gR2 on Solaris today, coding up some auditing scripts, which has gotten me back into Bourne Shell programming, ooh its been nice. I’ve also been using Git a lot more too, although mainly from the CLI rather than the QGit GUI – which is nice, but not my thing really, GUI’s just seem to slow me down.

Yesterday I finally got my RT2500 wireless card working in the laptop I’m giving Mum. I ditched Fedora9 and CentOS 5.2 as RedHat seem to be doing their best to destroy wifi – indeed a recent kernel patch stopped any wifi but Atheros chipsets from working at all. Anyway, I hate to say it, but I installed Ubuntu 8.04 LTS Desktop and wireless worked out-of-the-box.

I’ve been playing with OSXBMC on the Mac Mini – although now they’ve renamed it “Plex” and are merging with some skin-project called CenterStage. I played my first HD movie on my new 24″ widescreen monitor and its great!

I’ve converted my middle bedroom into a computer room, so now have half a dozen computers and God knows how much network equipment all buzzing away with their flashing LED’s in just one room – and not where I’m trying to sleep! I’ve been looking on eBay for a print server, but they’re either too expensive to ship from the UK or the French are dicking around, so I think I’ll give up.

When the shell’s finished (pretty soon) I’ll probably move one of the PC’s in there, with an old CRT or something – I’ve already wired up the CAT6 and a gigabit switch!