Resizing A BTRFS Filesystem

My OEL6 VM ran out of disk space in its root filesystem, so I had to resize the disk. Most of the howto’s are out-of-date or assume if you’re using cryptsetup you’re using LVM or use GParted which doesn’t handle LUKS, or assumes you’re using ext4 not BTRFS.

Actually it was pretty easy once you ignore all of those incorrect tutorials.

1. Resize the VirtualBox VDI file to 40Gb:

VBoxManage modifyhd oracle62.vdi --resize 40000

2. Boot into a Fedora20 LiveCD:

fdisk /dev/sda
delete partition 2 (d)
create new partition 2 (n)
write & quit (w)

3. When booted back into your VM, resize the BTRFS filesystem to fill you newly grown partition:

btrfs filesystem resize max /
btrfs filesystem defragment /

Full System Backup (and restore)

I thought I’d play around with backups as a continuation of my btrfs experiment. Well basically btrfs can’t do it without dd’ing the entire drive as the UUID’s can’t be changed.

So I thought I’d stick to ext4, and I’ve finally figured out how to do full root filesystem backup including LUKS encryption (without LUKS its easy, you could even do it with tar) using rsync of just the files instead of using dd to backup every bit (17Gb of files instead of 64Gb drive size in my case).

All of the backup section can be done from a live running system.

1. Backup the filesystem UUID’s:

blkid > uuids.txt

2. Backup full MBR (446 boot sector + 64 partition table + 2 sig) and extended partition info:

dd if=/dev/sda of=mbr.img count=1 bs=512
sfdisk -d /dev/sda > sda.sf

3. rsync the root filesystem to a remote server, or external disk, whatever. Obviously only do this to an encrypted filesystem as you’re copying the unencrypted files:

rsync -avp -P --numeric-ids --delete --delete-excluded \
    --exclude-from=excludes.txt / root@ip:/path/

My excludes.txt file looks like this:

# Include
+ /dev/console
+ /dev/initctl
+ /dev/null
+ /dev/zero

# Exclude
- /dev/*
- /proc/*
- /sys/*
- /tmp/*
- /media/*
- lost+found/
- .gvfs/
- .ccache/

That’s the backup done, now to restore to a new disk, do all of this from a live CD, you’ll have to scp the mbr.img and sda.sf files across.

4. Restore the MBR and partition info to a new blank disk:

dd if=mbr.img of=/dev/sda count=1 bs=512
sfdisk /dev/sda < sda.sf

If you want to restore just the boot sector and not the partition table, for example if your disks are different sizes, just change the block count, but you'll have to use fdisk to create the partitions then:

dd if=mbr.img of=/dev/sda count=1 bs=446

5. You'll have to format the filesystems and setup LUKS, but you don't need to create the partitions using fdisk:

mkfs.ext4 /dev/sda1
cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 sda2_crypt
mkfs.ext4 /dev/mapper/sda2_crypt

Now for the complicated part. If you just dd the entire disk, then all the partitions would keep the same UUID's, but as we've created new partitions on a blank disk, we have to reset them to the values we captured in #1, this is the part that's not possible on btrfs.

6. Reset LUKS UUID - for this your boot CD will need to have a newer cryptsetup than 1.1.3 found on F14 which doesn't have the ability to change UUID, cryptsetup 1.3.1 as on F16 LiveCD worked here:

cryptsetup luksUUID /dev/sda2 --uuid=12c92874-51ee-11e1-9c56-001d7d00626d

7. Reset root (/) ext4 partition UUID:

tune2fs /dev/mapper/sda2_crypt -U 19086ed4-51ee-11e1-864d-001d7d00626d

8. Reset /boot ext4 partition UUID:

tune2fs /dev/sda1 -U 23162d3a-51ee-11e1-b203-001d7d00626d

9. Finally we need to reinstall grub (I'm not sure why as we used dd on the MBR). So we mount root, and inside that mount /boot, and as we're not backing up /dev, we need to mount the live CD's /dev inside root too, then make that a chroot:

mount /dev/mapper/sda2_crypt /mnt
mount /dev/sda1 /mnt/boot
mount --bind /dev /mnt/dev
chroot /mnt
grub-install /dev/sda

That's it, done. Reboot into your cloned system.

Update: If you have an encrypted swap partition, on first boot you'll also need to run something like:

cryptsetup luksOpen /dev/sda3 sda3_swap
mkswap /dev/mapper/sda3_swap -U 9421cbe2-559f-11e1-9ec6-001d7d00626d

I've just restored my desktop machine into a VirtualBox VM using this method, including shrinking the disk from a 64Gb SSD to a 40Gb virtual disk (as only about 17Gb was used) and it works fine - encrypted swap and root (ext4). Screenshot.

On first boot I made some small modifications to /etc/hosts and /etc/sysconfig/networks to change IP, mac and hostname, and deleted /etc/udev/rules.d/70-persistent-net.rules so it would rescan for eth0 on reboot.

I also ran nvidia-installer --uninstall and deleted /etc/X11/xorg.conf to reconfigure Xorg. Of course if I rsynced the physical machine to the VM, I'd have to redo these steps again, so I've added some files to excludes.txt (such as guest additions).

Btrfs Experiment

After watching this video and reading this blog post, I decided to have a play with the new btrfs (ButterFS) filesystem.

So I downloaded OEL6u2 and made a minimal install in VirtualBox (which doesn’t even include wget or scp!)

I decided not to use LVM, but to use LUKS encryption for my 8Gb ext4 /, a 500Mb ext4 /boot with no encryption, and no swap partition.

Upon first boot I installed the Base Yum repo from here which is essentially just what’s on the DVD, and also the beta repo from here. I then installed the Unbreakable2 kernel and btrfs:

yum -y update kernel-uek
yum -y install btrfs-progs openssh wget

I then booted from the Fedora 16 LiveCD (not the install DVD) to do the ext4-to-btrfs conversion, as the OEL installer hasn’t yet been updated to include btrfs, although eventually it will allow direct btrfs root installation.

Next I had to decrypt the LUKS partition and do the conversion of the underlying ext4 filesystem:

cryptsetup luksOpen /dev/sda2 luks_root
btrfs-convert /dev/mapper/luks_root

As you need to edit the fstab, make a temporary directory to mount your root filesystem inside root’s $HOME directory:

cd /root
mkdir root
mount /dev/mapper/luks_root root/
vi root/etc/fstab

Simply replace “ext4” with “btrfs” on the “/” line and :wq

For some reason when I tried to boot at this stage I got all sorts of permissions issues and it wouldn’t boot. The fix is to disable SELinux:

vi root/etc/selinux/config

Prove we’ve booted into our btrfs root filesystem:

mount | grep btrfs
/dev/mapper/luks-f3b07133-ac92-4d75-a782-d0556e7655dc on / type btrfs (rw)

Remove the old ext4 snapshot and defragment:

btrfs subvolume list /
ID 256 top level 5 path ext2_saved

btrfs subvolume delete /ext2_saved
Delete subvolume '//ext2_saved'

btrfs subvolume list /
btrfs filesystem defragment /

Next I thought I’d try out the the yum plugin that takes a snapshot every time you run yum, so you can rollback (Debian equivalent is apt-btrfs-snapshot):

yum install yum-plugin-fs-snapshot.noarch

Then install Apache to try out the plugin:

yum install httpd
Loaded plugins: fs-snapshot
Setting up Install Process
Running Transaction
fs-snapshot: snapshotting /: /yum_20120204121740

Check for the snapshot, note its ID and reboot:

btrfs subvol list /
ID 256 top level 5 path yum_20120204121740


To boot into the snapshot made before you installed Apache, press “e” at grub prompt to edit the parameters and insert the following into the kernel line:


Then press “b” to boot. Once booted prove Apache isn’t installed anymore, and reboot back into the default snapshot:

rpm -qa |grep httpd

Check apache is installed again:

rpm -qa |grep httpd

Magic! So no we’ve converted an ext4 filesystem to btrfs on top of a LUKS partition, and have proven yum snapshots work and can boot into a snapshot without needing any backup/restore system.

The snapshot is just regular files, so could be tarred up and moved to an external drive for backup I guess:

ls /yum_20120204121740/
bin   dev  home  lib64       media  opt   root  selinux  sys  usr
boot  etc  lib   lost+found  mnt    proc  sbin  srv      tmp  var

I tried to simulate a catastrophic drive failure – i.e. if you’ve got a tarball of a snapshot, can you restore that to a blank disk with empty ext4 /boot and btrfs / partitions? Well the answer is no if you’re using LUKS. By using dd to backup the MBR and /boot partition I got somewhere, however when I restored the snapshot (untarred it into an empty btrfs filesystem) it booted without prompting for the LUKS key, and then died.

So it still seems a bit like btrfs is for rolling back to a previous point, or to be used with one or more drives in a RAID array, but no use when a single drive fails.

Update: I found that because btrfs stores the BLKID of the disk in every data block, you could never actually move a btrfs filesystem to another disk! The only way to do it (as I have) is to dd the entire disk to another disk, and then they’re BLKID’s will be the same, also it solves the problem of trying to piece together the /boot sector and grub in the MBR.

The problem however, is that as you can’t mount two devices with the same BLKID’s at the same time, you couldn’t use rsync to sync the two disks. So for backup of btrfs, you’re stuck with dd’ing the entire disk, or don’t use LUKS and use hardware RAID (not btrfs RAID or software RAID) instead, which is pretty crap.

Upgrades Galore

I fitted my new SSD to my fileserver yesterday as it was a rainy Sunday afternoon. Oddly enough the new 2.5″-to-3.5″ drive rails I got don’t fit in a floppy bay – well they do but the screw holes won’t line up, so I fitted it in my one remaining hard disk bay.

Anyway I was surprised how quickly I replaced the Ubuntu 9.10 setup with Debian 6.0.3 without losing any functionality. I decided to stick to Squeeze+Backports as Wheezy like on my desktop machine is way too much maintenance for a fileserver – I can’t cope with the “apt-get upgrade” fear! 😉

Speaking of backports, to replace with LibreOffice, you need to run this and answer “yes” to the dependency questions:

apt-get -t squeeze-backports install libreoffice libreoffice-gtk

Anyway the main thing I was worrying about – the printer/scanner was truly plug’n’play – I turned it on to do some scanning and CUPS automatically configured the printer part, and SANE just worked. None of the Epkowa (iscan+pips) Epson proprietary crap required.

I encrypted the boot drive using LUKS+LVM so I only need to enter the passphrase once, that seemed a lot easier than when I installed Wheezy and did multiple partitions.

I copied across the fstab and /etc/exports and all the various disks mounted and shared over NFS to the Mac seamlessly. I literally rebuilt the fileserver in two hours! Plus now it is all encrypted I can use it as a backup desktop machine for work.

Next up was the Mac Mini, currently running Leopard 10.5.8, I decided for £21 I might as well upgrade to Lion 10.7.2 as I already have 2Gb RAM and a Core2Duo, and apparently the new version of Plex doesn’t work on 10.5

Luckily I had a Snow Leopard 10.6.8 install in a virtual machine, so I bought Lion via the App Store (basically iTunes) using that. Wow the App Store is crap – I had to sign in about 6 times, I guess they’ve not heard of sessions at Apple.

I then used these instructions to create a bootable USB disk to do a fresh install of Lion – all within VirtualBox.

I’m actually dual booting Leopard and Lion using these instructions. Shrinking the disk so I could add a partition in the free space took the longest, installation was about 25mins. I’m glad I did it actually as although Lion runs fine (except it doesn’t like etherwake) the latest Plex is rubbish, so I’m booting Leopard and Plex at the moment.

Root Drive Encryption

I was pondering what to do about disk encryption on an unattended server. If you want an uninterrupted boot sequence, you can’t use full root disk encryption or it will ask you for your passphrase during init.

You can apparently do some fudge to put dropbear (mini SSH daemon and shell) into your initrd, so you can enter your passphrase over SSH, but that seemed hacky. I’d rather a serial console on a terminal server, for those servers with serial ports or LOM’s.

So I created a Fedora 12 virtual machine to play with my options. I decided to not encrypt the entire boot drive, only the /home partition, and then have a second backup drive entirely encrypted, neither of which are mounted at boot. That way you can use the machine as a remote server, reboot it, power off and back on using Wake-On-Lan, even allow other users to login; and when you have physical access you can use cryptsetup+mount to open your $HOME and backup partitions.

However, no matter what I tried, I kept getting prompted on boot for the passphrase, even though the fstab and crypttab were empty. Upon closer inspection of the files within initrd, it seems that the way of detecting root drive encryption and creating a passphrase prompt is not very picky. The routine ignores the fact that your backup drive does not have the bootable flag set, and is not the boot drive in the fstab or BIOS for that matter, and just assumes that if the entire drive is encrypted it must need opening to allow booting!

Short of recreating my initrd upon every new kernel install (don’t get me started on how shite dracut is compared to mkinitrd) the only way around it I found is to add the kernel parameter rd_NO_LUKS to the /boot/grub/menu.lst

I also header from one of the Fedora devs, that they’re going to move to upstart instead of init (see previous post) by Fedora 14. Hopefully the Ubuntu devs will have fixed the bugs by then!