Full System Backup (and restore)

I thought I’d play around with backups as a continuation of my btrfs experiment. Well basically btrfs can’t do it without dd’ing the entire drive as the UUID’s can’t be changed.

So I thought I’d stick to ext4, and I’ve finally figured out how to do full root filesystem backup including LUKS encryption (without LUKS its easy, you could even do it with tar) using rsync of just the files instead of using dd to backup every bit (17Gb of files instead of 64Gb drive size in my case).

All of the backup section can be done from a live running system.

1. Backup the filesystem UUID’s:

blkid > uuids.txt

2. Backup full MBR (446 boot sector + 64 partition table + 2 sig) and extended partition info:

dd if=/dev/sda of=mbr.img count=1 bs=512
sfdisk -d /dev/sda > sda.sf

3. rsync the root filesystem to a remote server, or external disk, whatever. Obviously only do this to an encrypted filesystem as you’re copying the unencrypted files:

rsync -avp -P --numeric-ids --delete --delete-excluded \
    --exclude-from=excludes.txt / root@ip:/path/

My excludes.txt file looks like this:

# Include
+ /dev/console
+ /dev/initctl
+ /dev/null
+ /dev/zero

# Exclude
- /dev/*
- /proc/*
- /sys/*
- /tmp/*
- /media/*
- lost+found/
- .gvfs/
- .ccache/

That’s the backup done, now to restore to a new disk, do all of this from a live CD, you’ll have to scp the mbr.img and sda.sf files across.

4. Restore the MBR and partition info to a new blank disk:

dd if=mbr.img of=/dev/sda count=1 bs=512
sfdisk /dev/sda < sda.sf

If you want to restore just the boot sector and not the partition table, for example if your disks are different sizes, just change the block count, but you’ll have to use fdisk to create the partitions then:

dd if=mbr.img of=/dev/sda count=1 bs=446

5. You’ll have to format the filesystems and setup LUKS, but you don’t need to create the partitions using fdisk:

mkfs.ext4 /dev/sda1
cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 sda2_crypt
mkfs.ext4 /dev/mapper/sda2_crypt

Now for the complicated part. If you just dd the entire disk, then all the partitions would keep the same UUID’s, but as we’ve created new partitions on a blank disk, we have to reset them to the values we captured in #1, this is the part that’s not possible on btrfs.

6. Reset LUKS UUID – for this your boot CD will need to have a newer cryptsetup than 1.1.3 found on F14 which doesn’t have the ability to change UUID, cryptsetup 1.3.1 as on F16 LiveCD worked here:

cryptsetup luksUUID /dev/sda2 --uuid=12c92874-51ee-11e1-9c56-001d7d00626d

7. Reset root (/) ext4 partition UUID:

tune2fs /dev/mapper/sda2_crypt -U 19086ed4-51ee-11e1-864d-001d7d00626d

8. Reset /boot ext4 partition UUID:

tune2fs /dev/sda1 -U 23162d3a-51ee-11e1-b203-001d7d00626d

9. Finally we need to reinstall grub (I’m not sure why as we used dd on the MBR). So we mount root, and inside that mount /boot, and as we’re not backing up /dev, we need to mount the live CD’s /dev inside root too, then make that a chroot:

mount /dev/mapper/sda2_crypt /mnt
mount /dev/sda1 /mnt/boot
mount --bind /dev /mnt/dev
chroot /mnt
grub-install /dev/sda

That’s it, done. Reboot into your cloned system.

Update: If you have an encrypted swap partition, on first boot you’ll also need to run something like:

cryptsetup luksOpen /dev/sda3 sda3_swap
mkswap /dev/mapper/sda3_swap -U 9421cbe2-559f-11e1-9ec6-001d7d00626d

I’ve just restored my desktop machine into a VirtualBox VM using this method, including shrinking the disk from a 64Gb SSD to a 40Gb virtual disk (as only about 17Gb was used) and it works fine – encrypted swap and root (ext4). Screenshot.

On first boot I made some small modifications to /etc/hosts and /etc/sysconfig/networks to change IP, mac and hostname, and deleted /etc/udev/rules.d/70-persistent-net.rules so it would rescan for eth0 on reboot.

I also ran nvidia-installer --uninstall and deleted /etc/X11/xorg.conf to reconfigure Xorg. Of course if I rsynced the physical machine to the VM, I’d have to redo these steps again, so I’ve added some files to excludes.txt (such as guest additions).

Btrfs Experiment

After watching this video and reading this blog post, I decided to have a play with the new btrfs (ButterFS) filesystem.

So I downloaded OEL6u2 and made a minimal install in VirtualBox (which doesn’t even include wget or scp!)

I decided not to use LVM, but to use LUKS encryption for my 8Gb ext4 /, a 500Mb ext4 /boot with no encryption, and no swap partition.

Upon first boot I installed the Base Yum repo from here which is essentially just what’s on the DVD, and also the beta repo from here. I then installed the Unbreakable2 kernel and btrfs:

yum -y update kernel-uek
yum -y install btrfs-progs openssh wget
reboot

I then booted from the Fedora 16 LiveCD (not the install DVD) to do the ext4-to-btrfs conversion, as the OEL installer hasn’t yet been updated to include btrfs, although eventually it will allow direct btrfs root installation.

Next I had to decrypt the LUKS partition and do the conversion of the underlying ext4 filesystem:

cryptsetup luksOpen /dev/sda2 luks_root
btrfs-convert /dev/mapper/luks_root

As you need to edit the fstab, make a temporary directory to mount your root filesystem inside root’s $HOME directory:

cd /root
mkdir root
mount /dev/mapper/luks_root root/
vi root/etc/fstab

Simply replace “ext4″ with “btrfs” on the “/” line and :wq

For some reason when I tried to boot at this stage I got all sorts of permissions issues and it wouldn’t boot. The fix is to disable SELinux:

vi root/etc/selinux/config
SELINUX=disabled
reboot

Prove we’ve booted into our btrfs root filesystem:

mount | grep btrfs
/dev/mapper/luks-f3b07133-ac92-4d75-a782-d0556e7655dc on / type btrfs (rw)

Remove the old ext4 snapshot and defragment:

btrfs subvolume list /
ID 256 top level 5 path ext2_saved
 
btrfs subvolume delete /ext2_saved
Delete subvolume '//ext2_saved'
 
btrfs subvolume list /
btrfs filesystem defragment /

Next I thought I’d try out the the yum plugin that takes a snapshot every time you run yum, so you can rollback (Debian equivalent is apt-btrfs-snapshot):

yum install yum-plugin-fs-snapshot.noarch

Then install Apache to try out the plugin:

yum install httpd
Loaded plugins: fs-snapshot
Setting up Install Process
....
Running Transaction
fs-snapshot: snapshotting /: /yum_20120204121740
....
Complete!

Check for the snapshot, note its ID and reboot:

btrfs subvol list /
ID 256 top level 5 path yum_20120204121740
 
reboot

To boot into the snapshot made before you installed Apache, press “e” at grub prompt to edit the parameters and insert the following into the kernel line:

rootflags=subvolid=256

Then press “b” to boot. Once booted prove Apache isn’t installed anymore, and reboot back into the default snapshot:

rpm -qa |grep httpd
reboot

Check apache is installed again:

rpm -qa |grep httpd
httpd-tools-2.2.15-15.0.1.el6.x86_64
httpd-2.2.15-15.0.1.el6.x86_64

Magic! So no we’ve converted an ext4 filesystem to btrfs on top of a LUKS partition, and have proven yum snapshots work and can boot into a snapshot without needing any backup/restore system.

The snapshot is just regular files, so could be tarred up and moved to an external drive for backup I guess:

ls /yum_20120204121740/
bin   dev  home  lib64       media  opt   root  selinux  sys  usr
boot  etc  lib   lost+found  mnt    proc  sbin  srv      tmp  var

I tried to simulate a catastrophic drive failure – i.e. if you’ve got a tarball of a snapshot, can you restore that to a blank disk with empty ext4 /boot and btrfs / partitions? Well the answer is no if you’re using LUKS. By using dd to backup the MBR and /boot partition I got somewhere, however when I restored the snapshot (untarred it into an empty btrfs filesystem) it booted without prompting for the LUKS key, and then died.

So it still seems a bit like btrfs is for rolling back to a previous point, or to be used with one or more drives in a RAID array, but no use when a single drive fails.

Update: I found that because btrfs stores the BLKID of the disk in every data block, you could never actually move a btrfs filesystem to another disk! The only way to do it (as I have) is to dd the entire disk to another disk, and then they’re BLKID’s will be the same, also it solves the problem of trying to piece together the /boot sector and grub in the MBR.

The problem however, is that as you can’t mount two devices with the same BLKID’s at the same time, you couldn’t use rsync to sync the two disks. So for backup of btrfs, you’re stuck with dd’ing the entire disk, or don’t use LUKS and use hardware RAID (not btrfs RAID or software RAID) instead, which is pretty crap.

Acer Revo 3610 mplayer 1080p Fix

I finally figured out how to get Full-HD to play on my Revo, that includes .m2t files from my HD-PVR card, not just .mkv files. Add this to ~/.mplayer/config

[default]
# Write your default config options here!
vo=vdpau,xv,
vc=ffh264vdpau,ffmpeg12vdpau,
demuxer=lavf

It forces VDPAU on for h.264 files, and falls back to X11 for DivX etc. Also LAVF as the demuxer seems to help with audio syncing. It’s not 100% smooth like 720p or below, but its 90% there.

As I put it in the default section, it also works for gnome-mplayer, smplayer and mplayer-gui, now works better than VideoLAN.

Still can’t get Flash 11 to play even 480p at any decent framerate, but we can blame Adobe for taking out Linux acceleration support – not that it worked well in Flash 10 anyway.

I also added this to /etc/X11/xorg.conf to make sure that Compiz wasn’t enabled:

Section "Extensions"
	Option "Composite" "Disable"
EndSection

Also added the highlighted line to the Device section to turn on triple-buffering to allocate more memory to 2D buffers:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "ION"
    Option	   "TripleBuffer" "True"EndSection

I also removed the RPMFusion-packaged Nvidia drivers and used the latest NVIDIA-Linux-x86-295.09.run

Updating The Sun JDK On Debian

The final version of sun-jdk-6 in Debian Wheezy before it was removed was 6u26. The latest from Oracle (excluding Java7) is 6u30, so I decided to manually install it myself.

In fact it couldn’t be easier, all you do is download the jdk-6u30-linux-x64.bin file and run it, which extracts it to a directory called jdk1.6.0_30, which you then rename to java-6-sun-1.6.0_30 and move to /usr/lib/jvm, you then delete the java-6-sun symlink and recreate it to point to the newer directory:

root@localhost:/usr/lib/jvm$ ln -s java-6-sun-1.6.0_30 java-6-sun

As alternatives is setup to point to /usr/lib/jvm/java-6-sun there’s no need to update anything for the files we need to build Android for example:

root@localhost:/usr/lib/jvm$ update-alternatives --get-selections | grep java
...
appletviewer                   auto     /usr/lib/jvm/java-6-sun/bin/appletviewer
firefox-javaplugin.so          auto     /usr/lib/jvm/java-6-sun/jre/lib/amd64/libnpjp2.so
java                           manual   /usr/lib/jvm/java-6-sun/jre/bin/java
javac                          auto     /usr/lib/jvm/java-6-sun/bin/javac
javadoc                        auto     /usr/lib/jvm/java-6-sun/bin/javadoc
javah                          auto     /usr/lib/jvm/java-6-sun/bin/javah

jar seems to have been replaced by fastjar on Debian, so there’s no need to link the Oracle version in /usr/lib/jvm/java-6-sun/bin/jar

Finally its a good idea to add this environment variable in ~/.bashrc – again pointing to the symlink, not the specific version:

export JAVA_HOME="/usr/lib/jvm/java-6-sun"

Apparently Cyanogenmod7 will compile using OpenJDK6 (and 7 with a patch) even though AOSP won’t compile with anything other than Sun JDK6.

I’ve also updated the blog to WordPress 3.3

Upgrades Galore

I fitted my new SSD to my fileserver yesterday as it was a rainy Sunday afternoon. Oddly enough the new 2.5″-to-3.5″ drive rails I got don’t fit in a floppy bay – well they do but the screw holes won’t line up, so I fitted it in my one remaining hard disk bay.

Anyway I was surprised how quickly I replaced the Ubuntu 9.10 setup with Debian 6.0.3 without losing any functionality. I decided to stick to Squeeze+Backports as Wheezy like on my desktop machine is way too much maintenance for a fileserver – I can’t cope with the “apt-get upgrade” fear! ;-)

Speaking of backports, to replace OpenOffice.org with LibreOffice, you need to run this and answer “yes” to the dependency questions:

apt-get -t squeeze-backports install libreoffice libreoffice-gtk

Anyway the main thing I was worrying about – the printer/scanner was truly plug’n'play – I turned it on to do some scanning and CUPS automatically configured the printer part, and SANE just worked. None of the Epkowa (iscan+pips) Epson proprietary crap required.

I encrypted the boot drive using LUKS+LVM so I only need to enter the passphrase once, that seemed a lot easier than when I installed Wheezy and did multiple partitions.

I copied across the fstab and /etc/exports and all the various disks mounted and shared over NFS to the Mac seamlessly. I literally rebuilt the fileserver in two hours! Plus now it is all encrypted I can use it as a backup desktop machine for work.

Next up was the Mac Mini, currently running Leopard 10.5.8, I decided for £21 I might as well upgrade to Lion 10.7.2 as I already have 2Gb RAM and a Core2Duo, and apparently the new version of Plex doesn’t work on 10.5

Luckily I had a Snow Leopard 10.6.8 install in a virtual machine, so I bought Lion via the App Store (basically iTunes) using that. Wow the App Store is crap – I had to sign in about 6 times, I guess they’ve not heard of sessions at Apple.

I then used these instructions to create a bootable USB disk to do a fresh install of Lion – all within VirtualBox.

I’m actually dual booting Leopard and Lion using these instructions. Shrinking the disk so I could add a partition in the free space took the longest, installation was about 25mins. I’m glad I did it actually as although Lion runs fine (except it doesn’t like etherwake) the latest Plex 0.9.5.1 is rubbish, so I’m booting Leopard and Plex 0.9.3.4 at the moment.