Compiling CyanogenMod 9

Posted on by

I initally added instructions here for compiling ColdFusionX as I was helping to maintain the project. Now that it looks like we’ll get an official CM9 for ZTE Blade, its been deprecated, so here’s the CM9 build instructions. It takes about 50mins to compile (including kernel) which is only about 10mins more than CM7+kernel would have taken on my 12Gb RAM, 3.2GHz Intel Core i5 Linux box with SSD.

1. Download CM9 (use repo sync to update) this is about 15Gb worth:

mkdir -p ~/cm9/vendor/zte/blade
curl https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo
chmod a+x ~/bin/repo
cd ~/cm9
repo init -u git://github.com/CyanogenMod/android.git -b ics

2. Download ZTE proprietary libs (use git pull to update) and copy into place. you’ll probably only have to do this once as they don’t get updated often:

cd
git clone https://github.com/koush/proprietary_vendor_zte.git -b ics
rsync -av --delete ~/proprietary_vendor_zte/blade/ ~/cm9/vendor/zte/blade/ --exclude='.git*'

3. Compile the rom – the first time you do this it will download the kernel source (you can add CM_EXTRAVERSION=-whatever on the end of the export line):

cd ~/cm9/vendor/cm/
./get-prebuilts
cd ~/cm9/
export USE_CCACHE=1 CM_SNAPSHOT=1
. build/envsetup.sh
lunch cm_blade-userdebug
make bacon -j8

Nessus 5.0 Review

Posted on by

Nessus 5.0 just got released, and if the forum is anything to go by, people are not impressed, me included!

First off its a major new version number, but appears to have no new functionality whatsoever. All that has changed are the report templates, and they’re totally screwed up. You’ve got HTML that doesn’t wrap properly and takes an age to render the XSLT, the Synopsis and Solution are no longer output at all (WTF?!) and the PDF export relies on Oracle Java, who knows why they didn’t use LaTeX or something.

There’s reports of people not being able to upgrade, downgrade or even install on various flavours of Linux, MacOSX and Windows; let alone do offline updates/activation.

There’s bugs in the Flash that prevent people even getting as far as the login screen, or being able to filter the plugin list – even after two updates in the feed already! I have no idea why it still uses Flash and not HTML5/Ajax like the rest of the planet.

The severity levels have changed – Low/None seem to become Info depending on what function you call and script attribute you set, they are different to 4.x and there’s a new Critical severity.

I know you should never use a x.0 release of any software, but seriously, this should really be called 4.5beta, not 5.0. If you thought upgrading from 4.2.2 to 4.4.1 was hardly worth it, then you won’t want to bother with this.

I’ve quickly grabbed all of the 4.4.1 installers before they remove them, as support for all but the very latest Linux distro’s has been ditched in 5.0, as have the generic tarballs.

Nessus is seriously going in the wrong direction, its trying to appeal to PHB’s when that should be the realm of Security Center. Users want new functionality like better IPv6/SCTP support and VoIP fuzzing, not bling.

If nmap‘s Lua scripting engine was more mature or OpenVAS was packaged a bit better, I’d be jumping ship.

Update: I’ve just tried 5.01 which I think has somehow managed to get even worse – report upload doesn’t show the upload window half the time (Chrome 20 or Firefox 13) and when installing for some reason it recreated the database cache (a long process) then fetched the new plugins and re-cached the database!

Full System Backup (and restore)

Posted on by

I thought I’d play around with backups as a continuation of my btrfs experiment. Well basically btrfs can’t do it without dd’ing the entire drive as the UUID’s can’t be changed.

So I thought I’d stick to ext4, and I’ve finally figured out how to do full root filesystem backup including LUKS encryption (without LUKS its easy, you could even do it with tar) using rsync of just the files instead of using dd to backup every bit (17Gb of files instead of 64Gb drive size in my case).

All of the backup section can be done from a live running system.

1. Backup the filesystem UUID’s:

blkid > uuids.txt

2. Backup full MBR (446 boot sector + 64 partition table + 2 sig) and extended partition info:

dd if=/dev/sda of=mbr.img count=1 bs=512
sfdisk -d /dev/sda > sda.sf

3. rsync the root filesystem to a remote server, or external disk, whatever. Obviously only do this to an encrypted filesystem as you’re copying the unencrypted files:

rsync -avp -P --numeric-ids --delete --delete-excluded \
    --exclude-from=excludes.txt / root@ip:/path/

My excludes.txt file looks like this:

# Include
+ /dev/console
+ /dev/initctl
+ /dev/null
+ /dev/zero

# Exclude
- /dev/*
- /proc/*
- /sys/*
- /tmp/*
- /media/*
- lost+found/
- .gvfs/
- .ccache/

That’s the backup done, now to restore to a new disk, do all of this from a live CD, you’ll have to scp the mbr.img and sda.sf files across.

4. Restore the MBR and partition info to a new blank disk:

dd if=mbr.img of=/dev/sda count=1 bs=512
sfdisk /dev/sda < sda.sf

If you want to restore just the boot sector and not the partition table, for example if your disks are different sizes, just change the block count, but you’ll have to use fdisk to create the partitions then:

dd if=mbr.img of=/dev/sda count=1 bs=446

5. You’ll have to format the filesystems and setup LUKS, but you don’t need to create the partitions using fdisk:

mkfs.ext4 /dev/sda1
cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 sda2_crypt
mkfs.ext4 /dev/mapper/sda2_crypt

Now for the complicated part. If you just dd the entire disk, then all the partitions would keep the same UUID’s, but as we’ve created new partitions on a blank disk, we have to reset them to the values we captured in #1, this is the part that’s not possible on btrfs.

6. Reset LUKS UUID – for this your boot CD will need to have a newer cryptsetup than 1.1.3 found on F14 which doesn’t have the ability to change UUID, cryptsetup 1.3.1 as on F16 LiveCD worked here:

cryptsetup luksUUID /dev/sda2 --uuid=12c92874-51ee-11e1-9c56-001d7d00626d

7. Reset root (/) ext4 partition UUID:

tune2fs /dev/mapper/sda2_crypt -U 19086ed4-51ee-11e1-864d-001d7d00626d

8. Reset /boot ext4 partition UUID:

tune2fs /dev/sda1 -U 23162d3a-51ee-11e1-b203-001d7d00626d

9. Finally we need to reinstall grub (I’m not sure why as we used dd on the MBR). So we mount root, and inside that mount /boot, and as we’re not backing up /dev, we need to mount the live CD’s /dev inside root too, then make that a chroot:

mount /dev/mapper/sda2_crypt /mnt
mount /dev/sda1 /mnt/boot
mount --bind /dev /mnt/dev
chroot /mnt
grub-install /dev/sda

That’s it, done. Reboot into your cloned system.

Update: If you have an encrypted swap partition, on first boot you’ll also need to run something like:

cryptsetup luksOpen /dev/sda3 sda3_swap
mkswap /dev/mapper/sda3_swap -U 9421cbe2-559f-11e1-9ec6-001d7d00626d

I’ve just restored my desktop machine into a VirtualBox VM using this method, including shrinking the disk from a 64Gb SSD to a 40Gb virtual disk (as only about 17Gb was used) and it works fine – encrypted swap and root (ext4). Screenshot.

On first boot I made some small modifications to /etc/hosts and /etc/sysconfig/networks to change IP, mac and hostname, and deleted /etc/udev/rules.d/70-persistent-net.rules so it would rescan for eth0 on reboot.

I also ran nvidia-installer --uninstall and deleted /etc/X11/xorg.conf to reconfigure Xorg. Of course if I rsynced the physical machine to the VM, I’d have to redo these steps again, so I’ve added some files to excludes.txt (such as guest additions).

Btrfs Experiment

Posted on by

After watching this video and reading this blog post, I decided to have a play with the new btrfs (ButterFS) filesystem.

So I downloaded OEL6u2 and made a minimal install in VirtualBox (which doesn’t even include wget or scp!)

I decided not to use LVM, but to use LUKS encryption for my 8Gb ext4 /, a 500Mb ext4 /boot with no encryption, and no swap partition.

Upon first boot I installed the Base Yum repo from here which is essentially just what’s on the DVD, and also the beta repo from here. I then installed the Unbreakable2 kernel and btrfs:

yum -y update kernel-uek
yum -y install btrfs-progs openssh wget
reboot

I then booted from the Fedora 16 LiveCD (not the install DVD) to do the ext4-to-btrfs conversion, as the OEL installer hasn’t yet been updated to include btrfs, although eventually it will allow direct btrfs root installation.

Next I had to decrypt the LUKS partition and do the conversion of the underlying ext4 filesystem:

cryptsetup luksOpen /dev/sda2 luks_root
btrfs-convert /dev/mapper/luks_root

As you need to edit the fstab, make a temporary directory to mount your root filesystem inside root’s $HOME directory:

cd /root
mkdir root
mount /dev/mapper/luks_root root/
vi root/etc/fstab

Simply replace “ext4″ with “btrfs” on the “/” line and :wq

For some reason when I tried to boot at this stage I got all sorts of permissions issues and it wouldn’t boot. The fix is to disable SELinux:

vi root/etc/selinux/config
SELINUX=disabled
reboot

Prove we’ve booted into our btrfs root filesystem:

mount | grep btrfs
/dev/mapper/luks-f3b07133-ac92-4d75-a782-d0556e7655dc on / type btrfs (rw)

Remove the old ext4 snapshot and defragment:

btrfs subvolume list /
ID 256 top level 5 path ext2_saved
 
btrfs subvolume delete /ext2_saved
Delete subvolume '//ext2_saved'
 
btrfs subvolume list /
btrfs filesystem defragment /

Next I thought I’d try out the the yum plugin that takes a snapshot every time you run yum, so you can rollback (Debian equivalent is apt-btrfs-snapshot):

yum install yum-plugin-fs-snapshot.noarch

Then install Apache to try out the plugin:

yum install httpd
Loaded plugins: fs-snapshot
Setting up Install Process
....
Running Transaction
fs-snapshot: snapshotting /: /yum_20120204121740
....
Complete!

Check for the snapshot, note its ID and reboot:

btrfs subvol list /
ID 256 top level 5 path yum_20120204121740
 
reboot

To boot into the snapshot made before you installed Apache, press “e” at grub prompt to edit the parameters and insert the following into the kernel line:

rootflags=subvolid=256

Then press “b” to boot. Once booted prove Apache isn’t installed anymore, and reboot back into the default snapshot:

rpm -qa |grep httpd
reboot

Check apache is installed again:

rpm -qa |grep httpd
httpd-tools-2.2.15-15.0.1.el6.x86_64
httpd-2.2.15-15.0.1.el6.x86_64

Magic! So no we’ve converted an ext4 filesystem to btrfs on top of a LUKS partition, and have proven yum snapshots work and can boot into a snapshot without needing any backup/restore system.

The snapshot is just regular files, so could be tarred up and moved to an external drive for backup I guess:

ls /yum_20120204121740/
bin   dev  home  lib64       media  opt   root  selinux  sys  usr
boot  etc  lib   lost+found  mnt    proc  sbin  srv      tmp  var

I tried to simulate a catastrophic drive failure – i.e. if you’ve got a tarball of a snapshot, can you restore that to a blank disk with empty ext4 /boot and btrfs / partitions? Well the answer is no if you’re using LUKS. By using dd to backup the MBR and /boot partition I got somewhere, however when I restored the snapshot (untarred it into an empty btrfs filesystem) it booted without prompting for the LUKS key, and then died.

So it still seems a bit like btrfs is for rolling back to a previous point, or to be used with one or more drives in a RAID array, but no use when a single drive fails.

Update: I found that because btrfs stores the BLKID of the disk in every data block, you could never actually move a btrfs filesystem to another disk! The only way to do it (as I have) is to dd the entire disk to another disk, and then they’re BLKID’s will be the same, also it solves the problem of trying to piece together the /boot sector and grub in the MBR.

The problem however, is that as you can’t mount two devices with the same BLKID’s at the same time, you couldn’t use rsync to sync the two disks. So for backup of btrfs, you’re stuck with dd’ing the entire disk, or don’t use LUKS and use hardware RAID (not btrfs RAID or software RAID) instead, which is pretty crap.

Acer Revo 3610 mplayer 1080p Fix

Posted on by

I finally figured out how to get Full-HD to play on my Revo, that includes .m2t files from my HD-PVR card, not just .mkv files. Add this to ~/.mplayer/config

[default]
# Write your default config options here!
vo=vdpau,xv,
vc=ffh264vdpau,ffmpeg12vdpau,
demuxer=lavf

It forces VDPAU on for h.264 files, and falls back to X11 for DivX etc. Also LAVF as the demuxer seems to help with audio syncing. It’s not 100% smooth like 720p or below, but its 90% there.

As I put it in the default section, it also works for gnome-mplayer, smplayer and mplayer-gui, now works better than VideoLAN.

Still can’t get Flash 11 to play even 480p at any decent framerate, but we can blame Adobe for taking out Linux acceleration support – not that it worked well in Flash 10 anyway.

I also added this to /etc/X11/xorg.conf to make sure that Compiz wasn’t enabled:

Section "Extensions"
	Option "Composite" "Disable"
EndSection

Also added the highlighted line to the Device section to turn on triple-buffering to allocate more memory to 2D buffers:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "ION"
    Option	   "TripleBuffer" "True"EndSection

I also removed the RPMFusion-packaged Nvidia drivers and used the latest NVIDIA-Linux-x86-295.09.run