Vagrant & Ansible

I’ve been getting back into Vagrant and Ansible lately, as I decided I needed a platform to do some Continuous Integration testing of Arduino and packaging of arduino-mk; and also building Kodi on a faster platform than my Atom HTPC.

As luck would have it, the Debian package for Ansible 1.9.2 just hit Sid, so I don’t have to build my own from git with “make deb”, which is a bit Ubuntu-centric and doesn’t work too well on Debian.

So I’ve made a Fedora 22 VM that I can make arduino-mk packages on, an Ubuntu 12.04 VM that I’m building Kodi 15 packages on, and an Ubuntu 14.04 VM that I’m using for both. I also made a Debian Jessie and another Ubuntu Trusty VM along the way.

As I am using the new VirtualBox 5.0rc2, Vagrant 1.7.2 needs this patch.

The Precise VM (with 8 cores and 4Gb RAM assigned) builds Kodi in 35mins – about half the time it takes the Atom, which isn’t that impressive, but its an improvement.

Setting up an Ubuntu 12.04 build environment for Kodi again was a right PITA though, mainly as “apt-get build-dep” doesn’t actually fetch half the dependencies, and you need newer versions of libyajl2, libyajl-dev, taglib and JsonSchemaBuilder, as well as now having to install gcc 4.9 as they’ve just switched to C++11 today!

So that’s basically this lot wrapped in an Ansible playbook:

add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-get build-deps xbmc
apt-get install libssh-dev libxslt1-dev git libmp3lame-dev swig libcec-dev openjdk-7-jdk gcc-4.9 g++-4.9
dpkg -i libyajl2_2.0.4-4_amd64.deb libyajl-dev_2.0.4-4_amd64.deb libtag1x8_1.8-0precise17_amd64.deb
cp /home/vagrant/kodi/packaging/deps/JsonSchemaBuilder /usr/local/bin/JsonSchemaBuilder
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 60
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 40
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 60
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.6 40
update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/gcc-4.9 60
update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/gcc-4.6 40
update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/g++-4.9 60
update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/g++-4.6 40

Ubuntu 14.04 is much easier, and you just need some basic tasks in your Playbook.yml

- name: install dependencies
  apt: name=xbmc state=build-dep

- name: install libxslt
  apt: name=libxslt1-dev state=present

- name: install git
  apt: name=git state=present

- name: install libcec-dev
  apt: name=libcec-dev state=present

Junos 10

Today I have been mostly installing Junos. Well actually I’ve wasted most of the day trying to get Junos 10.4 to work in Olive under VirtualBox. I understood that it required FreeBSD 7.1, so tried installing it under 7.1 and 7.4 to no avail.

In the end I cloned my Junos 9.0/FreeBSD 4.11 VM, allocated 512Mb instead of 256Mb and installed 10.4 as an upgrade, which also meant I didn’t have to bother removing checkpic.

I wasted a few rounds of installing due to using the export version, which doesn’t include SSH! Also part of the trick of getting it to work under VBox seemed to be to create a serial port as a named pipe – not sure why but that seemed to help get past the bootloader hanging, possibly as it had a TTY to allocate.

I also upgraded my 9.0 to 9.6 which has a bit of a more useful JWeb interface, and also requires 512Mb now.

All of this was to aide my development of a set of NASL scripts to do Junos security compliance auditing. It seems Tenable have worked around the UNIX-only limitation of Nessus’ ssh_cmd() function by putting in a special check for when uname -a fails – i.e. its either IOS or Junos (or unsupported). Of course in Junos shell mode, it will pass (as its FreeBSD) so you have to check that you’re in CLI mode to do the config checking.

Its only taken them four years of me asking for this, and I guess its come as a result of Nessus’s new IOS support for their own compliance plugin and local security checks for Junos patches etc.

Update: I’ve written 20 NASL plugins to do the Junos auditing now and I noticed I was hitting the SSH rate-limit setting in Junos, so my plugins were getting booted off. It was because for each plugin I was calling ssh_cmd() at least once and also a function that checks I could login with the correct level/privileges etc; so was making at least two SSH connection attempts per plugin, which soon hit the 10 connection attempts per minute limit that was configured.

So now I’ve moved all of my ssh_cmd() calls into one big include file which uses a single SSH connection to send 30 or so commands, and populates the knowledgebase with the results. The plugins then have that in their script_dependencies() and don’t use SSH at all, just a couple of calls to get_kb_item() which simplifies the code quite a lot and an entire scan can be done in 10secs!

Upgrades Galore

I fitted my new SSD to my fileserver yesterday as it was a rainy Sunday afternoon. Oddly enough the new 2.5″-to-3.5″ drive rails I got don’t fit in a floppy bay – well they do but the screw holes won’t line up, so I fitted it in my one remaining hard disk bay.

Anyway I was surprised how quickly I replaced the Ubuntu 9.10 setup with Debian 6.0.3 without losing any functionality. I decided to stick to Squeeze+Backports as Wheezy like on my desktop machine is way too much maintenance for a fileserver – I can’t cope with the “apt-get upgrade” fear! 😉

Speaking of backports, to replace OpenOffice.org with LibreOffice, you need to run this and answer “yes” to the dependency questions:

apt-get -t squeeze-backports install libreoffice libreoffice-gtk

Anyway the main thing I was worrying about – the printer/scanner was truly plug’n’play – I turned it on to do some scanning and CUPS automatically configured the printer part, and SANE just worked. None of the Epkowa (iscan+pips) Epson proprietary crap required.

I encrypted the boot drive using LUKS+LVM so I only need to enter the passphrase once, that seemed a lot easier than when I installed Wheezy and did multiple partitions.

I copied across the fstab and /etc/exports and all the various disks mounted and shared over NFS to the Mac seamlessly. I literally rebuilt the fileserver in two hours! Plus now it is all encrypted I can use it as a backup desktop machine for work.

Next up was the Mac Mini, currently running Leopard 10.5.8, I decided for £21 I might as well upgrade to Lion 10.7.2 as I already have 2Gb RAM and a Core2Duo, and apparently the new version of Plex doesn’t work on 10.5

Luckily I had a Snow Leopard 10.6.8 install in a virtual machine, so I bought Lion via the App Store (basically iTunes) using that. Wow the App Store is crap – I had to sign in about 6 times, I guess they’ve not heard of sessions at Apple.

I then used these instructions to create a bootable USB disk to do a fresh install of Lion – all within VirtualBox.

I’m actually dual booting Leopard and Lion using these instructions. Shrinking the disk so I could add a partition in the free space took the longest, installation was about 25mins. I’m glad I did it actually as although Lion runs fine (except it doesn’t like etherwake) the latest Plex 0.9.5.1 is rubbish, so I’m booting Leopard and Plex 0.9.3.4 at the moment.

Rawhiiiiiiiiide!

No, not the cowboy series, but the Fedora development repository.

As part of my investigation into why large transfers are hard locking my PC, I was advised to install the kernel from Rawhide, which also meant enabling the RPMFusion Rawhide repository to pull in the Nvidia modules. So first we install the repo:

yum install fedora-release-rawhide.noarch

Then we edit the repo files to enable them but limit them to kernel/nvidia RPM’s (we don’t want to upgrade to Fedora 15 Alpha!) and not debuginfo/source:

/etc/yum.repos.d/fedora-rawhide.repo

[rawhide]
name=Fedora - Rawhide - Developmental packages for the next Fedora release
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/development/$basearch/os/
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
includepkgs=kernel*

/etc/yum.repos.d/rpmfusion-nonfree-rawhide.repo

[rpmfusion-nonfree-rawhide]
name=RPM Fusion for Fedora Rawhide - Nonfree
#baseurl=http://download1.rpmfusion.org/nonfree/fedora/development/$basearch/os/
mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=nonfree-fedora-rawhide&arch=$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmfusion-nonfree-fedora-latest-$basearch file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmfusion-nonfree-fedora-rawhide-$basearch
includepkgs=*nvidia*

Then running “yum update” pulled in the 2.6.38-0.rc7.git2.3.fc16.x86_64 kernel from Fedora 16 (and Nvidia modules) and it does seem to have fixed the issue – I did a full rsync backup and a couple of 1.3Gb NFS transfers to test.

VirtualBox works fine too thankfully – dkms/akmod rebuilt the kernel modules upon reboot and my CentOS 5.5/Ubuntu 9.10/OEL6 guest VM’s work fine.

Mum’s thinking of getting an Amazon Kindle3 for her birthday, and it seems they now allow reading of regular PDF/epub files not just Amazon’s DRM-protected Mobi format, and there’s a Linux program called Calibre that enables you to convert between the ebook formats, including .cbz comic books.

Update: the new kernel doesn’t seem to have fixed the issue, I’ve had several crashes today without much network traffic. Starting to wonder if its an Nvidia driver issue or a hardware issue like HDD/PSU/RAM. I ordered a new PCIe NIC with Via Velocity VT6130 chipset instead of Realtek RTL8111D but doubt its going to fix anything now 🙁

I’ve also been playing with the Kindle and Kobo Android apps and downloading a shedload of free ebooks from Amazon to read on my phone – even got Calibre to convert useless PDF’s to more portable epub files. The Kobo website is useless as its throws DRM-protected PDF’s at you (for free books!) but the Android app just downloads unencrypted epubs.

Compiling VirtualBox OSE (updated)

I’ve built a default install of Fedora 13 64-Bit in a virtual machine for the purpose of figuring out what are the dependencies and workarounds required to compile VirtualBox OSE from Subversion.

1. Install make, gcc etc; as root:

yum groupinstall "Development Tools" "Development Libraries"

2. Install 32-Bit build tools and some Qt4 libraries, Java etc; as root:

yum install dev86 iasl qt4-devel pulseaudio-libs-devel glibc-devel.i686 libgcc.i686 texlive-texmf-latex java-1.6.0-openjdk-devel zlib-static glibc-static libstdc++.i686 libvncserver-devel libxslt-devel libIDL-devel SDL-devel libXmu-devel libstdc++-static

3. Symlink the Fedora-packaged JVM to where Oracle expect it to be installed, as root:

cd /usr/lib/jvm/
ln -s java-1.6.0-openjdk.x86_64 java-6-openjdk

4. Install the bera-mono font into LaTeX, as root:

cd /usr/share/texmf/tex/latex/
mkdir bera
cd bera/
wget http://www.tug.org/texlive/devsrc/Master/texmf-dist/tex/latex/bera/beramono.sty
texhash

5. Download makeself, as a regular user:

cd /var/tmp/
wget http://megastep.org/makeself/makeself.run
sh makeself.run

6. Install and rename makeself.sh to makeself as Oracle are expecting, as root:

cd /var/tmp/makeself-2.1.5
mv makeself.sh makeself
cp makeself makeself-header.sh /usr/local/bin/

7. Compile VirtualBox, UserGuide.pdf, the kernel modules and the Guest Additions as a regular user:

cd ~/vbox/
svn update
./configure --disable-hardening
source env.sh
kmk all VBOX_WITH_VNC=1
cd out/linux.amd64/release/bin/src
make
cd ~/vbox/
kmk packing

Update: The same setup seems to work for Fedora 14 too.

Update 2: might as well remove the VBOX_WITH_VNC=1 and installation of libvncserver-devel as VNC support in OSE is going to be broken now the new VRDE API is committed.