Wednesday, 20 April 2011

Development and build environments: the search -- part 1

One of the problems I often face is keeping my productive system I use for work, separate from my development environment with broken versions of programs and loads of extra packages installed.

For a while my solution was to create virtual machines in VirtualBox - which works great but has a very high overhead for the task. Especially when I am away from my desktop and only have my slightly underpowered laptop (no VT-x extensions, only 2GB RAM), running an entire virtualized system just to try out some packaging changes was painful.

So recently I started experimenting with some other solutions. My requirements were
  • low system requirements
  • easy full network access in the guest
  • able to run X apps (I like to have a development environment where I run KDevelop, plus most of the apps I develop are graphical)
  • process isolation (don't want a bad command ruining my host system)
  • file-level isolation (don't want my host polluted with loads of debuginfo, devel, and unstable packages/files)
  • at the same time, it should be able to share files if necessary
  • minimum of modification to the host system 
Probably the ideal solution would be something like FreeBSD jails, but I'm running Linux :) Of course I could just run many installations on the same computer, but rebooting all the time is very tiresome.

My first experiment was with Xen. Since my laptop doesn't have hardware virtualization extensions (even though it is a relatively recent Intel cpu, it's a cheap one), I had to use paravirtualization. Xen dom0 (host) needs a different kernel, but this was easy in openSUSE (Novell have traditionally supported Xen) install and setup went fine - in fact YaST has a module "Install Hypervisor and Tools". I was able to create and install a VM via virt-manager. But even paravirtualization was really, really slow on this machine - and interacting with the VM via vnc was quite awkward. All in all no better than the full virtualization I had before.

Next I tried LXC - Linux Containers. They are basically super-chroots, using the power of the new kernel cgroups. Initially it looked like exactly what I needed - similar to FreeBSD jails. Overall it is comparable to OpenVZ (I didn't try OpenVZ because it needs a special kernel), almost but not quite virtualization (also called "operating system level virtualization"). It is a very new project though, and it showed, as there were no easy frontends to configure and run them. libvirt / virt-manager was supposed to have support for LXC, but I couldn't get it to work quite satisfactorily. Also, the OS running inside it needs heavy modification as LXC doesn't present any hardware by default (just a chroot, remember). To get the installed guest OS to boot, all the init scripts need to be gutted to remove anything trying to interact with hardware directly (HW clock, console, display, udev, hal). This was a painful process, and I couldn't get it quite right even with some extensive (but out of date?) documentation on the openSUSE wiki and elsewhere. On the plus side the hardware requirements of this were very low, as you would expect for a chroot. Perhaps in the future, when easy frontends exist, and distro's have versions suitable for installation in LXC, this would be an excellent solution.

So the chroot-based LXC was almost what I needed - I looked around for other chroot-based systems. The openSUSE "build" script sets up chroots for build environments but is too focused on that, it doesn't give you much help for using the chroots for anything else. I found that ssh can automatically chroot users on login, so I tried setting up a "devel" user, configuring appropriately, and ssh'ing into localhost. But the purpose of ssh's chroots is security, so it doesn't help much with the task here either.

I tried writing my own chroot management scripts. What is needed is to mount various directories (/dev, /sys, /proc) into the chroot so applications have access to the kernel/hardware, and possibly copy over some settings (e.g. /etc/resolv.conf for network access). It should also have some cleanup functions so that when the work in the chroot is done, it removes these mounts, ensures there aren't any stray processes, and leaves the host system in a consistent state. I had a few attempts at this, but it turned out to be harder than it seemed. I figured someone else must have experienced this issue, this is Linux after all, there must be something that does what I need. After spending most of my time looking at RPM-based distro solutions to this problem, I had the idea to look at Debian - and they seemed to have exactly what I needed. One of the Debian buildd-tools is a program called schroot.

In part 2 I cover how I got schroot working on openSUSE and my experiences with it so far.

No comments:

Post a Comment

Creative Commons Licence
This work by Tejas Guruswamy is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.