Interview: Linux-VServer Project Leader Herbert Pötzl

Herbert PötzlI use both Linux-VServer and OpenVZ at work for server virtualization and they both work fantastically. Since I've already done an interview with the head of the OpenVZ project, I thought it was time for an interview with the head of the Linux-VServer project. I ran across Herbert Pötzl on the Linux-VServer IRC channel (#vserver on irc.oftc.net) and he was kind enough to grant me this interview. Herbert is hereafter referred to by his IRC nick, Bertl.

About the Linux-VServer

ML: Please tell me a little bit about yourself... education, hobbies, family, employment, etc?

Bertl: After finishing the Secondary School (focus on Natural Sciences). I started studying Physics and, once available, Computer Sciences. Finished with a DIPL.-ING. in CS (equivalent of M.S. degree). I'm still working on my PhD Thesis.

My hobbies include, but are not limited to (besides coding): Juggling, Billiards, The Movies, Electronics.

I'm currently self employed as IT Consultant, and lucky me, my beautiful wife is an artist and IT expert.

ML: For those unfamiliar with Linux-VServer, could you please provide a brief description of what it is?

Bertl: Linux-VServer is an isolation technique in concept very similar to BSD Jails or Solaris Containers, which allows multiple Linux environments to run on a single kernel side by side, with no measurable overhead.

ML: How long have you been working on Linux-VServer and how did you get started?

Bertl: I started as an simple user back when the project was called 'Linux Security Contexts', maintained by Jacques Gelinas. Everything back then was very rough and edgy, many possible exploits, no resource management, no SMP support.

But I liked the idea of the Project and soon I had a bunch of patches sitting on my desk, improving this behavior or adding that feature.

In Summer 2003, Jacques disappeared and the project completely stalled. A few months later, I volunteered to temporarily take over the Project Maintainership until Jacques' return. After he reappeared, and it became apparent that he didn't have the time to continue development, I officially became the new maintainer.

ML: I know that you prefer users to build their own Linux-VServer kernel and tools although the project does offer binaries for several Linux distributions... and Debian also offers binary kernels with Linux-VServer support. Do you think the build-it-yourself method increases the barrier of entry for new users and if not why not?

Bertl: There are binaries and pre-compiled kernel images available for most distros nowadays, but while a distro kernel definitely lowers the entry barrier, I'm not convinced that isolation or virtualization should be treated like a word processor or graphics program ... you definitely should know what you are doing if you set up a 150+ guest hosting platform :)

Besides that, a distro kernel has to handle quite a number of corner cases, and we already see where this leads to. If you look at typical kernel packages, they are not only huge but also come in 20+ flavors (desktop 2/4/16GB, server 4/16/64GB).

Compiling the kernel yourself not only increases the quality of the resulting kernel (because you can avoid all the corner cases), it also improves security and stability to some extent, and it's just great if you can tell your friends that you 'rolled your own kernel', isn't it? :)

ML: One of the unique features (in my mind) of Linux-VServer is unification. Could you please provide a brief description of what unification is and list any potential benefits or drawbacks?

Bertl: I think there are quite a number of 'unique' features in Linux-VServer, unification is only a small puzzle piece to achieve high density and efficient resource usage. ... but to get back to your question:

Typical hosting scenarios have a large number of quite similar guests — think fifty times Debian and forty times Fedora — so it can be assumed that they will share the very same files, like libraries and executables and only differ in configuration and 'user data'. Typically, those common files will not only take up a lot of disk space, each file will also consume a certain amount of memory (for caching and mapping).

Fortunately, both can be significantly reduced by using a well known Unix feature, called hard links — having more than one pointer to the very same inode not only reduces the amount of disk space but also eliminates the need for separate caches and mappings.

The idea was quite simple, but the implementation took a long time to get perfected. Today we not only have Unification, but also Copy on Write (CoW) Link Breaking.

ML: Have you had any discussions with any Linux distribution makers about including Linux-VServer support in their releases? What distros currently include a Linux-VServer kernel option?

Bertl: We are not approaching distribution makers to 'include' Linux-VServer (like companies would do), nevertheless, Tools and Kernel packages have made it into most Distros by now.

As far as I know, the following distributions have some kind of Linux-VServer package/option available:

ALT Linux, Arch Linux, Debian, Fedora, Gentoo, Knoppix, Mandriva, PLD Linux, Rock Linux, Slackware, T2, and Ubuntu

ML: Any idea how many people are using Linux-VServer? Care to mention any large deployments that you are aware of?

Bertl: I don't have any details here, but I think there are quite a number of users out there, mainly because Linux-VServer is really versatile and not limited to hosting, as it might seem at first glance.

Large Deployments are definitely PlanetLab and Lycos Europe, but I also heard rumors about Cisco and other larger companies. As we do not require any registration to use the software, there is no real way to tell, and personally, I do not care that much about the numbers.

ML: Can you tell me a little bit about how the development team works together? How does the team communicate? What version control software do you use?

Bertl: It's basically loose cooperation .. folks do what they like to do, with no real stress or pressure - after all, they are doing it for free.

Communication is done via IRC, in a shared user/developer channel (which allows to keep contact with the end-user).

For now Linux-VServer uses the good old Patch and Delta approach for the kernel although we have toyed around with the idea of switching to git. For the user-space tools, depending on the maintainer, various versioning systems are used.

ML: I've been monitoring the project IRC channel (#vserver on irc.oftc.net) for the past few weeks and am amazed by how active you and a few of your fellow developers (shoutout to daniel_hozac) are in the channel. I've often seen you personally greet people as they join the channel. Hmm, where's my question? I guess... it is... how have you been able to maintain the energy to be so active for so long?

Bertl: The #vserver IRC channel is a very friendly environment, more important it is the kind of work environment I always wanted, and finally IRC is my primary long distance communication medium ... so I think it comes quite naturally.

ML: Have you ever gotten burned out from IRC?

Bertl: No, I don't think that you can get burned out easily from talking with people.

ML: How do you feel about people who are working for commercial companies that seek free support via IRC?

Bertl: Strange question — the very same as I feel about people who are not working for commercial companies.

Of course, I appreciate it when folks working for commercial companies contribute back in some way.

ML: What do you think is the weakest area of the Linux-VServer right now?

Bertl: I think the weakest area is the memory management. I have quite a number of ideas on how to improve that, but it isn't that important in most cases, and it would consume quite some time for testing.

ML: If you suddenly had a few additional volunteers, what skills would you be looking for and how would you like to put them to work?

Bertl: Clustering. I think the future of isolation and virtualization is in larger installations spanning several hosts and moving applications around freely. Check-pointing and live migration have very limited use, if you want to do large scale load balancing or run 100,000 tasks in one environment.

The Linux Kernel and Security Contexts

ML: I see the term "security context" used a lot. What exactly is a "security context" and how does it relate and/or differ from a process container? Is it a functional difference or an implementation distinction?

Bertl: Basically "security context" is a relict from the time when nobody knew what a "container" would be :) but to be precise, nowadays a (security) context is only one piece of the big "container puzzle" — the mechanism which separates processes from each other in a secure way.

ML: Have you been working with the upstream mainline kernel developers to get the Linux-VServer code integrated into the mainline kernel? If so, how has that been going?

Bertl: We had a bunch of mainline folks hanging around in the channel, and some of them even tried to get Linux-VServer upstream, but personally I don't think that it makes too much sense to have Linux-VServer in mainline.

More important, mainline kernel development is mostly about political decisions, and I really don't like to invest any of my time in that.

ML: Andrew Morton said the following during his LinuxWorld Expo 2007 Opening Keynote:

"The one prediction I am prepared to make... is that over the next 1 to 2 years there'll be quite a lot of focus in the Linux kernel on... the core of the Linux kernel... on the project which has many names. Some people call it containerization... others will call it operating system virtualization... other people will call it resource management. It's a whole cloud of different features which have different applications."

How do you predict the mainstreaming of OS Virtualization will go?

Bertl: Currently it looks like the Linux Kernel OS Virtualization is trying hard to compete with Xen or KVM instead of focusing on efficiency and performance, i.e. the focus is on completeness and virtualization instead of lightweight isolation and resource sharing.

ML: Do you think all of the features provided by Linux-VServer will eventually make it into the mainline kernel?

Bertl: I really doubt that, there are so many different aspects of Linux-VServer which extend the Linux-VServer kernels beyond the simple aspects of OS Virtualization.

ML: What features do you anticipate will still have to be maintained outside of the mainline kernel?

Bertl: Depending on the future virtualization development, this will range from isolation mechanisms to resource sharing. In any case, current Linux-VServer APIs have to be maintained into the future, unless we drop backward compatibility, which isn't something that should be done lightly.

ML: When I did an interview with the OpenVZ Project Manager I mentioned the various stake holders in the OS Virtualization space... and for some reason, Linux-VServer didn't show up. Why is that?

Bertl: Not very surprising to me, as we always have been (and unfortunately still are) "the enemy" for the company behind OpenVZ. :)

ML: When I was researching container features and how various pieces and parts were making it into the mainline kernel, most of the information I found came from the Linux Weekly News site. There they listed who they thought were the stake holders and the Linux-VServer project didn't show up. Any ideas how that happened?

Bertl: I think the main reason for this is the fact that Linux-VServer is not actively pushing for mainline inclusion (compared to OpenVZ), basically because ...

1. I do not think that OS level isolation/virtualization is for everyone (similar issue as hard realtime)
2. Personally I do not see any advantage in getting 80% of Linux-VServer in mainline and still having to maintain 20% outside
3. Going mainline involves a lot of "useless" extra work (due to political issues) which I do not have the time for. :)

ML: I know that you and your team aggressively keep up with mainstream kernel development and take advantage of added features as they appear (especially those related to OS Virtualization). Why do you think Linux-VServer isn't mentioned very often in OS Virtualization discussions such as those seen on Linux Weekly News or in talks given by the likes of Andrew Morton?

Bertl: Linux-VServer developers do not really spend any time on marketing and public relations (this interview is the exception to the rule here) as we prefer to spend our time on design and development.

Maybe some day a Marketing / PR person will come along and start bringing Linux-VServer into the media. :)

ML: As OS virtualization features are adopted into the mainline kernel over the next year or two, how do you see it affecting the Linux-VServer project?

Bertl: As usual, we'll have a lot of work to do ... adapting code, fixing mainline bugs, etc.

ML: Would you care to mention some recent developments?

Bertl: We recently completed implementation and testing of IPv6 isolation which is a requirement in some parts of the world.

The CoW Link Breaking was improved too, so that unified files are almost like normal files now, except for the sharing benefit, of course :)

Device mapping is another area development was done for, but that still needs more testing.

ML: What other features/areas would you like to see Linux-VServer development move towards? Anything you've always wanted to add but just haven't gotten around to it yet?

Bertl: Yes, full cluster support for lightweight isolation.

Linux-VServer and OLPC's Bitfrost Security Platform

ML: It is my understanding that Linux-VServer has been adopted (or pieces of it) by the One Laptop Per Child project as part of the Bitfrost Security Platform. I have seen you chatting with a few of the OLPC developers in the IRC channel. What, if any relationship do the Linux-VServer developers have with the Bitfrost developers?

Bertl: I was hired by the OLPC folks to adapt Linux-VServer to the specific Bitfrost requirements, which I did.

Besides that, there is not more or less relation between OLPC and Linux-VServer than with any other project using Linux-VServer for specific purposes.

ML: Did you anticipate that Linux-VServer would ever be used to build a security platform?

Bertl: Linux-VServer drastically increases security if used properly, so yes, that is actually one of the main usage scenarios of Linux-VServer, although most folks will consider hosting and server consolidation the primary area of application.

ML: There were some recent discussions on the Linux-Kernel Mailing List about chroot. What are the basic differences between the standard chroot and that offered in Linux-VServer?

Bertl: chroot(), contrary to common belief, is not a security mechanism per se, it just changes the view of a process.

Together with security mechanisms like the barrier and vfs name-spaces in Linux-VServer, it can be used to isolate parts of the filesystem as it is done for typical containers (another piece of the big puzzle).

OS Virtualization vs. Other Methods

ML: OS Virtualization doesn't seem to get as much press coverage and some would say not as much respect as Machine Virtualization. Why is that?

Bertl: I think, in a world of closed source and badly coded commercial applications and operating systems, it is not very surprising that virtualization targets a complete and usually closed system, and that is exactly where Machine Virtualization is the primary choice.

ML: I understand that Linux-VServer has been able to cohabitate with the Xen for some time. Do you know anyone using Xen and Linux-VServer together?

Bertl: Yes, at least one person is actively using Xen and Linux-VServer together, but I guess there are more out there, especially as Xen and Linux-VServer go nicely side by side complementing each other, you won't use a fork when you need a spoon and vice versa.

ML: One of the main areas of concentration by many of the commercial vendors of virtualization products has been in the area of producing fancy GUI management tools. What GUI tools have you seen spring up for Linux-VServer (open source or proprietary) and do you recommend any of them?

Bertl: Yes, don't forget, the management tools are the main area where commercial vendors make money.

There are a bunch of proprietary GUIs I know of but do not really want to mention, as they have been developed by hosting providers for their personal use.

There are at least two FOSS products (OpenVCP, openQRM) and one proprietary solution I know about, but probably there are more out there. In the future, the libvirt project might bring a bunch of new GUI tools capable of managing Linux-VServer guests as well.

ML: Qumranet, the company behind KVM, recently announce a business strategy around desktop virtualization and a patent pending SPICE protocol. Do you think Linux-VServer and OS Virtualization in general have a role to play in desktop virtualization?

Bertl: Depends on how you define desktop virtualization. For example, there is a project called moreubuntu which uses Linux-VServer to build a multi-seat solution, where several users can share a single PC, with several keyboards and mice attached, without affecting each other.

ML: What if any add-on tools are being used by some of your users in the area of desktop virtualization? Do you think we'll eventually be looking for a patent unencumbered alternative to SPICE or do you think the protocols provided with X11 and VNC can be extended to meet future needs?

Bertl: I had to google for 'SPICE' in this specific context, as I only knew spice as being a simulation program for integrated circuits and wikipedia seems to agree with me here. :)

I think SPICE™ is a marketing gag, at least I do not see why a proprietary protocol should improve the (cit) 'interactive experience for virtual environments'.

ML: Given the VMware IPO and the buyout of XenSource by Citrix (both in the United States)... and the buzz word that virtualization is in the IT industry these days, have you given thought to the possibilities of a commercial flavor of Linux-VServer or offering commercial services? If not, why not?

Bertl: Virtualization might be a buzz word today, but the times where virtualization was new and shiny are long gone, mainline Linux always was behind in that area, and now it is slowly catching up.

Personally I (and probably other Linux-VServer developers too) provide commercial support and consulting for Linux and especially Linux-VServer, feel free to approach us.

In Closing...

ML: I greatly appreciate you taking the time to answer my questions. Are there any questions I missed... or anything additional you'd like to mention?

Bertl: You're welcome! No, nothing I can think of at the moment.

Linux-VServer Booth - LinuxTag 2006 in Wiesbaden / Germany
Back Row: Sven Kloe, Herbert Pötzl, Christian Heim
Front Row: Tux, Andreas John, Benedikt Böhm, Mike Schneider, Stefan Himpich, Soewono Effendi
Back to Camera: Björn Steinbrink

Bertl talks and everyone listens

Linux-VServer Official Logo