Interview: OpenVZ Project Manager Kir Kolyshkin

I had the opportunity to be part of the OpenVZ booth at the recent LinuxWorld Expo in San Francisco where I met the OpenVZ Project Manager, Kir Kolyshkin. He was kind enough to answer some questions for me via email.

About the OpenVZ Project

ML: Please tell me a little bit about yourself... education, hobbies, family... what jobs did have before SWsoft / the OpenVZ project?

Kir KolyshkinKir: I graduated in CS from Ukhta State Technical University, it's in the city where I was born and lived, somewhere 1500 km north from Moscow, the capital of Russia where I live now.

While in the university I had a chance to work and play with not only boring DOS/Windows, but also OS/2, HP-UX, SCO, Novell Netware, and some other operating systems, including Linux of course. My first Linux distro was some ancient version of Slackware, I only remember it came with kernel 1.0.9 and the CD also contained patches for up to 1.1.50. I immediately fell in love with Linux and free software model.

Before I became the project manager for OpenVZ, I worked at a few companies, including positions at Deutsche Bank and at the Russian natural gas giant Gazprom. As for SWsoft, I did a few projects there -- a search engine (ASPseek, now mostly in oblivion), a few projects for Virtuozzo (lead development of vzctl, then kernel testing, then template tools).

Now I live in Moscow with my wife and a 6-year-old son who will go to school in a few days. Of course he runs Linux, his favorite game was SuperTuxKart, now he's playing some Wesnoth. Aside from computers, he likes riding his bike, drawing, reading books, going to banya (russian sauna). That covers most of my hobbies as well -- let me just add skiing (cross-country, not mountain) and music (listening only, not playing).

ML: How did you get involved with the OpenVZ project?

Kir: Naturally, as I was working with different Virtuozzo subsystems I knew it rather well, and since I was a known proponent of free software and its philosophy I was invited to lead the project and met that with enthusiasm.

ML: Why did SWsoft decide to create OpenVZ as a separate project?

Kir: First, it was always been there since Virtuozzo (as the core of it is a modification of the Linux Kernel and thus must be under the GPL), it was just not existing as a separate entity, a separate project -- and now it is. Why? Because virtualization, and containers in particular, will become a commodity in a few years and we want to be a part of the process since we do have some expertise. Finally, this is our contribution back to Linux community.

ML: Who picked the name OpenVZ and who created the logo?

Kir: I do not remember who found the name -- a few people discussed different names on the mailing list. We tried a few names like VZOpen and Open Virtuozzo but didn't like them. OpenVZ seemed quite reasonable, being short, unique, and to the point (as we sometimes refer to Virtuozzo as just VZ). Finally, openvz.org domain was available.

As for the logo, I was in a rush to open the openvz.org to the world and mocked up a logo using the Inkscape editor in just a couple of hours, using its "star" tool. I chose the same greenish color as the Virtuozzo logo (which has changed since then). I thought it might be a temporary solution before some professional artist came out with something better for us, but apparently people liked it so it stayed :)

As for the green color -- OpenVZ helps to increase server utilization, thus saving power in a datacenter. By not adding much overhead per se it further helps to save more power. It's a green technology with a green logo!

ML: You are using GIT for source code management. Has the OpenVZ project used any other SCMs? Why did you decide to go with GIT?

Kir: We used CVS before (and still use it for some stuff) -- and it's just way too old and inconvenient for the 21st century! GIT is a few orders of magnitude more powerful and pretty damn fast. If you want to know why, see the Linus Torvalds talk about it: http://www.youtube.com/watch?v=4XpnKHJAok8

ML: Are there any areas in the OpenVZ project that you wish you had a bunch of volunteers to work on?

Kir: We have already seen some good contributions here and there, but there's always room for more! I would really like people to work more on tools, especially template tools and OpenVZ control libraries (a.k.a. vzctl-lib). A lot of people already contribute OpenVZ templates, and I'd like that to continue with not only OS templates, but also some kind of virtual appliances (i.e. a pre-installed set of applications for a specific purpose, like running a mail server).

I wish we could have some help with the mainstream integration -- if anyone would like to join the fun, start with subscribing to containers-at-linux-foundation-dot-org.

ML: Who is using OpenVZ and why? How many users do you think there are?

Kir: It's hard to know -- OpenVZ has users not customers, so we may not hear from users at all unless they have some problem with or a question about the software. I would guess -- tens of thousands of users. As for the technology itself, we should also include Virtuozzo here which is clearly the number two virtualization solution, seconded only by VMware.

ML: What direction do you see the project heading in the future? Are there any up and coming features we can expect to see in the not-too-distant future?

Kir: The port to the 2.6.22 Linux kernel just went out -- it was our short-term goal for the period. Our long-term goal is integrating OpenVZ into the mainline kernel; so far we have merged some tiny bits a pieces, and more will come soon (network virtualization, beancounters a.k.a. resource management).

ML: An Ubuntu user wanted me to ask if the OpenVZ project would eventually build kernels for the Ubuntu distribution... or if that would be something the Ubuntu project would be more responsible for?

Kir: I know the Ubuntu repository already contains packages for vzctl and vzquota; as for the kernel, one can use the OpenVZ kernel provided by Debian available from http://debian.systs.org/. Surely it won't hurt to ask Ubuntu/Canonical people to include OpenVZ in Ubuntu.

OpenVZ's Relationship to SWsoft and Virtuozzo

ML: Are all of the core OpenVZ developers employees of SWsoft? How much, if any, development work comes from outside of SWsoft?

Kir: Well, SWsoft's kernel hackers are working on OpenVZ and Virtuozzo at the same time, because Virtuozzo is a superset of OpenVZ.

As for the outside development, we greatly benefit from what you call the "outside" people. Let me use the chance to say thanks to the following people:

Benedikt Boehm, Christian Heim -- Gentoo ebuilds
Ola Lundqvist,Thorsten Schifferdecker -- Debian packages
Dmitry Levin, Konstantin A. Lepikhov -- ALTLinux packages

[img_assist|nid=382|title=|desc=LinuxWorld 2007|link=popup|align=right|width=100|height=75]
A lot of other people are quite active on our support forums and/or mailing lists. Some people provide download mirror services. Some people are blogging http://blog.openvz.org/. We had a great team of OpenVZ users at our booth at LinuxWorld Expo 2007 recently. (Editor's Note: Thanks, we really enjoyed it!)

Having said that, most of the OpenVZ code is the kernel, which is a complex thing, thus there are not that many kernel hackers on this planet. Still though, we had a few OpenVZ bugs fixes and features added by outside people and I hope this will increase in the future.

ML: What features separate OpenVZ and Virtuozzo?

Kir: Virtuozzo is for businesses, while OpenVZ is for geeks, thus the difference: Virtuozzo comes with a bunch of GUI-based and Web-based control panels, while OpenVZ sticks only to the command line tools.

ML: I've often heard that Virtuozzo scales even better and allows for even higher VPS/VE density than OpenVZ. Why is this true and what feature differences between OpenVZ and Virtuozzo account for this?

Kir: There is a VZFS filesystem in Virtuozzo but not in OpenVZ. It is similar to unionfs or aufs, the idea here is to make a single copy of files for applications, binaries, libraries and data. If there is a single copy of such files on disk for different VEs, there will be a single copy in memory holding the read-only code and data. It helps to save some memory and thus increase running VE density a bit -- but only the in cases where all (or most of) the VEs are identical (i.e. have the exactly same versions of programs installed).

So, VZFS is not a silver bullet and it comes with a price -- not conforming to POSIX by adding an ability to have additional permission and ownership attributes for symlinks. Because of that Virtuozzo ships with its own modified versions of tar, cpio, rpm, and deb which are patched to understand the added filesystem semantics. Besides, it complicates template tools. We do not want that sort of complexity in OpenVZ.

ML: How much financial support does SWsoft give to the OpenVZ project and do they ever expect OpenVZ to become a self-sustaining, truly independent project?

Kir: SWsoft sponsors pretty much all of OpenVZ, and there is no goal for the project to become self sustaining. Still, I can not say SWsoft (or anybody else) is controlling OpenVZ -- the code is all licensed under the GNU GPL and free/open to all.

ML: Do you think there is a chance that the OpenVZ community (and/or third-party commercial developers) will start adding features to OpenVZ that make it more competitive with Virtuozzo? ...and if that were to start happening, how would SWsoft react?

Kir: I think that would be just great, and I hope things like that will start happening.

As for the SWsoft, I'm probably not the one to ask, but still: I see SWsoft's future in providing high-level virtualization management and automation tools on top of technologies such as OpenVZ.

The Linux Kernel and Containers

ML: Much of the information I've found on the subject of OS virtualization and containers comes from Jon Corbet's Linux Kernel reports from the Linux Weekly News website. If I understand correctly there are at least three stakeholders in container technologies. Andrew Morton mentioned in his most recent talk about the Linux kernel that the only features he could predict for sure that were coming to the mainline Linux kernel were related to containers and he specifically mentioned the OpenVZ project.

Could you please list the various stake holders in container technology and how they relate to each other? From LWN's coverage I've seen:

IBM - Chandra Seetharaman
Google - Paul Menage
The OpenVZ Project / SWsoft

Kir: There are many more people at IBM from different countries involved in this: Cedric Le Goater, Daniel Lezcano, Dave Hansen, Serge Hallyn, Balbir Singh, Sukadev Bhattiprolu, Srivatsa Vaddagiri, maybe someone else I missed.

Also, a lot of work is done by Eric Biederman, and some wise fixes are coming from Oleg Nesterov. Surely, there are more people involved -- see containers@linux-foundation.org mailing list archives for more.

ML: Do each of the stake holders have a similar vision for containers or are they working in different areas? Is there much overlap between projects? Do they work together or are they completely independent?

Kir: Indeed, different parties have different views of this or that subtopic of containerization. Still, we all work together on the containers@linux-foundation.org mailing list, trying to find a common ground and consensus, by exchanging ideas, usage scenarios, actual code (in the form of patch sets). Such collaboration definitely helps to find the solutions all the interesting parties will like.

We will hold a containers mini-summit this September in Cambridge, UK (just before the Linux Kernel Summit which will also be in Cambridge), plus a couple of sections of the Kernel Summit itself will be devoted to containers-type virtualization.

ML: Often the mainline kernel seems to reject patches for a few common reasons:

1. They are too obtrusive/disruptive and change too much of the kernel at once and need to be rewritten to be more incremental and general purpose,
2. The kernel developers do not like the technical design of one or more pieces and request it be re-engineered
3. The patch is very special purpose and is not seen as a feature the general community would be interested in

What if any feedback have you gotten from the higher mainline kernel developers regarding the OpenVZ kernel patches?

Kir: I guess we met all the three reasons above before, but we are adaptive and responsive -- always reacting to the constructive comments in a positive way, fixing the issues, cleaning up the code etc.

ML: Can you update us on the current status of OpenVZ integration into the mainline kernel? Do you expect anything to happen in the near future regarding integration?

Kir: Most notable is the addition of the PID namespace patchset by Pavel Emelyanov into -mm (Andrew Morton's) tree -- it means the code will be in Linus' kernel in a few months. PID namespaces is a feature that makes it possible to have different sets of PIDs in different containers. The code was mostly developed by OpenVZ's Pavel Emelyanov, with some pieces from IBM's Sukadev Bhattiprolu. With the first version sent back in May, it was rewritten a few times to incorporate comments, suggestions and feature requests from everyone who was interested.

This is a good example of how we work together with other teams (this is truly a cooperative work between OpenVZ and IBM, with some help from other parties), and a good example of complexity -- after all, PID namespaces accounts for only about 5% of all OpenVZ code. IPC and utsname() virtualization which have been in the kernel since the release of 2.6.19, both account for another 5% or so. That means we have a long (and exciting!) way to go.

Anther good thing that happened was the acceptance of the user namespace patches from IBM (Serge Hallyn and Cedric Le Goater). This code is much smaller than the PID namespaces code -- what it does is add a basic ability for a container to have its own set of users, including root.

While those two patchsets will appear in the mainstream kernel soon, we have already backported those to our 2.6.22 kernel in order to give this new code a broader testing coverage. That actually helps -- we already found a small bug in the user namespaces code.

As for the future, I think that the rate of patches (and the rate of accepted patches!) will increase. I would guess that some kind of memory accounting (from OpenVZ beancounters) and some kind of network virtualization will be merged in a few months.

ML: If most of the functionality of OpenVZ does end up merging into the mainline Linux kernel, what if any changes to the project do you see happening? Will development be as active or do you anticipate it going into a maintenance only mode?

Kir: Your previous question itself gives us the answer -- I guess there will be some parts of OpenVZ functionality that will never be merged so we will keep maintaining those off the mainline tree. More to say, it's not just a kernel, it's also the tools which I guess will still live at openvz.org.

OS Virtualization vs. Other Methods

ML: Do you think that most users of virtualization products understand the benefits and drawbacks of OS Virtualization vs. Machine Virtualization?

Kir: I'm afraid most people don't, so here's a little explanation.

Virtual Machines (as in Xen, KVM, or VMware) are providing some kind of a virtual hardware, and you can run any OS on top of it. This is flexible, but it comes with a price -- virtualization overhead is still quite high (especially for I/O intensive apps like databases). From the other side, Virtual Environments (a.k.a. containers, as in OpenVZ) are much more efficient, having a native performance and negligible overhead, but from the other side it can only run one OS (in the case of OpenVZ it's Linux, in case the of Solaris Zones it's Solaris, etc.)

Additional reasons to use VEs rather than VMs (not counting the higher VE density and performance) are: dynamic resource management (you can change all of the VEs parameters like RAM limits during runtime) and no issues with scalability (you can make a VE which uses all of the hardware resources, like 64 CPUs and 64 GB of RAM. Besides that, container technology is pretty much hardware-independent and thus are easily portable to other architectures.

ML: Where do you see the future of virtualization going? Do you see any other virtualization types springing up?

Kir: The future of virtualization is in automation and management, i.e. making it simple and transparent to benefit from virtualization, whatever its type. As I said, different types of virtualization are suitable for different workloads and applications -- so there must be a way to manage physical machines, virtual machines and virtual environments in a uniform way not caring about the underlying virtualization technology (if any).

As for the other types of virtualization, there is on-going work on multi-server virtualization (such as in OpenMOSIX or Kerrighed), which is like containers inside out: instead of splitting a single physical server into many smaller virtual servers, such approaches try to consolidate the resources of a few physical servers to provide a bigger virtual server. One can dream about a mix of both technologies -- to combine a few servers into a single one, and then to partition it in any way. For now it looks more like a dream, inefficient from the practical standpoint -- but who knows what will happen in, say, 20 years?

ML: SWsoft is the parent company of Parallels, Inc. Do the Virtuozzo and Parallels teams collaborate at all... and do you think we'll ever see a hybrid product?

Kir: As I said, different virtualization types are for different workloads. As for the hybrid project -- Parallels is developing its server product, and Virtuozzo tools will be able to manage both Virtuozzo VEs and Parallels VMs.

ML: If I understand correctly, the OpenVZ patches do not conflict with the Xen patches... and it is perfectly fine to have a single kernel that supports both Xen and OpenVZ virtual machines... and I know that OpenVZ works fine on a fully virtualized Xen guest. What, if any, practical benefits do you see using the two together?

Kir: So far we have been successful at combining OpenVZ and Xen functionality in one kernel -- we were able to run Xen/OpenVZ kernel in both Xen Dom0 and DomU. That means you can have Xen guests and OpenVZ VEs at the same time on the same server.

From the one side, that means you can benefit from both technologies -- run Linux instances as OpenVZ VEs for better performance and density, and run other OSs as Xen guests. From the other side, there's some overhead even for OpenVZ because it still runs under a Xen hypervisor.

In Closing...

ML: I greatly appreciate you taking the time to answer my questions. Are there any questions I missed... or anything additional you'd like to mention... or address to the OpenVZ community at large?

Kir: If you haven't done it yet -- go to openvz.org and try this great virtualization technology for yourself, it's really easy to set up and get running. There are many different ways you can benefit from it -- even I may not know all of them.

If you are already using OpenVZ -- please tell me (kir-at-openvz-dot-org) how, I'm really interested in hearing your user story.