Installing and using OpenVZ on CentOS 5

I wrote up a HOWTO for the CentOS wiki entitled, Installing and using OpenVZ on CentOS 5, and thought I would share it here as well.

Please note that the OpenVZ kernel is a product of the OpenVZ Project 
and is NOT supported by CentOS. The OpenVZ Project follows the RHEL 
kernels closely and provides updates in a somewhat timely fashion 
after updated Red Hat (and CentOS) kernels are released. As a result 
the RHEL-based OpenVZ kernels are well suited for use on RHEL and 
CentOS hosts with support for (almost) all of the same hardware. 
Please note though that the OpenVZ kernel is less modular than the 
stock Red Hat / CentOS kernels with some hardware support being 
compiled in. 

It is recommended you read this HOWTO in its entirety before 
attempting any of the operations shown in it.

What is OpenVZ?

OpenVZ is operating system-level virtualization based on a modified Linux kernel that allows a physical server to run multiple isolated instances known as containers, virtual private servers (VPS), or virtual environments (VE). The preferred term these days is container. Containers are sometimes compared to chroot or jail type environments but containers are really much better in terms of isolation, security, functionality, and resource management.

OpenVZ consists of a custom Linux kernel (available from the OpenVZ Project) and some user-level tools. OpenVZ is very portable, does not rely on VT support in the CPU, and as a result it is available for a number of CPU families including x86, x86-64, IA-64, PowerPC and SPARC.

OS-level virtualization is quite different from machine / hardware virtualization products such as VMware Server, Parallels Workstation, VirtualBox, QEMU, KVM, and Xen in that with OpenVZ you can only do Linux on Linux virtualization.

OpenVZ modifies the Linux kernel to add advanced containerization features which allow for isolated groups of processes under a parent init along with about twenty dynamic resource management parameters for controlling container resource usage. The OpenVZ Project maintains three stable kernel branches:

1. RHEL4 / CentOS4 2.6.9 based
2. RHEL5 / CentOS 5 2.6.18 based
3. Vanilla 2.6.18 based

There are a number of unstable branches based on newer versions of the Linux kernel that may eventually reach stable status.

Why use OpenVZ?

Since it is relatively light weight, OS virtualization offers a number of benefits over machine / hardware virtualization:

1. It is much more efficient
2. It scales better
3. It offers much greater machine density
4. It offers a larger number of resource management parameters
5. Resource management is dynamic so no container restart is needed

OpenVZ is able to achieve better performance (so close to native it is hard to measure a difference), scalability and density because there is a single Linux kernel running on the physical host with each container only taking up the resources necessary for running the processes / services you want inside them without all of the overhead of a full operating system. A basic container might be between 8-14 additional processes on the host node. OpenVZ can also handle more advanced applications such as huge multi-threaded Java applications with hundreds of threads / processes given the appropriate amount of container resource management configuration.

Another advantage of OpenVZ is that it offers a wide range of dynamic resource management parameters including several for memory usage, number of processes, CPU usage, disk space usage, etc... all of which may be changed while the container is running. OpenVZ also supports container disk quotas as well as (optional) user and group disk quotas within the containers.

OpenVZ offers a number of advanced features including checkpointing and container migration from one physical host to another. Migration comes in two forms:

1. Live migration minimizes downtime (only a few seconds) and maintains machine uptime and network connections
2. Offline migration where the machine is stopped, migrated, and then started back up again

The migration features of OpenVZ do NOT require a shared storage solution and utilizes rsync to flawlessly copy container directory structures from one physical host to another.

When NOT to use OS Virtualization

While there are a large number of usage scenarios where you would want to use OS Virtualization, there remain a few scenarios where OS Virtualization is NOT suited and machine / hardware virtualization would be preferred:

1. When you need to run non-Linux OSes
2. When you want to run multiple kernel versions
3. When you need a highly customized kernel

OpenVZ History

SWsoft (now known as Parallels) initially released a product for Linux named Virtuozzo back in 2001. Their current product is named Parallels Virtuozzo Containers. In 2005 a version of Virtuozzo was released for Microsoft Windows. Also in 2005, SWsoft created the OpenVZ Project to release under a GPL 2 license the underlying technology upon which Virtuozzo builds.

While OS Virtualization does not seem to have garnered the press attention and excitement some of the machine / hardware virtualization products have gotten in recent years, having initially been released in 2001 (Virtuozzo) and 2005 (OpenVZ), they have both proven themselves to be efficient, stable, and secure workhorses on tens of thousands of servers around the world. Linux OS Virtualization (which includes Linux-VServer) is arguably the oldest and most widely deployed Linux virtualization platform to date.

OS Virtualization and OpenVZ Future

In a few presentations in 2007 and 2008 on the future of the Linux kernel, Andrew Morton identified containers as being the only thing he was certain of that was coming to the Linux kernel because there were a number of strong stake holders working on it including IBM, Google and the OpenVZ Project.

Container features started appearing in the mainline kernel starting with the 2.6.24 kernel and more have been added in subsequent releases. The joint effort is more commonly referred to as control groups (or cgroups for short) and a number of kernel subsystems have been modified (scheduler, memory management, etc) to be cgroup aware. It is not known how long it will take before the cgroup implementation in the mainline Linux kernel will be feature complete, stable, and in wide use... so it appears that OpenVZ will be around for some time to come.

There is another OS Virtualization product for Linux named Linux-Vserver. Linux-VServer is a quality product / project too but there are a number of differences between OpenVZ and Linux-VServer. Parallels / OpenVZ are working with the mainline Linux kernel developers to get container features into the mainline Linux kernel. The Linux-VServer developers are working independently and have decided that Linux-VServer will stay an out-of-tree patch for the foreseeable future. That is not to say that OpenVZ is going directly into the mainline kernel because it is not. The cgroup effort is a consensus of all of its stakeholders. It is clear however that the OpenVZ Project has contributed a lot of code to the mainline kernel.

Installing OpenVZ

The OpenVZ Project website (www.openvz.org) has a lot of quality documentation including a Users Guide PDF, a Quick Installation Guide, and a vast wiki of howto and troubleshooting articles. This article will briefly cover the installation process. You will need to be the root user for all of the following tasks.

Adding the OpenVZ yum repository

Installing OpenVZ on a CentOS 4 or CentOS 5 host is very easy because the OpenVZ Project provides an openvz.repo for use with yum. Simply download the openvz.repo file and place it in the /etc/yum.repos.d/ directory. Examine the openvz.repo file as it contains a number of repository definitions with two being enabled by default: 1) RHEL5-based kernel and 2) the OpenVZ utilities. Edit it to meet your needs.

Install the OpenVZ kernel

Depending on which kernel arch you want, simply do:

yum install ovzkernel.i386

or

yum install ovzkernel.x86_64

With the OpenVZ kernel installed it is almost time to reboot, but before rebooting a few additional details need to be taken care of: 1) Examine /etc/grub.conf to ensure the desired kernel is set to be the default, 2) Edit the /etc/sysctl.conf to enable some kernel features that are needed for OpenVZ and 3) Make sure SELINUX is disabled.

I will not cover how to edit /etc/grub.conf since that is a fairly common, non-OpenVZ specific task.

Editing /etc/sysctl.conf

If desired, backup your original sysctl.conf file (cp /etc/sysctl.conf /etc/sysctl.conf.original). Edit the /etc/sysctl.conf file with your preferred text editor and add the following:

# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# TCP Explict Congestion Notification
# net.ipv4.tcp_ecn = 0
# we do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0

Disabling SELINUX

Unfortunately SELINUX is not compatible with nor included in the OpenVZ kernel. Put the following line to /etc/sysconfig/selinux:

SELINUX=disabled

Now it is time to reboot your computer into the OpenVZ kernel.

Installing the utilities

To install the OpenVZ utility programs, simply do:

yum install vzctl vzquota

Once vzctl is installed you want to start up the OpenVZ service by doing the following:

service vz start

The vz service should be set to start up automatically but feel free to check it with:

chkconfig --list vz

Now OpenVZ is fully installed and you are ready to create your first container but first you need to get installation media for the desired Linux distribution.

Using OpenVZ

OS Templates - Linux Distribution Install Media

The vzctl command is used to create and configure OpenVZ containers. Before you can create a container, you need install media for the Linux distribution you want to install. OpenVZ can NOT use CD / DVD install media (NOR .iso disk image files). OpenVZ requires what it calls an OS Template to create a container for a given Linux distribution. You can download a number of pre-created OS Templates from the OpenVZ website. This is the recommended route for new OpenVZ users. Once you are more familiar with OpenVZ you may want to create your own OS Templates from scratch using a variety of recipes available on the OpenVZ wiki. You can find pre-created OS Templates provided by the OpenVZ Project here:

http://download.openvz.org/template/precreated/

You can also find community contributed OS Templates here:

http://download.openvz.org/contrib/template/precreated/

Download the desired OS Template file and place it in the /vz/template/cache directory on your CentOS 5 host node.

Creating Your First Container

To create a container you use the vzctl command with the create option and a few other parameters. You will need to decide on a unique container ID number (CTID). You will also need some additional information: What OS Template do you want to install from? What IP address will it have? What hostname do you want to give it? What initial resources do you want it to have (taken from a sample configuration template)? These are all set with the following parameters:

--ostemplate {template name}
--conf {config name}
--ipadd {nn.nn.nn.nn}
--hostname {FQDN}

Here is a complete example:

vzctl create 101 \
--ostemplate centos-5-i386-default \
--conf vps.basic \
--ipadd 199.199.199.199 \
--hostname mynew.container.com

That will create a directory named /vz/private/{CTID} and extract the OS Template into it. It will also copy the specified config file to /etc/vz/conf/{CTID}.conf to create a configuration for the container. Please note that the values given for --ostemplate and --conf do NOT match up to full OS Template nor the configuration filenames... and are somewhat truncated.

The vps.basic configuration is just that, very basic... with a somewhat conservative amount of resources given to the container. For a look at the various resource parameters and their values, feel free to have a look at the sample configuration files in the /etc/vz/conf directory. You can create your own configuration samples (as many as you like) using the vzsplit command (man vzsplit) or by copying existing config files to new files and editing them.

After you create a machine you need to set a few additional parameters using the vzctl set command. Here's an example:

vzctl set 101 \
--name mynew \
--nameserver "205.171.2.65 205.171.3.65" \
--diskspace 10G:10G \
--save

That will update your container's config file (in this example /etc/vz/conf/101.conf). Any time you run the vzctl set command if you don't include the --save parameter it will not save the configuration changes to your config file... but if the container is running it will dynamically change the parameters in the running container for that session. You will almost always want to include the --save as the last parameter for the vzctl set command.

Starting your new Container

Just issue the following command to start your new container:

vzctl start {CTID}

Or following our example

vzctl start 101

You should see a message about your container starting up. Assuming you didn't botch any of the parameters, it should start right up.

As you would expect, the keywords stop and restart also work as expected. If a container is not running you can destroy it which will remove its config file as well as delete its directory structure (/vz/private/{CTID}) from the host node filesystem.

Entering your new Container

Once your container is going, it should be accessible via the network just like a physical server would be. But since you are on the host node, you can use a vzctl shortcut to directly enter the container as root without authentication.

vzctl enter {CTID}

Following our example

vzctl enter 101

You should get a new command prompt inside of your new container. You will probably want to set a root password by using the passwd command OR you could do the following from the host node:

vzctl set --userpasswd {user}:{password} --save

It is assumed that you are a fairly good Linux system administrator who is NOT afraid of the command line. Have a look around your new system. It should look and act almost identically to a physical machine. You can install software (see section about yum below), create accounts, add new system services and modify their configurations.

You can view the resources given to a container within it by looking at the /proc/user_beancounters file. You can view all parameters given to all containers from the host node by looking at /proc/user_beancounters on the host node. Resource management is beyond the scope of this article so consult vzctl man page or the Users Guide PDF for complete information.

What if yum is missing in my container?

The pre-created OS Templates provided by the OpenVZ Project do not have yum installed (contrib OS Templates are more likely to have yum installed) and that seems to miff a lot of users. The reason is that some OpenVZ system administrators prefer to use a tool on the host node named vzyum that allows them to do yum type operations on containers from the host node. Why would you want to do that? Because the yum database and file caches can sometimes take up a significant amount of diskspace (and bandwidth) if each container has its own copy of yum. Using vzyum on the host node, there is a single yum database / cache and the software packages only have to be downloaded once.

There have been some technical issues using vzyum on CentOS 5 x86_64 host nodes (it worked flawlessly in CentOS 4) and as a result vzyum has fallen somewhat out-of-favor with CentOS 5... but the official pre-created OS Templates are still optimized for those using vzyum.

There is a wiki page on the OpenVZ wiki that explains how to install yum in a container if you want it. Basically you download all of the rpm packages needed for yum and install them with rpm.

In Closing

This article has only covered the very basics of OpenVZ. Topics not covered include container monitoring / resource management, configuration file editing / creation (see the vzsplit man page), the container filesystem, or any additional tasks necessary to become a productive OpenVZ system administrator. Also note that OpenVZ has two types of network adaptors and only the basic / default venet was covered. The veth network adaptor has a wider range of capabilities.

The documentation provided by the OpenVZ Project is really good and there is plenty of it. Be sure and read the Users Guide PDF guide for the most comprehensive documentation. The Users Guide is a little dated though as it was written before the addition of the veth network device, checkpointing and migration. Also available are the man pages for all of the various commands. When you want to get into container migration be sure and read the vzmigrate man page.

You are encouraged to get involved and participate in the OpenVZ project by visiting the OpenVZ website, using the forums, the IRC channel (#openvz on the Freenode IRC network), as well as filing bugs if you run across any at bugzilla.openvz.org. Get familiar with the OpenVZ website as there is a lot of material there.

Additional Resources

OpenVZ Project website
Interview with OpenVZ Project Manager
OS Virtualization vs. Hardware Virtualization video presentation from Linuxfest Northwest 2008
Container migration demo video
Performance evaluation of Xen vs. OpenVZ by HP Labs
OpenVZ blog
Parallels Inc.
Linux-VServer website
Interview with Linux-VServer Project Leader

This is the initial release of this HOWTO and I (Scott Dowdle) plan on updating it with additional information, especially if I get any reader feedback. Suggestions and comments are encouraged. Feel free to email: dowdle at montanalinux.org.

Please do not email the CentOS mailing lists or visit the CentOS 
IRC channel with OpenVZ related questions.  Use the OpenVZ IRC 
channel and the various other OpenVZ support resources instead.