Review: Proxmox Virtual Environment

| | |

Proxmox VEProxmox VEProxmox VE is a “bare metal” ISO Linux distribution that is a virtual machine platform. It is geared towards enterprise users and designed to be installed on enterprise grade hardware. The Proxmox VE distribution combines two virtual machine technologies; KVM and OpenVZ as well as a web interface to manage everything. Proxmox VE also integrates into its web interface a way to manage multiple computers as a cluster. For the rest of the article Proxmox VE shall be referred to as PVE. This article is written about PVE 1.1, the latest stable release.

Installation

PVE comes as an ISO installer and is not designed to be added onto an existing OS. PVE is built on a stripped down version of Debian. PVE does not come with any graphical interface other than the web based one. It requires a “big iron” server. The server needs to support Intel or AMD virtualization processor extensions and hardware RAID is recommended. The reason for using such high end hardware is each server may have many other mission critical virtual computers running on it. If the server goes down you don't just lose that server you lose every virtual computer that is running on it. The install is really straight forward you only need to answer a few simple questions. Everything is installed and ready to use on a single machine in about 10-15 minutes. Installing a couple of additional hosts and setting them up to act as a PVE cluster can be done in less than an hour.

Proxmox HomeProxmox HomeInterface

PVE's web interface is very elegant and exposes most underlying functionality of both KVM and OpenVZ. The interface is fairly intuitive but like all programs it has a learning curve. The interface gives basic information for your virtual machines, containers and for the host node. It provides stats on disk, memory and CPU usage. For the host node more advanced statistics are given that show a good overview of the host's health and potential bottlenecks.

The PVE web interface offers a Java-based VNC terminal so that you can log into your machines directly from the web interface. This method works well but has a few drawbacks. The most notable is there is no way to use the standalone VNC client to connect to a virtual machine. Also currently the open source Java releases do not support access to the host node's virtual consoles although that may change in the future.

The host node holds templates for the different types of OpenVZ containers that can be installed as well as ISO files for creation of KVM machines. PVE has a large selection of OpenVZ appliances that can be downloaded to do specific functions. Most of them are free however some like the Proxmox Mail Gateway are commercial offerings. Free offerings include Zenoss, Drupal, Joomla!, SugarCRM and Wordpress templates. You can also upload your own customized templates or ISO files. On PVE 1.1 (the latest version at time of writing) there is no way to set a description of the custom uploaded template so care must be used in creating the file name.

Create VMCreate VMCreating new virtual machines is a snap and happens in three clicks. You choose the type of virtual machine you want; either OpenVZ or KVM. Then you choose the template or ISO file to use. Then you can select how large of a disk the virtual machine is to have and set the amount of RAM it is allowed to use. For OpenVZ you can also setup the root password and network parameters. While the web interface does a good job of exposing the basics for virtual machine creation it does not allow you to do some of the more advanced things that are possible especially with OpenVZ. Fortunately if you need those settings you can locally login (or ssh) to your PVE host node and run the underlying command line tools.

Backup HomeBackup HomeThe web interface also provides a nice easy way to setup basic backups. There are three types of backups that can be made. They are snapshot which incurs no down time, suspend which incurs a short amount of down time and stop which incurs the most amount of down time. Snapshot is the default and there is not a compelling reason to use any of the other methods unless a machine fails to backup. The snapshot method works using LVM to copy currently running files that are in use by the OS at the time of backup. The suspend backup method works by backing up the files that it can on a running machine then it suspends the machine for a short time to get the files that were in use at the time of the backup. The stop method stops the machine completely and backs up all the files. The web interface allows you to schedule backups to happen at regular intervals. To make a non-scheduled backup you can use vzdump from the command line. The interface should make it easy to do a non-scheduled backup, hopefully this will be included in a future release. Backup restores are also command line only at this time, this needs to be changed as well. The backup leaves some things to be desired, I will discuss what I think needs to be added in the section entitled “Enhancements and Issues” below.

VM ListingVM ListingProxmox VE Cluster

PVE can be run as a cluster. Clustering brings several advantages. The biggest is the ability to migrate your virtual machines between cluster nodes, live without shared storage. Another benefit is the ability to manage multiple servers from one unified web interface. You are able to create virtual machines from the cluster master node on any of the cluster slave nodes. If your master node fails the slave nodes are still able to function although the cluster is down so migrations will no longer be available, however the virtual machines will still work as normal.

Cluster setup is simple, it must be done from the command line though. Hopefully in future releases cluster setup will be able to be completed from the web interface. Overall clustering is a powerful tool that makes PVE greater than the sum of it parts.

Enhancements and Issues

Proxmox VE is a great program “as is” right now, however there are some enhancements that I would like to see be made.

Changes to the way backups are made tops my list for enhancements. Backups are made with the program vzdump. Vzdump does basic backups very well however some enhancements are needed to really push it to the next level. One of the things that I would like to see changed is the way the file is saved. Currently the file is saved with a fairly generic file name (vzdump$VEID.tgz) the change would be to just add the date to file so you can know when the backup took place. I currently do this with a helper script but it would be nice to have it all integrated. Also a way to clean up outdated backup files would be nice. For example vzdump --clean $numdays would delete all backups that are older than a certain number of days. Again I do this with a helper script but it would be nice to have the entire thing integrated into the web interface.

Another issue with the backup is that there is no simple way to recover a single file easily. You can recover entire machines in seconds but single files are much much harder to recover. Currently the steps to recover a single file or set of files is to use vzdump to restore the machine. When the machine is brought on-line there is a IP address conflict causing a service outage until you can get the IP changed. You can then download the file(s) that you need and complete the recovery. In the future it would be nice to be able to mount the virtual hard drive in a read only state so you can then recover the files without having to bring up the entire machine and without risk of corrupting your backup since it mounted read only. Using LVM would be an easy way. Currently I use a separate system (backuppc) to be able to easily restore single files. This is not ideal because it doubles the storage requirements and adds to network complexity.

The last issue with backups is in a PVE clustered environment when a machine is migrated to a different host its backup schedule does not follow it, this is a small issue and the only work around is to make sure that you check your backup schedule when a host is migrated. Hopefully a way will be devised that the backup schedule can follow the virtual machine.

The next issue that affects only cluster users is the way networking is setup. Currently all cluster hosts must have the exact same network setup as the master host. This can be a major issue especially if you are using an advanced network setup. If all of the cluster nodes do not have the same exact same number of NIC's you are forced to use the lowest common denominator of the box with the least amount of network interface cards. The reason for this is when a new KVM machine or OpenVZ container using bridged Ethernet is created you select a NIC to attach to. If that NIC is not available on the host things will not work as expected. I have not figured out a reasonable work around. An enhancement would be to tailor the choice of NICs to what is available on physical hardware. There is still an issue as to what happens when the virtual machine is migrated to a host that does not contain the same number of NICs. How do other vendors like VMware or Xen handle this? Let me know in the comments.

Editors Note:
By default PVE does not come with a firewall setup. kapt2k has an additional blog posting describing how to make a firewall for the PVE host node and for all of the VMs and containers.

See: Proxmox VE Firewall

The Future and Support

PVE is a great and usable virtualization environment as is now. Verison 2.0 will have some neat things built in like firewall, advanced user management, better resource monitoring and DRBD support. It will also support more storage options like iSCSI.

The PVE community is not huge but it is growing. There is a mailing list and the developers are active on it. Also the wiki contains quite a bit of documentation as well as videos howtos. Also the KVM community and OpenVZ community can help with many issues. The company behind Proxmox VE also offers paid support on a per ticket basis.

PVE is a viable alternative to other products like VMware or Xen. I have been using PVE since the initial 0.9 public release. I currently run about 30 virtual machines on a Proxmox VE cluster including several mission critical servers and have had no issues with the core functionality in the 8 months that I have been using it. It is missing some features that the commercial offerings provide but no other open source application offers the functionality provided by Proxmox VE.

Backup SetupBackup Setup

AttachmentSize
archiveBackups.txt421 bytes

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

vm's keeps up and down...flapping ..any solution ???

vm's keeps up and down...flapping ..any solution ???


PVE rocks!

PVE rocks! And their support in the forums is absolutely amazing... I pay support for some companies and I never got better customer service than PVE guys give to me for free.

Thanks PVE guys!

Regards,
chopeta


Proxmox VE is great if you're familiar with Linux

Proxmox VE is great if you're familiar with Linux.

Basically, it's a normal Linux distribution (Debian) with a virtualization interface built around it.

Anything you're missing - firewall, iSCSI, DRBD? Just add it yourself, using apt-get etc.


Solution for 404 error

cttinsley

... and anyone else having problems with downloading templates.

Someone mentioned that Proxmox may have restructured their downloads. Indeed they have. To fix the problem with outdated links, log in to your Proxmox server using SSH as root, the issue:

# pveam update

Also, run apt-get update and apt-get upgrade to get the latest system updates.

Hope that helps.


bodhi.zazen's picture

Taking Proxmox for a spin :)

I had taken Proxmox for a spin earlier and am just getting to it again.

A few comments that I find makes Proxmox a little easier for me to use :)

Yes, it helps to be able to be somewhat knowledgeable in Linux as this is "bare metal" Virtualization.

1. To make my live easier, rather then having to type /var/lib/vz ... I make a link

ln -s /var/lib/vz /vz

2. I really like openvz , but there are times when I wish to use selinux in a guest, so that means KVM ;)

If you wish to use an iso , rather then a CDROM, to install a server, put it in /vz/template/iso

3. The console option on Proxmox requires you forward or allow connections to the (Proxmox) host on ports 5900 - 6000. It also requires up to date jre (java) on the client.

If you have these tools you can obtain a VNC connection to either KVM or OpenVZ guests 9openVZ = command line, so you may wish to use ssh instead). These connections are over ssl (https).

4. I did find a minor annoyance, you can not use .httaccess to require a password for the proxmox webmin page :( .

5. It would also be nice if the web interface allowed per-user configuration so one could allow a user access to his or her server without accessing the whole tamale :)


Good hints Bodhi, On points

Good hints Bodhi,

On points 4 and 5. .htaccess files do not really allow for the fine grained control of point 5 so the Proxmox devels made a good choice and chose to use a database of sorts for logging in. The database will then keep track of who is allowed to do what and that sort of thing. In the 2.0 release one of the planned features is for multiple users with differing permissions.

_
/-\ ndrew


First use of Proxmox

I went to applicances and tried to download a template, I get a 404 error and cannot download any templates. I logged in as root to the system, dns corrrectly resolved the download.proxmox.com URL to an IP address. Any thoughts? Before I can do anything with Proxmox, I need a template as it is my understanding that you have to have a template to create a VM. You writt that you can also create a VM from an iso, what kind of iso files can be used to create a VM? I also observed that OpenVZ does not require a 64 bit platform as does Proxmox and I assume the containers are interchangeable between them. OpenVZ only supports linux distros so I assume Promox is the same except that with KVM, there is support for that other operating system which we have to use from time to time. Even if I manually wget a template download, I wouldn't know what to do with it without some digging into the documentation and then when I did, I found references to files that don't exist on my 1.1 system. Can you provide some insigths getting up and running with Proxmox?

Adding to this, I found I could download the template on the computer where I am running the browser and then upload the template, and use it. But doing that, it doesn't have a name in template system, just the file. These templates are not particularly ready to use either, some command line configuration required.

Proxmox appears to require more than a bit of Linux expertise to use as nothing seems to work out of the box.

Clint


Proxmox Issues

Clint,

Not sure why the templates download did not work, I just tried it and was able to download the fedora-9-standard without any issues.

Creating a VM from an ISO you would need to use KVM as the virtualztion technology. You can just stick the install CD of the distro you want to use in the CD drive of the host and install off of that or you upload the ISO image of the distro and install from there. Either method works great.

The location to put uploaded ISO files is: /var/lib/vz/template/iso/

The uploaded template does not contain a description that is a known bug and will be fixed in the future however it is not a huge issue for the developers right now.

Proxmox is like any other program there is a learning curve to use it. Proxmox flattens this learning curve down substantially from using straight OpenVZ or KVM. I however disagree that it does not work out of the box. Had your template download worked you could have had a virtual machine working. I am not sure what when wrong there.

As far as the templates go. Some of those are command line only as lots of servers in the world are run CLI only. That is not really a function of Proxmox. However if you did desire a template with a GUI you can download one and use it. Although that would require some knowledge of the program so for out of the box experience I can see your point.

I hope that answers your questions. If you have more or need clarification please post another comment.

_
/-\ ndrew


Proxmox Issues

I confirm, that the download of the templates failed.
The reason is a broken url.
In the browser (virtual environment) the url is
"http://download.proxmox.com/.../centos-5-standard_5-2_i386.tar.gz",
but right is
"http://download.proxmox.com/.../centos-5-standard_5.2-1_i386.tar.gz"

Best Regards

Andreas


Scott Dowdle's picture

Where to get OS Templates

Clint,

OS Templates / Appliances are what are used to create OpenVZ containers.

For KVM machines, you boot an .iso image of an OS install media and install the OS into the KVM virtual machine just like you would on a physical machine. The only difference is that you can't boot from a CD (I don't think) so you have to use an .iso image of a CD instead. When you start the machine for the install you connect to it with the web/Java-based VNC client and see the VM start with the BIOS screen and the cd boot menu, etc. Then just install it like you would on a physical machine.

How do you get OS Templates and .iso images on your Proxmox VE server? There is an upload area in the web-based admin interface. I think there is an issue with uploading if your .iso is bigger than 2GB... in which case you'd want to use rsync or scp to transfer it from your machine to the Proxmox VE server. Where do you store it on Proxmox VE? I don't recall... so yeah, you'll have to look at the docs a little bit.

So far as the 404 error page you were getting when trying to browse and download OS Templates... perhaps they have reorganized their OS Template directory pages or were having a temporary issue with their web server. If you want to manually browse their site and find an OS Template you can download it to your local machine and then upload it to your Proxmox VE server. OS Templates, when compared ti .iso images, are a lot smaller and you should never run into the 2GB upload limitation.

I'm not currently running a Proxmox VE system so I can't look and give you the specifics but I bet Andrew will chime in with additional details if you need any more.

TYL, Scott Dowdle


WebMin & BackupPC on ProxMox

Additional management flexibility is gained on a ProxMox installation by installing WebMin on the hypervisor OS. If any Linux VZ containers are hosted WebMin can install separately on them. BackupPC can be installed as one of the virts. Between the two, extra hard drive space can be put to use, additional backup options are available, and management options outside of the ProxMox scope are readily available.

ProxMox Rox!


WebMin

I have never been a huge fan of webmin, I have not really had any trouble managing the server using traditional tool like ssh and such. But webmin does have a place and I can see how it would be useful as a web interface front end to do stuff that can't be done via Proxmox VE front end.

BackupPC could be run as a virtual. It made more sense for me not to run it as a virtual as I am backing up about 4TB of data and do not have enough space on my Proxmox cluster to do that. BackupPC is a great choice for ease of restores for single files. I have never tried a complete restore with BackupPC has anybody done that? Care to share the steps?

_
/-\ ndrew


Worzie's picture

Hardware Requirements

Andrew,
Without actually doing my own RTFM... is 64bit still a requirement?


64bit Requirment?

Yes it is a requirement, as well as having processor extensions. Intel VT or AMD-V is required to run KVM so by extension Proxmox.

_
/-\ ndrew


clustering

Does Proxmox VE clustering support automatic fail-over? In other words, if set up properly, can I pull the plug on one physical machine and have a virtual machine it contains continue to run? Obviously, this would require the VM to be replicated or mirrored on another cluster node in some way, but I'm just wondering if it can be done.


I doubt it's possible to do it in a reasonable way

What you're describing, from a technical point of view, would require a virtual guest to be constantly replicated between two virtualization hosts.
Although there is no problem in replicating storage that way (DRBD), having the guest to run interrupted if its host dies would require memory, CPU state etc. to be constantly replicated between two virtualization hosts.

In other words, a constant migration. To compare, RAM has speed of tens of thousands megabytes per second. Most you can get from 1 Gbit LAN is 100 megabytes per secons, and this is how slow your virtual guest's memory would be, plus other overheads to keep such constant migration. It would be extremely slow.

Certainly, KVM or OpenVZ (two virtualization technologies on which Proxmox VE is built) do not support it.

I'm also not aware of any other virtualization technology which is capable to do it.


Clusering Failover

Bernardo,

Proxmox VE in it's current state does not allow for automatic fail-over. This is a feature that has been discussed on the mailing list a couple times. The developers say we can expect this feature in Proxmox 2.0. It is slated for delivery in the first quarter of 2009. The Proxmox VE roadmap has the upcoming planed features listed.

_
/-\ ndrew


Proxmox

I would say this is a fair assessment of the product as of right now. As to popping an individual file. The approach we typically use is to convert the image to an iso. Then send it to a test box, unplug the cable, do an 'install' and pull the files. Drop the VM and send the files back via the host after the cable has been reconnected. Not clean but it goes pretty fast.

I have to say this is pretty stable. We have have 3 machines as well in a cluster and have had only minor issues with the environment. An additional feature I might add would be tooling to interact with Amazon S3 for storage of the backups. That and a cluster scheduler to manage it all in an automated fashion.


Backups

JohnMc

Your approach seems workable although seems very time consuming. Your way would solve the problem of IP conflicts though. We have to recover files on an almost daily basis for at least one of our users. If we had to make and ISO for each an every time that happened we would have to hire another person to just do that.

That is why it is worth the cost of an additional backup storage server for us to be able to recover files easily. Even if vzdump does integrate a way to mount read only or something similar we may keep our existing system just because it gets used so often.

Thanks,
_
/-\ ndrew


bodhi.zazen's picture

Thank You

kaptk2:

Thank you for this review. I am interested in Proxmox, but IMO it is a bit of a pain in that the installation takes an entire hard drive (which is a pain to say the least).

I am hoping the "fix" the installer or at least go with the default debian installer.

At any rate you can limit the size of the installation if you do not wish to devote an entire Tb to Proxmox, during the install, with :

linux maxroot=20 swapsize=4

create 20GB root and 4GB swap

I prefer to keep back ups and data separate from the main OS and the fact that Proxmox takes the entire disk is an irritant to me ;)

If it's not broken tweak it ... but remember if you break it you get to keep both pieces ;)


Proxmox VE Does That For You

Proxmox VE already splits the hard drive into an OS partition and a data partition so this step is not needed.

Thanks,
_
/-\ ndrew


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.