An OpenVZ Experiment - How many containers?

Creating 1,000 containers takes a while. You have to unpack the .tar.gz, copy a config, start up the container... and then in my script, as previously mentioned, I paused 60 seconds to give the processes time to start up and to give the disk a little break.

Now the bulk of the 1,000 containers have been running for a day. According to sar the iowait dropped from double-digit values to a very small single digit value around 6am. The iowait has been hovering between 1 and 2 ever since... making the average for the 24 hour period 9.79. Unfortunately I just let the container creation process run mostly unmonitored and I think a cron job for updatedb kicked in and I haven't excluded /vz so everything was getting indexed. I don't know if that was the cause for most of the iowait but whatever it was has passed. While the containers were still being created that certainly was the cause for the iowait.

Anyway, the machine's load is in the 0.2x range since the iowait went away with a 97% CPU idle.

I'll see if the smooth sailing continues for the duration. If so, I'll just make one more final comment and if not, I'll document that what happened.

I ended up running the 1,000 containers for about 1 week and then having to use the machine for a different purpose. I didn't really have anything to report other than the load stayed pretty low, the containers were responsive and I didn't notice anything unusual.

Sorry for the anti-climactic end.

When vzctl told me it couldn't start any more containers it was a scheduler error... a special scheduler used by OpenVZ in the 2.6.9 and 2.6.18 kernels. If I remember correctly there was about 16,088 processes so it didn't exactly stop at 16,000... and the host node could run additional processes it was just vzctl refused to start more containers.

You can start more than 16000 processes! Just increase sysctl's parameter

kernel.pid_max

Assume that half of pid_max - maximum processes for VEs.

Wow, how did I miss this? Actually, I'm sure I didn't but I must have been busy with other stuff and quickly shrugged it off thinking I didn't have time to do the whole thing over again at the time.

Now we know. :)

To me 638 containers is a substantial number. The error given really didn't tell me a lot so I'm not sure. Perhaps I should ask the OpenVZ folks and see what they say.

According to top I had plenty of memory available. Maybe it was related to memory that is usable by the scheduler? I don't know. I haven't found a good reference for the error. You have one?

ENOMEM just means something in the kernel ran out of memory.

Most likely you ran into memory fragmentation on the kernel stack. The kernel stack on x86-64 requires two continuous in physical memory pages and when memory fragments it's often difficult to get that after some time. You can check that by looking at /proc/buddyinfo: if column 6 is all 0 you ran out of order 1 memory blocks.

While I'd like to be able to do the same experiment with Linux-VServer, I'm not as familiar with it as I would like to be... and I don't currently have a server of the same class with Linux-VServer installed on it.

I certainly do welcome contributed articles to this site... hint, hint.

The ISPs of the world have been providing low cost operating systems level containers to their costumers for a while.

FreeBSD 4 (March 2000) opened up the possibilities of containers to the service providers. The technology was solid, but it did not come easy to use tools and it was technologically a bit limited compared to other solutions we see today.

"The FreeBSD jail mechanism is an implementation of operating system-level virtualization that allows administrators to partition a FreeBSD-based computer system into several independent mini-systems called jails.":
http://en.wikipedia.org/wiki/Freebsd_jail

"The jail utility appeared in FreeBSD 4.0.":
http://www.freebsd.org/cgi/man.cgi?query=jail&format=html

"4.0-RELEASE appeared in March 2000":
http://en.wikipedia.org/wiki/Freebsd#FreeBSD_4" target="_another2">http://en.wikipedia.org/wiki/Freebsd#FreeBSD_4

Yes, I have heard of FreeBSD jails too... but like you said, they aren't quite as feature complete as containers... and a bit difficult to work with. I wasn't trying to say that OpenVZ was "first". Heck, chroot would be first.

Yes, that comment ("Try that with VMware, VirtualBox or Xen") was made in jest. I certainly understand the differences between machine / hardware virtualization and OS virtualization. One point is that some people use the machine / hardware virtualization products for Linux on Linux hosting where OS virtualization clearly kicks its butt... although I do acknowledge there are some Linux on Linux virtualization scenarios that OS virtualization is not well suited for... but there are a lot of scenarios where it is well suited.

Anyway, I doubt that Solaris had Zones before SWsoft released Virtuozzo (from which OpenVZ comes). Virtuozzo was originally released in 2001 although I don't know the month. Perhaps I'm wrong about that but I haven't found any evidence of that.

Regarding filesystems, Linux/OpenVZ does not have anything like ZFS and the closest thing you can get to it would be using a combination of RAID and LVM. I use RAID and LVM but not LVM snapshots... so yeah, OpenVZ takes up lots more disk space. Virtuozzo uses vzfs which does pool files... so it saves disk space on common files between containers. OpenVZ does not have that feature. That would be nice to have in situations like my experiment where I create a lot of containers that are all the same.

I have looked into Solaris Zones but I've never quite understood them. Take for example this tutorial that I happen to have run across yesterday:

Devzones: Simple Use-and-throw Developer Environments

Granted it is about using Zones on Nexenta that is based on the OpenSolaris kernel... but as far as I know, it doesn't vary much from Zones in Solaris proper.

I need an article that better explains Zones because so far I've read several and I just don't get them. They sort of sound normal until I see all of these special commands run by users to create a zone... and then they destroy it? If you could point me in the right direction for a more comprehensive article, I'll check it out.

OpenVZ (and Linux-VServer) containers are just easier to understand... grouped processes into a container which is fuctionally a virtual machine with it's own resource parameters, distro, filesystem, root / users, etc... something you usually want to hang around because you created it for a reason.

A feature I believe that is missing in Solaris Zones that is avialable with OpenVZ is offline and live migration.

Well, you are right about the age bit - it was in 2004 that Zones in Solaris took off - with the Sol 10 release.

Solaris Zones are able to be migrated, but I've never used this feature.
http://www.sun.com/software/solaris/howtoguides/moving_containers.jsp
Not sure I'll want to with ZFS anyway. I'll just attach another array and add to the pool.

Of course I can always export the ZFS slice and then import it on another machine. Using AVS is quite a nice method of dealing with this - it's all included in OpenSolaris anyway.

Don't get me wrong, I like Linux - I'm using Linuxmint for my desktop at work right now. I work at a hospital though, and since I've had problems with ext3, reiser3, and of course ext2... well I'm a bit gunshy of using linux in mission critical production where my name is on the chopping block.

Having actually built a zpool out of 5 of those WD 500G USB attached drives - and having it hold 2TB data in a live backup situation - for over 3 years... and lose a drive, have it replaced, and all work - without rebooting, totally sold me on Solaris/ZFS.

For desktop, I use Linux, it's well suited. For server, I use Solaris, it's more powerful. This is just an opinion... but I'm likely to use the best tool for the job when allowed, and whatever the vendor specifies when I have to. Case in point, I mix Windows 2k3 environments running Tomcat with iSCSI mounts to Solaris ZFS exports - where they sit as drive D: - this is where I load Tomcat and the webapp. But because the company expects windows 2k3 with tomcat to do their upgrades, I have to live with that... doesn't stop me from using all the benefits of ZFS though.

Anyway, I used to be a die-hard Linux nut, avoiding all windows, all anything but Linux for about 5 years. Debian and Gentoo were my only flavors... it's funny, that kind of hermetic seal doesn't really do much to benefit a person either. Windows is a good system in its own way, and it's really nice to be able to subvert it when all the windows guys say it can't be done.

Well, if you look at the bottom of that link, you'll find tons of sun docs for zones. Head on over to cuddletech.com for good unix info. docs.sun.com are a good place to be. One thing to be said is that sun documentation is amazing and very detailed - but you WILL need it. If you don't like reading, you won't like Solaris.

well, I've written enough.
Jack

I attended a Sun sponsored event a couple of months ago and the message was clear... having been repeated over and over... Sun is an OpenSource company... and they want to be like Red Hat. Now they didn't exactly say that last part but the first part was like a mantra. Only problem is that they aren't open enough... at least for me. I'll give them props for FINALLY releasing Java under the GPL and all that they have contributed to the FOSS world. I don't know if GPL'ing ZFS would help them out or not... I'm not a stock holder and I don't care about their financials... but I know that is their number one concern.

To me Sun, regardless of the quality their software and hardware, is just another example of the bygone Unix days. They are like SGI... trying desperately to make it in a market place that seems to be lapping Linux up. They have been releasing their technology in order to please the FOSS mindshare but like with SGI, it hasn't helped much. I don't have any data to back this up but I believe Sun's marketshare has been shrinking for some time now. How important is that? For the short term not too important but for the long term... yeah, just ask SGI... or any other commercial Unix/BSD vendor. Why is Linux winning? I'm not sure but I think a lot of it has to do with the openness.

That isn't to say that I prefer inferior technology. I would love to have ZFS as a part of Linux but Sun has made sure that can't happen. Ok, they created ZFS, that is their right. I respect that. Linux doesn't have anything as good as ZFS at the moment, I concur... but what it does have is good enough for me. I haven't had any ext3 crashes in recent memory. The hardware RAID I have on my (important) servers has worked well for several years now... I keep good backups... and LVM is there but I haven't taken much advantage of that yet. I guess I'm just not that mature/skilled when it comes to storage technology. I do look forward to the maturization of the various up and coming filesystems that seem to have a chance at rivalling ZFS.

I'm pretty loyal to Linux but that doesn't mean I'm going to bury my head in the sand. If I reach a point where Linux isn't cutting it for my needs I will look elsewhere.

I've been using Linux longer than Solaris has been around on commodity hardware. When did Sun release Solaris for x86? I started using Linux in January of 1995. Things sure have changed since then.

I wish Sun well in its fight to stay alive but I won't be too surprised if it doesn't make it. Of course people have been predicting the end of Apple for like a decade or two but they seem to have pulled themselves out of the fire. Maybe Sun can too. Perhaps I'm even overstating it... maybe Sun isn't even in the fire to begin with.

I do somewhat resent someone telling me that I should use something else because it is better. Where have I heard that before? There are various ways to measure better. As you said, from a desktop users standpoint, there is a pretty strong case to be made for Linux. I think so on the server too... but I don't try to tell anyone they should use Linux (or a particular Linux distribution) because it is better. Much of "better" is about preference... and we all have our own.

Reading your statement about Sun been able to keep alive on 2008 and now bought by Oracle just make me laugh..

will keep reading this page, maybe i can make a buck in the future with your forecast about IT companies

:)

But I was defending my favorite OS. I really don't think that's at all wrong. And, yeah, I've been using Linux for a while myself - since around mid-year 96, when Win95 crashed on me, I gave up on M$ and tied out something called Debian 1.1... those were interesting times - I was in college working part time on the DEC Alphas, trying out Linux for the first time, trying to keep a dying HPUX machine alive, heh, the good ol days.

Anyway, if you've taken offence to something i said, well I guess that's the way it goes. If you think Sun is going away, well maybe it is, maybe it isn't. I do think they are restructuring their products to better serve a marketplace - take a look at the unified 7000 series of storage arrays.

Oh, and btw, sun didn't say Linux couldn't have ZFS... hell, FreeBSD, Mac OS/X and OpenBSD all use ZFS now. Don't tell me Sun is stopping Linux. Linux has ZFS in their Fuse methodology too - it's just crappy slow there. The problem is the licensing. Sun doesn't like GPL-V2 or GPL-V1. They do like GPL-V3 and have said they will re-release their software under that license as it's much more compatible with their CDDL license method.

I do get tired of the "Oh, it's a big company, and they want to make money... therefore they are evil. Screw them" mantra... Sun is a big company, yes, but are they flexible? yes? Tell me...is your CPU open-source? Oh, wait, you use x86... so, nope, your CPU technology is closed source. How odd is that... does open-source only apply to software? Or are open standards nice to apply to hardware too? If you are wondering why I even bring this up... take a look at the SPARC platform. It's an open-source or open standard CPU platform from which a few vendors build their server. Recall that Fujitsu makes CPUs as well - oddly enough they are based on the open standard called SPARC - and hey, they have put their improvements back into the standard. Look at how much more flexible the SPARC is vs. the x86. x86 is ubiquitous, common, and most of all cheap. It's hard to get inexpensive SPARC stuff... I don't even have one myself. I, too, use closed source hardware... but I can't wait to move to SPARC when I get the chance.

Anyway, I'm not trying to rant, just trying to show that there is always more than one side to the argument. If we end up in an all-Linux world... what kind of choice is that? We'll just have the one-size fits all crap again - and then there will be room for the pendulum to swing back to MS - or some other proprietary system. I'm all for variety - and not just more Linux distros... Variety means incompatibility sometimes, trade offs others, but overall variety increases competition, and competition breeds excellence.

well, have a good day!
Jack

Jack,

Thank you for your comments. They are great. I don't really argue or get offended... I discuss things... reasonably. You made some good points regarding SPARC.

I've read about their multi-core, 8-thread design and it is pretty darn awesome. They even include encryption features in the hardware. I wonder what software it takes to take advantage of that?

I'm not really anti-profit. Heck, I'm a Red Hat fan, right?

I'd really like to see Sun release ZFS under some license (or dual-licensed) that is compatible with the GPL-2 that Linux is licensed under. It was made fairly clear though, that Sun wasn't interested in that. Of course perhaps they'll change their mind.

I don't think Linux will last forever. Something has to take its place but what I'm not sure. Look at some of the stuff AST did after Minix... and the additions he has made to Minix 3. And of course there are a large number of OSes out there, mostly in niche areas.

I completely agree that competition is good.

I dunno about Linux not lasting forever, though of course eventually the sun will go nova...

Open source software has kind of a different life cycle from proprietary stuff. Consider GCC -- one of the earliest C compilers. It's still thriving, though I would bet that very few lines of code survive from version 1.0. OSS tends to adapt: I've been running Linux on my desktop since 1.02 and can hardly believe how it's exploded in 14 years. (Yeah, I ran it on a 386 with 4M of memory; Netscape was statically linked with the Motif libraries. When it started to load, get a cup of coffee. Hell, make a pot of coffee...)

Speaking of Sun going nova -- as to ZFS, I predict that either some of the ext4 geeks will read over the Solaris code and say, "Hey, I can do this a better way" -- which true geeks always say -- and then proceed to do it a better way, or Scott McNeally will get tired of being poked by IBM guys at Palo Alto cocktail parties who say, "Look, we gave 'em JFS, what are you gonna do?", and then poked by SGI guys who say, "Look, we gave 'em XFS, what are you gonna do?"

That said, though, I'll be interested to see what eventually happens with pthreads. Solaris has always been the champion of low-overhead thread scheduling and multiprocessing; Linux 2.6 is extremely good in that respect and has made incredible progress, but it's still not up to Solaris' level. If Sun would turn some of its programming talent in that direction, as IBM did with NUMA ...

I'm still not sold on the idea that Sun, because they wrote a nice goodie, should just give it all away. I'm glad they do, but I see the greedy behavior of the "I want this, I want that" Linux crowd to be distasteful. Linux seems to really really be slowing down in the innovation arena. Apache, Samba, Openldap, Glassfish, Tomcat, Java, MySQL, PostgreSQL, etc. All those are great opensource projects - not Linux. the fact they run on Linux is really nice... But where's the really cool neato breakthrough on Linux? They seem to not be happening? I recall the day when I finally got usb working - some 5 years ago, man was I excited and just unreal proud to be a linux user. What seems to be happening to me is that Linux is attracting a lot of the undesirable windows crowd, people who complain immediately about this or that without reading the man pages, or understanding the nature of the word "Google". So, here Linux is, another welfare nation, depending upon the handouts of others... Sun invests some $2B in R&D yearly for their projects... and Linux wants all their stuff for free. We all want I suppose. I just happen to look a Linux as a Ford Mustang, and Solaris as a Mercedes SL600. Both are powerful and fast, it's just that Solaris is way more luxurious - well written man pages, good documentation at docs.sun.com - every possible way for a person to figure it out for themselves. Declaring Sun dead is like saying transport by rail is going away. Good luck - I think you're in for a surprise in the coming years as to Sun's ability to change markets and grow.

In my opinion, we are in for a marked change in cpu usage - sparc is so much more powerful, so much more efficient than x86, it's about time AMD took on more of the Sparc methodology and produced a chip which could run at least 4 threads per core.

Well, opinions and butt holes, we all have them... hopefully we are not spewing too much waste.

Jack,

Sun has already given away ZFS to a certain extent... as several OSes that are compatible with the license they chose have already adopted it. What is annoying is Sun's mixed messages and constant direction changing. I guess that is supposed to be good and means that they are nimble and can change dynamically as the market changes... depends on ones perspective. Sun has decided to release some things under the GPL... take Java for example... although it took them a long time to do it.

While you see the Linux community as freeloaders let me remind you that there are a few commercial Linux companies that seem to be doing well even though they live (some better than others) the ethics of the GPL. Red Hat isn't as freedom oriented as the Free Software Foundation but they seem solidly rooted in the idea that what is good for the customer is good for them. To me Red Hat has a pure freedom model (notice the difference between free price and freedom) that is working for them.

Sun seems to kind of blow with the wind. Again I am thankful for and acknowledge their vast contributions to the FOSS community over the years. Check out the Open Source pony tail video for a satirical expression of Sun's business model. Having attended a Sun sponsored event recently I can tell you first hand that they are basically trying to morph themselves into a FOSS company... like they want to be Red Hat or something.

Linux isn't innovating but other projects are? I guess you just don't get it. Linux is a kernel. Distribution makers gather up the software. Various projects make the software. Sun understands this model very well because with OpenSolaris they are trying to adopt the Linux distro model... and are moving away from their Sun-only software model to everything that can possibly offer them and their customers more value. I think they have been choosing a lot of GPL'ed software and as a consumer of third-party software I would imagine they prefer GPL'ed stuff over less free software. If that isn't the case with the OpenSolaris distro makers, I'm sure it definitely is with OpenSolaris customers. The GPL protects the freedom of the user / customer.

So far as Linux as a kernel innovating... yes it is happening at a breakneck pace... in every direction. Linux is like a liquid and it flows in every direction that nature / gravity takes it. It is being used in embedded devices some of which are quite small both physically and with regards to resources... all the way up to the mainframe. There are a lot of extra flavors available to those that want them: hard real-time, soft real-time, advanced messaging, HA & HP clustering, grid computing, Linux native virtualization (KVM and in the not too distant future containers), security platforms (SELinux and AppArmor). Linux is moving in so many directions all at the same time that many see it as directionless and chaotic. The reason this is happening is because people can do with it as they wish.

Kernel development, after a certain level of maturity, is all about just maintenance and adding drivers for hardware. I think Linux totally kicks Solaris' butt with regards to hardware support. Not in the Sun hardware support area but all others.

The plain fact is that both Linux, Solaris, and third-party projects continue to innovate and borrow from others. If I had to judge I'd say that Solaris is borrowing more from Linux lately (not in the kernel space but in the distro space) than the other way around. I'd prefer to see the welfare used by individuals than by corporations. :) I'm not really complaining about Sun borrowing from Linux and third-party projects... it is about time. Most Sun users ended up manually borrowing anyway... and Sun has just decided that they can do more of that borrowing themselves rather than leaving it up to the customer.

Sun is very much a mono culture. I don't think you'll see them develop a filesystem that competes with ZFS. Linux? Well, they aren't too bothered by the existence of ZFS and the fact that they can't have it. Linux has a ton of filesystems and continues to develop more... and the internal competition is good.

As you know Sun has bought up a few projects (Star Division [StarOffice / OpenOffice.org] and MySQL for example) and I think the general consensus, after the initial injection of cash, the software projects have been somewhat mismanaged. There have been a number of articles claiming that OpenOffice.org as a project is not very healthy and that it is Sun's management style that has lead to that. I don't know enough about it to make a judgement.

So far as analogies go, I don't have a good one for you. Linux is so complex and amorphous and Sun is quite unique too... that any analogy I could come up with would only address one or two properties and fall apart with any serious scrutiny.

Why is the assumption that full virtualization and containers are mutually exclusive? There is at least one solution which provides the best of both worlds:

Proxmox VE provides the ability to run, manage, and migrate both fully virtualized virtualmachines via KVM, and containers via openVZ.

David,

Thanks for the comment. I'm very familiar with Proxmox VE. I gave a presentation on it at a BozemanLUG meeting a few months back and I have advocated it for use where I work... and it seems to be catching on.

I hope to have an in-depth article about the latest Proxmox VE release published in the not-too-distant future written by a BillingsLUG member (kaptk2). He has been working on it for a while but with the holiday season making us all rest and enjoy ourselves, it might be delayed until January. We'll see. In the mean time, enjoy the article he wrote some time ago about the initial Proxmox VE release.

I really like the Proxmox VE approach. I like the fact that KVM is in the mainline kernel and look forward to the day when Linux Native Containers is fully implemented in the mainline kernel.

Also if you check out the interview I did on The Linux Link Tech Show you notice I mention Proxmox VE (about 37 minutes into it) and speak quite favorably of it.

I also appreciate that the Proxmox VE folks give back a lot to the community. First of all Proxmox VE is GPLed.. and I've seen PVE developers be very active on the OpenVZ mailing lists... submitting patches to a few of the OpenVZ tools. They also wrote the VM backup utility vzdump.