OpenVZ

I was lucky enough to be a guest on the Sunday Morning Linux Review episode 115 to talk about OpenVZ. In prep for the show I wanted to provide the hosts with some recent, updated videos that show off OpenVZ. I made the following videos which are in webm format... so you can play them in your browser or download and play with a media player:

• 1-openvz-2013-intro.webm (Slides 18min/11MB)
• 2-minimal-centos-install.webm (Optional - Basic CentOS Install 5min/7.5MB)
• 3-openvz-install-on-centos.webm (Installing OpenVZ on CentOS 14.5min/21MB)
• 4-openvz-demo.webm (OpenVZ demo 41min/51MB)
• 5-openvz-gui-container.webm (How to make a GUI container 25min/82MB)

Related links:

openvz.org
OpenVZ Quick Install Guide
OpenVZ Stats
Blog post about 1,000 containers on a single host

Kir Kolyshkin from the OpenVZ Project talks about Linux Containers:

Breakfast - I had breakfast with Kir, his wife and daughter (the son must have been doing homework or playing video games back in the room?) at Dennys. I had a veggie-skillet. It was great.

OpenVZ Booth - Sunday was a much slower day. Activity in the larger exhibit room seemed to be much higher than it was in the Pacific room where the OpenVZ booth was. I spent quite a bit of time twittling my thumbs.

Daniel Robbins stopped by to see Kir again but he was temporarily away from the booth. When he got back I sent him over to talk to Daniel.

I talked to a long, blond-haired Google employee for an extended length of time... about fingernails. He had very long fingernails (for a dude) that kind of looked like claws so I asked if he played the guitar. He told me quite a bit about fleet programming and mapping memory to the disk... and mentioned that Google uses Ganeti and Xen for internal virtualization stuff... but not for running their external facing services. I learned more about fingernails than I thought possible.

OpenVZ Booth - KirTwo guys came up to the booth to ask if they could borrow my terminal window for a minute. I asked why and one said he wanted to show the other GNU Screen for the first time. I told them not to use screen but to use tmux instead... and then broke into an impromptu demo of tmux. They were amazed. Then a third friend came up who said he had been using screen for years but as a serial communications program (after having previously used minicom)... and that he had no idea that screen had the terminal multiplexing features that 99.9% of other screen users use it for. Showed him tmux too. I referred them to a UTOSC 2012 video on Screen vs. Tmux. They were very happy to learn it and thought it was awesome that it came from an OpenVZ booth and wasn't about OpenVZ. :)

There were about a dozen people who stopped by the booth to ask about OpenVZ. One guy said he had been to Bozeman before because he wife was on a trip skiing in the Bozeman area and broken her leg so they ended up visiting the Bozeman hospital.

Other Booths - Hard KernelI visited the Fedora booth and got the Multi-Desktop Live DVDs for Fedora 17 and 18 along with some stickers. I also got some media from the OpenSUSE booth. I really enjoyed several of the hardware related booths. For example there was a company there named ODROID Hard-Kernel that had a number of tiny Linux machines. The Tiny-Core booth had a few tiny machines as well. The server hardware guys were there with HP, ServerMicro and ServersDirect among others. Inkscape, KDE, LibreOffice and VideoLAN were represented. Several Linux distros had booths including Debian, Gentoo, Arch, Ubuntu, Fedora, OpenSUSE, and Tiny-Core. Did I leave anyone out? Cloud providers were there in numbers... as were storage solutions, databases, and backup solutions. The usual third-parties were there like the FSF, EFF, FreeBSD / PC-BSD, OpenBSD, OLPC, and Haiku. Linux box makers were there like System76 and Zareason. Various area user groups had booths. There were close to 100 booths in total. I believe I got pictures of everything.

OpenShift Presentation - I went to a presentation at 3PM entitled, Build Your Own PaaS using OpenShift Origin given by Red Hat's Steven Citron-Pousty (ODP slides). He was giving away 4GB USB thumb drives that were stainless steel with a bottle opener to those who asked good questions. I asked a good question but I didn't want the swag. OpenShift is available as an online hosted service (public cloud on Amazon EC2 known as OpenShift Online), as a pay service from Red Hat known as OpenShift Enterprise, or as a do-it-yourself upstream project known as OpenShift Origin. The only thing OpenShift requires is either Red Hat Enteprise Linux 6 (or clone) or Fedora. I learned that OpenShift uses SELinux and cgroups to make service gears which are functionally very similar to containers. OpenShift is an elaborately designed system whereby a wide variety of components can be mixed and matched quickly and easily to stand up almost any web-based service. Some of those services include databases, middle-ware, development languages, frameworks, developer tools, and some packaged web applications. Each category is a who's who of big name open source projects. OpenShift grabs what you want it to, creates a gear of out of it, and then auto-magically configures everything with a unique private IP address and uses HA-Proxy to tie it all together. Other, non-web-based, services are on the road map but do not currently have a target date. OpenShift looks very interesting for hosting companies or any mid-to-large company that is constantly deploying a number of new web-services. The fact that they have tied together SELinux and cgroups to make psydo-containers is very interesting. Someone in the audience asked if they considered OpenVZ and Steve said he didn't think so because he was unfamiliar with it. In summary with OpenShift, System Administrators can easily deploy anything that developers want and developers can do what they need to do and very little more.

Post Show - After the show I just went back to the room and watched TV. The Oscars came on. They come on much earlier in L.A. They ended at 8PM. I watched more TV (while uploading pictures I took from the show). Then the local news came on. It had the normal news stuff except they also included who was attending whos after oscars party. Being in L.A. on Oscars night is kind of cool.

Conclusion - This was my first visit to SCaLE and I must say I was impressed. The Hilton was a very fancy place to have it and the accommodations for the presentations were reasonable. SCaLE definitely was not the only stuff the Hilton had going on as I saw a number of other events in a few of the other conference rooms.

Wnen the show was over there was a team of about 10 people from SCaLE breaking down all of the network stuff. This was their 11th year and it went like clockwork. There was a significant amount of Linux luminaries and a wide variety of talks that ranged from beginner to kernel development and everything in-between. There were close to 100 exhibit booths and tons of swag. The exhibit floor was packed for much of Saturday but quite a bit thinner on Sunday. The exhibitors were made up of a good mixture of commercial venders as well as .org projects. While I was stuck in a booth for most of the show I did get to attend a few presentations. Supposedly most of the presentations were video recorded and I hope to see them posted online ASAP so I can check them out. I'd definitely attend a future SCaLE if given the opportunity.

OpenVZ BoothSaturday at SCaLE was fun. I staffed the OpenVZ booth most of the day... from 10 AM until 6 PM. I did get a few breaks and a break for lunch. I had a number of people stop by the booth who mentioned they were using Proxmox VE and really liking it. Quite a few people had no idea what OpenVZ was and so I would start off back asking them if they use any virtualization products and the answer was usually yes. If they were a hobbyist user, their answer was usually VirtualBox. If they were a business type person their answer was usually VMware or KVM. I was ssh'ed into a couple of remote machines that were OpenVZ hosts so I was able to show what containers looked like by doing a pstree on the host and showing multiple init processes.

A guy from IllumOS dropped by the booth to ask if anyone was running KVM virtual machines inside of OpenVZ containers. Not to my knowledge. He said they were using Zones isolate KVM VMs on IllumOS. He seemed to have some concern that KVM wasn't secure/isolated enough and that users might be able to break out... and that the zones would keep everything safe. I mentioned sVirt for KVM but I've not used it myself.

Kir gave his presentation at 3PM on Checkpoint and Restore In Userspace (CRIU) which is a sub-project of OpenVZ. He said it went well and he filled his time but he didn't get a chance to actually show a demo... which was a shame because he had a nice video that showed it from beginning to end. Hopefully I can get him to share that video online RSN.

Kir also mentioned to me the commercial containers product that Parallels has to do Windows containers on Microsoft Windows... and that it was a monumental product with a high level of Windows internals knowledge on display... and that they were trying to work with Microsoft to share information... but that Microsoft didn't seem to interested. They are toying with the idea of possibly releasing OpenVZ for Windows but it seems unlikely. Containers on Windows has to bypass some kernel anti-patching technology Microsoft has so it can install the container functionality and then it has to re-enable it to keep the bad guys out. I'm not much of a Windows person and I definitely don't know Windows internals at all... but it was interesting even when dumbed down for me. :)

For lunch I thought I'd head over to Carls Jr. again but that place was packed... mostly with SCaLE attendees... so I ended up walking about a mile to a Burger King. That was a nice bit of exercise. Speaking of exercise, I decided that I wasn't going to use the elevator and take the stairs... which is a bit of exercise because as you may recall my room is on the 12th floor. I did that about three times down and up on Saturday.

Daniel RobbinsA long time OpenVZ user named John Wenger from the L.A. Co-op stopped by to visit with Kir but he was away from the booth at the time. Late in the afternoon a guy from the Zenoss booth stopped by to say that he was going to give a demo using ZenPack (or something like that) that started up a few OpenVZ containers and showed off the monitoring capabilities of Zenoss. I commented on his "Bring IT" tee-shirt. After I had been talking with him for about 10 minutes it turned out that he was Daniel Robbins... who I have talked to a number of times on the #openvz IRC channel. Perhaps you recognize his name. Daniel was the creator of the Gentoo Linux distribution. These days he works on Funtoo. I jokingly asked him to sign my arm but he said I was weird. :)

Maybe it is because I'm in the L.A. area and I watch a lot of TV but one guy stopped by the booth who looked a lot like Hugh Laurie. If you don't know who that is, look it up. Anyway, he didn't know who Hugh Laurie was and hadn't been told he looked like him before. If only I had taken a picture. Anyway.

A few people reported that Java-based apps (like Jenkins I think) didn't perform well inside of a container and tended to spike I/O usage that would make the machine unresponsive for periods of time. Told them to check bugzilla.openvz.org and jump on an existing bug report if one existed or to file a new bug.

I got a chance to walk around both exhibit rooms and take pictures of all of the booths... but once I got them copied off of the SD card to my netbook I discovered that the vast majority of them were terribly blurry. I guess I was in too big of a hurry and wasn't holding the camera still long enough. The lighting in the place isn't that great but I didn't want to be pulsing a flash in everyone's eyes. Anyway, I dumped that set of pictures... so sorry, no pictures today like I had promised. I will make a concerted effort to take all new pictures.

I went by the OLPC booth and saw Caryl Bigenho. As you may recall, her and her husband Ed have a summer home outside of Bozeman and have visited every year. Caryl gave a presentation to the campus on the OLPC about two or three years ago... and she is the one that suggested we apply for a lending lab which netted us 10 OLPCs to share with the Montana community. I ended up going out to dinner at Denny's Restaurant (that is about half a mile from the Hilton) with Caryl, Ed, and a friend of theirs named Tony. Tony told me some about various OLPC deployments he has helped with in a few different countries in Africa. That was very interesting.

At 9PM I attended a Raspberry Pi Birds-of-a-Feather (Bof) get together. Caryl Bigenho lead it and asked everyone who had one or more Raspberry Pis and what they were doing with them. One guy who I think had an Australian accent but said he visited England at least once a month and that his son was involved with the MagPi magazine Kickstarter project. He said his son was now working on another Kickstarter project for an add-on power button for the RPi... and he showed a prototype microSD card adapter for the RPi that was different in that it was NOT the normal SDcard size. It was specifically made for the RPi SDcard slot so you could put a microSD card in it, plug it into the RPi and it does not stick out from the slot and is flush with the side of the board. I won't go into the various projects that people were doing because it was a very broad list (ham raid, First Robotics projects, co-location for RPi as a hosting server, etc). There were a couple of people from the Tiny-Core Linux booth and they said they had recently gotten Tiny-Core going on it and that it was the smallest and fastest Linux distro yet. They are showing it at their booth for anyone who wants to stop by and see it. Caryl mentioned that she had recently acquired an SDcard with Sugar (the OLPC learning environment) on it for the RPi. She hadn't had a chance yet to try it out but hoped to later in the day on Sunday.

There is a huge amount of interest in the RPi and even though other competitors may come and go in the space, the RPi has the numbers and add-ons and the people excited about it... that it should remain a viable platform for at least a few years.

This is my first time at the Southern California Linux Expo (SCALE) and it is their 11th year. You see, just a few days ago I had no plans to attend but then Kirill Kolyshkin contacted me via IRC asking if I was available to attend to help staff the OpenVZ booth. If you haven't heard of Kir before, he is the OpenVZ Project leader who is employed by Parallels. Having never attended SCALE I was very excited about going and checked with my two bosses (the wife and work) to see if I could go. Luckily both gave the thumbs up.

SCALE is actually FRIDAY - SUNDAY. Turns out that the Exhibit floor doesn't open until SATURDAY. Living in Montana my flight took me through Denver and by the time I got to Los Angeles and had ridden a free shuttle bus to the Hilton it was about 3PM PST.

I was so excited about going to SCALE, I had trouble sleeping Thursday night and add jet lag to that... I didn't have a whole lot of energy and went to bed around 8PM. Between hitting the hotel and going to bed though, I did do a few things.

1) I stopped by the exhibit floor to see how everyone was getting along setting up their booths. The exhibit floor is actually in two large rooms on the ground floor of the Hilton. One room is smaller than the other and the OpenVZ Booth (#93) is in the smaller of the two. I found the OpenVZ booth and I saw that Kir had already gotten it set up with a nice OpenVZ/CRIU banner and a large flat screen monitor.

2) I then checked in and got my room keys at the Hilton. I'm on the 12th floor. I went up to the room and got my netbook on the network to check my email and get on IRC. In IRC I saw Larry Cafiero. Larry is a SCALE promoter and PR person who is trying to work with a few of us in Montana to get a Montana Linux Fest in the not too distant future. Larry asked me to go to the Catalina C room to visit with him to touch base. I have visted with Larry several times at various shows (LFNW and UTOSC) when he was still associated with the Fedora Project. We chit chatted about the trip thus far and when I might start concentrating more on a Big Sky Linux Fest.

3) I attended Robyn Bergeron talk entitled, Managing the release and life cycle of an open source software project in a community. She jokingly said it as the longest presentation title ever.

4) Then I went to Carls Jr across the street from the Hilton and had one of their turkey burgers.

5) Then I went back to the room and the phone rang. It was Kir. His room is on the 3rd floor. He said he was doing some last minute work on his CRIU presentation (that is on Saturday at 3PM in the Century CD room) but that he wanted to get together to discuss the latest happenings in the OpenVZ world and what he has been working on... so I'd be better prepared for the booth. So, I went down to his room and we talked for about a hour or so. I got a gigantic brain dump worth of information. Turns out Kir (and his wife and two children) moved from the Moscow Russia area to the Seattle area sometime in December. Parallels has a small office there. Also in the same office is Linux kernel hacker and Parallels CTO of Server Virtualization, James Bottomley. You might have heard that James has been working lately on a secure boot setup for The Linux Foundation. Anyway, Kir mentioned that James wanted him to attend Matthew Garrett's Saturday morning keynote entitled, The Secure Boot Journey. Kir also wants to attend a presentation on Linux Native Containers (LXC) and of course he has his own presentation at 3PM. That means I'm going to be at the OpenVZ both for quite a while by myself. That's ok. Kir said that if there were any presentations I wanted to go to on Sunday, I could. I haven't really looked at the schedule yet.

Then Kir's wife and children (a boy and a girl) got back to the room after having toured around Hollywood. It just so happens that the Kolyshkin family had driven down from Seattle to L.A. which is quite the drive (about 1,000 miles or more?) although believe it or not, still shorter than some of Kir's flights over from Moscow to the US for various trade shows. They invited me to dinner but I was still full from the turkey burger and declined.

I went back to the room only to notice the Fedora Project had pushed out a considerable number of Fedora 18 updates including the 3.7.9 kernel and Firefox 19. I got my netwook all updated. I ssh'ed into my MontanaLinux build hosts at work and rebuilt with all of the updates. While that progressed I watched a little American Pickers on the History channel. It wasn't too long after that that I drifted of into the world of slumber.

I just woke up about 4AM PST and began writing this blog post. Getting up so early, I have about 5.5 hours before the opening of the OpenVZ booth. Wooo hooo.

I definitely have a lot to share from the talk I had last night with Kir but I'll wait until later to do so. Hmmm, I guess I do have to turn the light on in the room to get the coffee pot going. :)

I didn't bring my video camera because I thought I'd be stuck in the booth the whole time but I did bring a camera so expect lots of pictures from Saturday and Sunday. Assuming I have Internet connectivity (we as supposed to) at the booth, I'll be on IRC "live from the SCALE booth" just for the fun of it.

Oh, I guess I missed Jono Bacon's presentation late Friday about the Ubuntu Phone. In all honesty, I had no interest in attending.

It seems I've had a lot of questions about OpenVZ container migration lately on the #openvz IRC channel on the Freenode IRC network. While I made a silent screencast on that topic a few years ago, I thought it was time for a refreshed one so here it is. Enjoy.

What is an OpenVZ container? It is a form of virtualization where you can create a type of a virtual machine called a container that is basically a strongly isolated chroot environment with device and resource management features.

What is migration? It is the ability to easily move a container from one physical OpenVZ host to another. Live / online migration allows for no downtime and maintains existing network connections. Offline migration stops the container on the original host and starts it up on the destination host and as a result the containers uptime is reset and existing network connections are dropped. Watch the screencast for all of this in action.

You can also download this directly if desired. right-click, save link as:
openvz-vzmigrate.webm (12.8 MB)

In the last post I mentioned that I migrated from CentOS 4.9 to Scientific 6.1... and that certain aspects of this Drupal 4.7.x site were broken because of an incompatibility with PHP 5.3.x.

Downgrading a distro

Well, I decided to move from Scientific Linux 6.1 to Scientific Linux 5.7. EL5 offers both PHP 5.1.x and PHP 5.3.x and Red Hat announced a few weeks ago that they are extending the support lifecycle of both RHEL5 and RHEL6 from 7 years to 10 years. Migrating back to EL5 fixes the issues (knock on wood) that I was having with Drupal... but yet I can easily move to PHP 5.3.x at some point in the future if I so desire.

Doing EL major version upgrades

Two friends of mine happened to have CentOS 4.9 OpenVZ containers as well. They also run a number of services I'm less familiar with and weren't really versed enough with Linux to migrate their data like I did. In an effort to help them out, I looked into how to upgrade from EL4 to EL5. That really IS NOT supported or recommended but I thought I'd give it a try and see how it went. If it failed, I'd roll back to the original system. If it succeeded I'd keep it. After much work I *THINK* I figured it out. At least it worked for me in the particular situation I was dealing with. I started off with a page on the CentOS wiki about Upgrading from 4.4 to 5. I did not do a boot media based upgrade (I'm working with containers) so I did it strictly with rpm and yum.

I followed the instructions but they were written some time ago and were a bit outdated. So the first container I did took the longest because I was finding my way. Basically this happens in a few steps.

1. Install the EL5 repos
2. Manually download the core packages recommended and install them.
3. Hopefully when you are done rpm is still working. If yum is broken, manually install a few more packages to make it work.
4. With a working yum, upgrade everything else
5. Turn off any new services that happen to be on by default that you don't want
6. Find any stray packages left over from the previous release
7. Fix your service configs by comparing your original service configs with the new ones

Read on to find out more of the nitty gritty details.

I threw together a quick screencast for Dann from the Linux Link Tech Show explaining the history of and differences between Xen and KVM. Feedback is encouraged because I'm sure I have some mistakes in there.

xen-and-kvm-history.webm (36.2 MB)

I posted a contributed OpenVZ OS Template today. The contributed OS Template is Scientific Linux 6 32 bit and it was contributed by Vic from powerpbx.org (canuck15@hotmail.com).

I asked him to share information about he created it and this is what he replied back with via email:

I have no plans to create a x86_64 version or provide regular updates to the x86 version at this time. The only reason I created the x86 version is because I needed a RHEL (or clone) v6 template for my own use. It is easy enough to update/modify/copy by someone else now that this version is out there.

I created it using this procedure and rsync from VMWARE to OpenVZ. Then I manually went through all the installed packages and took out as much as I could to get the size down. When in doubt I compared to the installed packages in a CentOS 5 template.

Yum would not remove kernel so I had to do a "rpm -e --nodeps kernel"

In the newly rsync'ed OpenVZ container I create a file called "vz.repo" in /etc/yum.repos.d with the following text:

[vz-base]
name=vz-base
mirrorlist=http://vzdownload.swsoft.com/download/mirrors/centos-5
gpgcheck=0

[vz-updates]
name=vz-updates
mirrorlist=http://vzdownload.swsoft.com/download/mirrors/updates-released-ce5
gpgcheck=0

then "yum install vzdev vzdummy-apache vzdummy-jre-el5 vzdummy-kernel-el5"

Could not get "vzdummy-glibc" to work. It caused the template to not load on reboot. Someone smarter than me will have to figure that one out. Perhaps vzdummy-glibc needs to be updated for RHEL 6.

Additional things I ran into that appear to be RHEL v6 specific are as follows.

You must comment out "console" in /etc/init/rc.conf and /etc/init/rcS.conf

You must also delete or rename tty.conf and start-ttys.conf.

I noticed Kir's blog post about the updated vzctl today. Cool! Finally I can create Fedora 14 containers... and the container restart mechanism has been fixed up too.

I downloaded the beta OS Template that the OpenVZ Projects offers for Fedora 14, created a container, did all of the updates, removed the samba* packages, added a few packages I wanted (mc, screen, links), and modified the httpd.conf so it is more like factory. Then I disabled a few services that aren't really needed... after all, who needs xinetd running when it it doesn't have any services configured? Then I stopped the container, cleaned up the container filesystem some, and tar.gz'ed it up and uploaded it as a contrib OS Template.

I did this for both the 32-bit and 64-bit OS Templates. Enjoy!