Tales of Enterprise Linux Major Version Upgrading

In the last post I mentioned that I migrated from CentOS 4.9 to Scientific 6.1... and that certain aspects of this Drupal 4.7.x site were broken because of an incompatibility with PHP 5.3.x.

Downgrading a distro

Well, I decided to move from Scientific Linux 6.1 to Scientific Linux 5.7. EL5 offers both PHP 5.1.x and PHP 5.3.x and Red Hat announced a few weeks ago that they are extending the support lifecycle of both RHEL5 and RHEL6 from 7 years to 10 years. Migrating back to EL5 fixes the issues (knock on wood) that I was having with Drupal... but yet I can easily move to PHP 5.3.x at some point in the future if I so desire.

Doing EL major version upgrades

Two friends of mine happened to have CentOS 4.9 OpenVZ containers as well. They also run a number of services I'm less familiar with and weren't really versed enough with Linux to migrate their data like I did. In an effort to help them out, I looked into how to upgrade from EL4 to EL5. That really IS NOT supported or recommended but I thought I'd give it a try and see how it went. If it failed, I'd roll back to the original system. If it succeeded I'd keep it. After much work I *THINK* I figured it out. At least it worked for me in the particular situation I was dealing with. I started off with a page on the CentOS wiki about Upgrading from 4.4 to 5. I did not do a boot media based upgrade (I'm working with containers) so I did it strictly with rpm and yum.

I followed the instructions but they were written some time ago and were a bit outdated. So the first container I did took the longest because I was finding my way. Basically this happens in a few steps.

1. Install the EL5 repos
2. Manually download the core packages recommended and install them.
3. Hopefully when you are done rpm is still working. If yum is broken, manually install a few more packages to make it work.
4. With a working yum, upgrade everything else
5. Turn off any new services that happen to be on by default that you don't want
6. Find any stray packages left over from the previous release
7. Fix your service configs by comparing your original service configs with the new ones

Read on to find out more of the nitty gritty details.

More on what packages to manually download and upgrade with rpm

As the wiki says, download the EL5 repo packages and install them and import the signing key.

Just to be clear, a lot of this is really specific to what version of CentOS 4 you have installed and what version of CentOS 5 you are trying to upgrade to. If you are using a different release on one or both, the packages and package versions may change... but you can use this list as a loose guide. Please note the containers I was upgrading were 32-bit. If you are working with a 64-bit system, you'll need to alter the URLs below for the 64-bit packages.

wget \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/rpm-4.4.2.3-22.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/rpm-libs-4.4.2.3-22.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/rpm-python-4.4.2.3-22.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/yum-3.2.22-37.el5.centos.noarch.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/popt-1.10.2.3-22.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/glibc-2.5-65.i686.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/glibc-common-2.5-65.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/glibc-devel-2.5-65.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/glibc-headers-2.5-65.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/binutils-2.17.50.0.6-14.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/beecrypt-4.1.2-10.1.1.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/elfutils-0.137-3.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/elfutils-libelf-0.137-3.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/elfutils-libs-0.137-3.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-2.4.3-44.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-devel-2.4.3-44.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-elementtree-1.2.6-5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-sqlite-1.1.7-1.2.1.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-urlgrabber-3.1.0-6.el5.noarch.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/neon-0.25.5-10.el5_4.1.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/libxml2-2.6.26-2.1.12.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/libxslt-python-1.1.17-2.el5_2.2.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/db4-4.3.29-10.el5_5.2.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/libselinux-1.33.4-5.7.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/libsepol-1.15.2-3.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/mcstrans-0.2.11-3.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/m2crypto-0.16-8.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/krb5-libs-1.6.1-62.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/openssl-0.9.8e-20.el5.i686.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/readline-5.1-3.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/nss-3.12.8-4.el5_6.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/nspr-4.8.6-1.el5_5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-libs-2.4.3-44.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/openssl097a-0.9.7a-9.el5_4.2.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/libxml2-python-2.6.26-2.1.12.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/python-iniparse-0.2.3-4.el5.noarch.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/yum-metadata-parser-1.1.2-3.el5.centos.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/yum-fastestmirror-1.1.16-16.el5.centos.noarch.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/yum-utils-1.1.16-16.el5.centos.noarch.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/glib2-2.12.3-4.el5_3.1.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/keyutils-1.2-1.el5.i386.rpm \
http://mirror.spro.net/centos/5.7/os/i386/CentOS/keyutils-libs-1.2-1.el5.i386.rpm

I had to use the following rpm commandline to get everything to install. It doesn't hurt to run it without the --nodeps or --force just to see if there is a package or two that might be a good idea to remove before the operation as rpm will tell you about any conflicts it has. Of course don't remove any package that is going to take out a lot of stuff with it but if you must make sure to use --nodeps so it'll not take out the deps. With this recipe we are basically trusting our rpm skills to fix any problems that are encountered.

rpm -Uvh --replacepkgs --nodeps --force *.rpm

Again, that is a pretty brutal set of options there. You are basically saying you know what you are doing and telling rpm to abandon all safeguards.

Did you survive?

Ok after that operation finishes, do as recommended in the wiki page and clean up after rpm (rm -f /var/lib/rpm/__* ; rpm --rebuilddb). Does yum still work? If not you might need to manually install the python-sqlite package as mentioned on the wiki page.

Ok, NOW DOES YUM STILL WORK? Try wget. If wget doesn't work, I don't think yum is going to work. If you are missing a library on wget, fix it. Luckily since we are working in a container if something is broken and we can't download any more packages, we can use the host node to get the packages needed and then copy them into the container space. That really helps and wouldn't be much of an option on a physical machine... but I guess you could use sneakernet in that situation if necessary.

Ok, hopefully yum is working now. Now do the upgrade:

yum update

If yum tells you it can't find a needed package, try removing the package it is complaining about. Assuming it is a lesser important package (one that little or nothing else depends on), removing a package or two shouldn't hurt. You can try installing them later if desired. If using mysql, I believe I had to manually remove the mysql-libs package because that package has basically been merged into the mysql client package in EL5. Of course, removing it takes out other mysql packages but just add them back after the bulk of the upgrade is complete. If dummy-centos-4 is installed, remove it with rpm -e --nodeps.

Hopefully yum upgrade worked ok and you are almost done with upgrading everything. But wait, are there any older packages left behind? Maybe you have some stuff that came from some third-party repo or something. It doesn't hurt to look:

rpm -qa | grep el4 | sort

So, was there anything left behind? If it is from a third party repo, install the newer repo package for EL5. Now upgrade. Or remove and install. You have to be a master of yum. It took me a little work but I was able to get rid of the few lingering packages I ran into and get the EL5 versions installed.

Turn off unneeded services

For some reason, my upgrade pulled in a lot of dependencies and among them were some new services that I don't really care about. You can either try to remove those packages, but hey, they were brought in for dependencies so good luck with that. All I do is make sure to turn off anything I don't need / want:

chkconfig --list | grep '3:on'
chkconfig {undesired-service-name} off
service {undesired-service-name} stop (if it is running)

Fixing your service configs

There are some services you almost always alter / update the config files for, others you don't. When you upgraded your packages rpm and / or yum usually keeps the old configs and renames the new configs by tacking .rpmnew on the end of the file. Do an updatedb and then locate rpmnew. That will show you all of the services that you need to look at the configs of.

I must admit that I don't know every nook and crany of every service config but I can usually fake it pretty well. Step one in fixing your configs is to compare the original / old config to the new one using diff. Usually there isn't much one really cares about in the output unless it is something you yourself typed in by hand at some point in the past. Config changes you have manually made, you usually want to keep. For many services the differences between the old and new configs are not things you have changed. For those you can rename your old config and then move the rpmnew config to the original name. I like to add on .el4 to the end of the old config so I know where it came from.

For configs that I have a lot of customizations in, I use the new config and then copy and paste my additions from the old into the new. I think it is important to use the new config files and merge into it... rather than trying to take all of the changes from the new config and merge them into the old. This is especially true for apache. For sendmail, my old configs worked just fine. A lot depends on how big of a jump it is from the previous release to the newer release. Some services change more than others. Use your best judgement. My methods worked for me... but might not work for you.

Test it all out

Before you call it good, be sure to test everything and make sure it all works. Grep your apache config for all of your virtualhosts. Test them all out. You are used to always testing stuff out right?

If you find some broken stuff, fix it. I had to update my php.ini config some for the options I forgot to change when I was fixing my service configs. That was about it.

You did make a backup, right? If your upgrade fails horribly and there is just too much stuff to fix and you are overwhelmed and can't do everything that needs to be done, give up... and revert to your backup. That wasn't the case for me on the three OpenVZ containers I upgraded... they were all a great success.

I have to wonder if a physical machine would be harder to upgrade in this fashion. It probably would be... because there is a kernel and real hardware to contend with. The generalization a container provides makes this process easier.