Interview: Troy Dawson from Scientific Linux

Troy DawsonRed Hat Inc. rules the "enterprise" Linux market with their Red Hat Enterprise Linux (RHEL) product line. Novell Inc. (now owned by The Attachmate Group) is second with their SUSE Enterprise Linux product line. To the best of my knowledge, there aren't any free SUSE Enterprise Linux clones but there are a number of free RHEL clones. CentOS is the most well known RHEL clone but with the seeming unending delay of the 6.0 release (July 11th is my guess), CentOS has received quite a bit of criticism leading some users to investigate alternatives. As a result, Scientific Linux is getting a lot of long overdue attention given the fact that it too is a solid enterprise clone... that has been around for a long time... that has a lot of support behind it.

MontanaLinux is proud to present an interview that was conducted via email with Troy Dawson who is a long-time Fermilab employee and Scientific Linux developer.

About Troy Dawson

Montana Linux: Please tell us about yourself... as much as you feel comfortable with... as open or as closed as you want to be... family, education, work, hobbies, etc.

Troy Dawson: My name is Troy Dawson. I have a Bachelors degree in Physics and a Masters degree in Computer Science. I have worked at Fermilab since 1993. I was initially an accelerator operator, and then transferred over to computing in 1999.

I've been working with Linux since 1999.

I am married with two kids. I am very active in my church. I think my main hobbies are family, church, and computers.

About Scientific Linux

ML: How did Scientific Linux come to be?

Troy: There were three forces that came together and caused Scientific Linux to be made.

First, there is a High Energy Physics (HEP) computing collaboration called "Hepix". Every six months computing professionals from various HEP labs get together for a hepix conference to discuss HEP computing. (http://www.hepix.org/)

Second, Fermilab had been making a Linux Distribution called "Fermi Linux" since 1999. It was a modified version of Red Hat. CERN had also been making their "CERN Linux", which was also a modified version of Red Hat.

Third, Red Hat decided to end their free Red Hat Linux and only have RHEL.

At the October 2003 Hepix meeting, the various labs were debating what to do about Red Hat changing from Free to Subscription. Most HEP labs have very tight budgets for software, and large compute farms. They cannot afford to pay for a license for each machine.

During that conference RHEL 3.0 was released. Connie Sieh (who was at the conference) was able to download and recompile almost the entire release while she was at the conference. We had a working version of Fermi Linux based on this recompiled RHEL by November 11, 2003.

CERN was also trying the same thing. One of their managers sent Connie an email wondering about collaboration. Connie, I and our managers, thought this was a wonderful idea, and Connie set about customizing the installer so that it was easy for the various HEP labs to customize it and make their own distribution.

Connie and I managed to get the first release of Scientific Linux out just before the next Hepix meeting in May 2004. Connie went to the Hepix meeting expecting to meet resistance and was ready to persuade people to use the new distribution. What she found was that many of the labs had decided to use Scientific Linux, even several of the labs that were not Red Hat based.

And thus it began. I know that's a little bit longer than you'll see in Wikipedia, but it's more clear as well.

ML: Who contributes to Scientific Linux and in what ways?

Troy: The credits page isn't up to date (http://www.scientificlinux.org/about/credit)

With Scientific Linux 6 our community has grown rather fast and I haven't been able to keep up with the credits.

I think all of the contributors and developers are people who stepped forward and said "I can do this part" and then did it.

Stephan Wiesand from DESY has consistently done a wonderful job with our openafs. Wolfgang Friebel, also from DESY, has been wonderful in providing the alpine package.

Urs Beyerle initially hadn't contacted the main developers at all (as far as I know) but just showed up at one Hepix meeting with these Live Scientific Linux CDs. Ever since then he has been a wonderful asset to Scientific Linux.

Linux Ink (the company) has been wonderful. They have not only provided packages (kdeedu), but they are some of our best testers. When they see a problem, they usually have a solution and/or patch for it. They also setup the Russian translated www.scientificlinux.ru.

John Outlan stepped up and created a Scientific Linux forum. (http://scientificlinuxforum.org/)

He made sure we weren't going to be upset at him, and when he found out we weren't he set it up on his own. He and a few others moderate the forum and it's pretty good.

Shawn Thompson and his artwork has been wonderful. Since the very beginning I've done all the artwork except for the second logo. I do enjoy it, but I can't say I'm all that good. Shawn showed up and created a full set of themes and artwork for both SL6 and SL5. I look at it and just say "Wow".

And there are so many others who contribute in their own ways. Some people are very good at answering questions on the mailing list. Some give me patches and code. Many are great testers. I sort of feel bad having just the short list that I gave you, but if I listed everyone, your article would be too long to print.

ML: Scientific Linux seems to be a fairly open and transparent community. Is this a requirement of your funding model?

Troy: The Fermilab environment, and I believe High Energy Physics in general, is very open and transparent. These scientists know that in order to get the best results, they need to collaborate with others, be open and honest with what they found, and accept questions and criticisms.

That environment spills over in all that we do at Fermilab. It has nothing to do with funding.

For Scientific Linux, we've found that being open and transparent is much easier. There are a lot less questions that way, and you can spend your time doing whatever needs to be done instead of answering the same questions over and over.

ML: Approximately how many systems around the world are running Scientific Linux?

Troy: We don't know. We know the minimum, which you can see at the stats page. (http://www.scientificlinux.org/about/stats/)

According to those stats we have a minimum of 56,000 last month. I did some different number crunching and I believe that it is closer to 70,000 to 80,000 machines that get their security errata from us each month.

We use the numbers on those pages for trends, not real numbers.

If you want to count the number of iso downloads (which I very rarely do) in March we had 1,300,000 downloads of the SL 6.0 i386 install DVD, and 1,000,000 downloads of the SL 6.0 x86_64 install DVD.

ML: What do you think about Red Hat's decision to package the kernel sources differently for RHEL6? Did it affect SL at all?

Troy: If that's the way they want to package them, that's their choice. It doesn't affect SL at all. From the very beginning we said we would not change the kernel at all, and we haven't.

ML: Red Hat's Knowledgebase has also gone subscriber only. Does the Scientific Linux community see that as a loss?

Troy: Until I read your question I didn't even know about that. It is extemely rare that anyone in SL ever pointed one another to Red Hat's knowledgebase. So I have to say it isn't any great loss to us.

Building an enterprise Linux clone

ML: Please tell us a little bit about the build system you are using.

Troy: The build systems for each release is different, and has evolved over the years.

When we first started with SL3, we had one machine for i386 and one for x86_64. At some point we merged them, so that the base OS was x86_64, and the i386 part was built in a chroot area.

For SL4 we did the same as SL3, with one machine and both arches. But at that point, we had a very good backend filesystem, so all the build scripts and most everything was done on the backend filesystem, for both SL3 and SL4. That made it possible for us to reinstall the build machines if needed.

We use a BlueArc NFS file server. It's awesome. Say what you want about NFS, but when you have a backend as good as a BlueArc, NFS is a totally different beast.

Then along came SL5, and we started using mock to build packages. This was wonderful. It had it's own challenges and we had to rewrite many of our scripts, but it's much nicer than just straight building.

We tried to use mock for SL4, but found that Red Hat didn't package their packages correctly for RHEL4, to allow for a mock environment. Around that same time, we started having hardware problems with the SL3 build node. SL5 was out, and I managed to get my hands on a machine that did hardware virtualization (rare at the time) so I virtualized the build environment for both SL3 and SL4.

For SL6 we wanted something that would allow us to use other build machines when the load is high, like when we are building the distributions. We looked at a couple different techniques but really liked how Fedora does things, so we settled on Koji. There are a few things that make it better for Fedora than a RHEL rebuild environment, but we've worked around those problems. I really like how we've been able to add and remove machines depending on our load and the other machine availabilities.

Right now we are only using Koji for SL6, but we are thinking of migrating the builds for SL5 over to it. But for now, the environment for SL5 works, so we're keeping it.

ML: What hardware are you using now to build the various Scientific Linux releases?

Troy: Answered above for the most part, except the main SL6 build machine is currently a dual quad-core Intel machine with 24 Gig of memory. During the initial SL6 build there were a couple more dual quad-core machines that were new and needed to be burned in. We got to use them as build machines, but they are moving into production now.

ML: When you run into package build problems, how do you go about getting them resolved?

Troy: Unanswerable. If there was just one answer I could tell you, but there isn't.

I will say that for the SL6 rebuild, the SL community helped out with several of the more tricky problems. Especially those people that are part of the elrepo project.

ML: What is the source of most package build problems? Are they mainly caused by packager mistakes? Are any of them on purpose?

Troy: Most problems arise from figuring out what platform, or state of the platform, that Red Hat built their package on.

For example, with RHEL 6.0, There were many packages, build on Fedora 12, then 13, then RHEL6 Beta. And if we didn't build them on the same platform, they either wouldn't build on didn't pass our tests. But these were relatively few, less than 100 out of 2000.

Sure there are packages that just weren't packaged well. Those are quite rare. And in the more stable releases (RHEL4 and 5) they are extremely rare.

I don't think any of the mess-ups are on purpose. In fact, I think it is quite the opposite. There have been many packages that Red Hat has cleaned up because they didn't build very well. It takes years for them to get cleaned up, but they have been.

For more info, see these two pages regarding package build issues with SL 6:
   Problem RPMs By Catagory - http://www.scientificlinux.org/distributions/6x/build/problem
   Problem RPMs By Package - http://www.scientificlinux.org/distributions/6x/build/problembyrpm

ML: One of the common complaints about Red Hat-based distributions relates to "dependency hell". What other package management systems are you familiar with and how do you think rpm / yum compares these days?

Troy: That question is very much a "vi versus emacs" question and I'd rather not answer it.

ML: When Red Hat went from Red Hat Linux to Red Hat Enterprise Linux they intentionally reduced the number of software packages they offer in an effort to improve their ability to offer quality support. Some have suggested that Scientific Linux switch to Debian because of its large package repositories. Do you see the smallish size of RHEL's repositories (when compared to Debian and Fedora for example) as a problem?

Troy: Yes and No. There are several times that I am very envious of Debian and Ubuntu getting all the packages. Or when someone only packages up their programs for Ubuntu. But with the Rpmforge, EPEL, atrpms, elrepo and other repo projects, Red Hat is getting a lot more packages that Debian has. And it sort of creates this natural filter. If it isn't good enough for someone like Dag, or people on EPEL to put into a package, then a lot of times it isn't worth my time. There are exceptions, but it's not something I dwell on.

ML: Who does all of the graphic work for the Scientific Linux branding?

Troy: That would be me. Except for the second generation SL logo (for SL4 and SL5), I've done every bit of graphics.

We did have a logo contest for SL6. I entered along with everyone else, and I won. I had hoped that I would get help with SL6 because about a year before someone offered to at least do the backgrounds. They even sent me a small preview, which I loved, but it never materialized.

One of the funniest graphics stories I have is for the SL 5.0. I was very concerned about the background and the gdm login screen. I asked a lot of people both on the mailing lists, and by personally going to their machine and showing them. Everyone said they were "good enough" because they didn't really care about the background (it's the first thing they change) and the gdm screen they never see, because they are always logged in. So we released it as you now see it.

The very first review I read totally thrashed my graphics, both my background and the gdm login screen. I guess I should have been offended, but I thought it was hilarious. I thought "where was this guy two months ago when I was looking for someone to critique it?"

Since I originally answered this question, there has been a change. A fellow named Shawn Thompson has picked up this torch and run with it. He's reworking all the graphics for SL 6.1, as well as SL 5.

Scientific Linux Team: Urs Beyerle, Troy Dawson, and Connie Sieh

Relationships with third-parties

ML: What kind of a relationship (if any) do the Scientific Linux developers have with Red Hat? Have you received any unsolicited communications from them?

Troy: None and nope. I (or Connie) regularly attend the Red Hat Summit. They are very nice to us there. But we don't get any extra help, or criticism from them.

ML: Do you think Red Hat is better off with the existence of Scientific Linux and the various other RHEL clones or do you think it / they have a negative impact on Red Hat's bottom line?

Troy: This is something I feel strongly about. I think Red Hat is better with the clones, particularly Scientific Linux. Here is my reasoning. National Lab's like Fermilab run Scientific Linux, or a mixture of RHEL and SL. Universities come to these National Labs and in order to work with the infrastructure, they have to run RHEL or SL. The Universities and their students are very low on money (as are many of the National Labs) so they run SL on their machines.

The students at those Universities learn Scientific Linux. They graduate. They get a job in some corporation. The corporation asks which type of Linux they should use, and those graduates say "I know Scientific Linux and it is a very good and free clone of RHEL. Since we want support, we should use RHEL."

In short, the free clones end up giving RHEL customers.

What about the clone called Oracle Linux? Oracle Linux gave Red Hat more advertisement then they could have dreamed of. If you look at Red Hat's stocks and revenue before and after the initial Oracle Linux announcement, there is a definite change. Red Hat took Larry's announcement and used it to their advantage and it has paid off.

The one thing I'm afraid of is Oracle Linux hurting the free clones like SL. It hasn't happened yet, but it's something I worry about.

ML: CentOS is noted as being the most popular RHEL clone distros. Why do you think CentOS is more popular than Scientific Linux?

Troy: Two reasons.

1. Many people think you have to be connected to the government or research to use Scientific Linux. That's not true, but many people still feel that way.
2. Because we are associated with a government agency, we have a very hard time collaborating. We can't let people edit the web pages, log into the distribution servers, and a lot of other things that people want to do that makes them feel like a community.

ML: I've seen some folks asking if the CentOS and Scientific Linux communities could collaborate with each other. Do you already collaborate and if so how? If not, why not? I know that CentOS and Scientific Linux have different goals, but are there any areas that would be no-brainers to collaborate on?

We've tried collaborating with CentOS. It didn't work out like we hoped. In the end we do some collaboration with them. We're on their mailing lists, and many of their users are on ours. If we are having a problem with something, the odds are they are too, and we'll ask them, and occasionally they ask us.

I know when people ask this question they are meaning we should both use the same security errata, packages, and so forth. That just didn't work out.

ML: Views of Red Hat among the Linux user community seem to vary. What do you think Red Hat's strengths are?

Troy: Red Hat's biggest strength is their dedication to open source and keeping things open. They have gone to court many times and whenever possible, they have fought to have the outcome help the greater open source community and not just them.

Conclusion

ML: Thanks for taking the time to answer my questions!