A response to "The Fedora-Red Hat Crisis"

As seen on Slashdot and elsewhere is the Bruce Byfield article entitled "The Fedora-Red Hat Crisis".

I'd put this response as a comment to the article on the place where it was published but the site doesn't appear to have a comment system... but given all of the ads there, perhaps I missed it. Anyway... Bruce is inaccurate in a few points that I feel must be addressed.

Perhaps I should have done a better job with my references and as time passes I'll try to improve this... but I wanted to get it out there ASAP.

Corporate Interests before FOSS?

Bruce claims that Red Hat put corporate interests in front of FOSS interests. Ok, how did they do that? They have kept details of the Red Hat and Fedora break-ins private and have said that they have done so for legal reasons. It would appear that Red Hat is protecting themselves as well as the Fedora project. How exactly is that anti-FOSS? Part of the "legal" reasons is because there is going to be an investigation (probably already in progress) and perhaps, if the culprit(s) can be caught, a prosecution. It is like the police keeping certain details private until they are ready to file charges. That may or may not happen but it is understandable.

One of the differences between this and the Debian case was the nature of it. Debian's problem was of their own making (altering the source to a package) whereas Red Hat's / Fedora's was from an outsider. Debian had no reason to keep that under wraps because they weren't trying to find someone and prosecute them... and no legal organization that I'm aware of was going to do an investigation. So since they are completely different situations, I'm not sure if it is reasonable to compare them.

Security Through Obscurity?

How did Red Hat use security through obscurity? S-t-o is where there is an ongoing exploit or vulnerability and the people responsible for fixing it do not do so and rely on obscurity to keep it safe. How exactly did Red Hat/Fedora do that? I believe Bruce totally misuses the phrase in his article and that it is a non-issue.

Handled Horribly?

Bruce claims that Red Hat handled the situation horribly. Putting the PR issue asside, Red Hat / Fedora handled the situation in the best possible technical manner. I see no evidence nor claims otherwise.

a) With regards to Red Hat - They released a script that detects the faulty packages that were signed by the intruder... but never distributed via RHN. They released updated versions of ssh to conflict with the packages that were signed.

b) With regards to Fedora, they have decided to generate and use a new signing key even though... all evidence has shown that they original signing key was not compromised (they say the passphrase needed to use the key was never used during the period the breaking occurred) and switching keys for the end user might be a tedious process. Their {arch}.newkey repos just showed up today and they have a process for the transition. There has been some criticism of the method used to transition from one key to another but not really any better alternative.

So I ask, with the exclusion of the PR aspects, was their technical response "horrible"? To me the technical response is the more important of the two.

Given the fact that their PR response methods could be considered "horrible"... to me that is actually a good thing in a way. How so? Well, they didn't try to sugar coat it or spin it in some unreasonable way... like you would expect from Microsoft or almost any other proprietary software company. Bruce's comparing Red Hat to Microsoft was a huge stretch of the imagination in my opinion. Of course that comparison is controversial and would probably get a lot of page hits so more power to Bruce.

Not Protecting the Users?

Bruce said that Red Hat/Fedora left the users vulnerable. How so? While it is true that Fedora's updates were delayed during the period of time it took them to get to the new signing key process... and to get all of the old updates resigned with the new key... and the repositories synced across all of the mirrors... the users were never at risk as a result of the initial situation. I commend the warning made in the initial Fedora announcement about being wary of updates. It turns out that there was not a package threat so the initial warning was overkill... but they didn't know that at the time because their own internal assessment of the situation was ongoing.

I commend Fedora for releasing information to the users before they were completely sure of the situation. That is very much an indicator that the situation was not handled in a CYA corporate manner.

Standard practice for most proprietary software vendors is to say nothing until an issue finds its way into the press somehow... and then only respond with the minimal amount of information possible... usually denying everything until it can no longer be denied. That CERTAINLY WAS NOT the case with Red Hat/Fedora. They were out early with information... so early it was incomplete... and they did it of their own accord rather than as a mandatory response to some third party. Of course perhaps they were afraid that the intruder would somehow make some announcement before them and they were trying to get ahead of it. Who knows.

Bruce seems to also forget the two things that Red Hat did:

a) They released a script that detects the bad packages that were supposedly signed with a valid signing key by the intruder (supposedly in an automated process where the intruder did not have access to the passphrase for the signing key). The bad packages were never distributed by Red Hat's Red Hat Network system... and the only way someone could potentially install the packages is if they used some other source of binary update packages than RHN... which is, as I understand it, a violation of Red Hat's service agreement.

b) Red Hat also released updates to their ssh packages so they would be newer versions so there would be less potential confusion between the bad packages and the current ones provided by Red Hat.

In Conclusion

While it is true that the whole situation did show many areas where improvement is needed... and folks like Jon Corbet have already analysed the situation and shown how Red Hat / Fedora (and every other distro maker too) can learn from the situation and do better next time, I think they showed leadership in how they handled it and shouldn't be too ashamed of the "corporate speak" found in the information releases.