5106R/5107R/5108R YUM updates
In the lights of the recent cryptology crisis we have reviewed and modified some BlueOnyx related sources.
The following updates have been released:
PLEASE NOTE: The 5106R RPMs for base-admserv had a glitch that wasn't noticed for about an hour after release. In the file /etc/admserv/conf.d/ssl.conf on line 48 (the last line) there was a sneaky hard return between "SSLCipherSuite" and the first parameter that starts with "ECDHE-RSA-AES128-SHA256...". That caused AdmServ to fail.
As soon as that glicth was detected, we fixed and re-released base-admserv with a new version number. So if your AdmServ has failed and therefore your BlueOnyx GUI is unreachable, please log in by SSH and run "yum clean all" followed by "yum update" to fix the issue. Sorry about the troubles. The 5107R and 5108R RPMs of base-admserv were fine.
I usually don't make political statements or discuss world politics on the BlueOnyx project pages. But this time around this can't be avoided. Before I proceed, I need to make clear that these views are mine only and don't neccessarily represent those of the BlueOnyx project or its members.
The cryptology crisis:
You can call it whatever you want: "The Snowden affair", "The NSA affair" - in the end it doesn't matter. It affects anyone and everyone. Worldwide. Thanks to Edward Snowden we now all know that the worst conspiracy theories have become true and that they outrival the worst that George Orwell could think up for his masterpiece 1984.
"Big Brother" is watching us. The neo-fascistic US-regime has - for decades - done everything it could to subvert, infiltrate, intercept, analyse and store every form of electronic communication. Encrypted or not. Secret courts issue secret court orders to telecommunications companies, ISP's, service providers, software makers and OS vendors. Secret orders that the recipients are not allowed to talk about. They are not even allowed to confirm that they have received such secret court orders. How sick is that?
I'm of German origin and I know the German history. We've had two dictatorships on German soil (3rd Reich from 1933-45 and the communistic east German GDR from 1949-90). Both were police states with almighty security agencies like Gestapo or STASI. But these two horrible organizations fortunately have been swept away by history and they never had the unfathomable abilities that the US-regime has bestowed its own intelligence services with.
It doesn't matter if you're an US citizen, an ally or or an enemy: "big brother" is watching you. If they want to break into your cellphone, your PC, your laptop or server, they will get in. But they might not even have to, as they are globally wiretapping undersea cables, have installed bypasses into the worlds fibre communication lines, or have tabs into the large social networks, search engines, cloud services and application providers anyway. They're listening.
Now if they have such tabs into the communication lines, they can also write. They can replace data as it passes through. Them serving trojaned updates isn't all that unlikely all things considered. Man-in-the-middle attack anyone? There are technical ways designed to prevent these things. But they never were designed with an omnipotent foe in mind and might or might not just make the efforts needed to do so more difficult, but not entirely impossible.
OS related trust levels:
The next issue there is: All the big OS vendors are US based companies. Think Apple (iOS/MacOS), think Microsoft (Windows), think Google (Android). Anyone still using OS's from US based commercial outlets should think twice. Do you really think either one of these companies could afford to say "No!" to a secret court order that demands a government backdoor?
But hey, there are open source alternatives. Sure, but think twice again: RedHat, whose sources are used for CentOS, Scientific Linux and other clones. The Fedora Project - by itself firmly in the hands of RedHat as well. Can a RedHat say "No!" to a secret court order to install backdoors? At the end of they day they probably can't. As Open Source allows for a code review and audits it should theoretically be possible to detect such things. But at the end of the day: There is just so much code that it's impractical. And it could be hidden anywhere in plain sight.
Protocol related trust levels:
I touched the subject lightly already: Encryption might help a little here and there. We can sign updates with a checksum (we always did). But we have little control over the OS itself. There we have to trust "upstream", which in our case is CentOS and/or Scientific Linux. But they put the same trust in RedHat, from which they get their sources.
During the last few weeks I went through the sources with a fine toothed comb. Looking in likely and unlikely places. But hey, at the end of the day I'm just a geek who knows a little bit of everything and I'm not qualified to make a professional review of the RedHat sources. But yes, there are things that strike me as odd.
We know that the NSA also spends a fortune on subverting encryption standards, making sure that weaker encryption standards get selected by NIST, that random number generators in hard- and software generate not-so-random random numbers and what not. A lot of smart people are currently discussing, speculating and reviewing cryptology as we know it. To figure out where things might have gone wrong and which cryptology standards can still be trusted. As Bruce Schneier puts it: "The math is sane. The software isn't." So it could even be that some cryptology methods are excellent on paper, but the software implementation of them contains elements that make them less secure. In the light of theoretical advances in mathematics some older cryptology standards might also get soon near the point where they are a lot easier to subvert, or they might already have been broken by the "powers that be".
There are even active campaigns going where sources close to this or that government try to discourage the usage of this or that cypher, protocol or cryptological method. Either because it has already been broken by them, or they haven't and want to push people to use other - more easily - subverted protocols.
So at this time there is a lot of information flowing around. Some of it might be right, some might be almost right, others of it might be worse or better as it appears. And some of it might just be plain and utter bullshit. I don't know and only time will tell.
HTTPS and OpenSSL:
A lot of the big players (Amazon, Facebook, Google, Microsoft) have suddenly started to implement "perfect forward secrecy" (PFS) on their HTTPS servers. I don't want to bore you with the details, but allegedly PFS makes HTTPS more secure. An extra set of Diffie-Hellmann secrets is exchanged between client and server, which should prevent that someone records the connection and breaks it later. My explanation oversimplifies matters a bit, but that's it in a nutshell.
I looked at PFS and thought: Why the hell not? Anything might help and if they do it, let us do that for BlueOnyx as well.
So I also reviewed how we approach HTTPS in BlueOnyx. It's done via/etc/httpd/conf.d/ssl_perl.conf, with which we dynamically create Apache virtual host containers with all the elements needed to enable HTTPS for individual sites.
I used some external testing tools (www.ssllabs.com for example) to review the existing configuration to look for ways to improve it. SSLlabs gave a HTTPS site on BlueOnyx a C-rating, as we were still supporting all protocols that our OpenSSL makes available to mod_ssl:
That included some really old and weak stuff, too. Usually a client and server negotiate the strongest possible protocol and cipher that they both support. But with some tricks a weaker protocol can be used, too. Which is not what we want.
So I modified the SSL configuration and threw out all weak protocols. Then I tried to force SSL to preferably use the ECDHE cipher suites so that elliptic curve based PFS could be used. None of the ECDHE cipher suites seemed to work on any RHEL5 or RHEL6 clone (CentOS / Scientific Linux).
After a lot of digging it became clear why: RedHat is giving us the finger on that one. This OpenSSL related Bugzilla entry is from 2007. Let that melt on your tongue for a moment.
I'll give you the TL;DR version of the bug and the very revealing comments: Depending on who you ask, the elliptic curve cryptography stuff is fast, robust and pretty secure and everyone should use it. On the other hands there are those who question the NSA's involvement in its development and the selection of the curves that the elliptic curves use. In 2005 the makers of OpenSSL accepted the ECC implementation in a patch and it was included in OpenSSL 0.9.8 and all later versions. All Linux distributions (that I know) but those based on RedHat or Fedora provide elliptic curve cryptography and ECDHE ciphers in OpenSSL. But RedHat specifically removed that support from the OpenSSL that it ships. They claim it is for "patent reasons". The company Certicom (now owned by Blackberry, Inc.) holds over 130 patents relating to elliptic curves and public key cryptography in general. Sony was sued by Certicom over usage of ECC in 2007, but the case was dismissed when Sony claimed two cases of prior art.
In any case: RedHat is having qualms here, which apparently nobody else has. Draw your own conclusions from that, but in my eyes they are no longer the honest broker I believed them to be.
BlueOnyx and ECDHE ciphers:
So with the crippled OpenSSL that RedHat provided us with, we can't enable PFS in it's most modern fashion by using ECDHE ciphers. Backing out the forcible disabling of EC in OpenSSL-1.0.0 (CentOS6 and Scientific Linux 6) is not trivial. I can get it to work on a stock OpenSSL-1.0.0 without all the other RedHat supplied patches. But that would leave us with an OpenSSL implementation that has known security issues which some of those patches fix. So that's not a solution. Outright upgrading OpenSSL to the latest available version would be a way. But everything that uses cryptographic functions on Linux is compiled against OpenSSL and would need to be recompiled against the new version. The result would be a CentOS6 or Scientific Linux 6 which no longer would be compatible with the YUM updates from upstream. It would be a Linux distribution of its own and would need to be maintained in entirely by us. I don't think we want to go there - yet.
The next best thing would be to compile the latest OpenSSL into a separate directory and only recompile the network facing services against it: Apache, OpenSSH, Dovecot, ProFTPD and maybe (or maybe not) Sendmail. We already serve Dovecot and ProFTPD out of the BlueOnyx repository. So that would be doable.
However, the newer OpenSSL would also allow us to use TLS 1.2, which is a more secure protocol. At the moment we can only use SSLv3 and TLS 1.0. But that would also require that we use Apache-2.4 and no longer Apache-2.2. Which is tempting, but adds a new layer of complications.
In the end I decided to sit this one out for the time being. Because once we detour from the given update path, there is no going back.
Without elliptic curve and the ECDHE ciphers available, the next best thing that allows us to get "perfect secrecy forwarding" (PFS) working is using the EDH or DHE ciphers (Diffie-Hellman Exchange). However, the negotiation stage between browser and server is a bit slower compared to using ECDHE.
Now with that I could enable DHE based PFS for Chrome, Firefox, Java based clients, Opera and Safari browsers. That's pretty much everyone but Microsoft browsers.
Now IE 6 on XP and IE 8 on XP don't support PFS. But the rest pretty much do. They also support DHE, but in the default configuration DHE is giving such a low priority that it will almost never get chosen, or Microsoft only allows to use them with TLS 1.2, which we can't use either.
The net result is now that HTTPS sites on BlueOnyx (and the BlueOnyx GUI via HTTPS) will provide forwarding secrecy for all but Microsoft browsers.
Over all SSLlabs now gives a BlueOnyx HTTPS site an A-rating:
Now there is still some room for improvements and we will make them as we see fit. But for the best possible deal we will need an uncrippled and modern OpenSSL first of all. Especially in the lights of the ongoing Cryptocalypse a responsible Linux vendor would do the right thing and include the ECDHE ciphers.
What's more to come:
After dealing with HTTPS, our next focus is on reviewing the SSL implementation of the email related services on BlueOnyx, as well as OpenSSH and secure FTP.
While the Apache related updates we just released might make our OpenSSL implementation a bit more secure in general: Take it for granted that NSA and their friends are still stealing all your secrets. And nobody knows who else might be listening in. Because if protocols and methods have been subverted by them and/or US based companies supplied them with backdoor, then the information about how to exploit these might not remain in the hands of the new Gestapo Sturmtruppen forever. It might leak as well as other NSA secrets have been leaked in the last three months. With that I don't necessarily mean Edward Snowden, who seems to handle the releases in a very professional and ethical fashion. But there might be others who once worked for the NSA as well and made it off with just enough to know how to pull the right strings to get at your data.
Short of a bloody revolution and some good old fashioned hanging: If you're an US citizen, let your electorate know how you feel about the unlawful, unconstitutional and total spying that the Bush-regime and Obama-regime have unleashed on not only it's enemies, but its citizens, allies and friends alike.