5107R & 5108R SSL certificate issues

Looking at the SSL certificate import issue on 5107R and 5108R. A fact finding tour.

There had been numerous reports over time from users who had problems importing SSL certificates on 5107R and 5108R.

I already had set out to debug this repeatedly in the past. But usually when I tried, it worked fine for me. Well, I guess that's the so called "Murphy-Effect". \o/

But just today I had another client who had this problem and who offered me access to his server to troubleshoot the issue. And I think I finally tackled it.

First of all, the generic error message generated during the upload of the SSL certificate wasn't helpful at all and /var/log/messages did shed no light whatsoever on the nature of the problem. That was pretty odd to begin with.

So I started my hunt at the PHP page where the certificate is uploaded: /usr/sausalito/ui/web/base/ssl/uploadCert.php

The code in question uploaded the SSL certificate as $cert and then opened it through fopen(). It was then read line by line via fread() and written off again via the CCE method putFile().

This seemed to work fine on 5106R and I doubt that this code ever changed much from the original Sausalito code of the RaQ550 from way back.

But ... why the hell would you do that?

After adding a few debugging switches to the above code it became clear to me that the fread() was already chewing thin air and that the information about the certificate contends had been lost there. Not wanting to look much further, I set out to rewrite that section using the is_uploaded_file() and move_uploaded_file() functions specifically designed for such purposes. There is also no reason whatsoever to use fputs() to write that file off again. What for? We're not changing anything in it, after all.

So after cleaning up that code we now had a temporary file in /tmp with the uploaded certificate. The upload part therefore worked fine. But the import part handled through the following code still failed:

$ret = $helper->shell("/usr/sausalito/sbin/ssl_import.pl $tmp_cert --group=$group --type=serverCert", $output, $runas);

I tried to replicate that from the command line:

[root@raq6 ssl]# /usr/bin/openssl x509 -text -noout </tmp/file4pq4so
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 405423 (0x62faf)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=GeoTrust, Inc., CN=RapidSSL CA
        Validity
            Not Before: Apr 21 23:32:15 2012 GMT
            Not After : May 24 21:09:21 2014 GMT
        Subject: serialNumber=tYR07SdeHqeA/nns5RkZqZzUMnd9Z6Dk, OU=GT08456882, OU=See www.rapidssl.com/resources/cps (c)12, OU=Domain Control Validated - RapidSSL(R), CN=www.company.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:b0:df:2e:72:ba:c6:ca:71:26:45:e9:c5:69:ce:
                    [...]
                    72:30:87:3d:b1:98:40:3d:19:39:74:31:a5:70:6c:
                    64:91
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:[...]

Ok, so we got some info from the certificate. However, the script /usr/sausalito/sbin/ssl_import.pl also produced a ton of errors about usage of uninitialized values in CCE.pm. That certainly also contributed to the problem, as our PHP code interpretes the return values and doesn't want to see an error code other than "0", which means "no error".

So I looked at /usr/sausalito/sbin/ssl_import.pl and the first couple of lines already struck me as odd:

#!/usr/bin/perl -w
# Copyright 2001 Sun Microsystems, Inc.  All rights reserved.
# $Id: ssl_import.pl,v 1.8.2.1 2002/02/05 21:47:18 pbaltz Exp $
# import an uploaded signed certificate and optionally a private key

use strict; use lib qw(/usr/sausalito/perl /usr/sausalito/handlers/base/ssl); use Getopt::Long;
use CCE;
use Base::HomeDir qw(homedir_get_group_dir);
use SSL qw(ssl_get_cert_info ssl_create_directory);

The script uses "-w" (for enforcing 'strict' mode) in the interpreter line. Then it uses "use strict" again further down, which is redundant, but not really problematic.

However, what is problematic is that the interpreter line doesn't read like this:

#!/usr/bin/perl -I/usr/sausalito/perl

Sure, the libraries are loaded further down in the "use lib" line, but ever since Perl-5.10 this has been a bit of a hit and miss in my experience.

Once that was changed, the error messages went away. Now /var/log/messages finally showed some error messages that I could work with:

client 0:[0:31551]: SET  2377 . SSL importCert = 1335268840
client 0:[0:31551]: SET  2377 . SSL email = "" country = "" orgUnit = "Domain Control Validated - RapidSSL(R)" orgName = "" city = "" expires = "May 24 2014 21:09:21 GMT" state = ""
client 0:[0:31551]: SET 2377.SSL failed (-5)

There we go. We get a "-5" error from CCEd, which means that some mandatory information is missing. Replicating the transaction manually via cceclient revealed that "country" is mandatory information. And it's missing here.

A closer look at /usr/sausalito/sbin/ssl_import.pl revealed that this information is gathered via ssl_get_cert_info() from the SSL certificate. This is done through the Perl library /usr/sausalito/handlers/base/ssl/SSL.pm. And it fetches that info from the subject line of the decoded SSL certificate:

        Subject: serialNumber=tYR07SdeHqeA/nns5RkZqZzUMnd9Z6Dk, OU=GT08456882, OU=See www.rapidssl.com/resources/cps (c)12, OU=Domain Control Validated - RapidSSL(R), CN=www.company.com

Oh, look there: The subject line doesn't contain the country! Funny how that goes, because the SSL certificate from RapidSSL (same issuer!) for another domain of mine has it in the subject line of the decoded SSL certificate:

        Subject: serialNumber=N1l/jleSZBdpWm83Gu3qKOBZZCylzaf7, C=DE, O=www.mycompany.net, OU=GT62023476, OU=See www.rapidssl.com/resources/cps (c)11, OU=Domain Control Validated - RapidSSL(R), CN=www.mycompany.net

Funny how that goes!

So I changed the script to hard code the country to US and the state to "Other" if we cannot get that information from the decoded SSL certificate.

There we go. Problem solved!

Looking back at the whole mess and taking inventory, I can say that the problem was caused by a combination of the following issues:

• The GUI pages for uploading SSL certificates used PHP methods no longer supported that way on PHP >= 5.3
• The Perl script for decoding the SSL certificate and for shoving it into CCE was running on errors in newer Perl versions as it was not finding its support libraries.
• Some SSL certificate authorities changed the information in the "Subject" line of their SSL certificates and "country" is no longer mandatory information for them. But for us it was.

Just great <sigh>. Well, a patched base-ssl.mod is now in SVN and will be released today.