Update caused problems with PHP - fixed RPMS available
Last night's 'base-vsite' update contained several problems, which had a negative impact on PHP enabled sites. Fixed RPMs are now in the YUM repository.
Short summary of the problem and how to fix it:
The updated "base-vsite" was made available at 03:50 a.m. Central European Time and it contained several problems which have had a negative impact on PHP enabled websites.
At 10:30 a.m. Central European Time a fixed version of "base-vsite" (version number: 3.0-132BX99) was made available.
If your server is having problems with displaying PHP scripts, then please do the following to fix the issue:
1.) Login as "admin" by SSH and use "su -" to gain root access,
2.) Run the following commands:
yum clean all yum update -y
That will fetch and install the fixed "base-vsite" RPMs. A reboot or restart of any service is not required.
We aplogize for any problems that this update has caused.
How and why did it happen?
Again, my apology to everyone that was negatively affected by this update. I screwed up. That's for sure.
Thursday morning around 13:00 CET I finished the coding of the updates. The "base-apache" update was no problem and contained no surprise. The "base-vsite" update was a different story. I knew this update was potentially problematic, as it made some wide ranging improvements to our PHP security model. And to how the GUI handles PHP or suPHP.
But it also contained multiple really nice fixes that I was eager to publish.
Sometime during the late evening of Thursday and during the early morning hours of Friday I had tested the updates on 5106R, 5107R and 5108R test boxes.
At 03:50 a.m. CET on Friday 20th April I pushed the "Go" button and published the updates to the YUM repositories.
At shortly after 04:00 a.m. all the servers in the European timezone started to fetch their daily updates and that is when the problems started.
The problem:
The updated "base-vsite" prepends all PHP scripts on the server with /usr/sausalito/configs/php/set_php_headers.php to allow us the detailed logging of PHP related email traffic into /var/log/maillog.
For that purpose it stores some information about the executed PHP script into the environment variables, which we can poll and log.
So far so good. But the implementation I coded was flawed.
1.) PHP older than PHP-5.3 installed, PHP or suPHP enabled and 'safe_mode' enabled:
Enabled 'safe_mode' does not allow to change the environment variables. Unless you specify them explicitely in 'safe_mode_allowed_env_vars'.
Additionally: If 'safe_mode' is enabled, /usr/sausalito/configs/php/ is outside the path specified in 'safe_mode_include_dir'.
2.) Any PHP version, PHP or suPHP enabled:
If we prepend /usr/sausalito/configs/php/set_php_headers.php to all PHP scripts, then it is also required that the path to that script (/usr/sausalito/configs/php/) is specified in the 'open_basedir' directive.
The symptoms:
Depending on your PHP version and your 'safe_mode' settings you would get various undesireable results:
- Blank pages instead of the website.
- Safe Mode warnings like: 'Cannot set environment variable ...'
- Error messages like: 'open_basedir restriction in effect. File (...) is not within the allowed path(s)'
How did that make it through Q&A?
Simply put: Another screw up.
I was in a haste to release this and was the only one (as usual) who tested the updates. However, all my three test VPS's (one for 5106R, one for 5107R and one for 5108R) weren't really virgins anymore. They all were running PHP-5.3.10, which I had built and tested there two nights before. Additionally the Vsites on all boxes where I tested the updates had 'open_basedir' extended with a tiny little extra that is usually not present: I had allowed '/' to be within the 'open_basedir' restrictions.
That of course put /usr/sausalito/configs/php/set_php_headers.php within the allowed path. So anyone who didn't have '/' allowed in 'open_basedir' (like pretty much anyone) would have problems that I didn't see during testing.
The 'safe_mode' related problems (which only affected 5106R) were also invisible to me during testing, because I was running the Solarspeed PHP-5.3.10. PHP versions newer than PHP-5.3.0 no longer support 'safe_mode' and the BlueOnyx GUI then no longer shows the management pages for 'safe_mode'. Hence I didn't see these problems either.
So that's Q&A going at it with a blind eye, because the test boxes weren't really "stock".
That will be a lesson to be remembered: In the future updates will be tested on freshly installed systems.
The coding effort to fix this:
The first sign of troubles reached me at around 06:30 a.m. (CET), when my network monitor reported problems with www.solarspeed.net. A quick check showed a blank white page. A quick investigation showed that the most immediate problem there was that /usr/sausalito/configs/php/set_php_headers.php was not within the allowed paths of 'open_basedir'.
I quickly set out to fix the code and to release another update. Because I thought it could be fixed quickly and that the problem was less serious than it looked. In retrospect I should have immediately done a rollback as mentioned before.
Fix #1: v132BX97
After some furious hacking I had a fix ready at 08:22 a.m. (CET) and immediately released it to YUM after the briefest testing I could muster.
It had taken roughly two hours to code, because it was a bit more complicated than anticipated. I had to make sure that 'open_basedir' contained everything that had to go in there. Nothing more, nothing less. And some new defaults had to be enforced. That involved some heavy tossing around and merging of arrays, eleminating double entries and enforcing some checks. The way how I coded it is also not terribly efficient, but it was the quickest that I could come up with.
At 08:25 a.m. (CET) my phone started to ring constantly with reports about the issue and shortly thereafter there was the first report of the problem on the BlueOnyx mailing list.
After installing the fix #1 on a client's server, I noticed all the 'safe_mode' related problems.
Fix #2: v132BX98
Coding for this started at 08:35 a.m. (CET) and was finished at 09:15 a.m. (CET), when that fix hit the YUM repositories.
However, this only dealt with adding /usr/sausalito/configs/php/ to the 'safe_mode_include_dir'.
After I installed that fix on a client's server, I realized the problem with the 'safe_mode_allowed_env_vars' that needed to include the variable names that our PHP prepend script uses.
Fix #3: v132BX99
Coding for this started at 09:30 a.m. (CET) and was finished at 10:25 a.m. (CET), when this fix was published to the YUM repository.
The coding was slighly less problematic, as I could re-use code fragments from Fix #2, but the testing was a bit more complicated. In the middle of this my in-house buildbox locked up hard and had to be rebooted. Splendid timing for that!
After relasing Fix #3 I took all mirrors out of the loop that don't have the latest updates yet.
Fix #4: Squirrelmail
Squirrelmail also did run into issues on 5106R where it is still installed. A fix for that was rolled up and released at 12:18 p.m. (CET).
Fix #5: phpMyAdmin
Anyone who was running a phpMyAdmin different from the "stock" BlueOnyx phpMyAdmin would also have problems with their Apache phpMyAdmin.conf for the public Apache webserver. If there was such a config for the public Apache webserver. Because usually on BlueOnyx we run phpMyAdmin only on the AdmServ. So the fix for that was a bit more complicated, but it was finally published to the YUM repositories at 13:30 p.m (CET).
Fix #6: base-subdomains
At 16:40 p.m. (CET) I got notified that subdomains also had some PHP related issues. Yeah, that's another place where PHP settings get written off. So I started to tackle that as well and published this fix to the YUM repositories at 18:20 p.m. (CET).
Final status:
As of 18:20 p.m. (CET) on 20th April 2012 we seem to have watered the storm and all issues appear to be wrapped up.
← Return