Developer talk: CCEd and i18n

This article is a somewhat longer detour through the ins and outs of the BlueOnyx architecture and the how's and why's of how things work. And where - over the years - fractures occured that had to be patched over to keep things running. These work arounds weren't always ideal, but given the code complexity we sometimes didn't have much choice. 

The introduction of AI aided coding has allowed us to address these issues and this explains how that was accomplished.

In the beginning ...

Back in the day when the Cobalt Networks engineers introduced the Sausalito GUI interface? They did a lot of smart things. CCEd itself? The security model it provides? Here we are - almost 30 years later - and we are still using a fairly large chunk of their code as a foundation and as building blocks for BlueOnyx. 

You also have to recall that back then the code was designed to run on Intel 486 CPUs and with less RAM than your average smartphone today uses as CPU cache. There is also the old wisdom that compiled code runs faster than anything written in a scripting language. 

The original Sausalito developers were well aware of that and the architecture they designed offloaded the heavy duty tasks into compiled code:

Cobalt Configuration Engine Daemon: CCEd

This is the daemon that was the heart and the brain of Cobalt Qube's and RaQs and it also drove BlueQuartz as much as it now drives BlueOnyx.

It provides a Unix Socket with which one can interact to "talk" to CCEd and access the "Cobalt Object Database" (CODB). The format of the database itself is defined in XML Schema files and their presence and content shapes and structures the CODB database into Classes and Namespaces with key/value pairs. Each of which have ACLs, have input validation to make sure only data matching the desired input of each field is stored into the backend. Invalid data is outright rejected. For example: A field that only accepts a single IPv4 IPs will not take anything that isn't a valid IPv4.

CCEd is also is event driven. If Classes or Namespaces are created, changed or deleted? Or even individual values within Classes and Namespaces are changed? Then event handlers can be launched that perform limited tasks with elevated privileges. Like changing service configurations, creating users and performing service restarts. Some handlers just validate conditions and run during configure stages of transactions, other execute or perform tasks in the cleanup stage.

CCEd itself is written in C and is pretty fast and rock stable. Over the decades we managed to port it forward and make it run on each subsequently newer OS that BlueOnyx runs on. We brought it forward from 32bit to 64bit when BlueOnyx 5108R (on EL6) came out and also tacked on some extra features over the years or squashed the odd bug whenever we found it. But all in all? CCEd is and always was solid. Kudos to the original developers!

Shortcomings of CCEd:

As great as CCEd is: You also have to understand its development history. It was created sometime between 1996-1998 at a time when Cobalt Networks was in a boom-time and under a lot of developmental pressure. Tight deadlines, too many Chiefs, too few Indians, long hours and a lot of constraints where it is surprising that they managed to do what they did. Then they were bought up by Sun Microsystems, people left, new people came aboard, things stagnated and then - as quickly as Sun had "fallen in love" with Cobalt it also "fell out of love" and binned and EOL'ed the entire product line.

The readme's and comments in the CCEd source code and some relevant documents from that time show the ideas that the original enginners had. And the things that were still on the to-do list when the hammer came down and development just stopped.

We had to live with the code that we inherited and for the most part it was beyond our scope and ability to do more than "bring it forward" and "keep it compiling and running". Really doing fair bit of true modernization wasn't doable, as none of us are proficient enough in C to deal with expert level code like this. The risks of bodging it were just too great.

But we always knew where CCEd struggled:

• Flat file database
• No proper database indexing 
• FIND transactions resulted in linear searches over almost ALL objects (slow)
• File based session management
• Sub-par memory management when compared to modern standards (malloc!)
• Somewhat exotic dependencies by modern standards

Last week I threw a shell based AI agent on a development box and pointed it at the SVN checkout of CCE. It identified all the shortcomings and suggested solutions. After some careful directions and with proper instructions I let the AI do the work to modernize CCEd.

Ideally I wanted to move it from a flat file database to SQLite. The idea being: Then we finally have proper database indexes and FIND transactions will be so much faster. I didn't expect fast results, as such a complex rework is anything but trivial. I expected this to be multi-week if not multi-month endeavor. 

As it turned out? The AI informed me that we HAD database indexes to begin with. And that changed the entire equation drastically. The original developers had already built in database indexes from the get go. But: The implementation was partial and broken in several ways. It became clear that apparently they had set out with the intent of doing full database indexes, but time constrains reduced the (working) scope dramatically. The intent was probably to fix this later, but due to the "end of times" at Sun/Cobalt this never happened.

What we had?

• Class Indexes
• NameSpace Indexes
• Indexes for every key/value of all Classes

During a "FIND Vsite" or "FIND User" the index would be used to return the OIDs of all Objects of said Class. Which is great.

If you did a "FIND Vsite name = site1"? That would also use the database index to quickly return the OID of the single Class that matched this query.

But: If you searched for multiple arguments OR used regular expressions as a search-argument? Then the index handling was just plain broken. Especially past the search for the first argument CCEd did a linear search over ALL Objects of the same Class. There were also other edge cases where a linear search over all Objects was the default. And these linear searches? They are just painfully slow and cause a lot of I/O traffic.

So we had AI fix that for us and we now have proper database index handling. Which greatly speed things up, as basically any CCEd transaction is initiated by a FIND transaction to figure out the Object ID of the Class we need to work with. Even during a CREATE we first use FIND to check if a matching Object of sufficient relevance already exists. 

Next issue on the list was the memory management, which was switched to a memory pool. While doing that, AI also identified and fixed several 24-28 year old bugs in the code where file descriptors were not properly closed. 

The final issue was the file based session management of CCEd, which stores session files in /usr/sausalito/sessions/. And every call to CCEd needs to revalidate the session information, so these quickly add up as well. We already have Redis (5210R and 5212R) or Valkey (5212R) installed on BlueOnyx and use it to speed up the GUI. Hence it was the most logical choice to use Redis/Valkey for CCed session management as well. This then got implemented, too, with file based session store being a fallback in case Redis/Valkey becomes unavailable.

Interfacing with CCEd:

Now this is where the wheels hit the asphalt and it is easy to loose traction: Software that needs to interface with CCEd. May that be the GUI, the API or scripts.

Handlers and Constructors are usually written in Perl, but they can basically be in any scripting language or even be binary if need be. But they need to be able to communicate with CCEd in order to read or write CODB data and perform tasks. For Perl we have a Perl module that provides this interface. For Python we once had such a module, too. But as we no longer use Python for Handlers/Constructors? We deprecated it. 

But generally: A Handler or Constructor or any other application just needs to connect to the CCEd Unix Socket and follow protocol, succeed authentication and needs to do things in a documented way. That's it.

GUI access to CCEd:

And there is the speed bump: The GUI is written in PHP. How can it access CCEd? Well, of course via the Unix Socket of CCEd. But having a scripting language like PHP "talk" to a Unix Socket has a few caveats: You can't "talk" and "listen" at the same time. You need to talk first. Then stop talking and need to listen to the response before you can talk again. Much like radio or phone communication. When the two sides "talk over each other", you get a mumbo-jumbo that hardly makes any sense. Likewise: A scripting language doing this complex input preparation and interpretation for each transmission or receipt? That gets complex pretty fast. And complex means: Slow and CPU intensive. 

The original Cobalt developers therefore chose to offload the PHP-to-CCEd-Interface into a Zend API Module for PHP. They wrote a neat 948 line C source code that compiles into a binary cce.so that can be loaded as module into PHP. And it provides PHP with (then) built in functions to directly communicate with CCEd. 

This is as neat as it comes and it's light-weight, fast and really easy on the resources.

The Struggles:

Back in the "old" days on Cobalt RaQs and Qubes? Later on BlueQuartz and early versions of BlueOnyx? This dependency of the GUI on a certain OS provided PHP version was mandated by cce.so (and i18n.so). Because such PHP modules HAVE to be compiled against the exact PHP version that the GUI is using. For the most part. Some minor version changes are not a big issue, but going from (initially) PHP-4.1.2 (which cce.so was initially designed for!) to say PHP-4.2.0? Or anything more modern than that? PHP would then fail to run for as long now incompatible *.so modules were being loaded. And without these required modules loaded into PHP? The GUI would flat out refuse to work.

This was then overcome by "side-loading" a newer PHP somewhere on the sides, while the OS and the GUI could happily continue to use the OS provided PHP with cce.so support.

When we started the BlueOnyx project and launched BlueOnyx 5106R we modernized cce.so to work with the PHP that shipped with CentOS 5. And during each subsequent new OS that BlueOnyx was based on? We had to do it again. Eventually with BlueOnyx 5209R (CentOS 7) and PHP-5.4 we hit the end of the road and managed for the last time to bring cce.so forward. On all subsequent BlueOnyx versions we were (for the longest time) unable to "revive" cce.so and "make it work" with whatever PHP the OS or the GUI were using. 

Basically the Zend API for PHP modules had drastically changed and even if cce.c (the sources) were only 948 lines? It was beyond our ability to "fix it" and "make it work" again with more modern PHP versions.

As a replacement the PHP CCEd interface was rewritten to use a PHP Class that communicated with the Unix Socket of CCEd and that was ... slow. Like 2-5x slower than the old cce.so PHP module. But in absence of a better solution it was the only viable way forward.

Enter CCEd-API:

Eventually we introduced the CCEd-API. Which is v2 of a BlueOnyx API and a binary coded entirely in Golang. It also "talks" to CCEd via the Unix Socket and provides a TCP/IP interface which applications can interact with. The BlueOnyx GUI was switched over to use CCEd-API as primary method and only if the API wasn't available it would fall back to use the PHP Class to "talk" with CCEd.

Usage of CCEd-API was almost as fast as cce.so, but not quite. There was still extra pre- and post-processing communication overhead required in PHP. 

I18n Language Support

The GUI uses i18n to display all texts in one of the nine supported GUI languages. And for that the original Cobalt devlopers also had a clever way of doing it long before PHP itself inherited native i18n support. And even then: What the original Cobalt developers did was (IMHO) a much better implementation than what natively ships with PHP these days. And they also wrote it in C as Zend API module and the compiled i18n.so was loaded into PHP to provide native GetText support. Fast, smart, low overhead. Simply out: Perfection.

As with cce.so: Eventually we were unable to bring i18n.so forward to newer PHP versions and - again - had to default to use a PHP Class to "emulate" i18n support. Now imagine doing that for nine languages, one of them using EUC-JP, while all others use UTF-8. As soon as you hit text that has Kanji, Umlauts or Accents in it? Things get "interesting" pretty quick. And doing that in PHP? Let's just say: It plain sucks and it's slow. Especially when the average GUI page has northwards of 200 localized strings in it. And if it's just the menus and their help texts.

AI to the rescue again:

Way back when ChatGPT came out cce.so and i18n.so were my prime candidates that I wanted to have "fixed" by AI. But: It's not a simple matter. Attempting a complex task like that in a browser driven AI interface is a real struggle and the context windows were usually not big enough to get anywhere near a working prototype. And expecially cce.so is so crucial, that we cannot have a half-assed version that "mostly works". It has to be perfect. We cannot compromise there.

Last week I sat down and used Hermes with kimi-2.6 and/or GLM and Qwen3-coder-next as LLMs. Hermes was installed on a 5212R development box and I simply pointed it at the SVN code tree of the 5212R version of CCEd and told it what I wanted it to do. It analyzed the entire code, the build architecture and hammered out a modernization plan, where it outright said: This HAS to be a complete rewrite to cover all bases of the PHP-8.3 Zend API. And I then insisted it having 1:1 the same feature set - with no compromises. Hermes started to work on a first draft and after around six hours it threw the towel with kimi-2.6. I switched to GLM. That gave up after four hours. Qwen3-code-next? For nine hours it insisted that it was working on the new code, but didn't churn out a single line of code.

I then used Codex, the CLI agent from OpenAI with GPT-5.5. I gave it the "strategy paper" that Hermes/kimi-2.6 had drafted up, Codex agreed after a brief code analysis and started hammering out a new cce.c and did a test compile. It failed. Without input from me it fixed the code, compiled it again and we had a first working version of cce.so working. This took eight minutes. I kid you not!

It did take two additional days to streamline the build process, so that a "make rpm" in the code tree would generate sausalito-cce-* RPMs with cce.so included and to iron out a couple of minor issues that popped up during testing. But the heavy lifting? That had taken only eight bloody minutes. 

Further code work was required in base-alpine-* to "teach" our CceClient.php Class (again) to use the finalized new cce.so for communication with CCEd. With the API and the Unix Socket method serving as 1st and 2nd fallback in case cce.so somehow "goes walkies" and stops responding.

i18n also got the same treatment and likewise a combination of Hermes and Codex was used to create a modernized version of it.

Final result:

The combined code changes undertaken during this BlueOnyx modernization drive may not look like much. But they serve the distinct purpose to make the GUI faster, more reliable and more predictable.

And from objective speed tests I can say: It delivers.

Certain pages that needed +3 seconds to load via PHP over Unix-Socket? And still took 2.2 seconds via CCEd-API? With cce.so and i18n.so they load in 0.88 seconds.

Of course it all depends on how many translated locale strings and CCEd requests a page has to make before it can be displayed. On the average we see speed increases of 10-50% across the board and the GUI feels quite a bit snappier just because of that. 

Another important point is predictability. Run-time issues during one-directional Unix socket communication are no longer an issue when using cce.so. And likewise: Using i18n.so for the translations saves us from a lot of transpositional errors. Like when the GUI is set to English and we look at a user's full name that has Umlauts, Accents or Kanji. The amount of code to make sure that a straight up PHP implementation gets it right is ridiculously complex. Wherwas in i18n.so it's effortlessly simple to do it right.

And that concludes this round of development and it shifts BlueOnyx back onto a really solid and reliable foundation that quickly delivers reproduceable results.