5212R - Custom AlmaLinux 10 ISO building

Developer diary entry about re-spinning an AlmaLinux 10 ISO to provide us with a custom BlueOnyx 5212R installation ISO.

Preface:

Our BlueOnyx 5212R (based on AlmaLinux 10) is almost finished and ready for release. So it was time to revisit ISO building again to see what needed to be done to create an ISO built that installs AlmaLinux 10 and the roughly 1300 RPMs of BlueOnyx - in one go.

We usually base the BlueOnyx ISO's on the AlmaLinux-*-x86_64-minimal.iso and for this purpose I naturally fetched the AlmaLinux-10.0-x86_64-minimal.iso and set up the usual CD build environment on a build box by copying over the one for the 5211R ISO and starting to make the necessary adjustments. 

How we do it:

We mount the Alma minimal ISO and copy all data over to a separate directory which will be the root directory of the ISO. The included /Minimal repository on the CD is emptied out and /EFI and /isolinux get the grub2 bootloaders for EFI and Bios adjusted to present our custom boot-menu with the different install options. Branding is added as well as the custom kickstart configs. Which is simple.

Then we run some clever scripts with commands centered around YUM/DNF. These pull in all the required RPMs from the AlmaLinux and BlueOnyx repos right into the /Minimal repository of the build directory. With an adjusted comps.xml and modules.yaml the modular repository metadata is then recreated. This usually requires the most legwork first time around. But once its set up? Then it can be fully automated until the version jump of the OS and the ISO.

And then it's just running mkisofs, making it a hybrid boot (bios/uefi) and implanting the checksum. That part usually never changes. How we do this for the AlmaLinux 8 based BlueOnyx 5210R is detailed here. And for the AlmaLinux 9 based BlueOnyx 5211R it's not much different.

That ain't working anymore!

Well, it is a bit different now with EL10. "Upstream" is now building the ISO images in a new way. /isolinux is gone, /boot has been added and instead of using mkisofs they're now using xorriso. Which has its advantages, but it makes my work a a bit more complicated.

Because it's not only got BIOS/UEFI/Isohybrid, different boot loader mechanism and different 1st and 2nd stages for the install.

Bottom line: It is no longer *that* easy to take an existing ISO, rip it off, remix and "re-bake" it.

So I examined the original AlmaLinux ISO with commands such as these:

xorriso -indev *.iso -report_system_area as_mkisofs
xorriso -indev *.iso -report_el_torito as_mkisofs
xorriso -indev *.iso -report_el_torito cmd

Which helped me to figure out exactly how they wrote out the actual ISO so that I can replicate the options. That took a LOT of fiddling, but eventually I got a combination that results in a working ISO.

New build commands for straight-up remixing:

1.) Extract the hybrid MBR from the original ISO:

dd if=/home/build_cd.5212R/Import-ISO/AlmaLinux-10.0-x86_64-minimal.iso bs=1 count=432 of=/home/build_cd.5212R/isohybrid-mbr.bin

2.) Remix the ISO, adjust bootloader and repos, add kickstarts

3.) "Bake" it again with correct options AND the extracted hybrid MBR:

LC_ALL=C xorriso -as mkisofs -o /home/build_cd.5212R/BlueOnyx-$VER.iso -b images/eltorito.img -J -R -l -c boot.cat -hide boot.cat -hide-joliet boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -append_partition 2 0xef /home/build_cd.5212R/5212R/images/efiboot.img -appended_part_as_gpt -graft-points -V "AlmaLinux-10-0-x86_64-dvd" --grub2-mbr --interval:local_fs:0s-15s:zero_mbrpt,zero_gpt:/home/build_cd.5212R/isohybrid-mbr.bin --protective-msdos-label -partition_offset 0 -partition_cyl_align off -partition_hd_cyl 84 -partition_sec_hd 32 /home/build_cd.5212R/5212R

TOTALLY self evident what all of this does, right? <sigh>

This is also where it'll most likely breaks when the 10.1 ISO comes out. In which case the fiddling to re-determine some of these options starts from scratch again.

4.) Implant checksum:

LC_ALL=C /usr/bin/implantisomd5 /home/build_cd.5212R/BlueOnyx-$VER.iso

At this time I just rebranded the original AlmaLinux 10 Minimal as proof-of-concept and that reconstructed ISO boots fine via BIOS and EFI.

So that's a start. The rest is filling in the gaps and plenty of testing.

Plan B (Code-Name: "Bullsh*t")

If this had not worked? Building RHEL/Alma/Rocky images "the correct way" from scratch is a REAL pain in the ass:

• Install Ansible
• Install Docker
• Get a git repo set up with the build configuration
• Get a full Pungi-Docker for AlmaLinux from various separate repos
• Set up a Pungi-Docker instance with the configs from git
• Build ISO

The instructions and repos I found only cover AlmaLinux 9 so far:

https://wiki.almalinux.org/development/building-almalinux-iso-locally.html#setting-up-an-ansible-inventory

The Pungi side of things:

https://git.almalinux.org/almalinux/pungi-almalinux
https://github.com/AlmaLinux/pungi-node-public
https://github.com/AlmaLinux/pungi-scripts-public

That's a variable with at least four unknowns for me and getting any step wrong will result in failure. And it's not like AlmaLinux is openly advertising which build config and they used for the latest ISO. I probably can be found somewhere, but have fun digging. Or piecing that information together by trial and error.

Another fun anecdote: I used Grok4 to present options, alternatives and do sanity checks. It shows which web sources it's checking to fetch the info its basing its answers on. The first URL it checked was over at Almalinux.org. The second one a RHEL page. The third one? Was this. 

Reason for publishing this:

In the last (almost) 20 years BlueOnyx has built ISOs various different ways. Back in the days of old Brian Smith (NuOnce Networks) did the BlueQuartz ISOs and rigged up the initial build environment for BlueOnyx ISOs. Eventually newer OS releases warranted changed approaches. So we used whatever we could get working: Revisor, Pungi, spit and duct tape, home-grown scripts and sweat and tears. One fact hasn't changed over the last two decades: The "upstream" documentations about how to do your own remixes of RHEL clones have always sucked, were outdated, intentionally vague or hidden in the basement in a locked closet with the lights turned off and sewage seeping in. Nobody has an interest in making this easy. I even get why that is the case.

So when I started to publish HOW we built the EL7 and later on the EL8 ISO's (first CentOS 8 and then AlmaLinux 8) these pages were almost always in the top 10 "Vsits from Search Engines" in our visitor statistics. That should tell you something about how desperate some people can get and how badly it IS documented elsewhere.

If you got here looking for how to roll up your own AlmaLinux 10 remix and need further pointers? You're welcome to ask and I'd be glad to share further information.