My First illumos Build

23 Oct 2014

My first illumos build took over 30 hours, spread out over 3 days. Not because the actual build took that long, but because I hit so many snags along the way. I learned a lot in the process, but boy was it painful. I want to share some lessons learned in hopes that it may reduce your pain during your first illumos build.

The first thing to tackle is nomenclature. There is some daunting jargon surrounding illumos. But once you understand a few key terms tutorials become easier to follow and communication with illumos veterans flows better.

Nomenclature

Keep in mind, I’ve stitched together these terms in my own way and may have some details wrong. They are more of a (hopefully) good approximation than an authoritative definition.

ON (OS/Network)

The kernel, libc, network, system libraries, and commands that make up the core illumos system. To be pedantic, this is what illumos is. It is not an operating system itself.

Distribution

The operating system built on top of ON (illumos). This includes the userland software and whatever else the distribution decides to add. The three most common distributions are OmniOS, SmartOS, and OpenIndiana.

nightly

nightly, the command used to perform ON builds.

ONU (ON Update)

The onu command which takes a build of ON and creates a new BE so that you can boot into it and test your changes. You may also hear about packages and an image. The end goal of nightly is to build the collection of packages which form the new ON image. This image is installed in the new BE created by ONU.

BE (Boot Environment)

The environment from which the entire operating system runs. It is made up of packages which are installed by the IPS system when running the ONU script.

osnet consolidation

The list of packages that make up ON. ONU tells IPS to update this consolidation package which causes all the appropriate packages to upgrade in lock step so you can test your new ON.

IPS (Image Packaging System)

The program that manages the packages.

beadm, BE admin

The command to administrate boot environments. Needed to get back into the old BE after testing the new one.

Build Instructions

This is only one way to perform the build, but not the only way. This follows the instructions found on the illumos wiki as closely as possible. As you become more comfortable with the build you’ll want to learn about more advanced techniques like incremental builds, multiple workspaces, and testing without booting into a new BE.

I wrote these instructions specifically for a fresh install of OmniOS or OpenIndiana. I recommend using OmniOS. If you must use OpenIndiana then make sure you use Hipster (The build fails on older OI releases because localedef relies on the host's xlocale.h). You can perform a traditional install, create a Virtual Box VM, or do like I do and run it on a SmartOS KVM instance.

Setup

The first step is to prepare the box for doing a nightly build. You only need to do this once. I even wrote a script to do all the work for you. This script is idempotent, if it fails halfway through you can just re-run it with no ill effect. If you pass the -p option it will print build instructions similar to this post.

$ wget https://zinascii.com/pub/rpz-misc/latest/rpz-misc.tar.gz $ tar -zxvf rpz-misc.tar.gz $ ./rpz-misc/bin/nightly-setup

Run Nightly

Setup configures an illumos.sh environment file for you to pass to nightly; this environment file contains many environment variables which control various aspects of the build. For example, NIGHTLY_OPTIONS controls what stages of the build are run. If you remove the l flag then the lint stage will not run drastically reducing the build time. This is useful for initially testing a patch before doing a full lint.

$ cd /code/illumos $ /opt/onbld/bin/nightly illumos.sh || echo "BUILD FAILED -- CHECK LOGS"

The nightly command does not output to stdout. To monitor progress you must tail the nightly log file.

$ tail -f log/nightly.log

When finished the script moves the nightly log file into a timestamped directory and compiles the mail_msg summary. If the build fails you’ll find the exact error in the nightly log file. It's best to focus on the first error as it is often the cause of subsequent erros.

$ grep '\*\*\*' log/log.<ts>/nightly.log

ONU—Create Nightly BE

If the build goes well then a packages dir should exist.

$ ls packages/i386/nightly/repo.redist/ catalog cfg_cache file index pkg tmp trans

This is your new ON. To use it you need to create a new BE based off your current one, create a new IPS repo, and then update the osnet-incorporation package on the new BE. Luckily you don’t have to understand what any of that means because the onu script does all the work for you.

$ sudo /opt/onbld/bin/onu -t nightly -d packages/i386/nightly

The -t option is the name of the new BE and -d is the path to your nightly packages dir. On success you can run beadm list to verify the new nightly BE is there. The value of R in the Active field indicates that on reboot the nightly BE will become the active one.

$ beadm list BE Active Mountpoint Space Policy Created nightly R /tmp/onu.KLaa4j 15.3G static 2014-10-23 01:31 openindiana N / 7.98M static 2014-10-20 20:56

Boot Nightly BE

To get into your nightly BE you simply reboot.

$ sudo reboot

Once back into the system convince yourself that you are indeed running the nightly BE by using beadm again.

$ beadm list BE Active Mountpoint Space Policy Created nightly NR / 15.3G static 2014-10-23 01:31 openindiana - - 33.6M static 2014-10-20 20:56

Now you may test your changes. After testing you may find more adjustments need to be made. In that case activate the old BE and reboot again. This returns you to the previous environment, exactly as it was. Any changes you make on that nightly ZFS dataset will be lost on reboot. I.e., changes under /code/illumos, while running the nightly BE, will be lost after reboot. Any files that you want to persist when booting backwards from nightly to openindiana should be kept in your home directory ($HOME exists on a different dataset than the BE).

$ sudo beadm activate openindiana $ sudo reboot

Once back in the original BE you destroy the nightly BE, make your tweaks, run nightly, and create a new nightly BE with ONU.

$ sudo beadm destroy nightly $ cd /code/illumos $ vi ... $ /opt/onbld/bin/nightly illumos.sh || echo "BUILD FAILED -- CHECK LOGS" $ sudo /opt/onbld/bin/onu -t nightly -d packages/i386/nightly $ sudo reboot

Getting Help

If you have trouble don’t hesitate to reach out on the #illumos IRC channel or on the illumos-discuss mailing list. There are friendly people willing to help. But understand that this is my take on building illumos as a beginner; the veterans may point you in different directions. They are not necessarily familiar with this blog post or my script.

Further Reading

How to Build illumos

Basic illumos Workflow

For illumos newbies: On developing small