My First illumos Build
23 Oct 2014
I switched to OmniOS as my illumos-gate build platform. I
updated my setup script accordingly and changed the name
nightly-setup. Finally, I updated this
article to reflect usage of the latest script. The shell
snippets still refer to my original OpenIndiana environment.
OmniOS may have small differences in output.
My first illumos build took over 30 hours, spread out over 3 days. Not because the actual build took that long, but because I hit so many snags along the way. I learned a lot in the process, but boy was it painful. I want to share some lessons learned in hopes that it may reduce your pain during your first illumos build.
The first thing to tackle is nomenclature. There is some daunting jargon surrounding illumos. But once you understand a few key terms tutorials become easier to follow and communication with illumos veterans flows better.
Keep in mind, I’ve stitched together these terms in my own way and may have some details wrong. They are more of a (hopefully) good approximation than an authoritative definition.
- ON (OS/Network)
- The kernel, libc, network, system libraries, and commands that make up the core illumos system. To be pedantic, this is what illumos is. It is not an operating system itself.
- The operating system built on top of ON (illumos). This includes the userland software and whatever else the distribution decides to add. The three most common distributions are OmniOS, SmartOS, and OpenIndiana.
nightly, the command used to perform ON builds.
- ONU (ON Update)
onucommand which takes a build of ON and creates a new BE so that you can boot into it and test your changes. You may also hear about packages and an image. The end goal of nightly is to build the collection of packages which form the new ON image. This image is installed in the new BE created by ONU.
- BE (Boot Environment)
- The environment from which the entire operating system runs. It is made up of packages which are installed by the IPS system when running the ONU script.
- osnet consolidation
- The list of packages that make up ON. ONU tells IPS to update this consolidation package which causes all the appropriate packages to upgrade in lock step so you can test your new ON.
- IPS (Image Packaging System)
- The program that manages the packages.
- beadm, BE admin
- The command to administrate boot environments. Needed to get back into the old BE after testing the new one.
This is only one way to perform the build, but not the only way. This follows the instructions found on the illumos wiki as closely as possible. As you become more comfortable with the build you’ll want to learn about more advanced techniques like incremental builds, multiple workspaces, and testing without booting into a new BE.
I wrote these instructions specifically for a fresh
install of OmniOS
or OpenIndiana. I
recommend using OmniOS. If you must use OpenIndiana then make
(The build fails on older OI releases because
localedef relies on the
xlocale.h). You can perform a traditional
Box VM, or do like I do and run it on a
The first step is to prepare the box for doing a nightly
build. You only need to do this once. I even wrote a
to do all the work for you. This script is idempotent, if it
fails halfway through you can just re-run it with no ill
effect. If you pass the
-p option it will print
build instructions similar to this post.
$ wget https://zinascii.com/pub/rpz-misc/latest/rpz-misc.tar.gz $ tar -zxvf rpz-misc.tar.gz $ ./rpz-misc/bin/nightly-setup
Setup configures an
illumos.sh environment file
for you to pass to
nightly; this environment file
contains many environment variables which control various
aspects of the build. For example,
NIGHTLY_OPTIONS controls what stages of the build
are run. If you remove the
l flag then the lint
stage will not run drastically reducing the build time. This
is useful for initially testing a patch before doing a full
$ cd /code/illumos $ /opt/onbld/bin/nightly illumos.sh || echo "BUILD FAILED -- CHECK LOGS"
The nightly command does not output to stdout. To monitor progress you must tail the nightly log file.
$ tail -f log/nightly.log
When finished the script moves the nightly log file into a
timestamped directory and compiles the
summary. If the build fails you’ll find the exact error in the
nightly log file. It's best to focus on the first error as it
is often the cause of subsequent erros.
$ grep '\*\*\*' log/log.<ts>/nightly.log
If the build goes well then a packages dir should exist.
$ ls packages/i386/nightly/repo.redist/ catalog cfg_cache file index pkg tmp trans
This is your new ON. To use it you need to create a new BE
based off your current one, create a new IPS repo, and then
osnet-incorporation package on the new
BE. Luckily you don’t have to understand what any of that
means because the
onu script does all the work
$ sudo /opt/onbld/bin/onu -t nightly -d packages/i386/nightly
-t option is the name of the new BE
-d is the path to your nightly packages dir.
On success you can run
beadm list to verify the
new nightly BE is there. The value of
Active field indicates that on reboot the
nightly BE will become the active one.
$ beadm list BE Active Mountpoint Space Policy Created nightly R /tmp/onu.KLaa4j 15.3G static 2014-10-23 01:31 openindiana N / 7.98M static 2014-10-20 20:56
To get into your nightly BE you simply reboot.
$ sudo reboot
Once back into the system convince yourself that you are
indeed running the nightly BE by using
$ beadm list BE Active Mountpoint Space Policy Created nightly NR / 15.3G static 2014-10-23 01:31 openindiana - - 33.6M static 2014-10-20 20:56
Now you may test your changes. After testing you may find more
adjustments need to be made. In that case activate the old BE
and reboot again. This returns you to the previous
environment, exactly as it was. Any changes you make on that
nightly ZFS dataset will be lost on reboot. I.e., changes
/code/illumos, while running the nightly
BE, will be lost after reboot. Any files that you want to
persist when booting backwards from nightly to openindiana
should be kept in your home directory (
exists on a different dataset than the BE).
$ sudo beadm activate openindiana $ sudo reboot
Once back in the original BE you destroy the nightly BE, make your tweaks, run nightly, and create a new nightly BE with ONU.
$ sudo beadm destroy nightly $ cd /code/illumos $ vi ... $ /opt/onbld/bin/nightly illumos.sh || echo "BUILD FAILED -- CHECK LOGS" $ sudo /opt/onbld/bin/onu -t nightly -d packages/i386/nightly $ sudo reboot
If you have trouble don’t hesitate to reach out on the #illumos IRC channel or on the illumos-discuss mailing list. There are friendly people willing to help. But understand that this is my take on building illumos as a beginner; the veterans may point you in different directions. They are not necessarily familiar with this blog post or my script.