Goodbye Joyent
12 Feb 2020
Going to Joyent was a dream come true. I went from aspiring kernel dev to the mac expert. I learned to prevail against bugs that made me wanna throw my computer out the window, such as i40e ring freezes and inside baseball semantics between vfork and signal delivery. I reviewed a lot of code, including this absolutely beautiful patch by Patrick Mooney to implement a faster clock_gettime. I read a lot of code too, which wasn’t always easy, but sometimes I was lucky enough to find myself treated to one of Robert’s epic block comments. I fixed some bugs, including a few brutal deadlocks that would lockup compute nodes weekly and crash customer manta jobs. I created bugs of my own. I panicked a lot of kernels (*slaps /var/crash/volatile*...I can fit so many cores in here). I touched lx brand, zones, dladm, dlmgmtd, mdb, mac, dls, dld, ip, aggr, ixgbe, i40e, bufmod (friggen bufmod!), viona (bhyve networking), simnet, test-runner, and even sdev! I was honored to give a talk about one of the founders of this great tradition that we call the Unix Time-Sharing System. But above all, I got to work with and learn from some truly great people. I feel like I packed a decade of learning into three-and-a-half years.
At this point, you can’t send or receive a packet in SmartOS or OmniOS (and hopefully all illumos distros after some upstreaming) without crossing my code. In fact, this blog itself is hosted on OmniOS and was delivered to you by my networking bits. I think that's pretty rad, but also scary as hell (networking gods, I tempt ye not). I've come a long way from recovering database developer.
But like all good parties, it had to come to an end; and it did, on Feb 03, 2020. I was let go as part of a RIF (Reduction In Force). It’s not a fun way to go. I’m bummed. I feel like I was just starting to ease into my all-night rager as an illumos kernel developer, only to have 5-0 break it up before I could jump off the roof into the pool. I’ve looked up to the Sun developers since Surge BoF 2012, when I watched Bryan and Brendan use DTrace to debug latency outliers in the cloud. I’ll never forget that evening; I had never seen two people with such command and poise over the operating system — they were brilliant to watch. After becoming addicted to SmartOS that year, I knew that somehow, someway, I wanted to be a part of it. I wanted to join the team at Joyent and further their mission of bringing a better cloud to the world. It was going to be my own version of Sun, in a world where the original had long since been swallowed up by the event horizon Oracle.
And it was just that, for a while. My first day at Joyent is one I’ll cherish forever. Sitting in a room overlooking the San Francisco Bay with Bryan, Robert, and Angela, with Jerry and Patrick on the phone, discussing my potential future contributions — I had arrived. I was exactly where I was meant to be, with the people I was meant to be with. The lx branded zone was alive and well, containers were looking more promising by the day, and I was ready to dive head-first into the depths of the kernel I loved so much.
But as any software veteran knows, projects often don’t survive the whims of management. No one is fired for picking Linux (these days), but they might be for picking something else. I already experienced this once before, as a core developer of the Riak database. We were rigorous, paying homage to the theoretics of distributed systems, but with a focus on bringing that theory to the masses. So much so that our last CEO said we had to stop doing so much “computer science”. He meant it as an insult, but we wore it as a badge of honor. But hey, MongoDB had a sweet API and BJSON, who cares if it lost your data occasionally [1]. I understand that people like to stick with what is popular. I respect that decision — it is theirs to make. But I’ll never be a part of that crowd. I want to use software that speaks to me, software that solves the problems I have, software guided by similar values to my own. For me, no project does this more than SmartOS and the illumos kernel. It is my Shawshank Redemption in a sea of MCU.
I don’t know what the future holds. I still believe in the illumos kernel and the operating systems based on it (I’m not alone). I’m going to do my best to continue working on it — paid or not. But for now, I’ve got a mountain to snowboard, and a winter to enjoy (hello Winter of Z). To my fellow Joyeurs, past and present: it was an honor. I’ll see you around the bend.