Going to Joyent

5 Aug 2016

Four weeks ago I accepted a job at Joyent to work full time on SmartOS. It feels beyond good to say that. It marks the end of my two year journey of becoming an OS developer.

It started in the summer of 2014: I had just left Basho after 3.5 years of working on Riak, when I decided I wanted to become an OS developer. I purchased Solaris Internals, cloned illumos-gate, fired up cscope, and got to work. I hardly knew any C, x86 might as well have been Brainfuck, and, frankly, I knew shit about operating systems. But I was determined.

I’ve always learned best by beating my head against something until it makes sense. I’m not a fast learner; I’m persistent. What others have in ability I make up for in effort. And when it comes to OS internals it’s all about work ethic. The more you look, the more you realize it’s just another program. The main difference being: it’s the program all the other programs run on.

My strategy: to pick something, anything, that looked interesting, and write a post describing how it works. I wrote several of these posts in 2014 and 2015. The most popular post was on the semantics of POSIX realtime signals. It generated over 40,000 hits in 3 days. More important, it put me in touch with Roger Faulkner: the creator of truss(1), the Solaris process model, and the real /proc filesystem. At the time I didn’t like my interaction with Roger. He explained, in what I would later find out to be his typical gruff manner, that I was wrong; so I concluded he is a prick. But over the years I realized that I was being a brat—he was trying to teach me something and I let my ego get in the way. I’ve come to view that interaction as a blessing. I interacted with one of the greats, a mentor of my mentor’s mentor (a Great Great Mentor).

A couple of weeks later something even more surreal happened, at illumos Day 2014. Bryan Cantrill was the last speaker of the day. One of my mentors and someone I admire greatly. He was there to regale us with the story of Joyent’s resurrection of lx-branded zones: Linux system call emulation on top of the illumos kernel. But before he would do that he decided to speak about me! I couldn’t believe it. I was so overwhelmed that I don’t remember most of what he said. I was too busy flipping shit—Bryan Cantrill is on stage, in front of other kernel developers I look up to, saying my name. I was in a dream. It turns out, unknown to me at the time, that he wrote the POSIX queue code for both Solaris and QNX, which I wrote about. He compared me to the great expository technical writers Elliott Organick and Richard Stevens. And it was at this moment that I knew I could do this: I could become an OS developer.

Never underestimate the effect kind words can have on someone that looks up to you.

Later that same day I met Garrett D’Amore and Jacob Loveless. Garrett is an exceptional kernel hacker and the founder of illumos. Jake, at the time, was the CEO of Lucera and a big fan of SmartOS. A month later they made me a job offer as a “kernel developer”. I hadn’t earned that title, but I would. This job would provide me with the challenges I needed and solidify my passion for OS development. While there I became familiar with lx-brand zones, the TSC register, the ixgbe driver, process binding, and scheduling. I was a kid in a candy store. Jake was the best CEO I’ve ever worked for (or met). Not only did he genuinely care about me, but we also talked about everything from arbitrage, to compiler research, to OS network internals. We still talk to this day. I’m also grateful to Garrett, who took a chance on me and gave me room to make mistakes. I learned so much.

While at Lucera I would also get to know Dan McDonald. The maintainer of OmniOS and a core illumos contributor. It was Dan who bootstrapped me on lx-brand zones. He is a wealth of knowledge and a good human being.

I also met other illumos community members like Nahum Shalman, Albert Lee, and George Wilson. Albert and I worked the illumos booth at USENIX LISA 2015. I was honored, as such a new member, to represent illumos at LISA. That day I interacted with many Unix people. Especially memorable was drinking with Andrew Hume. A boisterous Australian who gifted us with funny stories about Dennis Ritchie and Ken Thompson from the old Bell Labs days. He is the author of the Unix 10th edition manual and creator of Mk (Plan9’s successor to make).

My work at Lucera, on ixgbe and lx-brand zones, naturally led me to Robert Mustacchi and Patrick Mooney, both Joyent kernel engineers. I would often discuss my findings with them on IRC. Their guidance accelerated my learning and helped me greatly. After a while I wished I could work for Joyent so I could interact with them every day. When I first joined Lucera we talked about building an illumos team, but over time circumstances changed and that was no longer in the cards. Then I let myself go too far down a rabbit hole and I didn’t feel Garrett, by himself, could offer me the support I needed. I let myself burn out.

After much consideration, I left Lucera.

At that point I took a detour and went to Digital Ocean. I first talked to Joyent, but their pending deal with Samsung (I didn’t know that was the reason at the time) put them in a hiring freeze. I decided to talk to DO because my old manager from Basho, Dave “Dizzy” Smith, was there. I loved working with Dizzy at Basho so I figured it was worth checking out. Dizzy sold me on a position involving their virtualization and I enthusiastically said yes. Sure, it was Linux, but I liked the team and I figured I’d grow into it. But it turns out I was lying to myself. My heart is in illumos, and I can think of nothing more fun than to get paid to make it better. So after only four months, I left DO. I’ve never done that before, and I felt bad about it. But I had to follow my gut. DO is a great place to work, and I would recommend it to anyone looking. But it’s not the place I want to work.

The gears started turning when I saw Bryan in Denver, shortly after Joyent announced their acquisition by Samsung. I was there on a company retreat, but I knew he was in town to keynote USENIX ATC. I messaged him on Twitter and we met at The Oxford. After my team roped him into a game of werewolf (a popular game at DO), he opened his laptop and gave me a runthrough of his keynote. It was your typical Cantrill talk: a series of rants forming another larger rant. This particular rant focused on ATC’s decline of relevance to the practitioner. While watching a spastic Cantrill fire metaphorical shots at USENIX ATC in the middle of The Oxford lobby at 12 in the morning, I was reminded of something: this is who I want to work with. People with strong opinions that aren’t afraid to share them. People that still believe in OS innovation. People with some fucking passion for their craft. Anyone who knows Cantrill should know his hiring story at Sun. His epiphany moment in Jeff Bonwick’s office: when Jeff agreed that the M:N scheduling model is crap. I was having the same moment that night. I had to work at Joyent. We said nothing about employment that night, but we agreed we should “talk” soon. We had an understanding; and I had some hard thinking to do.

The week after I took some time off work to stay with my friend (another former Basho colleague) Jared Morrow. The plan was to lift weights, chill at the reservoir, go off-roading, and maybe shoot guns. I hadn’t accounted for the gin-filled evenings and “real talk”. I knew I wanted to work at Joyent but I was timid about leaving DO so abruptly. Jared made me realize that I had to stop focusing on what other people wanted and do what I wanted. He helped me realize that the time to do this was now. I had to shit or get off the pot. I’ve wanted to work on this team for two years. What was I waiting for? So I “pulled the trigger” and called Bryan (I literally called Bryan from the gun range).

And so here I am. A week from now my two year journey to become an OS developer comes to an end; and a new chapter begins. I don’t know what specific things I’m going to work on, but I’m sure it will push me to the limit. I look forward to the challenge. I look forward to learning from people like Bryan Cantrill, Robert Mustacchi, Patrick Mooney, Josh Clulow, Jerry Jelinek, and Max Burning. I like to tell people that the 3.5 years I spent at Basho were some of the best of my life. I hope my time at Joyent is like that—just with different people on a different problem. I’m going to Joyent.