Home opinion-and-analysis Open Sauce Linus Torvalds: looking back, looking forward

Most years, Linus Torvalds comes to Australia. He apparently likes the place, so the creator of the Linux kernel makes his way to the Australian national Linux conference in January.

He's been doing this since 2003, when the affable James Bromberger and Tony Breeds (corrected) organised the LCA in Perth, only missing out in 2010 when the event moved across the Tasman to Wellington.

Torvalds is an excellent subject for an interview; he never evades a question, not even if he has a single word to offer as reply. And despite claiming to have a big ego, he is indeed very approachable.

He couldn't sit down for an interview as he was leaving the conference the morning after I approached him; hence this was done by email.

iTWire: It's been nearly 20 years now since that now-famous message posted to comp.os.minix announced the arrival of something that has grown beyond anyone's imagination. Have there been any occasions when you've thought of walking away from it?

Linus Torvalds: Walking away? No. There have certainly been times of frustration, and times when I've really wanted to take a break, but quite frankly, even then it's been "I need to get away from this for a few hours (or days) to relax and do something else" rather than anything more than that.

And I can't imagine doing it now either. I'd just get really bored very quickly. There have been stressful times, but in hindsight even the stressful times have usually been things that really were worthwhile in the end. And interestingly (and maybe not all that surprisingly, but it always took me by surprise every time it happened anyway), most of the ones that have been the most stressful have been not really about technical issues. They've always fundamentally been about development model. So there's been several times when the way I did something just didn't work out well as the kernel project grew, and I (and others) needed to change how we worked in Linus Torvaldsorder to streamline the process and make it work better in the face of many new developers.

As an example, we had the multi-year release cycles, which really led to a lot of pain and caused all the distributions to have to back-port a lot of code to the "stable" version, which got increasingly hard as the development tree diverged quite radically over the years. And people still remember that model, and the whole "even release numbers" (for stable, ie 2.4.x) vs "odd release numbers" (for development, ie 2.5.x) got so ingrained that a few other projects started doing the same thing. And it caused lots of problems, and we had to change how the whole release cycle looked. Because it was _incredibly_ painful, and making ready for 2.6.0 was what led me to say that I needed to take a year of unpaid leave from Transmeta exactly because it was getting to be such a big load (that's how I ended up at OSDL, which is now the Linux Foundation - it started out as a way to pay the bills and have health insurance during my year off).

Or all the changes we've done to our source code management. I stayed at patches + tar-balls for the longest time, because it used to work. But I couldn't imagine doing that any more, and we had some painful times with the networking tree using CVS for a while, and trying to sort out the results of that.

So there's been times when it wasn't as much fun, but never really a "walk away" moment, no.

iTWire: What about the BitKeeper episode back in 2005? Was that one of the more stressful moments? And how is your relationship with Andrew Tridgell after that?

LT: So having to drop BK was annoying, but it actually was not nearly as stressful as the events that led up to me _starting_ to use it.

To explain the background, one of the less happy times in kernel development around 2001, with the release of 2.4.0 (and subsequent events). It was one of those major releases, and we'd been doing 2.3.x for a long time, and that was also when the interest in Linux was really taking off in a big way. So things were changing a lot, and the whole development process was really feeling some pain. There were long discussions about patch acceptance (or rather, lack of it - google for "Linus doesn't scale" etc) and we had some serious problems with the VM (largely about that HIGHMEM thing, actually), that were very, very painful.

So that was, I think, the timeframe where the kernel development process _really_ could have failed, and people were unhappy (for various good reasons).

And that timeframe is what explains why I love BitKeeper so much, and have so much respect for Larry McVoy. We ended up solving the VM problems (more or less - there's no way to ever really get the Intel PAE model to work perfectly), but what solved many of the fundamental development problems was getting a source code management system that was really distributed.

That may sound obvious _today_, when people are finally getting pretty used to 'git', and it's widely used even outside the kernel, but in early 2002 when I then started using BitKeeper, not only were there no open source alternatives that were close to viable, almost nobody out there even really _understood_ the point of distributed SCMs. I give Larry McVoy a lot of kudos for pushing BitKeeper, and teaching the world how things should be done. Sure, nothing comes from a vacuum, but still - BK was simply on a different level from anything I had ever seen. To the point that several years later, when I had to drop using it, most everybody _still_ didn't get the point. Due to the licence issue, BK didn't get huge traction even in the kernel community, and so only a fairly small subset of maintainers really "got it" (but enough that during the BK years we had several major subsystems that could be merged through BK, and that made a lot of the scalability issues go away).

So on a stress level, 2001 and 2002 were pretty high. We had serious development flow problems, and me learning the distributed SCM model and just telling people that that's how we're going to do things was certainly not pain-free. But despite all the crap people still give about BK, I'm 100 percent convinced it was absolutely the right thing to do. Because people really don't remember how big a development issue we had before we could have submaintainers that could maintain their own distributed trees and handle distributed merging.

In comparison to that, the couple months of pain in 2005 when BK went away was a walk in the park. It was an annoyance, it wasn't a big fundamental problem with the development model, or a time when I was worried about fundamentals. I even had a ton of fun doing git for a few months.

And yeah, it resulted in awkwardness with Tridge. For a while I really tried very hard to try to find some acceptable common ground where people would be happier with BK, and looking at some way to get Tridge and Larry to come together at some middle ground. For a while I thought that if I could get some nice export format for the BK data, we could get around the major issues. Because at the time there _still_ was no even remotely acceptable open source alternative to BK. And during that time of me looking for solutions, I ended up feeling that "Free Software" (Tridge) vs "Open Source" (me) really hit home at a very practical level.

But as mentioned, I did have fun with git. It resulted in kernel development being shut down for a couple of weeks (and then several months of learning experience and pain for early git adopters), but the end result obviously turned out ok. But I did end up despising most SCM developers out of it all.

iTWire: The number of people and companies that depend on Linux keeps growing year by year. Is this in your subconscious and is there a burden of expectation because of this?

LT: It's not something I really think about all that much, but it does end up impacting the development process. We simply aren't as crazy and care-free as we used to be - it used to be that we could literally just "break the world", and then worry about fixing things up later. These days, we simply can't afford to do that any more. We just need to be more careful, and not as wild and crazy as Linux development used to be.

So it's sometimes less spontaneous, and perhaps less "fun" in a sense. On the other hand, one of the things I've always enjoyed in Linux development has been how it's stayed interesting by evolving. So maybe it's less "fun" in the crazy-go-lucky sense, but on the other hand the much bigger development team and the support brought in by all the companies around Linux has also added its own very real fun. It's a lot more social, for example. So the project may have lost something, but it gained something else to compensate. And that's been true about development in general: development today is still very interesting, even if it's a totally _different_ kind of interesting from what it was almost 20 years ago.

And I wouldn't say it's a "burden". It's resulted in us changing how we do things. These days, exactly to handle the fact that we have all these "serious" users, we have multiple layers of stabilization to try to let people hook into the process at the point that makes sense for them. That starts within the development tree itself (ie the part I maintain, the "upstream kernel"), with the whole notion of a merge window and a separate stabilisation process for each release, but then there's the "stable tree" that is a further stabilisation point, followed by the "long-term releases" which are usually the starting point for "distribution kernels". And then the distros themselves have layers of stabilisation on top of that.

Now, what this all results in is that the actual kernel developers don't need to feel like they have to worry about the staid enterprise users, because there are several layers of quality control and stabilisation between the code jockeys and the people who really don't want to take any risk at all. At the same time, we do try to make sure that the distance between the two points ("developer with new code" and "enterprise user who really doesn't like risk") never gets _too_ wide, because a quick feedback cycle helps everybody. Our fairly short three-month development cycle is geared to that: trying to make it as easy as possible for people to stay as close to the bleeding edge as they are comfortable with.

iTWire: How did this model come about? Is it something that evolved from kernel community discussions or is it something you thought up, suggested and then put into practice?

LT: Pretty much every single development model change has come about as a slow gradual evolution of how we do things, and mostly it has never really been "discussed" or "designed". The shorter development cycle was something that was discussed at one of the kernel summits several years ago, but while it was a big change to how we did things, it was a fairly direct result of everybody realising that we really couldn't continue with multi-year development trees. So it was literally a case of "ok, if two years is too long, how about trying to radically shorten it to two months instead?". And trying it out, and it worked (two months is still the official goal, although everybody knows that in reality it drags out to three months).

So it's not like it was rocket science, it's more a "let's try this and see if it works". The successful models stay around. The "layers of maintenance/stability" is another successful model that we've pretty much always had, it's just that there's more of them. We used to have "Linus does the development tree, Alan Cox does the stable tree, and distributions do their own trees". Now we just have more layers (and the shorter development windows make for more active maintenance trees).

And we have more process in place. We used to have trouble with fixes in the stable tree not always making it to the development tree, so when people started a new stable tree you ended up with old bugs resurfacing that had been fixed once already in the _previous_ stable tree. So now we have the "stable tree patches need to have been in the development tree" rule so that you don't end up in that situation anymore.

But no, there is very little "intelligent design". Most of our process has been small incremental improvements that didn't really need a lot of deep thinking.

iTWire: You seem to be a very low-key person despite all that you've done. How do you manage to keep your feet on the ground?

LT: I've got a huge ego, so I don't know about the whole "low-key" thing.

At the same time, I'm a geek, and not all that social and definitely not interested in the whole public speaking etc side, so I do tend to keep a fairly low profile. And I really enjoy working from home. I also never had any real wish to change the world, or be a visionary - I just wanted to get an OS, and do interesting programming. I never wanted to fly, in other words. I really do enjoy plodding around in the details. Not because I'm humble, but simply because it's what I'm interested in. Maybe that accounts for you thinking I have my feet on the ground.

iTWire: How do you go about prioritising the features for your next release? Is there a lot of lobbying or is it solely an engineering decision?

LT: I'm very seldom feature-oriented. There are some specific cases where I have cared about very specific features (in the current merge window, I made sure that the new filename lookup code got merged, for example), but on the whole what tends to be the most important thing about features are simply whether they are ready, and whether they have actual real users.

A lot of that is tied to our release process - it's largely based on _timing_ rather than on features. If something is ready to be merged, works, and has real users, it gets merged. I will not hold up a release in order for some feature to make it into it, and I don't plan "those features will get into the next release". It's purely a matter of "ok, if was ready for the merge window, and it passed all the criteria, so I'll merge it". And the biggest criterion tends to be that the maintainer of whatever subsystem is happy with it - so it's almost purely an engineering decision.

I say "almost purely", because there is a form of lobbying that works very well. It's simple - companies that have vested interest in certain areas simply pay engineers to get their specific feature done. And obviously engineers are very human, and get attached to and interested in specific issues, and those are often related to things the company they work for is interested in. So I'm not claiming that all decisions are purely based on technical merit, but I think we do have a rather good balance between the pure technical merit, and the wishes of participating companies.

Of course, one thing I've always tried to make sure of is that I'm personally not too tightly associated with any particular Linux player. It's why I've never worked for a Linux distribution, or for any of the big companies making a Linux push. Exactly so that people could trust that at least the top-level maintainer is fairly unbiased and there's no hidden agenda.

iTWire: In your mind, where do kernel features that cater more to the desktop sit?

LT: So to me, personally, the desktop is still where all the interesting stuff is. It's how I personally interact with Linux every day, it's the part of Linux I see most, and it's the area that I think is both the most complex and most influential. It's the one that has the most far-flung problems.

So a kernel feature that helps desktop users is right at the top for me.

That said, there are seldom kernel features that matter hugely (most desktop issues are on the application level), and while the desktop is really important to me, I do have to balance it out with other needs. So if some kernel algorithm doesn't really scale to hundreds of CPUs, we simply can't use it - even if it would work well on the desktop. And I detest "tuning" (because I don't think normal people really ever want to do it, or would know enough even if they wanted to), so it's almost never an issue of "just desktop" vs anything else. Things really do need to work across the board, it's just that desktop testing is one thing I do personally.

But in the last merge window, one of the cool new features we did was this automatic task-grouping for the CPU scheduler, which is pretty much a desktop-only thing. The funny thing was, we actually ended up using code that was designed for specific server setups to implement it. And that's often been the case: features that were _meant_ for one particular niche actually end up being useful for other areas. The whole SMP code, for example, was just for servers not that long ago - very few people had all that many cores on their desktop. These days, of course, if you buy a new machine, you will be hard pressed to find one with just a single CPU in it, even at the low end.

iTWire: What's the one feature you've added or else incorporated into the kernel over the entire development process that's given you the most joy? And what's the one feature that you never liked much but still had to include for reasons beyond your control?

LT: I really can't say that some particular feature has been particularly great or bad, because I've just been doing Linux for too long. We've had lots of great features, and we've had some clunkers.

One thing I still think worked out really well was how we do filename lookup caching (the so-called "dentry" layer). It started out as a patch from a PhD student that did something totally different (it was designed to do file namespace replication over multiple machines) and was really a classic university research project - pretty odd, rather hacky, and not really useful in general. And I just got really excited about the patch, and took the crazy idea and made it do things it was never really meant to do. And these days our whole filename cache is based on that, and the original reason for it is just an obscure detail and was never actually used by anybody afaik.

So that particular thing gave me much joy, because it really worked out so well, and it came from such an odd situation.

When it comes to "feature I had to include for reasons beyond my control", it tends to be about crazy hardware doing stupid things that we just have to work around. Most of the time that's limited to some specific driver or other, and it's not something that has any relevance in the "big picture", or that really affects core kernel design very much. But sometimes it does, and then I really detest it. The whole "support memory past the 4GB mark on a 32-bit architecture" thing is something I absolutely hate. It impacted _everything_. It was really painful, and people trying to run 32-bit x86 processors with 32GB of memory was just a total disaster.

We could do it, but it was a memory management nightmare. And I was really unhappy with how Intel seemed to drag their feet on the whole 64-bit side, pushing instead their crazy and fundamentally flawed Itanium architecture. Then AMD came around with x86-64, and there was finally light at the end of a long, dark, nasty tunnel.

iTWire: From all accounts, you are not only a good manager but also a good family man. How do you manage your time so that nobody feels neglected?

LT: I think it's more that I neglect everybody equally.

I work from home and have very flexible work hours, which makes it a lot easier to be a good dad and help out when help is needed. But at the same time, I do have to say that we have a pretty traditional family model at home, and the real reason my kids don't starve to death is that my wife stays at home and is the one who really takes care of them. I sit in my office and work. A lot. It's not a 9-5 job, it's not even a Mon-Fri job. It's weekends, and it's sitting here answering emails from journalists at 9:30pm.

iTWire: Touche. Do you get lots of requests for interviews? Are there ever times when the Linux Foundation requests you to make a public appearance on their behalf?

LT: Heh. No. I obviously do email interviews, but I don't do a ton of them (it seems to come and go in waves). But it's part of what I consider my job, and I really don't mind it if the questions aren't _too_ inane (which sometimes happens - don't worry, your's haven't been).

And my contract with the Linux Foundation explicitly states that they can't make me do public appearances, or even influence my technical decisions. It's a great contract, there's basically one paragraph saying what I should do (boils down to "maintain the kernel, and everything done on LF time needs to be open source"), while the rest of the contract is about things that I don't have to do.

And the funny part is that I think everybody _likes_ it that way. I obviously do, but even the companies involved in LF and paying dues really all prefer me to be neutral, and not involved in politics. Everybody really is happier knowing that I'm a neutral party, both inside and outside of LF.

Of course, sometimes I still go to LF events. Usually it's somebody saying "Oh, btw, there's world-class diving in xyz, so if you come to the conference you could do some scuba afterwards".

iTWire: Living in America for so many years, you must have remarked on occasion about the differences in attitudes between the people in the US and those in your own native land, Finland. What do you like most about your adopted country? And what do you dislike?

LT: The dislike is the easier one to answer - it's the huge disparity in social and economic circumstances. Coupled to some seriously odd societal hangups ("social safety-nets are bad" or "guns are good, sex is bad" and the occasional "religion is good, science education is bad" crazies).

So one thing I really miss from Finland is the egalitarian society, with strong safety nets, and less of the crazy "punishment" mentality, and more of the "let's help people get away from bad patterns" social model. A couple of years ago, the New York Times ran a big article about the Finnish prison system, and that thing just made me proud to be from Finland. The US really is a third-world country in some respects.

That said, I obviously live here, and not in Finland. Why? Partly simply because the US is bigger. When we moved, we moved because we were young and could try something new, but we moved to the US (and Silicon Valley in particular), because there was an interesting startup doing things that people weren't doing in Finland. I simply wasn't interested in cell-phones, and that's a _big_ deal in Finland. I really do believe that most of the people I went to university with ended up working for Nokia or some related company. A _lot_ of technology in Finland is about mobile phones. Maybe you've played the "Angry Birds" game? That's a Finnish company.

So I moved to the US because I could, and because there was more choice there. And I can talk about "good safety nets" etc all I want, but I'm a programmer, and I make good money. So I'm not personally affected by the lack of safety nets, I just find it disturbing how many people are affected by it, and how everybody still seems to take it for granted. And are often even proud of it.

iTWire: Do you teach your kids about their heritage and encourage them to learn their mother-tongue?

LT: We speak Swedish at home, although with the kids going to regular English-speaking school all their friends obviously speak English, and it has become the stronger language for them, even if Patricia (our eldest) didn't really hear any English at all until she was two or three years old.

They don't even know any Finnish - both me and my wife are of the Swedish-speaking minority in Finland ("finlandssvenskar") and in fact my English is much stronger than my Finnish. I went to Swedish-speaking school all the way to high school, and did 90 percent of my university studies in Swedish too. So to me Finnish was always a distant second language that I'd use mainly when shopping or something like that. My wife went to Finnish-speaking school, but spoke Swedish at home, so while her Finnish is rather stronger than mine, Swedish is her "emotional" language too.

But "teach them about their heritage"? Outside of the language we speak at home, not really.

iTWire: Where do you stand on this whole debate about multiculturalism? Both the US and Australia are melting pots with lots and lots of cultures so the debate is very much alive here. As I guess it must be there too, especially during a time like this when the economy is shot to pieces.

LT: Well, I think one reason I've avoided having to think too much about that discussion is that realistically, it's not like I come from some wildly different cultural background. It's not like the whole "move from Western Europe to the US" was a huge shift of culture. Sure, there are differences, but we're still fundamentally talking "Western culture" and it's not exactly hard fitting in with light skin, blue eyes and brown hair.

And I'm not a huge fan of some rigid multiculturalism and some hardline "you should protect your own culture". If you move to a new place, you certainly don't need to adopt every single cultural notion of that place, but I also think it's crazy and inflexible to think that you should try to take your culture with you. You moved. Deal with it. Don't try to re-create your homeland in the new place. I think it's really sad to see all those "enclaves" where people of the same background move together to make it easier for them to stay the same and keep their old culture. They may make for some great tourist areas or interesting food places, but still...

But as mentioned, it's easy for me to say. I didn't feel (or look) all that out of place.

iTWire: America is a highly religious country. You are, one hears, an agnostic. Does the predominance of religion in everyday life ever prove confronting to you or your family?

LT: There's almost no daily religion that I see, and when I do see any, I'm actually a bit surprised. A large part of that is probably the places we've lived in: neither Silicon Valley nor Portland could even remotely be described as "bible belt", and are both very liberal by US standards - as an European you fit in very well. Yes, there are more churches, and you meet more people who go to church each week (Finland may be 80 percent Lutheran, but people who go to church each week? I don't think I knew many). But religious? Not so you'd really notice it.

Obviously, other parts of the US are different. It _is_ a big country. With a wide disparity in not just social and economic status, but in culture too.

iTWire: So there is no pressure at school for the kids to learn intelligent design and the like?

LT: Creationism/intelligent design? Hell, no. We made sure that our kids go to good schools. The crazies would get laughed out of the place if they tried to bring it up where we live.

iTWire: Have the kids ever asked you questions that would lead to a religious discussion or are they too young for that? If it came up, how would you handle it?

LT: It's come up, but it hasn't been a big deal. The view that all the US is a bible belt really isn't true. Especially not in urban, well-educated areas.

iTWire: Do you ever find yourself reading the source code to other open source operating system (kernels), e.g., NetBSD, to see how a particular feature is implemented? What I mean is, as an author do you have/make time to read code other than kernel code, either for amusement or education?

LT: I haven't ever really found it useful to read other people's code for ideas - source code is a singularly bad medium for transferring high-level concepts, since the whole point is to tell a really stupid computer exactly what to do, rather than explain it at any human level.

So no, I wouldn't read code to see how something is implemented. I'd read code to see why something doesn't _work_ - but then it wouldn't be another OS, it would be something like reading the source code of zlib when I wondered why git spent so much time in some particular library routine ;)

In general, I'd much rather read a book meant for humans than code meant for computers. It's how I started - I still remember Bach's "The Design of the Unix Operating System" fondly. That's how I learnt about how Unix worked. Not to mention Tanenbaum's "Operating Systems: Design and Implementation" book.

These days? No, I don't read OS books any more. I actually seldom read computer books at all.

iTWire: What kind of reading do you do? My special subject for my master's was evolution and I've read that Charles Darwin, after he had worked on The Origin of Species, could not read anything serious in his free time; all that he liked to read was romantic fiction.

LT: Heh, I do read some "serious" books, but they tend to be on biology, human behavior, or physics. But yes, most of the reading I do is really pretty much "crap fiction". It used to be hard science fiction, but these days it's mostly just pretty regular fantasy.

iTWire: What do you do when family or friends ask you to help them fix a problem with their computer, and that computer's not running Linux?

LT: Umm.. Nothing? All the computers in our house are running Linux, so I take care of them, but no, I don't do general computer support.

iTWire: Have you tried to teach any of your children how to program and, if so, in what language?

LT: They haven't shown a lot of interest. If they do, I think I'd start with Python, but who knows? I suspect they'd be better off with just about anybody else teaching them: I don't really have the patience to be a good teacher to begin with, and it's worse when it's your own kids and you get frustrated with them not immediately catching on to some issue. Especially if it's something I consider trivial.

So I occasionally have to go through their math homework with them (because the wife really can't help them), but let's face it - both I and they are happier when I don't need me to.

iTWire: The 2.6 kernel was released way back when. Does the fact that we have yet to see a 2.7 development kernel reflect the fact that there are no new "big ideas" left to explore in the 2-series kernel?

LT: It's more indicative of the whole change of development model. We've been able to introduce pretty big new ideas, but we've done it while not breaking everything, and without having to create a whole new "unstable tree". So we really don't need the pain of having the kind of separate "development kernel" that the 2.1.x, 2.3.x and 2.5.x series were.

We _have_ talked about changing the version numbering, because the "2" at the front seems to be pointless, and the "38" number is starting to be a bit cumbersome (people really aren't very good at remembering the difference between "35" and "36" - it's much easier to keep track of "4" vs "5"). So at the last kernel summit, some people argued for simply dropping the "2" prefix, and turning the "3x" into "3.x", and thus making the numbering a bit more relevant to how we do things. But we did a vote, and there wasn't enough support for it, so nothing happened. But maybe next year more people will have come to the conclusion that the current numbering system is just baroque.



Download an in-depth guide to managing a healthy, motivated and energetic workforce without breaking the bank.


Sam Varghese

website statistics

A professional journalist with decades of experience, Sam for nine years used DOS and then Windows, which led him to start experimenting with GNU/Linux in 1998. Since then he has written widely about the use of both free and open source software, and the people behind the code. His personal blog is titled Irregular Expression.






Join the iTWire Community and be part of the latest news, invites to exclusive events, whitepapers and educational materials and oppertunities.
Why do I want to receive this daily update?
  • The latest features from iTWire
  • Free whitepaper downloads
  • Industry opportunities