The Wonderful World of Linux 2.6
Joseph Pranevich - jpranevich <at> kniggit.net

Although it seems like only yesterday that we were booting up our first Linux 2.4 systems, time has ticked by and the kernel development team is nearing the completion of the v2.6 kernel. This document will attempt to describe many of the new features of the 2.6 kernel (with a heavy bias toward the i386 port of Linux.) Unlike the prerelease announcements of closed-source software, all of the features described here are available now (some more bug-free than others) in the Linux 2.5 development kernel. (By Linux tradition, even-numbered kernels are "stable" and for production systems, while odd numbered kernels are for the intrepid only.) That being said, some of the features described here could still be removed or marked as "experimental" in the actual 2.6 release. The kernel development is currently in a feature-freeze however and it is likely that the final release will not deviate much from what is described here. Please also be aware that some of the "new" features discussed here may have been back-ported to Linux 2.4 after first appearing in Linux 2.6, either officially or by a distribution vendor.

To make this document somewhat more complicated, the exact numbering of the upcoming version of Linux has not yet been decided upon. The most likely revision number is v2.6, but there have been suggestions that v3.0 may be chosen due to the inclusion of NUMA and MMU-less support in this upcoming version (more on that later). For convenience, this document will always be referring to the upcoming version of the kernel as v2.6 and in the present tense.

What you are reading is the first draft of this document, dated 7/13/2003 and based on the 2.5.75 development kernel. There will be other drafts released as the final work on the kernel is completed, but this document will not be "officially" closed until after the final release. As it is the first draft, there may be errors or omissions; if you notice either then let me know, I will be happy to fix them.

The Story So Far...

The Linux kernel was started in 1991 by Linus Torvalds as a Minix-like Operating System for his 386. Linus had originally wanted to name the project Freax, but we are all fortunate that he decided against doing that. The first official release of Linux 1.0 was in March 1994, which included official support only for the i386 and supported only single-processor machines. Linux 1.2 was released in March of 1995 and was the first version to include official support for different system types (specifically, Alpha, Sparc, and Mips). Linux 2.0 arrived in June of 1996 and included support for a number of new architechtures, but most importantly was the first revision to support multi-processor machines (SMP). Linux 2.2 arrived in January of 1999 as an incremental improvement which delivered significantly better performance on multi-processor machines and again supported a wider range of hardware. And finally, Linux 2.4 was released in January of 2001 as another major scalability improvement with SMP, but it also featured the integration of many desktop features into the mainline kernel including USB, PC Card (aka PCMCIA) support, internal plug-and-play, etc. Linux 2.6 stands not only to build on these features, but also to be another "major leap" with improved support for both significantly larger systems and significantly smaller ones (PDAs and other devices.)

Multiple Platform Support

One of the many strengths of the Linux-powered Operating Systems (such as the GNU Operating System, the OS and environment so commonly distributed with Linux that it is often not recognized as a separate entity) is that they support such a wide range of hardware and platforms. Every version since 1.2 has included support for new processor types and features. The Linux 2.6 kernel release is no exception to this trend, and while this may not directly impact the use of Linux on Intel, it's very important for Linux to be as widely available as possible.

Scaling Down -- Linux for Embedded Systems

One of the major ways that Linux 2.6 has grown its support for multiple platforms is through the acceptance of much of the uClinux project into the mainstream kernel. The uClinux project (possibly pronounced "you-see-Linux", but more properly spelled with the Greek character "mu") is the Linux for Microcontrollers project. For many years, this offshoot of Linux has been a driver of support for many embedded processors and it is wonderful to get more of it integrated into the mainline kernel.

Unlike normal Linux ports, the embedded ports described here do not have all of the same features as normal Linux due to hardware limitations. The primary difference is that these ports feature processors that do not feature an MMU. (In the Intel world, MMUs were introduced with the 386.) While these systems are true multitasking Linux systems, they do not have memory protection (so any program can crash any other program) and some system calls dealing with spawning processes are disabled. And because they do not have memory protection (or indeed, much if any security), that also implies that they are not useful for multi-user systems.

There are four major new lines of embedded processors supported by Linux 2.6. The first of these new ports is for Motorola's newer line of embedded m68k processors. These processors go by names such as Dragonball and ColdFire and are included on systems and evaluation boards manufactured by Motorola, Lineo, Arcturus, and others. Most Linux users will be most familiar with these processors as they are the guts underneath Palm Pilots starting with the first (the Palm 1000), up until the Palm III. Sadly, support for older m68k processors without MMUs (such as the 68000s used in early Macintoshes) is not yet covered by this software. Other new supported embedded platforms include Hitachi's H8/300 series (not yet including H8S, but may be integrated soon) and the NEC v850 processor.

It's hard to over-emphasize the major architecture shift that is taking place with support for MMU-less systems under Linux 2.6. All previous versions of Linux were derived, however indirectly, from the limitations inherent with Linus's initial work on an Intel 80386. Extrapolating in this direction, it is possible that other older hardware could also be supported in the future (indeed, there are projects that are working toward this goal) and many new and exciting doors have been opened. Unlike the work on the modern and still-in-production embedded processors included here though, support for older processors would be largely considered a "hobbyist" goal and most likely not useful for end users (and possibly not deemed important enough to include in the official Linux releases of the future.)

Although not technically a part of the uClinux merge (because it includes a MMU), recent versions of Linux also include support for Axis Communications' ETRAX CRIS ("Code Reduced Instruction Set") processors. (Specifically, the ETRAX 100LX and newer are supported.) Actual support for this processor arrived during the 2.4 development cycle, but it was introduced after 2.4.0 so it deserves mention. It is an embedded processor that is primarily for use with network hardware. The ETRAX100 is a related but MMU-less processor that is supported by uClinux, but support for that in the mainline Linux kernel tree has not been integrated.

Opteron Support - 64-bit Linux for the Consumer

Another processor which was integrated during the 2.4.x development cycle but deserving of mention here is Linux's new support for the AMD Opteron chips (based on the AMD64 architecture.) This is a new chip with back-compatibility with existing Intel-clone processors and that may even receive backing from Microsoft. Whether this processor or a member of Intel's Itanium family becomes the de facto standard for 64-bit consumer products remains to be seen.

While later editions of the 2.4 kernel did support running on this processor, there were limitations preventing it from being ready for production use. Most critical for high-end uses may be that applications were individually limited to 512 MB of RAM each. Additionally, support for running x86 (32-bit) binaries on the platform has been improved.

Subarchitecture Support

In addition to the several new processor architectures that Linux 2.6 supports, the new version of Linux also includes a new concept called a "subarchitecture". Previously, Linux often had the underlying assumption that processor types and hardware types went hand in hand. That is, that i386-descendant processors are only used on PC/AT-descendant servers. In Linux 2.4, this assumption was broken for i386 with the addition of support for SGI's Visual Workstation, a "legacy-less" platform running with an Intel chip. (And in fact, it was broken long before on many other architectures. For example, m68k has long supported Amigas, Macintoshes, and other platforms.) The big change in Linux 2.6 is that this feature and concept was standardized so that all architectures handle this in a similar and saner way that allows for more clear separation of only the components that need separated.

With this standardization comes two new platforms to support for i386. The first is NCR's Voyager architecture. This is a SMP system (developed before the now-standard Intel MP specification) supporting 486-686 processors in up to 32x configurations. The actual number of configurations that were sold with this architecture is relatively small, and not all machines are supported yet. (The oldest ones are unsupported.) The second architecture supported is the more widespread PC-9800 platform developed by NEC into the (almost) dominant PC platform in Japan until relatively recently. The original PC-9800 machines shipped with an 8086 processor and the line eventually evolved and matured (in parallel with the AT-descendants) until they featured Pentium-class processors and SMP support. (Of course, the support for Linux is limited to 386 or better.) Although completely unknown in the US, versions of Microsoft products up until Windows 95 were ported to run on this hardware. The line has been officially discontinued by the manufacturer in favor of more "standard" PCs.

By formalizing Linux's support for these "slightly different" hardware types, this will more easily allow the operating system to be ported to other systems, such as dedicated storage hardware and other components that use industry-dominant processor types. To be clear though, one should not take this subdivision too far. These subarchitecture have been separated because very low-level components of the system (such as IRQ routing) are slightly or radically different. This is different than running Linux on an X-Box, for example, where relatively little other than hardware drivers and some quirks separate the system from being a "generic" i386 system. Support for the X-Box would not be a subarchitecture.

Scaling Up -- NUMA and Bigger Iron

In addition to supporting these new hardware types, the new release of the Linux kernel also includes features that make it more acceptable on larger and larger servers (some running i386 processors, some not.) As support for this is very new to Linux, much optimization work remains to be done. This is an area where Linux is very rapidly growing and maturing and we can expect Linux to be a strong contender in this space relatively soon.

One of the biggest changes in this respect is Linux's new support for NUMA servers. NUMA (or "Non-Uniform Memory Access") is a step beyond SMP in the multi-processing world and is a step forward for efficiency on systems that have many processors. SMP systems were designed with some of the same limitations as their uniprocessor counterparts. One of the most limiting ramifications of this design choice is that it is based on having a single pool of memory that is equally accessibly from all processors. On a many-processor system, there is an extremely high contention rate between the multiple processors onto the single memory bus, leading to performance bottlenecks. NUMA servers leap beyond that by introducing the concept that, for a specific processor, some memory is closer than others. One easy way (and not terribly technically incorrect) to imagine this is that you have a system with expansion cards containing CPUs, memory, and possibly other components (I/O, etc.) There are many of these cards in a system and while they can all talk to each other, it's pretty clear that the CPUs will have the easiest time talking to the local memory (the memory on the card rather than on a separate card.) In many ways, the new NUMA architecture is an example of a very tight-knit cluster.

To properly support these new NUMA machines, Linux had to adapt in several respects to make the new model efficient. To start with, an internal topology API was created to actually let the kernel internals understand one processor or one memory pool's relations to I/O devices and each other. Supported from that, the Linux process scheduler now is capable of understanding these relationships and will attempt to optimize tasks for best use of local resources. Additionally, many NUMA machines are built in such a way that they have "holes" in the linear memory space "between" nodes. The new kernel is able to deal with those discontiguous cases in a reasonable way. There are many other internal changes which were made to allow Linux to support these new high-end machines, and this is definitely an area of growth for the kernel as a whole. Over the course of the next year, we can expect to see many more efficiencies and other improvements in Linux's support for these very high-end systems.

Linux Internals

In addition to just flatly supporting more and new kinds of hardware, Linux 2.6 also offers improvements in support for the existing platforms as they (and Linux) improve on their own hardware capabilities. This includes CPU-specific optimizations for Transmeta's Crusoe; Intel's Pentium 4 Xeon, Pentium 3-M, and Pentium 4-M; and mobile AMD processors. The new versions of Linux are also capable of working around a bug in some Athlons. And while most users did not see the bug, Linux 2.6 also resolved a problem where some systems with more than 16 processors would sometimes hang on boot.

Hyperthreading

Although not new in Linux 2.6, the kernel started including support for "hyperthreading" on Intel P4 processors with the 2.4.17 release. (This is included here both because it was not a part of the initial release of Linux 2.4 and also due to the major changes since then.) Hyperthreading is the ability for a single processor to actually masquerade as two (or more) processors from the operating system perspective. What is absolutely the most amazing thing about this feature is that Linux was the first OS to bring the features to market, despite compatible processors being released by Intel almost a year ago. The rumor has it that Microsoft is uncertain whether to charge a license fee for a hyperthreaded processor as one or two CPUs. Linux's open model (and the lack of per-CPU licenses) allowed the Linux OS to be the first to support this new feature. Of course, a single processor that pretends to be two is still a single processor and performance won't really improve much.

One thing that is new in 2.5 is that the scheduler and other components have been optimized so that hyperthreading on a cpu can now actually be a win. Under 2.4, this was not always the case and some workloads would actually be slower with this enabled.

Size Matters -- Scalability Improvements

In addition to NUMA support, Linux 2.6 also has other changes for Intel servers at the top of the food chain. First and foremost is full support for Intel's PAE ("Physical Address Extension") which allows most newer 32-bit x86 systems to access up to 64GB of RAM, but in a paged mode. Additionally, through improved APIC support and other changes, IRQ balancing has been significantly improved on multiprocessor systems.

In many other respects, internal limits have been increased when possible. The number of unique users and groups on a Linux system has been bumped from 65,000 to over 4 billion. (16-bit to 32-bit). This will make Linux more practical on large file and authentication servers where it might be possible to hit the previous limit. Similarly, The number of PIDs (Process IDs) before wraparound has been bumped up from 32,000 to 1 billion. This change, combined with other efficiencies in the PID subsystem, will help to improve application starting performance on very busy or very long-lived systems. Although the maximum number of open files has not been increased, Linux with the 2.6 kernel will no longer require you to set what the limit is in advance; this number will self-scale. And finally, Linux 2.6 will include improved 64-bit support on block devices that support it, even on 32-bit platforms such as i386. This allows for filesystems up to 2TB.

Kernel Interactivity and Responsiveness

One of the areas of focus for Linux 2.6 has been in making the system more responsive for desktop users and other uses that need a high degree of control over when events happen. Each of these target uses have very different challenges, but there have been many changes that can benefit both.

On major internal change present in the 2.6 kernel that should not be understated is that the kernel itself is now preemptible. In all previous versions of Linux, when the OS is currently doing something in the kernel, it can't be interrupted (and on multi-processor machines, this was true on a per-CPU basis.) As of Linux 2.6, the kernel now allows itself to be interrupted mid-task so that user applications can continue to run even if the kernel is doing something complicated. (In order to avoid the obvious race conditions that this can cause, the kernel does have certain sections of the code locked so that they can not be interrupted while in progress.) The primary benefit of this change is that interactive performance (for desktop users, for example) has been given a boost and so the system will "feel" faster for things like user input.

One other change that will help make Linux into a more responsive operating system with applications that support it is the inclusion of support for new "futexes" (or "Fast User-Space Mutexes", if that helps.) Futexes are a way in which multiple processes or threads can serialize events so that they don't trample on each other (a "race condition"). Unlike the traditional mutex operations that most threading libraries support, this is partially kernel based (but only in the contention case) and it also supports setting priorities to allow applications or threads of higher priority access to the contested resource first. By allowing a program to specify that one waiting task is more important than another, it allows for more responsiveness in what may be the more timing-critical areas of an application.

Linux's I/O subsystems has also undergone major changes to allow it to be more responsive under all sorts of workloads. This change includes a complete rewrite of the I/O scheduling subsystem, the code of the kernel that determines what processes get to read from devices and when. The newly rewritten layer is now better capable of ensuring that no processes get stuck waiting in line for I/O for too long, while still allowing for the older optimizations which made sure that reads, for example, happened in the most efficient order for the hardware.

While developers of "real time" (RTOS) applications will benefit from these changes, Linux 2.6 will not be a fully real-time kernel. However, this and other changes lay the ground work to make a fully RTOS version of Linux possible, and external patches have been put out (but not yet approved and integrated into an official kernel release) which can provide that support for users and developers that need it now.

Module Subsystem - Device Drivers

The module subsystem is another area which has received a major boost with the advent of Linux 2.6. Much of the code has been rewritten with the intention to improve stability and to make the system more transparent. In addition to these obvious external changes, many more things have changed "under the hood" with how the kernel sees and uses modules.

The first and most obvious (although least functional) change to kernel drivers in Linux 2.6 is that the file extension has changed. Instead of being ".o" (a common extension for "object" files, generally files created during the compilation of any program, before linking is performed to make an executable application), the new extension is ".ko" ("kernel object"). Other than making it more clear that modules are not truly intermediary files, this is just a cosmetic change.

Completely non-cosmetic is the extensive work to eliminate race conditions that have been present in the code for many revisions. The crux of the problem is that it was possible to have a device start using a module while it is being unloaded, but after the module checks to make sure no one is using it. The new kernel module coding should make this condition much harder to trigger. To take this solution a step further, it is also now also possible to simply disable unloading of modules altogether.

More transparency is another feature of the new module subsystem. In nearly all previous versions of Linux, a module was smart enough to detect devices that it could support by scanning busses (such as PCI, ISA PnP, and PC Card) for device IDs that it recognized. In Linux 2.6, much of this support has been standardized and made external to the kernel so that it will be easier for external programs and module loaders can see what devices a particular module would support. This will allow the various hardware management programs (such as RedHat's "kudzu") to make intelligent decisions about hardware, even when they are not themselves familiar with the devices that they are working with. And in the event that you know better than the current version of the driver, it is still possible (via an interface in the new 'sys' filesystem, see below) to force a driver to try to work on a specific device, even if it does not know it is supported.

Other Improvements

In addition to the many changes described above, there are a number of other general Linux internal changes which will improve performance in many cases. These include more removals of the "Big Kernel Lock" (non-fine-grained locks which were used in the early days' of Linux's support for multiple processors), optimizations of filesystem readahead, writeback, and manipulating small files, and others.

There is one further stability issue that has been resolved with Linux 2.6: it is no longer possible to allocate more than the maximum amount of RAM (plus swap) you have on a system. Previously, Linux would allow the malloc() ("memory allocation") system call to succeed in some cases, even when memory is exhausted. The overcommitment logic has been revised and this case should now be impossible. (Of course, if you run out of RAM on the system-- even without exceeding the maximum-- you have worse problems to worry about.)

Linux has also always been a trumpet-bearer for the benefits of open standards. One of the big internal changes is that the internal Linux threading infrastructure has been rewritten to allow the Native POSIX Thread Library (NTPL) to run on top of it. This is a major performance boost for Pentium Pro and better processors in heavily threaded applications and many of the top players in the "enterprise" space have been clamoring for it. (In fact, RedHat has backported the support to Linux 2.4 and includes it starting with RedHat 9.0) This includes new concepts to the Linux thread space including thread groups, local memory for individual threads, POSIX-style signals, and other changes. One of the major drawbacks is that applications (such as Sun Java) not written to spec that rely on old Linux-isms will break with the new support enabled. As the benefits overwhelm the cost (and with so many large players in the game), it's clear that most important applications will support the changes before too long after the new kernel is released.

And finally, although there is no direct benefit for most applications of Linux, the 2.6 kernel now includes the ability to completely disable support for swap at compile time. This will allow Linux to run in a slightly smaller memory footprint and may be useful for embedded devices where it is unlikely that a swap device would be used at all.

Unified Device Model

Often over-looked as a separate component of an operating system, a device model is nearly essential for an OS that is designed to run on a wide variety of hardware. In a nutshell, the device model is the infrastructure within the kernel for detecting and determining the resource usage of all of the optional components of the system. All operating systems (and versions of Linux) intrinsically have some device concept. Older versions of Linux (2.2 and previous), had only a bare-bones approach to handling devices. There were drivers for different types of hardware busses and the various device drivers knew how to probe the device busses that they supported to find their hardware. This system was very spread-out and the various busses had various and incompatible APIs for handling the various operations. Linux 2.4 was the first stage in a "unified" device model by joining PCI, PC Card, and ISA Plug-and-Play into a single device structure with a common interface. Linux 2.6 has vastly expanded on those efforts to produce a completely new view of how the Linux kernel sees the hardware it is running on that is universal across the system.

Kernel Object Abstraction

The central core of the new device model infrastructure is a new object-oriented interface that all of the lower-level device types need to use. This kernel device object structure (called "kobject") includes all of the interfaces for reference counting and for handling sub-devices. The lower level devices (such as system busses) now use this common layer to present a unified view of the system both in the kernel and in user-space. Now that all this has been centralized, this has allowed Linux to do a number of useful things with this information.

With this new information fully in-place in the kernel, this has allowed Linux to better support systems where a in-depth knowledge of hardware is required. An obvious example of where this information is useful is with power management. The new power management standard in recent years is ACPI. ACPI, the "Advanced Configuration and Power Interface", support was first introduced in Linux 2.4. Unlike APM, systems with this new interface are required to individually tell all compatible devices on the system that they need to change their power states. The new kernel system is just what the doctor ordered to allow the subsystem to keep track of devices that need to be powered down or have other changes made. A second example where this is useful is with busses that support "hot plug". The ability for a device to be added to a system after boot now seems pretty common and straightforward, but true Linux support for these devices did not appear until Linux 2.2. Since Linux 2.4, this support has been improved and support was added for hot-plugging PCI, PC Card, USB, and Firewire devices. The new revision of the kernel's centralized device system expands on this support by essentially removing the differences between hot-plug and legacy devices. When you boot your computer, the device detection routines systematically "plug in" all of the devices on your system. Since a kernel object is created for all objects in the system when they are discovered, and since that is the same regardless of whether the devices are discovered at boot time or later, much of the infrastructure for dealing with pluggable devices has been simplified.

The System Filesystem

Possibly, the most obvious user-visible ramification to this new driver model is the creation of a new system filesystem (to join 'proc' for processes, 'devfs' for devices, and 'devpts' for UNIX98 pseudo-terminals) called 'sysfs'. This filesystem (intended to be mounted on '/sys') is a visible representation of the device tree as the kernel sees it. The kernel gets this information entirely through a very close coupling with the kernel object subsystem: when a kernel object gets created, so too does an appropriate file or directory. (It is possible for a kobject to explicitly not get a record in the sysfs, if there is a reason to do so.)

Since each device (or rather, each kobject) on the system is given a unique directory structure, the next logical step was to export the various known attributes (name of the device, power mode, irq, and that sort of thing) of the device into the tree so that they could be read from and written to by a system administrator. This has beneficial (but initially confusing) side-effect that many of the device-specific uses of /proc/sys may be moved into /sys either already, or at a future date.

Core Hardware Support

As Linux has moved forward over the years and into the mainstream, each iteration of the kernel seemed to be a leap and bound better than the previous in terms of what types of devices it can support both in terms of supporting emerging technologies (USB in 2.4) and supporting older "legacy" technologies (MCA in 2.2). As we arrive at the 2.6 kernel however, the number of devices that Linux does not support is relatively small. There are few, if any, major branches of the PC hardware universe yet to conquer. It is for that reason that most (but certainly not all) of improvements in PC hardware support (including the device model, describe above) have been related to strengthening the support that has already been present.

Internal Device Busses

Arguably as important as the processor, the underling bus(es) in a system are the glue that holds things together. The PC world has been blessed with no shortness of these bus technologies, from the oldest ISA (found in the original IBM PC) to modern external serial and wireless busses. Linux has always been quick to adapt to a new bus and device type as they have become popular with consumer devices, and significantly less quick adapting to technologies that get relatively little use.

One example where Linux came to support a type of bus late was when it finally included support for the ISA Plug-and-Play (PnP) extension into Linux 2.4, much later than the popular commercial OSes supported it. (You could get it working using some kludgy user-space utilities prior to that.) Linux 2.6 includes a major update to this subsystem, making it nearly feature-complete and better integrated into the rest of the new device model. New features include full PnP BIOS support, a device naming database, and other changes to make support more robust. The sum of all of those modifications, is that now Linux is now a "true" Plug-and-Play OS and may be set as such in a compatible machine's BIOS.

The (relatively unpopular) alternatives to PnP in the ISA-era were MCA (or Microchannel Architecture) and EISA (or Extended Industry Standard Architecture). Both of these subsystems have received some updates during the 2.6 development cycle to support the new device model. In addition, EISA has been further standardized with the other subsystems through the inclusion of a device name database.

Other than just the big bullet points, there have also been a number of other noteworthy changes to Linux's support for hardware busses. Linux's PCI bus, the most prevalent and important of all of the busses, support was given a major boost during the development of Linux 2.6 including improved hot-plug and power management support. In a similar vein, the new release will also now include support for systems with multiple AGPs ("accellerated graphics ports" -- a separate high-speed bus that also uses the PCI protocol), such as high-end graphical workstations. In terms of support for these types of PC internals, Linux is keeping up with the hardware trends in the marketplace.

In addition to all of the "real" device busses, Linux 2.6 has added (at least internally) the concept of a "legacy" bus. This is a bus type that is specific to each architecture that contains all of the assumed devices that you would expect to find. On a PC, for example, this may include on-board serial, parallel, and PS/2 ports-- devices that exist but are not enumerated by any real busses on the system. On some platforms, this legacy support may involve more complicated things (such as querying firmware), but in general this is just a wrapper to allow these devices to be handled in a standard way in the new driver paradigm.

External Device Busses

While these older standards of devices have matured but added relatively few new features, that is not the case with USB. The Universal Serial Bus has seen numerous improvements during the most recent development cycle. The most notable of these changes is that the new kernel will support USB 2.0 devices. USB2 is the new standard supporting a device bandwidth of up to 480 megabits per second. (Compared to 12 mbit/sec of current USB.) The devices that support this standard are generally referred to as "high speed" USB devices are are gradually taking over the marketplace. A related new standard, USB On-the-Go (or USB OTG), a point-to-point variant on the USB protocol for connecting devices directly together (for example, to connect a digital camera to a printer without having a PC in the middle) is not currently supported in Linux 2.6. (Patches for this feature are available, but not yet rolled into the official release.) In addition to device support, much of the way USB devices have been internally enumerated has been revised so that it is now possible to have many more devices of the same type all accessible from within Linux. This would primarily be of benefit to large print or storage servers (although the latter would be more likely to use a dedicated storage bus.) This is definitely an area that the technology has grown significantly in the last few years, and Linux is working to keep pace with the devices on the market.

Wireless Devices

Wireless technology has really taken off within the public in the past several years. It often seems as if cords (except power!) will be a thing of the past within a handful of years. Wireless devices encompass both networking devices (the most common currently) and also more generic devices such as PDAs, etc.

In the wireless networking space, devices can generally be divided into long range (for example, AX.25 over amateur radio devices) and short range (usually 802.11, but some older protocols exist.) Support for both of these has been a hallmark of Linux since the early days (v1.2) and both of these subsystems have been updated during development of 2.6. The largest change here is that major components of the short range subsystems for the various supported cards and protocols has been merged into a single "wireless" subsystem and API. This merge resolves a number of minor incompatibilities in the way different devices have been handled and strengthens Linux's support for the subsystem by making a central set of userspace tools that will work with all supported devices. In addition to just standardization, Linux 2.6 introduces a number of overall improvements including better capability to notify in the event of a state change (such as a device that has a "roaming" state) and a change to TCP to better handle periodic delay spikes which occur with wireless devices. Due to the press of better supporting wireless devices on Linux 2.4, many of these changes are already present in the stock kernel.

In the wireless devices space, there have been similar major advancements. IrDA (the infrared protocol named for the Infrared Data Associates group) has received some advancements since the last major release such as power management and integration into the new kernel driver model. The real advancements have been made in providing Linux support for Bluetooth devices. Bluetooth is a new wireless protocol that is designed to be short range and low on power consumption, but does not have the line of sight limitations that IrDA has. Bluetooth as a protocol is deigned to go "anywhere" and has been implemented in devices like PDAs, cell phones, printers, and more bizarre things such as automotive equipment. The protocol itself is made up of two different data link types: SCO, or "Synchronous Connection Oriented", for lossy audio applications; and L2CAP, or "Logical Link Control and Adaptation Protocol", for a more robust connection supporting retransmits, etc. The L2CAP protocol further supports various sub-protcols (including RFCOMM for point-to-point networking and BNEP for Ethernet-like networking.) Linux's support for the things that Bluetooth can do continues to grow and we can expect this to mature significantly once more devices are in the hands of the consumers. It should also be mentioned that initial support for Bluetooth has been integrated into later editions of the 2.4 kernel.

Block Device Support

Storage Busses

Dedicated storage busses, such as IDE/ATA and SCSI, have also received a more major update during the 2.6 cycle. The most major changes centered around the IDE subsystem which has been rewritten (and rewritten again) during the development of the new kernel, resolving many scalability problems and other limitations. For example, IDE CD/RW drives can now be written to directly through the real IDE disk driver, a much cleaner implementation than before. (Previously, it was required to also use a special SCSI-emulating driver which was confusing and often difficult.) The IDE layer is also now capable of querying a machine's BIOS when it comes across a controller that it does not recognize so that it can use reasonable values for timings and other required data. On the SCSI side, there have also been many small improvements scattered around the system both for wider support and scalability. One specific improvement for older systems is that Linux now supports SCSI-2 multipath devices that have more than 2 LUNs on a device. (SCSI-2 is the previous version of the SCSI device standard, circa 1994.) Another important change is that Linux can now fall back to test media changing like Microsoft Windows does, to be more compatible with devices that do not completely follow the specification. As these technologies have stabilized over time, so too has Linux's support for them.

Although not a storage bus in itself, Linux now includes support for accessing a newer machine's EDD (Enhanced Disk Device) BIOS directly to see how the server views its own disk devices. The EDD BIOS includes information on all of the storage busses which are attached to the system that the BIOS knows about (including both IDE and SCSI.) In addition to just getting configuration and other information out of the attached devices, this provides several other advantages. For example, this new interface allows Linux to know what disk device the system was booted from, which is useful on newer systems where it is often not obvious. This allows intelligent installation programs to consider that information when trying to determine where to put GRUB (the Linux boot loader), for example.

In addition to all of these changes, it should be stressed again that all of the bus device types (hardware, wireless, and storage) have been integrated into Linux's new device model subsystem. In some cases, these changes are purely cosmetic. In other cases, there are more significant changes involved (in some cases for example, even logic for how devices are detected needed to be modified.)

Filesystems

The most obvious use of a block device on a Linux (or any other) system is by mounting a filesystem on it, and Linux's supports for filesystems have been vastly improved since Linux 2.4 in a number of respects. Key among these changes include support for extended attributes and POSIX-style access controls.

The second (and third) extended filesystems, the system(s) that most Linux installations use as the default, has been the most improved during the development of Linux 2.6. Principal among these changes is support for "extended attributes", or metadata that can be embedded inside the filesystem itself for a specific file. Some of these extended attributes will be used by the system and readable and writable by root only. Many other operating systems, such as Windows and the MacOS, already make heavy use of these kinds of attributes. Unfortunately, the UNIX legacy of operating systems have not generally included excellent support for these attributes and many user-space utilities (such as 'tar' etc.) will need to be updated before they will save and restore this additional information. This is another area of growth for Linux, and support for extended attributes is maturing.

The first use of the new extended attribute subsystem is to implement POSIX access control lists. POSIX access control is a superset of standard UNIX permissions that allows for more fine-grained control. When necessary (for example, exporting files under NFS) , these controls can be mapped (albeit cautiously) onto the standard user/group permissions.

In addition to these changes for ext3, there are several other smaller changes. The journal commit time for the filesystem can now be tuned to be more suited for laptop users (which might have to spin up the drive if it were in a power save mode.), default mount options can now also be stored within the filesystem itself (so that you don't need to pass them at mount time), and you can now mark a directory as "indexed" to speed up searches of files in the directory.

Linux has also made a number of improvements in the filesystem layer to improve compatibility with the dominant PC operating systems. To begin with, Linux 2.6 now supports Windows' Logical Disk Manager (aka "Dynamic Disks"). This is the new partition table scheme that Windows XP and later have adopted to allow for easier resizing and creation of multiple partitions. (Of course, it is not likely that Linux systems will be using the new scheme for new installations anytime soon.) Linux 2.6 also features improved (and rewritten) support for the NTFS filesystem (which is now also used in consumer products) and it is now possible to mount a NTFS volume read/write. Writing support is still experimental and is gradually being improved; it may or may not be enabled for the final kernel release. And finally, Linux's support for FAT12 (the DOS filesystem used on really old systems and floppy disks) has been improved to work around bugs present in some MP3 players which use that format. Keeping up with the other PC technologies is a major area of focus as new revisions of the Linux kernel.

Compatibility with other operating systems have also been improved. The extended attribute support in the HPFS filesystem (used by OS/2 and others) have been updated. OS/2-style attributes have been separated into a different name-space. The XFS filesystem has also been updated to be on-disk compatible with the IRIX operating system.

In addition to these changes, there have been a large number of more scattered changes in Linux's filesystem support. Quota support has been rewritten to allow for the larger number of users supported on a system. Individual directories can now be marked as synchronous so that all changes (additional files, etc.) will be atomic. (This is most useful for mail systems and directory-based databases, in addition to slightly better recovery in the event of a disk failure.) Transparent compression (a Linux-only extension) has been added to the ISO9660 filesystem (the filesystem used on CD-ROMs.) And finally, a new memory-based filesystem ("hugetlbfs") has been created exclusively to better support shared memory databases.

Input / Output Devices

On the more "external" side of any computer system is the input and output devices, the important bits that never quite seem as important. These include the obvious things like mice and keyboards, sound and video cards, and less obvious things like joysticks and accessibility devices. Many of Linux's end-user subsystems have been expanded during the 2.6 development cycle, but most of the common devices were pretty mature already. Largely, Linux 2.6's improvements to these devices are derived directly from the kernel's improved support for external busses, such as Bluetooth wireless keyboards and similar. These are however a number of areas where Linux has made larger improvements.

Human Interface Devices

One major internal change in Linux 2.6 is the reworking of much of the human interface layer. The human interface layer is the center of the user experience of a Linux system, including the video output, mice, and keyboards. In the new version of the kernel, this layer has been reworked and modularized to a much greater extent than ever before. It is now possible to create a completely "headless" Linux system without any included support for a display or anything. The primary benefit of this modularity may be for embedded developers making devices that can only be administrated over the network or serial, but end-users benefit as many of the underlying assumptions about devices and architectures has been modularized out. For example, it was previously always assumed that if you had a PC that you would need support for a standard AT (i8042) keyboard controller; the new version of Linux removes this requirement so that unnecessary code can be kept out of legacy-less systems.

Support to Linux's handling of monitor output has also received a number of changes, although most of these are useful only in configurations that make use of the kernel's internal framebuffer console subsystem. (Most Intel Linux boxes are not configured this way, but that is not the case for many other architectures.) In my humble opinion, the best feature is that the boot logo (a cute penguin, if you've never seen it) now supports resolutions up to 24bpp. That aside, other new features for the console include resizing and rotating (for PDAs and similar) and expanded accelleration support for more hardware. And finally, Linux has now included kernel support for querying VESA monitors for capability information, although XFree86 and most distributions installation systems already have covered this detail in user-space.

In addition to the big changes, Linux 2.6 also includes a number of smaller changes for human interaction. Touch screens, for example, are now supported. The mouse and keyboard drivers have also been updated and standardized to only export a single device node (/dev/input/mouse0, for example) regardless of the underlying hardware or protocol. Bizarre mice (with multiple scroll wheels, for example) are now also supported. PC keyboard key mappings have also been updated to follow the Windows "standard" for extended keys. Joystick support has also been improved thanks not only to the addition of many new drivers (including the X Box gamepad), but also to include newer features such as force-feedback. And finally (but not least important), the new release also includes support for the Tieman Voyager braille TTY device to allow blind users better access to Linux. (This feature is important enough that it has been back-ported to Linux 2.4 already.)

As a side note, Linux has also changed the "system request" interface to better support systems without a local keyboard. The system request ("sysrq") interface is a method for systems administrators at the local console to get debugging information, force a system reboot, remount filesystems read-only, and do other wizardly things. Since Linux 2.6 now supports a completely headless system, it is now also possible to trigger these events using the /proc filesystem. (Of course, if your system hangs and you need to force it to do things, this may not be of much help to you.)

Audio

One of the most anticipated new features of Linux 2.6 for desktop users is the inclusion of ALSA (the "Advanced Linux Sound Architecture") in lieu of the older sound system. The older system, known as OSS for "Open Sound System", has served Linux since the early days but had many architectural limitations. The first major improvement with the new system is that it has been designed from the start to be completely thread and SMP-safe, fixing problems with many of the old drivers where they would not work properly outside the expected "desktop-means-single-cpu paradigm." More importantly, the drivers have been designed to be modular from the start (users of older versions of Linux will remember that modularity was retro-fitted onto the sound system around Linux 2.2), and that this allows for improved support for systems with multiple sound cards, including multiple types of sound cards. Regardless of how pretty the internals are, the system would not be an improvement for users if it did not have neat new whiz-bang features, and the new sound system has many of those. Key among them are support for newer hardware (including USB audio and MIDI devices), full-duplex playback and recording, hardware and non-interleaved mixing, support for "merging" sound devices, and other things. Whether you are an audiophile or just someone that likes to play MP3s, Linux's improved sound support should be a welcome step forward.

Multimedia

Beyond simple audio these days, what users want is support for the really fancy hardware like webcams, radio and TV adapters, and digital video recorders. In all three cases, Linux's support has been improved with the 2.6 release.

While Linux has supported (to a greater or lesser extent) radio cards (often through userspace) for many iterations, support for television tuners and video cameras was only added within the last one or two major revisions. That subsystem, known as Video4Linux (V4L), has received a major upgrade during the work on the new edition of the kernel including both an API cleanup and support for more functionality on the cards. The new API is not compatible with the previous one and applications supporting it will need to upgrade with the kernel.

And on a completely new track, Linux 2.6 includes the first built-in support for Digital Video Broadcasting (DVB) hardware. This type of hardware, common in set-top boxes, can be used to make a Linux server into a Tivo-like device, with the appropriate software.

Software Improvements

The improvements to Linux are not solely focused on hardware support and infrastructure. Without supporting software (such as filesystems and network protocols), supported hardware is just so much dead wood.

Networking

Leading-edge networking infrastructure has always been one of Linux's prime assets. Linux as an OS already supports most of the world's dominant network protocols including TCP/IP (v4 and v6), AppleTalk, IPX, and others. (In fact, the only common one that comes to mind is Microsoft's old and tangled NetBIOS/NetBEUI protocols.)

Like many of the changes in the other subsystems, most networking hardware changes with Linux 2.6 are under the hood and not immediately obvious. This includes low-level changes to take advantage of the device model and updates to many of the device drivers. For example, Linux now includes a separate MII (Media Independent Interface, or IEEE 802.3u) subsystem which is used by a number of the network device drivers. This new subsystem replaces many instances where each driver was handling that device's MII support in slightly different ways and with duplicated code and effort. Other changes include major ISDN updates and other things.

On the software side, one of the most major changes is Linux's new support for the IPsec protocols. IPsec, or IP Security, is a collection of protocols for IPv4 ("normal" IP) and IPv6 that allow for cryptographic security at the network protocol level. And since the security is at the protocol level, applications do not have to be explicitly aware of it. This is similar to SSL and other tunneling/security protocols, but at a much lower level. Currently supported in-kernel encryption includes various flavors of SHA ("secure hash algorithm"), DES ("data encryption standard"), and others.

Elsewhere on the protocol side, Linux has improved its support for multicast networking. Multicast networks are networks where a single sent packet is intended to be received by multiple computers. (Compare to traditional point-to-point networks where you are only speaking to one at a time.) Primarily, this functionality is used by messaging systems (such as Tibco) and audio/video conferencing software. Linux 2.6 improves on this by now supporting several new SSM (Source Specific Multicast) protocols, including MLDv2 (Multicast Listener Discovery) and IGMPv3 (Internet Group Messaging Protocol.) These are standard protocols that are supported by most high-end networking hardware vendors, such as Cisco.

Linux 2.6 also has broken out a separate LLC stack. LLC, or Logical Link Control protocol (IEEE 802.2), is a low-level protocol that is used beneath several common higher-level network protocols such as Microsoft's NetBeui, IPX, and AppleTalk. As part of this change-over, the IPX, AppleTalk, and Token Ring drivers have been rewritten to take advantage of the new common subsystem. In addition, an outside source has put together a working NetBEUI stack and it remains to be seen whether it will ever be integrated into the stock kernel.

In addition to these changes, there have been a number of smaller changes. IPv6 has received some major changes and it can now also run on Token Ring networks. Linux's NAT/masquerading support has been extended to better handle protocols that require multiple connections (H.323, PPTP, etc.) On the Linux router front, support for configuring VLANs on Linux has been made no longer "experimental".

Network Filesystems

Overlaid on top of Linux's robust support for network protocols is Linux's equally robust support for network filesystems. Mounting (and sometimes exporting) a network filesystem is one of the very few high-level network operations that the kernel cares about directly. (The most obvious other, the "network block device", did not receive many changes for 2.6 and is generally used in specialized applications where you end up doing something filesystem-like with it anyway.) All other network operations are content to be relegated to user-space and outside the domain of the kernel developers.

In the Linux and UNIX-clone world, the most common of the network filesystems is the aptly named Network File System, or NFS. NFS is a complicated file sharing protocol that has deep roots in UNIX (and especially Sun Solaris' excellent implementation). The primary transport protocol can utilize either TCP or UDP, but several additional sub-protocols are also required, each of which also run on top of the separate RPC ("remote procedure call") protocol. These include the separate "mount" protocol for authentication and NLM ("network lock manager") for file locking. (The common implementation is also tied closely to other common RPC-based protocols, including NIS-- "network information service"-- for authentication. NIS is not commonly used for authentication on Linux machines due to fundamental insecurities.) It is perhaps because of this complexity that NFS has not been widely adapted as an "Internet" protocol.

In Linux 2.6, this core Linux filesystem received many updated and improvements. The largest of these improvements is that Linux now experimentally supports the new and not widely adopted NFSv4 protocol version for both its client and server implementations. (Previous versions of Linux included support for both v2 and v3 of the protocol.) The new version supports stronger and more secure authentication (with cryptography), more intelligent locking, support for pseudo-filesystems, and other changes. Not all of the new NFSv4 features have been implemented in Linux yet, but the support is relatively stable and could be used for some production applications. In addition, Linux's NFS server implementation has been improved to be more scalable (up to 64 times as many concurrent users and a larger request queues), to be more complete (by supporting serving over TCP, in addition to UDP), to be more robust (individual filesystems drivers can adapt the way files on those systems are exported to suit their particularities), and more easily maintainable (management though a new 'nfsd' filesystem, instead of system calls.) There have also been may other under the hood changes, including separating lockd and nfsd and support for zero-copy networking on supported interfaces. The NFS client side has also benefited from a number of improvements to the implementation of the underlying RPC protocol including a caching infrastructure, connection control over UDP, and other improvements for TCP. Linux's support for using NFS-shared volumes as the root filesystem (for disk-less systems) has also been improved as the kernel now supports NFS over TCP for that purpose.

In addition to improving support for the UNIX-style network filesystems, Linux 2.6 also delivers many improvements to Windows-style network filesystems. The standard shared filesystem for Windows servers (as well as OS/2 and other operating systems) has been the SMB ("server message block") protocol and the Linux kernel has had excellent client support of the SMB protocol for many revisions. Windows 2000 however standardized on an upgraded superset of the SMB protocol, known as CIFS ("common internet filesystem.") The intention of this major update was to streamline and refine certain aspects of SMB which had at that point become a complete mess. (The protocol itself was loosely defined and often extended to the point that there were cases even where the Win95/98/ME version was incompatible with the WinNT/Win2k version.) CIFS delivered on that intention and added UNICODE support, improved file locking, hard linking, eliminated the last vestiges of NetBIOS dependencies, and added a few other features for Windows users. Since Linux users do not like to be kept in the dark for long, Linux 2.6 now includes completely rewritten support for mounting CIFS filesystems natively. Linux 2.6 also now includes support for the SMB-UNIX extensions to the SMB and CIFS protocols which allows Linux to access non-Windows file types (such as device nodes and symbolic links) on SMB servers which support it (such as Samba.)

Although not as commonly seen today, Linux has not completely forgotten about the Novell NetWare users. Linux 2.6 now allows Linux clients to mount up to the maximum of 256 shares on a single NetWare volume using its built in NCP ("NetWare Core Protocol") filesystem driver.

Linux 2.6 also includes improved support for the relatively new domain of distributed network filesystems, systems where files on a single logical volume can be scattered across multiple nodes. In addition to the CODA filesystem introduced in Linux 2.4, Linux now includes some support for two other distributed filesystems: AFS and InterMezzo. AFS, the Andrew filesystem (so named because it was originally developed at CMU), is presently very limited and restricted to read-only operations. The second newly supported filesystem, InterMezzo (also developed at CMU), is also newly supported under Linux 2.6 and it allows for more advanced features such as disconnect operation (so you work on locally cached files) and is suitable for high-availability applications where you need to guarantee that storage is never unavailable (or faked, when down). It also has applications for keeping data in sync between multiple computers, such as a laptop or PDA and a desktop computer. Many of the projects providing support for these new types of filesystems are initially developed on Linux, putting Linux well ahead of the curve in support for these new features.

Miscellaneous Features

Security

Another of the big changes in Linux 2.6 that does not receive enough attention is the wealth of new security-related changes. Most fundamentally, the entirety of kernel-based security (powers of the super user under a UNIX-like operating system) has been modularized out to be one out of a potential number of alternate security modules. (At present however, the only offered security model is the default one and an example how to make your own.) As part of this change, all parts of the kernel have now been updated to use "capabilities" as the basis of fine-grained user access, rather than the old "superuser" system. Nearly all Linux systems will continue to have a "root" account which has complete access, but this allows for a Linux-like system to be created which does not have this underlying assumption. Another security-related change is that binary modules (for example, drivers shipped by a hardware manufacturer) can no longer "overload" system calls with their own and can no longer see and modify the system call table. This significantly restricts the amount of access that non-open source modules can do in the kernel and possibly closes some legal loopholes around the GPL. The final change that is somewhat security-related is that Linux with the new kernel is now able to use hardware random number generators (such as those present in some new processors), rather than relying on a (admittedly quite good) entropy pool based on random hardware fluctuations.

Virtualizing Linux

One of the most interesting new features in Linux 2.6 is its inclusion of a "user-mode" architecture. This is essentially a port (like to a different hardware family) of Linux to itself, allowing for a completely virtualized Linux-on-Linux environment to be run. The new instance of Linux runs as if it was a normal application. "Inside" the application, you can configure fake network interfaces, filesystems, and other devices through special drivers which communicate up to the host copy of Linux in a secure way. This has proved quite useful, both for development purposes (profiling, etc.) as well as for security analysis and honeypots. While most users will never need this kind of support, it is an incredibly "cool" feature to have running on your box. (Impress your friends!)

Laptops

In addition to all of the other general purpose support described above (improved APM and ACPI, wireless support improvements, etc.) Linux also includes two other hard-to-classify features that will best assist laptop users. The first is that the new edition of the kernel now supports full software-suspend-to-disk functionality for the Linux user on the go. The new version also supports the ability of modern mobile processors to change speed (and power requirements) based on whether you are plugged in or not.

Legacy Support

Although Linux 2.6 is a major upgrade, the difference to user-mode applications will be nearly non-existent. The one major exception to this rule appears to be threading: some applications may do things that worked under 2.4 or 2.2 but are no longer allowed. Those applications should be the exception to the rule however. Of course, low-level applications such as module utilities will definitely not work. Additionally, some of the files and formats in the /proc and /dev directories have changed and any applications that have dependencies on this may not function correctly. (This is especially true as more things shift over to the new /sys virtual filesystem. In the /dev case, backwards-compatible device names can easily be configured.)

In addition to those standard disclaimers, there are a number of other smaller changes which may affect some environments. First, very old swap files (from Linux 2.0 or earlier) will need to be reformatted before they can be used with 2.6. (Since swap files do not contain any permanent data, that should not be a problem for any user.) The kHTTPd daemon which allowed the kernel to serve web pages directly has also been removed as most of the performance bottlenecks that prevented Apache, Zeus, et. al. from reaching kernel speeds have been resolved. Support for old DOS "disk managers" such as OnTrack and EzDrive for very old harddisks and DOS-compatibility has been removed. Support for using a special kernel-included boot sector for booting off of a floppy disk has also been removed; you should use SysLinux instead. And finally, the "soundmodem" support (for playing acoustic games with your soundcard to make it into a modem) has been removed; a user-space version is better maintained and more up to date.

Stuff At The Bottom

This document was assembled primarily from long hours looking at BitKeeper changelogs, looking at and playing with the source, reading mailing list posts, and lots and lots of Google and Lycos searches for documentation about this and that. As such, there are likely places where something could have been missed or misunderstood, and places where something could have been backed out after the fact. (I have been especially careful of the two versions of IDE support that were worked on during this time period, but there are other examples.) I have also tried to be particularly careful abuse using project web pages for projects that have been integrated into the kernel as they often are "further ahead" than what is officially supported. If you see any errors in this document or want to email me to ask me how my day is going, you can do so at jpranevich <at> kniggit.net.

For more technical minded people that just want a bullet list, Dave Jones has put together another excellent summary of Linux 2.6 development progress at http://www.codemonkey.org.uk/post-halloween-2.5.txt. Any similarities between this document and his work is purely coincidental since we are both pulling from original sources and we're both i386-bound.

This document is Copyright 2003, Joseph Pranevich. Redistribution online without modification is permitted, but please email me if you are interested in offline distribution (magazines, etc.) As a matter of courtesy, I would appreciate an email in either case as a newer version may be available.