Solaris 10 and Net-SNMP

Net-SNMP is provided in Solaris 10 release now, and how to use it is descripted in this article.

To disable Sun's SNMP
#svcadm disable svc:/application/management/snmpdx:default

the Net-SNMP daemon program locates at /usr/sfw/sbin/snmpd, and configure file of snmpd saved with the name snmpd.conf at /etc/sma/snmp. Use svcadm to start of stop snmpd:
#svcadm enable svc:/application/management/sma:default
#svcadm disable svc:/application/management/sma:default

Log File
if there's any error or warning messages, see the file /var/svc/log/application-management-sma\:default.log. The default log file of snmpd is /var/log/snmpd.log.

Configuration file
We may use /etc/sma/snmp/snmpd.conf.save as the working configure file to test wheather our snmpd daemon is ok.
#cp /etc/sma/snmp/snmpd.conf.save /etc/sma/snmp/snmpd.conf
#svcadm restart sma

P.S. There's another file also named as snmpd.conf stored in /var/sma_snmp directory, and which is called persistent storage file. The persistent storage file, /var/sma_snmp/snmpd.conf, contains USM security information and any MIB components that are set for persistent storage. This file also contains the engineID and the engineID boots. This persistent storage file is automatically updated when the System Management Agent starts. When the System Management Agent stops, the snmpusm and snmpvacm utilities write user security information to this storage file.

Using the AgentX Protocol
The AgentX protocol is supported in the System Management Agent. By default, the System Management Agent ships with a secure profile, that is, read–only access. AgentX allows interaction with third party subagents, provided that these subagents support AgentX over Unix Domain Sockets. For security reasons, AgentX is not supported over TCP/UDP. For more information on the AgentX protocol, see http://www.ietf.org/rfc/rfc2741.txt Edit the main /etc/sma/snmp/snmpd.conf configuration file to configure the System Management Agent to use the AgentX protocol. By default, the AgentX protocol is disabled. The following procedure describes how to enable the AgentX protocol. To enable the AgentX protocol add the following to /etc/sma/snmp/snmpd.conf file:

master agentx

restart the sma (system management agent).
# svcadm restart svc:/application/management/sma:default

Using the SMA
To view the status of the agent
# svcs svc:/application/management/sma:default

To See Which MIBs Are Initialized
# /usr/sfw/sbin/snmpd -Dregister_mib -Dmib_init -L

To Check the Disk Space and Network on a Local or Remote Machine
# /usr/sfw/bin/snmpwalk -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageIndex

# /usr/sfw/bin/snmpget -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageSize.1

# /usr/sfw/bin/snmpwalk -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageUsed

# /usr/sfw/bin/snmpnetstat -v 2c -c public -a localhost

if all these works, congratulations, the sma is configured and brought on properly.

A Comparison of Solaris, Linux, and FreeBSD Kernels

by Max Bruning

I spend most of my time teaching classes on Solaris internals, device drivers, and kernel crash dump analysis and debugging. When explaining to classes how various subsystems are implemented in Solaris, students often ask, "How does it work in Linux?" or, "In FreeBSD, it works like this, how about Solaris?" This article examines three of the basic subsystems of the kernel and compares implementation between Solaris 10, Linux 2.6, and FreeBSD 5.3.

The three subsystems examined are scheduling, memory management, and file system architecture. I chose these subsystems because they are common to any operating system (not just Unix and Unix-like systems), and they tend to be the most well-understood components of the operating system.

This article does not go into in-depth details on any of the subsystems described. For that, refer to the source code, various websites, and books on the subject. For specific books, see:

• Solaris Internals: Core Kernel Architecture by McDougall and Mauro (visit Solaris Internals)
• The Design and Implementation of the FreeBSD Operating System by McKusick and Neville-Neil (visit The Design and Implementation of the FreeBSD Operating System)
• Linux Kernel Development by Love (visit Linux Kernel Development, 2nd Edition) and Understanding the Linux Kernel by Bovet and Cesati (visit Understanding the Linux Kernel, 2nd Edition)

If you search the Web for Linux, FreeBSD, and Solaris comparisons, most of the hits discuss old (in some cases, Solaris 2.5, Linux 2.2, etc.) versions of the OSes. Many of the "facts" are incorrect for the newest releases, and some were incorrect for the releases they intended to describe. Of course, most of them also make value judgments on the merits of the OSes in question, and there is little information comparing the kernels themselves. The following sites seem more or less up to date:

• "Solaris Vs. Linux" is pretty one-sided for Solaris 10 over Linux.
• "Comparing MySQL Performance" on Solaris 10, Linux, FreeBSD, and others.
• "Fast Track to Solaris 10 Adoption" has some comparisons between Linux and Solaris.
• "Solaris 10 Heads for Linux Territory" is not really a comparison, but reviews Solaris 10.

One of the more interesting aspects of the three OSes is the amount of similarities between them. Once you get past the different naming conventions, each OS takes fairly similar paths toward implementing the different concepts. Each OS supports time-shared scheduling of threads, demand paging with a not-recently-used page replacement algorithm, and a virtual file system layer to allow the implementation of different file system architectures. Ideas that originate in one OS often find their way into others. For instance, Linux also uses the concepts behind Solaris's slab memory allocator. Much of the terminology seen in the FreeBSD source is also present in Solaris. With Sun's move to open source Solaris, I expect to see much more cross-fertilization of features. Currently, the LXR project provides a source cross-reference browser for FreeBSD, Linux, and other Unix-related OSes, available at fxr.watson.org. It would be great to see OpenSolaris source added to that site.

Scheduling and Schedulers

The basic unit of scheduling in Solaris is the kthread_t; in FreeBSD, the thread; and in Linux, the task_struct. Solaris represents each process as a proc_t, and each thread within the process has a kthread_t. Linux represents processes (and threads) by task_struct structures. A single-threaded process in Linux has a single task_struct. A single-threaded process in Solaris has a proc_t, a single kthread_t, and a klwp_t. The klwp_t provides a save area for threads switching between user and kernel modes. A single-threaded process in FreeBSD has a proc struct, a thread struct, and a ksegrp struct. The ksegrp is a "kernel scheduling entity group." Effectively, all three OSes schedule threads, where a thread is a kthread_t in Solaris, a thread structure in FreeBSD, and a task_struct in Linux.

Scheduling decisions are based on priority. In Linux and FreeBSD, the lower the priority value, the better. This is an inversion; a value closer to 0 represents a higher priority. In Solaris, the higher the value, the higher the priority. Table 1 shows the priority values of the different OSes.

Table 1. Scheduling Priorities in Solaris, Linux, and FreeBSD

Solaris
	Priorities	Scheduling Class
	0-59	Time Shared, Interactive, Fixed, Fair Share Scheduler
	60-99	System Class
	100-159	Real-Time (note real-time higher than system threads)
	160-169	Low level Interrupts
Linux	Priorities	Scheduling Class
	0-99	System Threads, Real time (SCHED_FIFO, SCHED_RR)
	100-139	User priorities (SCHED_NORMAL)
FreeBSD	Priorities	Scheduling Class
	0-63	Interrupt
	64-127	Top-half Kernel
	128-159	Real-time user (system threads are better priority)
	160-223	Time-share user
	224-255	Idle user

All three OSes favor interactive threads/processes. Interactive threads run at better priority than compute-bound threads, but tend to run for shorter time slices. Solaris, FreeBSD, and Linux all use a per-CPU "runqueue." FreeBSD and Linux use an "active" queue and an "expired" queue. Threads are scheduled in priority from the active queue. A thread moves from the active queue to the expired queue when it uses up its time slice (and possibly at other times to avoid starvation). When the active queue is empty, the kernel swaps the active and expired queues. FreeBSD has a third queue for "idle" threads. Threads run on this queue only when the other two queues are empty. Solaris uses a "dispatch queue" per CPU. If a thread uses up its time slice, the kernel gives it a new priority and returns it to the dispatch queue. The "runqueues" for all three OSes have separate linked lists of runnable threads for different priorities. (Though FreeBSD uses one list per four priorities, both Solaris and Linux use a separate list for each priority.)

Linux and FreeBSD use an arithmetic calculation based on run time versus sleep time of a thread (as a measure of "interactive-ness") to arrive at a priority for the thread. Solaris performs a table lookup. None of the three OSes support "gang scheduling." Rather than schedule n threads, each OS schedules, in effect, the next thread to run. All three OSes have mechanisms to take advantage of caching (warm affinity) and load balancing. For hyperthreaded CPUs, FreeBSD has a mechanism to help keep threads on the same CPU node (though possibly a different hyperthread). Solaris has a similar mechanism, but it is under control of the user and application, and is not restricted to hyperthreads (called "processor sets" in Solaris and "processor groups" in FreeBSD).

One of the big differences between Solaris and the other two OSes is the capability to support multiple "scheduling classes" on the system at the same time. All three OSes support Posix SCHED_FIFO, SCHED_RR, and SCHED_OTHER (or SCHED_NORMAL). SCHED_FIFO and SCHED_RR typically result in "realtime" threads. (Note that Solaris and Linux support kernel preemption in support of realtime threads.) Solaris has support for a "fixed priority" class, a "system class" for system threads (such as page-out threads), an "interactive" class used for threads running in a windowing environment under control of the X server, and the Fair Share Scheduler in support of resource management. See priocntl(1) for information about using the classes, as well as an overview of the features of each class. See FSS(7) for an overview specific to the Fair Share Scheduler. The scheduler on FreeBSD is chosen at compile time, and on Linux the scheduler depends on the version of Linux.

The ability to add new scheduling classes to the system comes with a price. Everywhere in the kernel that a scheduling decision can be made (except for the actual act of choosing the thread to run) involves an indirect function call into scheduling class-specific code. For instance, when a thread is going to sleep, it calls scheduling-class-dependent code that does whatever is necessary for sleeping in the class. On Linux and FreeBSD, the scheduling code simply does the needed action. There is no need for an indirect call. The extra layer means there is slightly more overhead for scheduling on Solaris (but more features).

Memory Management and Paging

In Solaris, every process has an "address space" made up of logical section divisions called "segments." The segments of a process address space are viewable via pmap(1). Solaris divides the memory management code and data structures into platform-independent and platform-specific parts. The platform-specific portions of memory management is in the HAT, or hardware address translation, layer. FreeBSD describes its process address space by a vmspace, divided into logical sections called regions. Hardware-dependent portions are in the "pmap" (physical map) module and "vmap" routines handle hardware-independent portions and data structures. Linux uses a memory descriptor to divides the process address space into logical sections called "memory areas" to describe process address space. Linux also has a pmap

command to examine process address space.

Linux divides machine-dependent layers from machine-independent layers at a much higher level in the software. On Solaris and FreeBSD, much of the code dealing with, for instance, page fault handling is machine-independent. On Linux, the code to handle page faults is pretty much machine-dependent from the beginning of the fault handling. A consequence of this is that Linux can handle much of the paging code more quickly because there is less data abstraction (layering) in the code. However, the cost is that a change in the underlying hardware or model requires more changes to the code. Solaris and FreeBSD isolate such changes to the HAT and pmap layers respectively.

Segments, regions, and memory areas are delimited by:

• Virtual address of the start of the area.
• Their location within an object/file that the segment/region/memory area maps.
• Permissions.
• Size of the mapping.

For instance, the text of a program is in a segment/region/memory area. The mechanisms in the three OSes to manage address spaces are very similar, but the names of data structures are completely different. Again, more of the Linux code is machine-dependent than is true of the other two OSes.

Paging

All three operating systems use a variation of a least recently used algorithm for page stealing/replacement. All three have a daemon process/thread to do page replacement. On FreeBSD, the vm_pageout daemon wakes up periodically and when free memory becomes low. When available memory goes below some thresholds, vm_pageout runs a routine (vm_pageout_scan) to scan memory to try to free some pages. The vm_pageout_scan routine may need to write modified pages asynchronously to disk before freeing them. There is one of these daemons regardless of number of CPUs. Solaris also has a pageout daemon that also runs periodically and in response to low-free-memory situations. Paging thresholds in Solaris are automatically calibrated at system startup so that the daemon does not overuse the CPU or flood the disk with page-out requests. The FreeBSD daemon uses values that, for the most part, are hard-coded or tunable in order to determine paging thresholds. Linux also uses an LRU algorithm that is dynamically tuned while it runs. On Linux, there can be multiple kswapd

daemons, as many as one per CPU. All three OSes use a global working set policy (as opposed to per process working set).

FreeBSD has several page lists for keeping track of recently used pages. These track "active," "inactive," "cached," and "free" pages. Pages move between these linked lists depending on their uses. Frequently accessed pages will tend to stay on the active list. Data pages of a process that exits can be immediately placed on the free list. FreeBSD may swap entire processes out if vm_pageout_scan cannot keep up with load (for example, if the system is low on memory). If the memory shortage is severe enough, vm_pageout_scan will kill the largest process on the system.

Linux also uses different linked lists of pages to facilitate an LRU-style algorithm. Linux divides physical memory into (possibly multiple sets of) three "zones:" one for DMA pages, one for normal pages, and one for dynamically allocated memory. These zones seem to be very much an implementation detail caused by x86 architectural constraints. Pages move between "hot," "cold," and "free" lists. Movement between the lists is very similar to the mechanism on FreeBSD. Frequently accessed pages will be on the "hot" list. Free pages will be on the "cold" or "free" list.

Solaris uses a free list, hashed list, and vnode page list to maintain its variation of an LRU replacement algorithm. Instead of scanning the vnode or hash page lists (more or less the equivalent of the "active"/"hot" lists in the FreeBSD/Linux implementations), Solaris scans all pages uses a "two-handed clock" algorithm as described in Solaris Internals and elsewhere. The two hands stay a fixed distance apart. The front hand ages the page by clearing reference bit(s) for the page. If no process has referenced the page since the front hand visited the page, the back hand will free the page (first asynchronously writing the page to disk if it is modified).

All three operating systems take NUMA locality into account during paging. The I/O buffer cache and the virtual memory page cache is merged into one system page cache on all three OSes. The system page cache is used for reads/writes of files as well as mmapped files and text and data of applications.

File Systems

All three operating systems use a data abstraction layer to hide file system implementation details from applications. In all three OSes, you use open, close, read, write, stat, etc. system calls to access files, regardless of the underlying implementation and organization of file data. Solaris and FreeBSD call this mechanism VFS ("virtual file system") and the principle data structure is the vnode, or "virtual node." Every file being accessed in Solaris or FreeBSD has a vnode assigned to it. In addition to generic file information, the vnode contains pointers to file-system-specific information. Linux also uses a similar mechanism, also called VFS (for "virtual file switch"). In Linux, the file-system-independent data structure is an inode. This structure is similar to the vnode on Solaris/FreeBSD. (Note that there is an inode structure in Solaris/FreeBSD, but this is file-system-dependent data for UFS file systems). Linux has two different structures, one for file operations and the other for inode operations. Solaris and FreeBSD combine these as "vnode operations."

VFS allows the implementation of many file system types on the system. This means that there is no reason that one of these operating systems could not access the file systems of the other OSes. Of course, this requires the relevant file system routines and data structures to be ported to the VFS of the OS in question. All three OSes allow the stacking of file systems. Table 2 lists file system types implemented in each OS, but it does not show all file system types.

Table 2. Partial List of File System Types

Solaris	ufs	Default local file system (based on BSD Fast Filesystem)
	nfs	Remote Files
	proc	/proc files; see proc(4)
	namefs	Name file system; allows opening of doors/streams as files
	ctfs	Contract file system used with Service Management Facility
	tmpfs	Uses anonymous space (memory/swap) for temporary files
	swapfs	Keeps track of anonymous space (data, heap, stack, etc.)
	objfs	Keeps track of kernel modules, see objfs(7FS)
	devfs	Keeps track of /devices files; see devfs(7FS)
FreeBSD	ufs	Default local file system (ufs2, based on BSD Fast Filesystem)
	defvs	Keeps track of /dev files
	ext2	Linux ext2 file system (GNU-based)
	nfs	Remote files
	ntfs	Windows NT file system
	smbfs	Samba file system
	portalfs	Mount a process onto a directory
	kernfs	Files containing various system information
Linux	ext3	Journaling, extent-based file system from ext2
	ext2	Extent-based file system
	afs	AFS client support for remote file sharing
	nfs	Remote files
	coda	Another networked file system
	procfs	Processes, processors, buses, platform specifics
	reiserfs	Journaling file system

Conclusions

Solaris, FreeBSD, and Linux are obviously benefiting from each other. With Solaris going open source, I expect this to continue at a faster rate. My impression is that change is most rapid in Linux. The benefits of this are that new technology has a quick incorporation into the system. Unfortunately, the documentation (and possibly some robustness) sometimes lags behind. Linux has many developers, and sometimes it shows. FreeBSD has been around (in some sense) the longest of the three systems. Solaris has its basis in a combination of BSD Unix and AT&T Bell Labs Unix. Solaris uses more data abstraction layering, and generally could support additional features quite easily because of this. However, most of the layering in the kernel is undocumented. Probably, source code access will change this.

A brief example to highlight differences is page fault handling. In Solaris, when a page fault occurs, the code starts in a platform-specific trap handler, then calls a generic as_fault() routine. This routine determines the segment where the fault occurred and calls a "segment driver" to handle the fault. The segment driver calls into file system code. The file system code calls into the device driver to bring in the page. When the page-in is complete, the segment driver calls the HAT layer to update page table entries (or their equivalent). On Linux, when a page fault occurs, the kernel calls the code to handle the fault. You are immediately into platform-specific code. This means the fault handling code can be quicker in Linux, but the Linux code may not be as easily extensible or ported.

Kernel visibility and debugging tools are critical to get a correct understanding of system behavior. Yes, you can read the source code, but I maintain that you can easily misread the code. Having tools available to test your hypothesis about how the code works is invaluable. In this respect, I see Solaris with kmdb, mdb, and DTrace as a clear winner. I have been "reverse engineering" Solaris for years. I find that I can usually answer a question by using the tools faster than I can answer the same question by reading source code. With Linux, I don't have as much choice for this. FreeBSD allows use of gdb on kernel crash dumps. gdb can set breakpoints, single step, and examine and modify data and code. On Linux, this is also possible once you download and install the tools.

Max Bruning currently teaches and consults on Solaris internals, device drivers, kernel (as well as application) crash analysis and debugging, networking internals, and specialized topics. Contact him at max at bruningsystems dot com or http://mbruning.blogspot.com/.

Limitations of File System for Solaris OE

Description

This document will describe the limitations of the filesystem for the Solaris [TM] Operating Environment.

The effect of OpenBoot[TM] PROM (OBP) revisions on filesystem size limitations is also discussed.

Definitions:

1. 1 Mbyte (megabyte) 2^20 bytes (1,048,576 bytes)

2. 1 Gbyte (gigabyte) 2^30 bytes (1,073,741,824 bytes)

3. 1 Tbyte (terabyte) 2^40 bytes (1,099,511,627,776 bytes)

In general if this document refers to a limitation of (for instance) 2 Gbyte, ¨2 Gbyte  1 byte〃 (1,073,741,823 bytes) is meant.

To improve readability, this has been abbreviated to ¨2 Gbyte〃.

The symbol ¨~〃 is used to denote ¨approximately〃

Maximum size of a single file and a filesystem.

This also applies to Solaris x86, but there may be some issues with disk drives larger than 30 Gbyte. This is due to hardware limitations with some PC motherboard/disk configurations.

OS Release	Single file	File system
Solaris 2.5.1	2 Gbyte	1 Tbyte
Solaris 2.6 - 9 12/02 (U3)	~1012 Gbyte	1 Tbyte
Solaris 9 08/03 (U4) - Solaris 10 FCS	~1023 Gbyte	16 Tbyte

1. A single file in Solaris 2.6 through Solaris 9 (U3) is limited to approximately 1012 Gbyte because the file must fit inside the 1 Tbyte filesystem. The filesystem is nominally 1 Tbyte, but in fact, due to the overhead in such a large filesystem, the largest single file ends up being about 1012 Gbyte. (part of this is a bug, but even if bug-free, a single file can't be 1 Tbyte due to the filesystem overhead). The overhead in the filesystem includes amongst other things, items such as superblock backups and inode tables. The example given here is using 1024 Kbyte (1 Mbyte) for the number of bytes per inode (nbpi) within the UFS filesystem. If nbpi is set to a lower value, more filesystem space will be allocated to inode tables and less will be available to store data. With nbpi=8 Kbyte, the maximum single file size would be smaller than 1012 Gbyte. The file in this example cannot use any of the minfree area set up on the filesystem (which is setup for ¨root only〃 use). On a 1 Tbyte filesystem minfree is set to 1% which translates to about 10 Gbyte.

2. A safe assumption here would be that the limit on the size of a single file is the size of the filesystem,minus 1% to 2% overhead. In Solaris 2.6, the swap and tmpfs filesystems are still limited to 2 Gbyte. This is not the total amount of swap, it is a limit per swap slice or per swap file. A swap slice or file may be defined as larger than 2 Gbyte, but any space above 2 Gbyte in that slice or file will not be accessible and the size of the slice or file will be reported by the swap command as 2 Gbyte. There can be multiple swap slices or files totaling more than 2 Gbyte. Any later release of Solaris running 32 bit kernel has the same limitation. Later releases of Solaris running a 64 bit kernel do not have this limitation. See the ¨USAGE〃 paragraph in the Solaris 8 swap(1M) manual page for the new limits.

3. Solaris 9 Update 4 introduced multiterabyte UFS. The maximum individual file size is still the same as before (~1 Tbyte ), as increasing it would require radical on-disk format changes. The total filesystem size can now be up to 16 Tbyte. The -T option is specified to the newfs command to create such a filesystem. See the newfs(1M) manpage for additional information. There is also a limit of 1 million files per Tbyte, for instance. A 4 Tbyte UFS filesystem would have a 4 million files limit). This is done to keep fsck times reasonable (even when logging is enabled).

4. Multiterabyte UFS functionality can also be added to earlier releases of Solaris 9 by installing the UFS patch 113454-09 or later. See the Special Install Instructions in the patch README for a list of additional patches required to get the full Multiterabyte functionality.

5. The maximum single file size in a multiterabyte filesystem, which is greater than 1 Tbyte is 1 Tbyte minus 500 Mbyte or 1023.5 Gbyte As a rule of thumb this should be taken as 1023 Gbyte.

6. A multiterabyte UFS filesystem is not bootable (This means the root filesystem cannot be a multiterabyte filesystem).

7. A multiterabyte UFS filesystem is not mountable in any version of 32bit Solaris kernel.

Limitations in combination with the OBP.

Excluding the multiterabyte root filesystem limitation detailed above, the root filesystem has limits in Solaris[TM] 2.x that are not imposed on any other filesystems. This limiting factor is a combination of two things, the OS release and the OBP (Open Boot Prom) level.

To establish the OS release, examine the /etc/release file To establish the OBP level, use the command prtconf -V (capital ¨V〃).

Here is a list of the various possible configurations.

OBP level	OS Release	Max root filesystem size
OBP 3.1beta1 or newer	Solaris 2.5.1 Kernel Patch 103640-08 or newer	No limit
OBP 3.1beta1 or newer	Solaris 2.6	No limit
OBP 3.0 or earlier	Solaris 2.5.1 Kernel Patch 103640-07 or older.	2Gbyte
OBP 3.0 or earlier	Solaris 2.6 on an Ultra (sun4u)	4Gbyte

Solaris 10 and ZFS

ZFS is now fully integrated in Solaris 10 U2 and above.

Limitations of a single file and filesystem in ZFS:

http://www.opensolaris.org/os/community/zfs/faq/

What limits does ZFS have?

- The limitations of ZFS are designed to be so large that they will never be encountered in any practical operation.

- ZFS can store 16 Exabytes in each storage pool, file system, file, or file attribute.

- ZFS can store billions of names: files or directories in a directory, file systems in a file system, or snapshots of a file system.

- ZFS can store trillions of items: files in a file system, file systems, volumes, or snapshots in a pool.

Mpxio and LSI storage

Two Sun cluster boxes(Sun v440 and Sun Fire 490) running solaris 10 connect to LSI storage box using fibre channel switches. One LUN on the LSI box was created to supply the two server box storing 1.9TB data. When mpxio is disabed on both servers, only one server can recognize the LUN correctly while the other regards the LUN as unformatted disk. When trying to format or label the LUN using format utility, illegal ASC write errors ocurred.

When mpxio is enabled on both servers through the command:
#stmsboot -e

problem described above solved. Both server can access the LUN with no errors occuring. I don't know why. Anybody knows?

Why ZFS for home

--from uadmin.blogspot.com

I’m getting annoyed at people that keep saying ZFS is okay for servers but I don’t need it for home. ZFS scales from one drive to an infinite number of drives and has benefits for all of them.

Let’s take a look at the average home computer a single drive holding a mix of files, up to 300GB drives are common. That is a lot of data to lose and its getting easier lose data these days. Further more new hard drives aren’t getting any more reliable with time. Of course you can lose things on new hard drives by just misplacing them in one of the 1000’s of directories you can use in an attempt to organize your files.

What do other operating systems and file systems provide to fight this situation? In Linux you can use raid, redundant array of inexpensive drives, then if a hard drive fails your data is safe. Okay the fun part begins when you try to enable raid, the obvious choices are raid 1 (mirroring your data) or raid 5 (that uses part of your drives as parity protecting your data uses less space but requires a minimum of 3 drives to work). I won’t bore you with the technical details I will just show a small sample of the commands to create a raid 1, a mirror image of one drive onto a second drive.

These instructions were taken for http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO-5.html#ss5.6

You have two devices of approximately same size, and you want the two to be mirrors of each other. Eventually you have more devices, which you want to keep as stand-by spare-disks, that will automatically become a part of the mirror if one of the active devices break.

Set up the /etc/raidtab file like this:

raiddev /dev/md0
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
persistent-superblock 1
device /dev/sdb6
raid-disk 0
device /dev/sdc5
raid-disk 1

If you have spare disks, you can add them to the end of the device specification like

device /dev/sdd5
spare-disk 0

Remember to set the nr-spare-disks entry correspondingly.

Ok, now we're all set to start initializing the RAID. The mirror must be constructed, eg. the contents (however unimportant now, since the device is still not formatted) of the two devices must be synchronized.

Issue the

mkraid /dev/md0

command to begin the mirror initialization.

Check out the /proc/mdstat file. It should tell you that the /dev/md0 device has been started, that the mirror is being reconstructed, and an ETA of the completion of the reconstruction.

Reconstruction is done using idle I/O bandwidth. So, your system should still be fairly responsive, although your disk LEDs should be glowing nicely.

The reconstruction process is transparent, so you can actually use the device even though the mirror is currently under reconstruction.

Try formatting the device, while the reconstruction is running. It will work. Also you can mount it and use it while reconstruction is running. Of Course, if the wrong disk breaks while the reconstruction is running, you're out of luck.

Looks like fun right? Before ZFS the situation wasn't much better in Solaris. A typical home user will see this and say I will do this next week. Then next week never comes. Of course doing raid5 only gets more complex in Linux at least, for ZFS its just a slight change to the commands used to create a mirror. In ZFS we execute two or three commands and we are done.

# zpool create data mirror c0t0d0 c0t1d0
# zfs create data/filesystem

Done. The only complex part is getting the last two entries and you can find those by running the Solaris format command. The red is added to help readability

#format < /dev/null Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0
1. c1t2d0
/sbus@1f,0/SUNW,fas@1,8800000/sd@2,0
#

That takes care of drive failure; another problem is accidental deletion, accidentally installing a broken application or any change you would like to undo. Linux’s answer to this is backups, either on optical media, tape or perhaps another harddrive. This is expensive or time consuming, choose one. So the typical home user will most likely put this off to till another day and won’t have a backup for there data.


ZFS has snapshots, they are easy and painless and have a very low cost in resources to create. Snapshots are basically a picture of your data; these are taken in real time and are nearly instant in ZFS, to get these in any other OS you need to buy expensive raid hardware or an expensive software package something that no home user will want to buy.

For example I want to protect my mp3 collection so I put it on file system all its own.


# du -sh /mp3
17G /mp3
#

And then I took a snapshot of it for protection.

# time zfs snapshot data/mp3@may-1-2006
real 0m0.317s
user 0m0.017s
sys 0m0.030s
#

Not bad, 1/3 of a second to protect 17 Gigabyte of data. That can easily be restored should I make a mistake and delete or corrupt a file or all of them.

And here is a little script I created to take snapshots of all my zfs file systems and puts a date stamp on each one. Each snapshot takes very little space, so you can make as many as you need to be safe.

#!/bin/sh
date=`date +%b-%d-%Y`

for i in `/usr/sbin/zfs list -H -t filesystem -o name` ;
do /usr/sbin/zfs snapshot $i@$date ;
done

A few minutes in crontab or your desktop graphical crontab creator and you can have this script execute daily with no user intervention. Below is a sample line to add to your crontab that that takes snapshots at 3:15 am

15 3 * * * /export/home/username/bin/snapshot_all

To see your snapshots, is easy you just look in .zfs/snapshot that is in each zfs filesystem. You can even see individual files that make up a snapshot by changing directories further into the snapshot. This even works if the file system is shared via NFS.

Now let’s take a look at how to recover from mistakes using snapshots. First lets create filesystem, and populate it with a few files.

#zfs create data/test1
#cd /data/test1
#mkfile 100m file1 file2 file3 file4 file5
#ls
file1 file2 file3 file4 file5
#

We now have 5 files, each 100 megabytes, lets take a snapshot, and then delete a couple files.

# zfs snapshot data/test1@backup
# rm file2 file3
# ls
file1 file4 file5

The files are gone. Oops a day later or a month later I realize I need those files.

# cd ..
#zfs rollback data/test1@backup

So all we do is rollback the using a saved snapshot and the files are back.

# ls
file1 file2 file3 file4 file5
#

ZFS makes it easy to create lots of filesystems, in Linux you are limited to 16 file systems per drive (yes I know you can use the Linux volume manager but of course that adds even more complexity to the raid setup outlined above, as drives get bigger you end with hundreds or even thousands of files and directories per drive making it easy to lose files in the levels of directories. With ZFS there is no real limit to the number of filesystems and they all share the storage pool, they are quick and easy to create.

#time zfs create data/word_processor_files
real 0m0.571s
user 0m0.019s
sys 0m0.040s
#

Little over half a second to create a filesystem and you can create as many as you like.

The next problem the home user may face is running out of space. Typically the user heads down to the local electronic or computer shop and gets another hard drive or two if they want to be safe and use raid, so they get to head back to the raid setup guide, of course. Depending on the filesystem you may be able to grow the filesystem with more cryptic commands turning your filesystems into a raid 1+0, but its pretty complicated, so most people resort to keeping them simple and moving files back and forth between the filesystems to get the space they need.

With ZFS it is only one command to add the drive(s) to the pool of storage.

#zpool add data mirror drive3 drive4

Afterward all your filesystems have access to the additional space. If money is a little tight, you can turn on compression on any filesystem you like with a simple command then all files that are added to the filesystem are compressed possibly using less space. Note this usually doesn’t slow down IO at all, on some systems and workloads it actually speeds up data access.

# zfs set compression=on data/filesystem

ZFS is so simple you can talk your grandmother through the process of creating filesystems or restoring data. This is just a small sample of what ZFS can do, but it’s all just as simple as what I have shown you in this document. Even if you are more advanced, you can still benefit from ZFS’ ease of use. No more hitting the web to study how-to's to setup raid or LVM. Even if you can't afford two drives in your home box ZFS will be perfectly happy with one drive, though you do lose hardware redundancy, snapshots are still there to take care of software or user introduced filesystem problems.