upgrade to OpenIndiana

Reading the requirements to upgrade from opensolaris to OpenIndiana was done following suggestions at http://wiki.openindiana.org:8080/display/oi/Installing+or+Upgrading

With one difference; It won't let me update even though no obsolete or failed packages were found.
The solution was to start from the commandline using the -f option: 
pfexec pkg image-update -fv

DOWNLOAD                    PKGS       FILES    XFER (MB)
Completed                  846/846 39788/39788  583.4/583.4

Removal Phase                            17782/17782
Install Phase                            26718/26718
Update Phase                             46466/46466
PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                            846/846
Indexing Packages                            846/846
Optimizing Index...
PHASE                                          ITEMS
Indexing Packages                            837/837

A clone of opensolaris-6 exists and has been updated and activated.
On the next boot the Boot Environment opensolaris-7 will be mounted on '/'.
Reboot when ready to switch to this updated BE.

sawtooth like data transmission

To improve performance it was mentioned to disable autotuning in Windows
(source: http://www.curuit.com/disable-tcp-auto-tuning-to-solve-slow-network-%28vista-%10-windows-7-%10-windows-2008%29-20070621145/)

The answer was to turn off tcp autotuning by running a cmd prompt as administrator & entering netsh tcp interface set global autotuning=disabled.

zdb command

Typing “zdb” with no arguments produces a list of pools and the storage devices they are using. Its a quick way to find out the serial number of a drive in a zpool

ZFS Raid10

Creating a Mirrored Storage Pool

To create a mirrored pool, use the mirror keyword, followed by any number
of storage devices that will comprise the mirror. Multiple mirrors can be
specified by repeating the mirror keyword on the command line. The
following command creates a pool with two, two-way mirrors:

# zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0

The second mirror keyword indicates that a new top-level virtual device is
being specified. Data is dynamically striped across both mirrors, with data
being replicated between each disk appropriately.

ZFS snapshot

machine0$ zfs snapshot mypool/myfs@now
machine0$ zfs send mypool/myfs@now > myfile
machine1$ zfs receive anotherpool/anotherfs@anothersnap < myfile

machine0$ zfs send mypool/myfs@now | ssh machine1 zfs receive anotherpool/anotherfs@anothersnap

Remove snapshot
zfs list -H -t snapshot
zfs destroy extra/backup@now

Windows recognised iscsi drive as RAW (no longer as NTFS)

After connecting my iscsi F-drive from one pc to another it was no longer recognised being formatted as NTFS so Windows asked if I wanted to format the F-drive (strange because I did not touch the data ...)
I went to the command prompt and did:
D:\> CONVERT F: /fs:ntfs
Where F: is a name of the drive you want to convert. It reported back that the drive was already NTFS.
I than ran: chkdsk F: /F which fixed it. After that it was immediately accessible as a normal Drive including all the data.

realtime monitoring your zfs pool

I use the following command to see real time transaction per 5 sec on my pool extra:

# zpool iostat -v extra 5

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
extra       8.87G  28.4G    134      0  9.47M      0
  c7d0      8.87G  28.4G    134      0  9.47M      0
----------  -----  -----  -----  -----  -----  -----

(note: this was during a file copy over 100Mb/s LAN, reaching the max throughput of such a LAN)

To check local disk performance on my zpool I used:
# time (mkfile 2g /extra/foo)
real    2m7.349s
user    0m0.017s
sys     0m1.247s
(this creates a random file foo of 2GB  in my zpool extra and shows the time to complete)

package manager failed with update from snv_129 to snv_130

I had my pkg publishers as followed:
dev                   http://pkg.opensolaris.org/dev/   
opensolaris.org  http://pkg.opensolaris.org/release/

But this just results in pkg refresh failing:
-To correct the repository origin, execute the following command as a privileged user:

pkg set-publisher -O opensolaris-dev

To add a new publisher using this publisher's repository origin, execute the following command as a privileged user:

pkg set-publisher -O http://pkg.opensolaris.org/dev/

After the new publisher has been added, this one should be removed by executing the following command as a privileged user:

pkg unset-publisher opensolaris-dev

I tried to follow the suggestions but to no avail.
Then I read it was mentioned to have only 1 publisher opensolaris.org as Preferred and delete dev resulting in the following pkg publisher:
opensolaris.org   http://pkg.opensolaris.org/dev/ 

But still the package manager failed to upgrade to 130 as did pkg refresh
In the forum (http://www.pubbs.net/opensolaris/200912/32545/) it was mentioned to use "pkg image-update -f"  due to errors related to release snv-128 and 129 (Option -f bypasses SUNWipkg)

Update succeeded to 130.

3com 3C200-T Gigabit network card (tcge)

I was not happy with the throughput using the onboard NIC so I installed the 3com Gigabit NIC which supports jumbo frames. NIC is not recognized by OS as standard so you have to get the driver from Asustek:
http://dlsvr01.asus.com/pub/ASUS/lan/3com/3c940/046_Solaris.zip

Just follow the instructions provided and your NIC will be recognized and fully functional.

iscsi performance

Besides the settings mentioned in the iSCSI uguide from Microsoft: Setting TcpAckFrequency =1

Someone mentioned lack of performance was solved by disabling the "receive Windows auto tuning". See: http://support.microsoft.com/kb/947239

Server refused to allocate pty snv_128 and 129

after upgrade to snv_129 I was not able to login using ssh.

this is a known bug which can be solved by editing /etc/minor_perm
add the following:
"clone:ptmx 0666 root sys" to /etc/minor_perm

After reboot you can login again with ssh
(source: http://defect.opensolaris.org/bz/show_bug.cgi?id=12380)

Nagle Algorithm

Please take a look at the node where the iSCSI Initiator is running and determine if the following parameter is being set.

[ /kernel/drv/iscsi.conf ] add the following:

#

# Disable Nagle within the iSCSI Initiator

#

tcp-nodelay=1;

The tcp-nodelay setting, alters the iSCSI Initiators socket options by setting the well-known option TCP_NODELAY, or more specifically turning off the "Nagle Algorithm". For performance reasons, this setting is hard-coded in the iSCSI Target and also Availability Suite's SNDR, both being TCP/IP based data movers of disk blocks.

For both iSCSI and SNDR, the Nagle algorithm interacts badly when using TCP delayed acknowledgments, of which both iSCSI and SNDR relies upon. These performance issues have been noted by many others, including the author John Nagle. See link at bottom of this page: http://en.wikipedia.org/wiki/Nagle's_algorithm

For Windows check the Microsoft ISCSI User Guide to change TCPIP parameter
in registry: "TcpAckFrequency=1"
(http://download.microsoft.com/download/A/E/9/AE91DEA1-66D9-417C-ADE4-92D824B871AF/uGuide.doc)

iscsi performance

(Source: http://opensolaris.org/jive/thread.jspa?threadID=78965)
Sidenote: Today we made eight network/iSCSI related tweaks that, in aggregate, have resulted in dramatic performance improvements (some I just hadn't gotten around to yet, others suggested by Sun's Mertol Ozyoney)...

- disabling the Nagle algorithm on the head node

- setting each iSCSI target block size to match the ZFS record size of 128K

- disabling "thin provisioning" on the iSCSI targets

- enabling jumbo frames everywhere (each switch and NIC)
- raising ddi_msix_alloc_limit to 8
- raising ip_soft_rings_cnt to 16
- raising tcp_deferred_acks_max to 16
- raising tcp_local_dacks_max to 16

Rerunning the same tests, we now see...

[1GB file size, 1KB record size]
Command: iozone -i o -i 1 -i 2 -r 1k -s 1g -f /data-das/perftest/1gbtest

Write: 143373
Rewrite: 183170
Read: 433205
Reread: 435503
Random Read: 90118
Random Write: 19488

[8GB file size, 512KB record size]

Command: iozone -i 0 -i 1 -i 2 -r 512k -s 8g -f /volumes/data-iscsi/perftest/8gbtest
Write:  463260
Rewrite:  449280
Read:  1092291
Reread:  881044
Random Read:  442565
Random Write:  565565

[64GB file size, 1MB record size]
Command: iozone -i o -i 1 -i 2 -r 1m -s 64g -f /data-das/perftest/64gbtest

Write: 357199
Rewrite: 342788
Read: 609553
Reread: 645618
Random Read: 218874
Random Write: 339624

Thanks so much to everyone for all their great contributions!
-Gray

Using POPtray to check your Axigen Mail

To check your Axigen mail using POPtray you need to tick the following
box in Axigen Console: Services: POP3
On UNSECURED connections allow authentication with:
Normal login (unsecured)
( Otherwise you get "Invalid Authentication Type" message  within POPtray when you check your account)

script for fmadm alerting

source: http://prefetch.net/code/fmadmnotifier

From above site I found the following script
#!/bin/bash
#
# Program: E-mail fault manager errors
#
# Author: Matty < matty91 at gmail dot com >
#
# Current Version: 1.1
#
# Revision History:
#
# Version 1.1
# Avoid the use of temporary files -- Michael Shon
#
# Version 1.0
# Initial Release
#
# Last Updated: 08-18-2006
#
# Purpose:
# Fmadm.sh queries the fault manager to see if errors have been
# generated. If an error is detected, the script will email the
# admininstrator defined in the ADMIN vairable with the error
# details.
#
# License:
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# Installation:
# Copy the shell script to a suitable location
#
# Usage:
# To check for events once per hour, add a cron job similar to the following:
#
# $ crontab -l | grep fmadmnotifier.sh
# 0 * * * * /etc/scripts/fmadmnotifier.sh
#

PATH=/usr/bin:/sbin:/usr/sbin:/usr/sfw/bin

# Who to E-mail with new updates
ADMIN="root"

# Location of binaries
AWK=$(which awk)
FMADM=$(which fmadm)
HOSTNAME=$(which hostname)
MAIL=$(which mailx)
MKTEMP=$(which mktemp)

# Check to make sure the mail binary exists
if [ ! -f ${MAIL} ]
then
echo "Cannot find ${MAIL}"
exit 1
fi

# Check to make sure the fmadm utility exists
if [ ! -f ${FMADM} ]
then
echo "Cannot find ${FMADM}"
exit 1
fi

# Verify that mktemp exists
if [ ! -f ${MKTEMP} ]
then
echo "Cannot find ${MKTEMP}"
exit 1
fi

# Run fmadm faulty to check for hardware errors
FMADMOUTPUT=$(${FMADM} faulty | ${AWK} '$0 !~ /STATE/ && $0 !~ /^----/ { print $0 }')

if [ -n "${FMADMOUTPUT}" ]
then
(
echo "The fault manager detected a problem with the system hardware."
echo "The fmadm and fmdump utilities can be run to retrieve additional"
echo "details on the faults and recommended next course of action. "

echo ""
echo "fmadm faulty output:"
echo ""

${FMADM} faulty
echo ""
) | ${MAIL} -s "Hardware fault on $($HOSTNAME)" ${ADMIN}
fi

And some fmadm details:
The fmadm utilities "config" option can be used to view the list of diagnosis engines and agents that are active on a system:
i $ fmadm config
MODULE cpumem-retire disk-transport eft fmd-self-diagnosis io-retire snmp-trapgen sysevent-transport syslog-msgs zfs-diagnosis zfs-retire VERSION 1.1 1.0 1.16 1.0 2.0 1.0 1.0 1.0 1.0 1.0 STATUS active active active active active active active active active active DESCRIPTION CPU/Memory Retire Agent Disk Transport Agent eft diagnosis engine Fault Manager Self-Diagnosis I/O Retire Agent SNMP Trap Generation Agent SysEvent Transport Agent Syslog Messaging Agent ZFS Diagnosis Engine ZFS Retire Agent Fault manager logs
· The fault manager maintains two log files: ­ The error log contains a list of errors events that have been sent to the fault manager daemon ­ The fault log contains a list of problems that have been diagnosed and repaired · The fault log can be viewed by running fmdump:
$ fmdump · The error log can be viewed with fmdump's "-e" option:
$ fmdump -e · Fmdump also has a "-u" option to limit the output to a specific UUID, a "-T" option to display events that occurred during a specific timeframe, and "-v" and "-V" options to display verbose output Viewing faulty components

zpool monitoring

The second script checks the current state of the zpools, looking for degraded arrays (caused by failed drives), unavailable spares and unrecovered errors. Because it keeps a state file in /etc/zfs, it would need to be run as root. I run this hourly. It should be possible to update this script to also check for ZFS checksum errors, but I haven't taken the time to do it. The reminder code hasn't been tested, as I haven't had a failure since the code was put in place.

#! /bin/sh

STATEFILE="/etc/zfs/chk.state"
ALARMUSER="root@localhost"

zpool status 2>&1 | \
egrep -i '(degraded|unavail|unrecover)' > /dev/null

STATE=$?

if [ -f $STATEFILE ]
then
LASTSTATE=`cat $STATEFILE`
else
LASTSTATE=1
echo $STATE > $STATEFILE
fi

#
# Error is currently set.
#
if [ $STATE = 0 ]
then

#
# Error wasn't set previously. Send out the error message.
#
if [ $LASTSTATE = 1 ]
then
HOSTNAME=`uname -n`
zpool status -x | \
mailx -s "ZFS.error.on.$HOSTNAME" $ALARMUSER
echo $STATE > $STATEFILE
exit
fi

#
# Send out a reminder every other day.
#
FOUND=`find $STATEFILE -mtime -2`
if [ -z $FOUND ]
then
exit
fi
HOSTNAME=`uname -n`
zpool status -x | \
mailx -s "ZFS.error.reminder.on.$HOSTNAME" $ALARMUSER
echo $STATE > $STATEFILE
exit
fi

#
# Error was set, but is no longer. Send out the fixed message.
#
if [ $STATE = 1 -a $LASTSTATE = 0 ]
then
HOSTNAME=`uname -n`
zpool status -x | \
mailx -s "ZFS.error.fixed.on.$HOSTNAME" $ALARMUSER
echo $STATE > $STATEFILE
exit
fi


EDIT: Updated above script to look for unrecovered errors, thanks to information in this post by nhamilto40. To reset the error counts, the "zpool clear pool" command can be used.

I scanned this thread, and see no scripts. Perhaps this will be more useful than I thought.

ZFS tutorial using files instead of disks

Using Files
To use files on an existing filesystem, create four 128 MB files, eg.:

# mkfile 128m /home/ocean/disk1
# mkfile 128m /home/ocean/disk2
# mkfile 128m /home/ocean/disk3
# mkfile 128m /home/ocean/disk4

# ls -lh /home/ocean
total 1049152
-rw------T 1 root root 128M Mar 7 19:48 disk1
-rw------T 1 root root 128M Mar 7 19:48 disk2
-rw------T 1 root root 128M Mar 7 19:48 disk3
-rw------T 1 root root 128M Mar 7 19:48 disk4

This is easy for testing you don't need real disks or partitions
(source: http://flux.org.uk/howto/solaris/zfs_tutorial_01)

Remote powerdown from Windows with plink.exe

Because I am using a private network without connection to the outside world,
I am not concerned about security.
Usink plink.exe as part of the puTTY package I managed a remote shutdown:
Create a shortcut on windows with the following command:
(I use the user admin for logon)

plink.exe -ssh admin@hostname -pw password -m shutdown

The tricky bit is the remote command with the option -m,
(you have to use the full path because no profile is loaded)
so my "shutdown" saved on the windows box contains this line:

/usr/bin/pfexec /usr/sbin/init 5

very similar to the 'pfexec init 5' you use when logged on in Solaris

Autologon

Go to /etc/X11/gdm/custom.conf and manually modify the required fields in the [daemon] section and add the following;

[daemon]
AutomaticLoginEnable=true
AutomaticLogin=<username>
Note: You cannot auto-logon as root.

Bare in mind that I’m not concerned about security at this point as the box is not exposed to the open world, and is somewhat protected by the fact it lives on a “private” LAN. (i.e. my home lan)

If you want to run something automatically on logon you can put the commands into the /etc/profile file. Since I’ve been looking for a way to autostart virtualbox machines this will be the place i do it. (mainly because it is one of the easiest way to do it)
(source: http://sigtar.com/2009/01/29/open-solaris-autologin/)

Wireless using WUSB54GC (usb13b1,20)

At first I had a WUSB54GC based on Ralink RT3070 chipset recognized as usb1737,77 but so far no driver exists so I swapped it for the well-known older version being usb13b1,20 which uses the rum driver

It was now quite easy to get it up and running

If no wired link is available, a scan for wireless LANs is done, and the resulting list offered via a GUI popup window prompts the console user to select a preference. If a successful connection is made, the WLAN in question is stored in the plain text file /etc/nwam/known_wifi_nets and the daemon may connect to any WLAN in that list without prompting the user again. If a user wishes to add other preferences or revoke existing ones, he can do so by bringing up the NWAM Manager menu with right-click on the icon, and then selecting "Manage Favorite Wireless Networks...". A user can also edit the known_wifi_nets file directly. This interface is volatile and might change in a future release

admin@opensolaris:/etc/nwam# more known_wifi_nets
denver 0:18:39:ad:ce:36

Note: My network was recognized after I enabled broadcast again on the wireless modem and choosing the option within the Gui of NWAM to put wireless as preferred connection.

/etc/nwam/llp contains:
admin@opensolaris:/etc/nwam# more llp
bge0 static 10.0.0.4/24
rum0 static 10.0.0.10/24

In order to have wireless standard as preferred put rum0 first in llp after bge0

Helpful commands:
ifconfig -a
dladm show-wifi
dladm scan-wifi
dladm show-secobj
(Note: I removed all the profiles with dladm delete-secobj)
svcadm restart nwam