OpenSolaris 2008.11

OpenSolaris 2008.11 was just released for x86, it is a huge improvement over the previous release. I'm not really a desktop user but I think those features seem to be at least on pair with Ubuntu and other similar environments. It has automatic network management, Songbird, Compiz,Transmission, OpenOffice and a graphical package/update manager among other features.

On top of that it has a unique feature for the end users called "Time slider" which uses continuous ZFS snapshots to browse changes or restore lost data. It also has Netbeans 6.5, SunStudio, Ruby, Python, PHP and Java 1.6 available in the repositories making it an attractive choice for developers who now can utilize DTrace in a modern desktop environment.

This all is built on top of snv101b with ZFS as root filesystem with additional storage features besides ZFS (COMSTAR, iSCSI, CIFS) and virtualization (xVM, Solaris zones, VirtualBox).

The packaging system(IPS) has been long overdue, with OpenSolaris we now have network based repositories from which software and updates can be installed. You don't need to download all those packages yourself and resolve dependencies.

Take a peak here before downloading it here.

Perhaps it sounds like I'm working for marketing at Sun, but it's all that good ;)

SYSV Packages fundamentals

If you are new to Solaris or never bothered looking at how the SYSV packages are built up, i found a good article at bigadmin that could be worth a read:

Introduction to Package Components for the Solaris OS

I found this article while researching why a file was not updated when patching and upgrading a system. It turned out that the file is flagged as a editable file, which means it will not be overwritten by the package system. Such files should be handled by a class action script that are responsible for making the required changes to the file.

In this particular case we had problems with OPL servers (M5000,M8000) that did not have a working console login. We had installed them with a flash archive created with a Solaris release predating OPL support, so the /etc/ou.ap used by autopush was not updated with information required for OPL hardware.

Follow the white rabbit

Yesterday i encountered a strange problem which was very entertaining to debug and dtrace came to the rescue as as always. I had done a fresh install of Solaris Nevada build 101 on a AMD64 machine at home. The installation went fine and i started to configure on the host, but after a while i begun to get error messages like: "couldn't set locale correctly". I checked the SYSV package containing my locale, en_US.8859-1:

# pkgchk SUNWlang-en-extra
ERROR: /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
pathname does not exist

The installation logs showed that the package had been installed correctly. I tried to reinstall the package which temporarily fixed the problem but after 10 minutes or so it the file was missing again. I repeated this twice to see if there was any time pattern, but it gave me no clue to why the files was removed. I decided to use both BSM auditing and dtrace to see what might be causing the removal of the file. I waited and waited, but the file stayed there so i started to use the system as ordinary, that somehow triggered the removal of the file!?

A dtrace script created by Chris Gerhard showed me what caused the removal of the file:

# ./delete.d
dtrace: script './trace.d' matched 2 probes
CPU ID FUNCTION:NAME
0 75683 unlink:return man prctl
UID 0 PPID 11244 sh

Audit also logged the unlink of the file, following the trail it could have shoved me that man was responsible:

header,242,2,unlink(2),,ollespappa,2008-11-03 02:52:01.096 +01:00
path,/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
attribute,100555,root,bin,65538,320199,18446754073709451615
subject,arne,root,root,root,root,15984,2799982011,1330 5632 xx.xx.xx.xx
return,success,0

So, man(1) unlinked the file, that did not make any sense at all. A dtrace one liner gave me more information:

# dtrace -n 'pid$target:libc:unlink:entry { printf("unlink(%s)\n",copyinstr(arg0)); ustack(); }' -c 'man ls'
CPU ID FUNCTION:NAME
1 76165 unlink:entry unlink(/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3)
libc.so.1`unlink
man`format+0xd54
man`searchdir+0x14d
man`mandir+0x194
man`manual+0x1c1
man`main+0x5d4
man`_start+0x7d

The unlink was called from the format function of man, Looking at the source I identified which line was responsible for the unlink:
2754 (void) unlink(tmpdir);

After some more investigation I realized that the argument string to the unlink command might not get initialized, it then points out in space. For this system it pointed at a string which represented the path to a file which could be unlinked. This occurs only if the manual page is in another format than SGML since the variables are initialized inside an SGML-related if-statement :

2612 if (sgml_flag == 1) {
2613 if (check_flag == 0) {
2614 strcpy(tmpdir, "/tmp/sman_XXXXXX");

This bug seems to have been in Solaris for ages, but the uninitialized string might never have come to point to anything useful to unlink, it could perhaps be the switch to SunStudio 12 in snv100 that triggered this behavior.

This is now CR 6767074.

Update 6 released and attach/detach (updated)

After a long wait update 6 of Solaris was released a few days ago and is available for download. All new features is listed here, as i have mentioned earlier the bulk of the changes are in the major ZFS update, including ZFS boot. This is a great update to a already good OS, but i was extremely disappointed to see that they have not fixed the attach/dettach bug related to patch history yet, it have been waiting for this fix for well over a year. In this update bugid 6637869 (zone attach doesn't handle obsolete patches correctly) is fixed but a new one is listed in known issues, bugid 6710545 (fix for 6550154 doesn't always ignore older patches).

What this means is that it is still not enough to be on the exact same patch level on both machines when moving zones in a supported way, all patches have to be applied in the same way, have the same history. Otherwise it might not work. I don't know what update on attach can do for this, but that is not the point, attach is still broken.

Update, Gerald kindly informed me that this is indeed fixed in Solaris 10 10/08 (137137-09) But it is stated as an issue in the release notes, which i based this post on.

Too tight diamond shoes

I've encountered a terrible problem today, one of our hosts had more resources that it should, fortunate it was a T5120 so it was easy to disable hardware from the SP:

For (strand=P63;strand >= P32, strand ):
Remove half of all the strands 32 ( 4 cores ), P64 down to P32:
-> SYS /SYS/MB/CMP0/P63 component_state=Disabled

Remove half of the memory, disable memory branch BR3 and BR2:
-> set /SYS/MB/CMP0/BR3/CH0/D0 component_state=Disabled
-> set /SYS/MB/CMP0/BR3/CH0/D1 component_state=Disabled

Reset the system to apply the changes:
-> reset /SYS

Sun is readying Solaris 10/08

The release of Solaris 10/08 is getting close, the sun.com website has some updates to what new features we can expect in this release. I have mentioned some of them earlier, but this is what Sun want to highlight:

xVM guest support
Major ZFS update including ZFS boot
New HW support including Intel Xeon 7400
LDOM enhancements (MPxIO, dynamic I/O reconfiguration)
X64 Fault management enhancements
NVIDIA SATA enhancements

I sure hopes bugid 6637869 is finally fixed in this release so migrating zones will be practical in a supported way. (6637869 zone attach doesn't handle obsolete patches correctly)

This release will be a great leap forward for patching and upgrading when used together with ZFS snapshots and live upgrade. To always be able to roll back changes in a matter of minutes when upgrading or patching without adding extra disk and copying data for hours will be of tremendous value in an production environment. This update also address the need to be able to place Solaris Zones on a ZFS filesystem and still be able to upgrade and patch without restrictions.

I'll post an update when i get my hands on this new release mid October.

iPhone software update

Apple has released a update to the iPhone OS, 2.0.1, it seems to have fixed most of the problems i have had with the device. It have not crashed in two days now, and applications also work much better. The only problem so far is Safari which has gone down about once a day. Hopefully they also update iTunes soon, there are some very irritating problems with syncing and applications updates. iTunes wants to update all of my applications for no reason and there are several instances of the same applications which gets downloaded.

Too bad the sync process takes such a long time to complete with 2.0, it could probably have been at least a bit better if the had not abandoned firewire for the iPhone, it is much faster than USB 2.0 in data transfer.

But overall i am am happy, i now have a device i can depend on again, it's a nightmare to be on vacation with a flaky smartphone;)