No more public bug information

A tremendously useful site for Solaris/OpenSolaris/Solaris 11 operators, users and developers have been bugs.opensolaris.org. I don't know how many problems I have solved by using this site and I am sad to report that it has now been down for a well over a month. Now even I have given up hope that Oracle will restore the service, all information available is the message on the site: "This service is temporarily unavailable.". It no longer seems like a temporary outage and I would not be surprised if Oracle have cut this source of information too and is trying to do it quietly without any announcement. If that is not the case I love to be proven wrong.

This is another change that will make it harder for me to promote future Solaris releases and write about interesting technical details.

Solaris updates in 2012

In the Solaris online forum last week there where refereces to another scheduled Solaris 10 update, update 11 (s10u11), schedule for release the second half of 2012.

There where no other information regarding the release but it's good to see that Solaris 10 will continue to be updated a while after Solaris 11 have been released.

The scheduled also included the expected Solaris 10 8/11 release and the Solaris 11 release which probably will be available at Oracle World 2011. There was also an update scheduled for Solaris 11 the first half of 2012.

Update: You can look at the roadmap here.

Joyent illumos tree

Joyent which builds their cloud service on zones and ZFS just announced a new repository with patches against illumos that provides new features and bug fixes:

"Some highlights of the illumos-joyent tree include:
* ZFS I/O fair-share scheduling for zones
* the Joyent brand, which can be used as a template for other non-SysVR4 or IPS zone brands
* Reintroduction of sparse zone images
* Crossbow vnics on demand for zones & non-unique vnic naming (unique per zone, not per system)
* svcs enhancements ( svcs -Z/-z for interrogating zone services, -L for outputting log files directly (no more ls /var/svc/log | grep... ))
* vfsstat and iostat tweaks and ziostat, iostat(1M) for ZFS I/O
* more per-zone IO kstats
* the zonemon utility for zone kernel state troubleshooting
* DTrace enhancements such as llquantize"

I'll guess many of these changes will find it's way into illumos in a near feature. The whole mail from John Sonnenschein can be read here.

Joyent have always been an interesting company and have recruited very talented people after Sun settled. Both Bryan Cantrill and Brendan Gregg are now employed by Joyent so I would not be surprised if they will become one of the larges contributors to illumos.

illumos-joyent at github

Solaris Online forum

A rare opportunity is coming April 14, Oracle is sharing information on their Solaris strategy and upcoming Solaris 11 features.

The agenda taken from the registration site:

"9:00 a.m. - 9:45 a.m. PT
Oracle Solaris Strategy Overview
Bill Nesheim, VP Oracle Solaris Engineering

9:45 a.m. - 10:00 a.m. PT
An Industry Analyst's View of the Operating System Market
Gary Chen, IDC

10:00 a.m. - 10:45 a.m. PT
Manage Your Deployments With Image Packaging System and the Automated Installer
Bart Smaalders, Oracle Solaris Engineering
Dave Miner, Oracle Solaris Engineering
Glynn Foster, Oracle Solaris Product Management
Isaac Rozenfeld, Oracle Solaris Product Management

10:45 a.m. - 11:30 a.m. PT
Get More out of Your Oracle Solaris Environments With Virtualization
Dan Price, Oracle Solaris Engineering
Nicolas Droux, Oracle Solaris Engineering
Duncan Hardie, Oracle Solaris Product Management

11:30 a.m. - 12:15 p.m. PT
Learn How All New Features in Oracle Solaris 11 Raise The Bar For Operating Systems
Markus Flierl, Sr. Director Oracle Solaris Engineering
Liane Praza, Oracle Solaris Engineering
Joost Pronk, Oracle Solaris Product Management

Register and get more information here.

Sort memory capping out

I had a curios problem today, on a otherwise lightly loaded machine with plenty of memory left login as the Oracle user took tens of seconds to complete. Since it was the souring of a shell script that was stalling identify where the time was spent:

$ set -x
$ . /path/to/oraenv
...
+ nawk { print $2 }
+ sort
...

The whole delay was spent in sort(1) for no obvious reason. Truss shows the system call:
24.8167 sysconfig(_CONFIG_AVPHYS_PAGES) = 1500736
0.0011 sysconfig(_CONFIG_PAGESIZE) = 8192
0.0008 getpid() = 24178 [24177]

It took 24 seconds to get the number of available memory pages, a operation that worked fine in the global zone. The sysconfig source shows us that the call is very different for a global zone and for a memory capped zone:

if (!INGLOBALZONE(curproc) &&
  curproc->p_zone->zone_phys_mcap != 0) {
  pgcnt_t cap, rss, free;
  vmusage_t in_use;
  size_t cnt = 1;

  cap = btop(curproc->p_zone->zone_phys_mcap);
   if (cap > physinstalled)
    return (freemem);

   if (vm_getusage(VMUSAGE_ZONE, 1, &in_use, &cnt,
    FKIOCTL) != 0)

If there is a physical memory capping set for the zone that is less than the amount of physical memory in the machine vm_getusage will be called, it will in turn look at every memory segment for every process , this can take quite a while on larger if the zone is a heavy allocator of memory, in this case the zone was using about 50GB of memory. This is not something you want to do every time a shell script calls $(sort). If you have ever used prstat -Z with large local zones you have seen the effects of this, it can take a long time.

Comment from the source:
"This file implements the getvmusage() private system call.
getvmusage() counts the amount of resident memory pages and swap
reserved by the specified process collective. A "process collective" is
the set of processes owned by a particular, zone, project, task, or user."

The source of the problem in sort was in utility.c:
size_t phys_total = sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE);
It seems like no other utilities in solaris uses sysconf(_SCPHYS_PAGES) which is why we had no other problems.

The short time solution was to disable the physical memory cap for these zones:
# rcapadm -z zone01 -m 0

FUSE in Solaris 11

It looks like the final version of Solaris 11 will get FUSE (PSARC 2010/456). It was previously under development for OpenSolaris but never integrated and the project has not been updated for at least year. It now seems like Oracle internally have resumed the FUSE development for Solaris 11 and plans to integrate it into the internal snv_163 build which opens in about a month.

"The FUSE project (PSARC/2010/456) introduces the UVFS file system so we need a uvfs.7fs to briefly describe this feature. Content will be provided by the
FUSE team.

Projected Nevada integration build is snv_163."

OpenSolaris FUSE project

Future features of Solaris 11

There is not much available information on the ongoing development of Solaris 11. I have scanned the bug database and summarized the sparse information available. Much of the work remaining before a final Solaris 11 release is fixing bugs found in Solaris 11 Express and making the new packaging/install system fully production ready, but there will also be features added that are not available in the Express release. Here are a few of the more noticeable changes:

• Zones should be able to be NFS servers
  Solaris 11 will be able to share NFS filsystems from within a local zone.
• User names longer than 8 characters needed
  The Solaris kernel is already capable of user with names longer than 8 characters but userspace tools needs to be updated to support this.
• EOF 32-bit x86 kernel support
  The final release of Solaris 11 will only support 64-bit x86 processors, this is already the case for SPARC since Solaris 10.
• Xorg 1.9 integration
  The X Server will be updated from current 1.7 skipping version 1.8.

A list with bugid of other changes that have integrated or is about to become integrated including the above:

6981088 PSARC 2010/368 EOF 32-bit x86 kernel support
6983602 Xorg 1.9 integration [PSARC/2011/008]
6625651 Support flowadm from non global zone
7006556 need partitioning support for EFI system partition and BIOS boot partition
4109819 user names longer than 8 characters needed
6946425 reduced copy support to comstar iSCSI target
6982476 SCSI PassThrough in Solaris Local Zone
6714685 Intel Advanced Vector Extensions (AVX)
6802568 PCI-Express Framework support for SR-IOV
6944824 Intel e1000g NIC support for Leweisville (82579LM and 82579LF)
6828868 smbadm support to Display Server Service (SRVSVC)
6981644 Update bash to version 4.1
6394308 Multiple Connections / Session (MC/S) support for iscsi initiator
6946536 zvol support in non-global zones
6654408 Include FreeGLUT 2.4.0 in Nevada.
6957488 PSARC/2010/035 Support for GVRP (Dynamic VLAN)
6983112 Need a PAM module for mounting per user encrypted ZFS home directories
6516586 DMX support for Solaris x86 and SPARC [PSARC 2010/176]
6982811 64 bit Xvnc server and Tiger Vncviewer support
6940855 Zero-Copy for iSER