Joyent presentations @ FISL 13

Joyent had a several speakers at the FISL 13 conference and presentations/slides are now available online.

Bryan Cantrill, Corporate Open Source Anti-Patterns: Doing It Wrong
video, slides
Bryan speaks his mind about corporate open source patterns with insights from Sun, the OpenSolaris project and Joyent. He does a bad job hiding what he thinks of Oracle ;)

Brendan Gregg, Performance analysis, the USE method
slides, video
Brendan on performance analysis using the USE method with good examples.

Update: Added Brendans video.

illumian available for download

A new illumos based distribution is available for download, illumian. illumian is the based on the APT packaging system and is a successor to Nexenta Core Platform (NCP) which was built by nexenta using the OpenSolaris source and APT packages.

The next version of NexentaStore (4.0) should also be built upon illumian, previous versions was built on NCP.

There is currently one image available for server text install on X86 hosts.

illumian.org

OpenSolaris build 134b

A bit ironic but as Oracle released Solaris 11 Express, the build that was once supposed to become Opensolaris 2010.05 was also put out the gates. The opensolaris.org/release repository was updated to be used as a platform for upgrading existing OpenSolaris installations to Solaris 11 Express. You can now upgrade to this version anyway:

                      OpenSolaris 2010.05 snv_134b X86
  Copyright (c) 2010, Oracle and/or its affiliates.  All rights reserved.
                           Assembled 28 May 2010

OpenIndiana announced

Today project OpenIndiana was announced, it is continuation of OpenSolaris created by the community. It's currently pretty much OpenSolaris build 147 but in the near future it will be based on the illumos gate.

It's nice to see a distribution that is similar to OpenSolaris not everyone wants to go the illumos/debian way like Nexenta and OpenIndiana aims to be a free drop-in replacement for Solaris 11/Oracle Solaris Express. It can be used for free in production unlike the new Solaris 10/Solaris 11 licensing from Oracle. OpenSolaris installation should also be able to upgrade to OpenIndiana.

The slides from the announcement are available here:
http://dlc.openindiana.org/tmp/slides.pdf

Iso Images are also available, but this is the first bits they ever release and they call this release a bit "messy" so use it with caution:
http://dlc-origin.openindiana.org/isos/147/

Oracle pulls ON source cord

Oracle have now pulled the plug on the ON source updates, the external hg repository will no longer be updated by Oracle (or anyone else), all Oracle Solaris development will be internally and code only released when a Solaris 11 release is made. The last complete build tag became onnv_147 which closed a few days ago.

May god have mercy on their souls.

This will change my habits a bit, everyday went through the changes in the ON gates to see what have happened. But this might not be the and of days, Garrett D'Amore write about this and the affects on Illumos in his blog. He states that from now on ON and Illumos will go different ways but is very enthusiastic about it. I really hope the Illumos will get momentum and attract enough engineers to stand on it's own. There must be a free Solaris alternative or it will become an Enterprise only option with no community involvement.

I would expect that binary distributions based on the latest source available will become available from some place quite soon. A newer version than the snv_134 which contains a few nasty bugs is long overdue.

The new place to look for changes in the open ON source:
http://www.illumos.org/projects/illumos-gate/repository

Service auto discovery in future Solaris

While looking at changes for the future packaging system in Solaris I stumbled on a putback that included pybonjour in the pkg(5) gate. Apparently Solaris of the future is going to be using zeroconf for auto discovery of packages repositories. Looking through other OpenSolaris project it seems like this and NWAM also will be using auto discovery. OpenSolaris have had support for mDNS discovery since build 72 but the pkg use-case is what got me interested.

This will allow clients on the same network to discover mirrors of the repository and use them to ease the load on the primary repository. The mirroring service is easily enabled:

# svcadm enable application/pkg/dynamic-mirror

From pkg.depotd(1M):

"The pkg depot is also able to act as a mirror server for local client images from pkg(5). This allows clients that share a subnet on a LAN to mirror their file caches. Clients may download files from one another, thereby reducing load on the package depot server. This functionality is available as an alternate depot service configured by smf(5). It uses mDNS and dns-sd for service discovery."

From pkg(1):

"mirror-discovery
(boolean) Mirror-discovery tells the client to discover
link-local content mirrors using mDNS and DNS-SD. If this is
set to True, the client will attempt to download package content
from mirrors it dynamically discovers. To run a mirror that
advertises its content via mDNS, see pkg.mdnsd(1m)."

# pkg set-property mirror-discovery True

Network Auto Magic (NWAM) will include service auto discovery of DNS and naming services:
http://hub.opensolaris.org/bin/view/Project+nwam/service-discovery

ZFS crypto lifesign

Aside from BP-rewrite, encryption is probably the most awaited ZFS feature, but there have been sparse information regarding the status of both projects for about a year.

The ZFS crypto project which are going to deliver on-disk encryption for ZFS have now at least showed a small life sign. This is after a long time of radio silence from Darren, Sun and Oracle regarding the ZFS crypto project since May last year when "PSARC/2009/443 ZFS Crypto Updates" was published. Now there are references to a new PSARC, but it's not public so we have only the name to to look at: PSARC/2010/329 ZFS Crypto additional keysource types.

The last crypto releated PSARC from laste year, which was public:
PSARC/2009/443 ZFS Crypto Updates

zonestat implemented

We are not left out in the dark yet, the gates are still open and we can see changes made to the OpenSolaris source, the source that is going to be used for Solaris 11.

Today PSARC/2010/29 zonestat was implemented, it provides enhanced observability for Solaris zones. prstat -Z can provide a nice overview of all zones on a system but it does not tell you anything about any resource capping of memory or CPU resources. There have been a perl-script with the same name available for some time which is able to extract and present a nice summary of zones and their capping and usage. This is not only a reimplementation in C, it is also architectures the new observability feature into three layers, a collection daemon, an API and a command line utility. This case has requested patch binding and it should not be hard or include any significant risk to backport it to a Solaris 10 update.

Examples from the PSARC:

Example 1: Summary of cpu and memory usage over a 5 second interval:
    # zonestat 5 1
    SUMMARY
                -----CPU------------- ----PHYSICAL--- ----VIRTUAL----
           ZONE USED %PART %CAP %SHRU USED  PCT  %CAP  USED  PCT %CAP
        [total] 9.74   30%    -     - 7576M  23%    - 11.6G  24%    -
       [system] 0.28  0.8%    -     - 6535M  19%    - 10.4G  21%    -
         global 9.10   28%    -     -  272M 0.8%    -  366M 0.7%    -
      kodiak-ab 0.32  1.0%    -     -  256M 0.7%    -  265M 0.5%    -
      kodiak-dp 0.00  0.0%    -     - 77.6M 0.2%    - 71.1M 0.1%    -
kodiak-gjelinek 0.00  0.0%    -     - 58.7M 0.1%    - 59.3M 0.1%    -
     kodiak-edp 0.00  0.0%    -     - 53.0M 0.1%    - 58.9M 0.1%    -
 kodiak-johnlev 0.00  0.0%    -     - 51.9M 0.1%    - 57.4M 0.1%    -
  kodiak-jordan 0.00  0.0%    -     - 51.7M 0.1%    - 56.8M 0.1%    -
   kodiak-steve 0.00  0.0%    -     - 51.5M 0.1%    - 56.2M 0.1%    -
   kodiak-susan 0.00  0.0%    -     - 48.9M 0.1%    - 55.7M 0.1%    -
kodiak-batschul 0.00  0.0%    -     - 48.5M 0.1%    - 49.5M 0.1%    -
 kodiak-garypen 0.00  0.0%    -     - 46.3M 0.1%    - 49.5M 0.1%    -
     kodiak-rie 0.00  0.0%    -     - 22.7M 0.0%    - 49.4M 0.1%    -

PSARC/2010/291 zonestat

OpenSolaris dead as a distribution

A supposedly internal memo from Oracle has been posted to the opensolaris-discuss list. The most important part of this is probably that it finally states that OpenSolaris as a distribution is dead, Oracle will no longer make OpenSolaris available in binary form as a release or in bi-weekly development builds. The source will for most parts continue to be open, but will not be pushed out in realtime as it does today, it will be posted after a Solaris 11 release has been made available. This way Oracle will always be first in line to productize their latest innovations and fixes. neither will Oracle continue to post most of the PSARC externally. There will however be options for partners to gain early access to source and binaries from Oracle. Solaris 11 will be made available in an Solaris 11 Express form as a preview to the next release of Solaris 11. It will be free to use for developers and has an optional support plan.

As am enterprise customer the message is great, to know that Oracle really believes and invests in Solaris and that Solaris 11 will be made available early. The availability of the source after a release is also very important. I also feel that Oracle genuinely are going to make Solaris even better and are willing to put in the resources to do so. Looking at all the new features available in OpenSolaris since Solaris 10, Solaris 11 will be a huge leap forward for Solaris.

As someone involved and personally interested in the OpenSolaris community this is not very good. I will be cut of from the day-to-day changes in the source and no free OpenSolaris distribution will be available to attract new customers outside of the large enterprises. I think there will be some other distribution that will try to take the place of OpenSolaris, but with the access to the source cut, except for snapshots from releases it will not be as interesting as the weekly builds have been. It would also stall months behind Solaris 11 as it has to wait for the release to be made available.

If Oracle just would make Solaris 11 available at a lower price with obviously less support commitment it might come something good out of this even for businesses with smaller budgets. I think this is really something Oracle must do, otherwise the adoption rate of Solaris will decline drastically. It would neither undercut the sales of the enterprise support contracts, they will still be sold, mission critical deployments are not run with patch or mail-only support options. I've been involved with several smaller startups which love Solaris and like to use it, but neither need or can afford the support contracts for Solaris today, it would be sad if there was no open option for them to continue to use Solaris for a, for them, reasonable price.

So in short, I think that this will make Solaris even better for the large customers, Oracle will put much effort in keeping Solaris the most capable and advanced operating system available. But smaller companies, startup and hobbyist will have a harder time to adopt Solaris 11. I will continue to use Solaris 11 express for my private servers if possible, but I can obviously not be as involved as before since I get cut off from frequent builds and changes to the source.

The full memo is available here.

Oracle confirms Solaris 11

John Fowel, executive vice president of system, now at Oracle have disclosed some a timeframe for the Solaris 11 release. It should be released in 2011 and contain many of the new features and enhancements that has been implemented in OpenSolaris. It should even contain a superset of what is currently in OpenSolaris, with new technologies created after the Oracle acquisition.

I've previously discussed that Oracle might be focusing on Solaris 11 instead of OpenSolaris, and that seems to have been correct. They does not mention any new release of OpenSolaris in this web event.

This also confirms the name "Solaris 11" for the next Solaris release which always officially have been Solaris next or Solaris.next since marketing wants to have the options open for the name of the next release. Many of you probably remember the change from Solaris 2.7 to Solaris 7.

I will write more when I and if more information is made available, the web event will be made available online later.

You can read the whole article from Serverwatch here.

Update: Eweek has some more information regarding this while awaiting the webcast. Oracle plans to double the performance of SPARC every year through 2015: Oracle Outlines SPARC, Solaris 11 Plans.

Read-only zpools implemented

Read-only zpools have now been implemented in the source as proposed in PSARC/2010/306.

Previously individual filesystem have could be mounted as read-only, but now the whole pool can be imported this way:
# zool import -o readonly=on zpool01

When importing a pool with read-only, no writes will be performed to any of the devices in the pool, they are even opened read-only.

From the PSARC:
"When the 'readonly' property is 'on' (read-only) the following actions are performed:

1). pool is loaded but transaction processing is disabled
2). all filesystems and zvols are mounted in read-only mode
3). any intent-log replays are deferred (any pending synchronous
writes will be replayed once the pool is imported read-write)

The following restrictions apply when a pool is read-only:

1). Attempts to set additional zpool-level properties during import
are ignored.
2). All filesystem mounts are converted to include the 'ro'
mount option.
3). Additional mount options may be set on a temporary basis.

A pool which has been imported read-only can be restored back to read-write by exporting and reimporting it."

PSARC/2010/306 Read-only ZFS pools

ZFS diff

A really nice feature have been implemented for ZFS, which allows you to list all file changes between two snapshots of a ZFS filesystem. The PSARC was filled in March but it has now been implemented. It will have a simple syntax just like diff(1):
# zfs diff zpool01/fs01@snap01 zpool01/fs02@snap02

With output in this format:

M       /myfiles/
M       /myfiles/link_to_me   (+1)
R       /myfiles/rename_me -> /myfiles/renamed
-       /myfiles/delete_me
+       /myfiles/new_file

From the PSARC:
" There is a long-standing RFE for zfs to be able to describe
what has changed between the snapshots of a dataset.
To provide this capability, we propose a new 'zfs diff'
sub-command. When run with appropriate privilege the
sub-command describes what file system level changes have
occurred between the requested snapshots. A diff between the
current version of the file system and one of its snapshots is
also supported.

Five types of change are described:

o File/Directory modified
o File/Directory present in older snapshot but not newer
o File/Directory present in newer snapshot but not older
o File/Directory renamed
o File link count changed

PSARC/2010/105 zfs diff

Lots of enhancements, no build available

Since build 134 of OpenSolaris which once was destined to become the OpenSolaris 2010.1H release quite a lot have changed. If there was a release based on that build today it would feel a bit old giving all the changes now available. I took a look on some of the larger or in my opinion more noticeable changes and made a list. It makes you want to get your hands on a Solaris Next developmen build soon, or at least update the home rolled builds to b146 once it is finished. The development efforts does not seem to have decreased, it's a shame we still don't have any public builds to test all these new features.

This list is just a overview of the enhancements, there have been over 1300 putbacks containing over 2200 bugfixes including many new features. This is only the OS/Net consolidation with the core part of OpenSolaris not including PKG, X, JDS or the many free software packages available for installation from the package repositories.

PSARC/2010/014 Open Source elxl replacement
PSARC/2010/042 increase max TCP_INIT_CWND
PSARC/2008/532 NWAM Phase 1
PSARC/2009/576 pam_krb5 PKINIT support
PSARC/2010/004 Logical Domains Information API and ldminfo
PSARC/2010/067 Interim modernization updates
PSARC/2010/045 x86gentopo enumeration of direct attached SATA
PSARC/2009/354 Always on / no reboot Solaris Audit
PSARC/2009/364 dlstat and flowstat
PSARC/2010/041 USB CDC ECM driver
PSARC/2010/062 increase number of realtime signals
PSARC/2009/689 Audio DDI Simplifications
PSARC/2009/534 SMB/CIFS Standalone DFS
PSARC/2010/009 Modified ZFS passthrough-x ACL inheritance
PSARC/2010/032 EC and SHA2 for KMF
PSARC/2010/013 SMF Early Manifest Import
PSARC/2009/104 Hot-Plug Support for ACPI-based Systems
PSARC/2009/306 Brussels II - ipadm and libipadm
PSARC/2009/421 Open Fabrics User Verbs (OFUV) - primary kernel components
PSARC/2009/019 SAS Management Protocol library
PSARC/2010/102 ikeadm dump algs
PSARC/2010/085 IPoIB Administration Enhancement
PSARC/2010/029 Improved ACL interoperability
PSARC/2010/101 in.iked preshared key file extensions
PSARC/2010/048 zfs-based ndmpd backup
PSARC/2010/119 "Console User" assignment, logindevperm and virtual console update
PSARC/2010/043 Reliable Datagram Service v3
PSARC/2009/377 In-kernel pfexec implementation.
PSARC/2010/108 zil synchronicity
PSARC/2010/127 ipadm hostmodel property
PSARC/2010/155 sshd(1M) PAM Service name options
PSARC/2010/174 TMPDIR support for Solaris tar
PSARC/2009/646 bd - generic block device driver
PSARC/2010/028 BSD List Interfaces
PSARC/2010/181 PRIV_SYS_RES_BIND privilege
PSARC/2010/106 DTrace TCP and UDP providers
PSARC/2010/151 new socket options for TCP timers
PSARC/2010/083 Kerberos Profile API
PSARC/2010/188 PKCS#11 URI parser for libcryptoutil
PSARC/2010/177 KMF Certificate Name mapping extensions
PSARC/2010/144 lofi(7D) in non global zones
PSARC/2009/590 Socket Filter Framework
PSARC/2007/477 Drivers registering devids
PSARC/2010/222 Relaxed Type Requirements for SMF Profiles
PSARC/2010/215 Add libstmf interfaces to set/get default stmf properties
PSARC/2009/505 IRM Framework Extension(s)
PSARC/2009/430 Default system CA (X.509) Certificates
PSARC/2010/195 PKCS11 KMS Provider
PSARC/2009/042 max-processes rctl
PSARC/2010/240 Allow ranges for the 'cpus' datalink property
PSARC/2010/166 layer-3 net properties for exclusive-IP zones
PSARC/2010/239 OFUV Userland Interface
PSARC/2010/235 POSIX 1003.1-2008 *at(2) syscalls
PSARC/2010/241 userattr
PSARC/2010/111 exclusive IP for s10c
PSARC/2010/269 KMF Certificate Validation Enhancements
PSARC/2010/164 interfaces for basic install network configuration
PSARC/2010/113 Boot Block Downgrade Avoidance
PSARC/2010/234 IBTF 2010.Q2 Enhancements
PSARC/2009/617 Software Events Notification Parameters CLI
PSARC/2009/618 snmp-notify: SNMP Notification Daemon for Software Events
PSARC/2009/619 smtp-notify: Email Notification Daemon for Software Events
PSARC/2010/278 FMA/SMF integration: instance state transitions
PSARC/2010/279 Modelling panics within FMA
6914369 enable s10brand on system booted under xvm
6889694 COMSTAR kstat to track IO statistics at the ITL
6655655 Single phase COMMIT processing for NFS3 and NFS4 server
6830088 Fast reboot support for SPARC platforms
6871356 Sata framework should support INQUIRY VPD page 0xB1
6782858 Solaris needs Broadcom 5718 network support
4404421 RFE: tar command -z and -j support
6202362 Solaris tar can't unpack files created with GNU tar
6922272 SATA framework does not handle >2TiB disks
6939349 RFE: Update ksh93 to ast-ksh.2010-03-09
6923815 Implement FMA functionality in qlge driver
6889322 Add NLP2020 PHY support to nxge
6874797 Solaris needs to support I/O subtree reparenting
6945131 SMF should support multiple administrative profiles
6951174 Intel microcode 2010-02-09 update
6923890 Add FMA to SiliconImage si3124 sata driver
6946100 Add machine check and performance counter support for new member of Nehalem family
6937229 Solaris needs to support BCM5717 and BCM5724
6953601 improve logical unit creation time for zvol backed logical units
6910752 mpt_sas driver performance can be improved
6494473 ZFS needs a way to slow down resilvering
6917066 zfs block picking can be improved
6932959 ZFS allows too many devices to be faulted
6716117 ZFS needs native system attribute infrastructure
6675946 'zpool status' should show the progress of resilvering for individual disk.
6668666 zpool command should put a bootblock on a disk added as a mirror of a root pool vdev
6733267 Allow a pool to be imported with a missing slog
6959846 DMU traverse prefetch size should be a global tunable
6866610 Add SATA TRIM support
6908227 pyzfs needs to move to python 2.6

If any of the links point to nowhere or in the case of PSARC:s empty or almost empty directories there are no public information available besides the source.

Backporting

Last night i spend a few hours backporting recent changes from the ON gate to my own gate that is based on ONNV_134. I wanted to get a build that is as close to OpenSolaris b134 as possible while pulling fixes and a few enhancements for ZFS, COMSTAR, SATA and NFS. I went quite easy, the main problem was that i did want to get as close to the latest ZFS bits without including the changes that removed the iSCSI daemon (not because I will use it but it was part of 134). With a few modifications to the patches i now have a ONNV_134 build with a much more current ZFS. My plan is to test this for a while and then use it for my ZFS storage nodes at home, where of course ZFS is the most important part. There have been quite a few fixes for ZFS since February when build 134 closed.

This is obviously only something I will be playing with until Oracle decides to release a Solaris version based on a more current snapshot of the OpenSolaris source. Private builds and independent distributions can be both fun and useful but does not go trough the amount of testing as the "official" releases does, that kind of fragmentation decreases the testing for all distribution.

In the case of ZFS i like to use OpenSolaris and preferably a more current one. It is the growing ground of Fishworks, the software part of the S7000 series which I use for some production environments. The ZFS code in those should go trough a fair amount of testing both by Oracle and all of the thousands of enterprise deployments.

All this said, times goes by and with no binary release in sight a community effort to build and deliver binaries for the community for testing, preparation for the next release and recruit new users might be needed. But it would be sad if we ended up divided between an Oracle paid distribution and a community one with no involvement by Oracle other than providing almost all of the source and new features of the ON code, which they could decide to stop delivering or only deliver parts of it. But all this is in the hands of Oracle.

Notification enhancements

A common request from smaller shops or private users of Solaris is that they want email notifications when something is wrong with their system. This is most common with ZFS filers, they might not have any surveillance systems implemented but want to be sure to get a notification if something goes wrong with a zpool. A quick fix to this is of course a small script which can dump zpool or FMA errors over email but it is not a standardized or complete solution.

Now there have been a large putback in OpenSolaris that among other things delivers this functionality. FMA faults and SMF events can with these change be configured to deliver notifications over mail or SNMP. It also enables the fault management daemon in local zones so that they can deliver SMF state changes from within the zone. Using this framework you can also configure any of your own SMF services to deliver state change notification over any of these two protocols.

PSARC/2009/617 Software Events Notification Parameters CLI
PSARC/2009/618 snmp-notify: SNMP Notification Daemon for Software Events
PSARC/2009/619 smtp-notify: Email Notification Daemon for Software Events
PSARC/2010/225 fmd for non-global Solaris zones

SATA Trim command in B146

The SATA TRIM command have been implemented in the OpenSolaris source, build 146. This can be used by filesystems to prevent performance degradation over time on SSD:s by giving the garbage collector feedback on which blocks are no longer in use. This is bugid 6866610. It it still not implemented in ZFS but this is the ground stone for making that possible and there is a feature request for that also 6957655.

Looking into the future the similar SCSI UNMAP command would be of great use for thin provisioning, both for thin devices located on a ZFS LUN and for ZFS residing on a thinly provisioned LUN for example in a SAN.

The later is described in this bug report 6913905.

New ZFS recovery option

Currently if you loose a non-redundant log device in your zpool it becomes impossible to import, there is no way to exclude the log device and import the rest of the pool. There are however light on the horizon, PSARC 2010/292 proposes a solution to this problem with a new switch for the zpool import command.

From the PSARC:
"This fast-track introduce a new command line flag to the
'zpool import' sub-command. This new option, '-m', allows
pools to import even when a log device is missing. The contents
of that log device are obviously discarded and the pool will
operate as if the log device were offlined."

PSARC 2010/292 zpool import despite missing log

Update
This has now been implemented in the source, the changeset is here.

NFS Instances/Zones with NFS shares

There is a PSARC awaiting approval for providing on of the most long standing enhancements requests for Solaris Zones: NFS Shares from within a local zone. This functionality will not be limited to NFS and will initially also support CIFS. It seems to be planned for Solaris Next only, so no backport to Solaris 10.

From the PSARC:
"Patch binding is requested; however, there are no plans to backport any of
the proposed changes.

The proposed changes are needed to support an NFS server in a non-global
zone; however, the changes are not specific to NFS. They would also be used
when enabling any file sharing protocol (ie. CIFS) server for non-global zones."

PSARC/2010/2080

Various ZFS news

Adam leventhal of the Fishwoks team have tested the DDRDrive X1 as a ZIL in the hybrid storage pool. It is 4GB of NV-RAM backed by flash, which is quite a good match for the ZIL requirements. You can read about his finding on his blog. For a good explanation on ZIL and synchronious writes go and read Constantin Gonzalez latest entry in his blog.

Mirrored root pools been available since the introduction of ZFS boot, but the step for writing the placing a boot block on the mirror to make it usable after a disk failure have remained a manual step, but the automatic fix for this have now mad it into on_145:

6668666 zpool command should put a bootblock on a disk added as a mirror of a root pool vdev.

Desktop updates for Solaris Next

Good things come to those who wait? I've seen some indication that the latest release of Gnome (2.30) have been integrated into the snv_140 build. It's tagged as OS-2010.06, but the OpenSolaris 2010.06 name seems to have been in use since April so that might not mean anything. Firefox also have been upgraded to 3.6.x in build 135 and onwards. Time slider have got an update which includes replication to external filesystems (PSARC/210/129). These updates are available in Solaris Next Development builds post snv_134 so they have not been released even in the development repository for OpenSolaris.

A desktop related thing that I've totally missed is that there are ongoing work for integrating Xorg 1.8.x. The source is available here.

It seems like Oracle still are aiming to reach developer and some desktop users with Solaris. I think this is good since people use what the know when comfortable and perhaps have seen the advantages they go on and use it on their servers. This is also good news for a future modern SunRay environment.