ZFSSA/S7000 major update

The first major software update of S7000/ZFSSA/Fishwork in over a year is now available. With the original version "2011.Q1" it seems a bit delayed, perhaps due to the departure several key persons behind the software post Oracle acquisition of Sun. New features in this release:

• ZFS RAIDZ read performance improvements
• Significant fairness improvements during ZFS resilver operations
• Significant zpool import speed improvements
• Replication enhancements - including self-replication
• Seval more including bug fixes.

There are alsoa few integration features for Oracle including RMAN and Infiniband if you happen to have a Exadata around.

The improved RAIDZ performances is the hybrid raidz/mirror allocator in zpool version 29.

The ZFSSA is a fantastic product with probably the best interface and analytics available. But the development seems to have stagnated a bit the last year, so have the blog post with useful information and performance comparison by the people behind it. And I still miss one feature badly; synchronous replication of datasets, continuous replication is not always good enough.

Release Notes

Both Mike and Brendan quits Oracle

The last of the tree DTrace creators are now leaving Oracle, Mike Shapiro, who also was part of the FishWorks core team. Another member of the FishWorks team was Brendan Greg who also resigned from Oracle. Those of you who didn't now about Bredan before probably got to know who he was when he became famous for screaming at disks in a online video.

DTrace and ZFS is the core foundation of the S7000 storage appliance, now the whole team who invented DTrace is gone and so are the two core persons behind ZFS, Jeff Bonwick and Bill Moore.

There are a lot of other great people behind these products, but the ones listed above are rare and extremly talented engineers and have all been part of breaking new ground in their area of expertice. The FishWorks team even created a usable web interface, a rare thing comming from Sun.

Mike Shapiro: End of file
Brendan Gregg: G'Day and Goodbye
Brendan Gregg screaming, Unusual disk latency
FishWorks, Now it can be told

Solaris 11 Express 2010.11 and ZFS

Besides ZFS data set encryption the upcoming Solaris Express release (Which seems to be tagged 2010.11) will contain several enhancements to ZFS besides dataset encryption that was mentioned earlier:

• RAID-Z/mirror hybrid allocator
"Normally mirror vdevs contain mirrored data, RAID-Z vdevs contain RAID-Z data, etc. However, for latency-sensitive metadata, we can use a mirrored layout across the children of a RAID-Z vdev. This ensures that such metadata can be read in a single I/O."


• Support for decompression, checksumming and raidz in zdb -R will make it much easier to debug ZFS errors on disk or just to learn more about the on-disk format of ZFS.


• Performance improvements for listing ZFS filesystems.

The downside of this is that these are features only present in Oracles internal gate and not other distribution besides Solaris 11/Express and the S7000/FishWork appliances can use them until and if Oracle releases the source after Solaris 11 is released.

I hope the Solaris 11 Express license will allow me to use Solaris 11 for my private storage needs, S7000 are nice and deployed at work but still a bit too expensive for my datacenter at home ;)

6977913 RAID-Z/mirror hybrid allocator
6757444 want zdb -R to support decompression, checksumming and raid-z

FishWorks 2010.Q3 release

A new major update to the FishWork appliance software have been released, 2010.Q3. This is the first release to support the new S7000 appliances released at Open World. It also contains bug fixes and a few enhancement, nothing exiting but it includes new Infiniband administration, better AD support, NDMP stream support and routing enhancements.

These are areas with bug fixes according to the release notes:

• device resilvering following failure and replacement
• snapshot management and deletion
• remote replication
• upgrade, factory reset, and rollback
• Active Directory interoperability
• shadow migration

ak-2010.08.17.1.0 Release Notes
FishWorks software updates

Another bright mind leaves Oracle

I can't say it comes as a surprise but another great mind is leaving Oracle. This time it's Jeff Bonwick, the man behind the slab kernel memory allocator in Solaris and more recently (together with others) ZFS. It is indeed disturbing to see that so many of the innovators from Sun, who stayed with the company thought terrible financial times are now leaving.

This means that most high profile persons behind Fishworks (S7000 appliance), DTrace and ZFS have left the red ship. Jeff Bonwick, Adam Leventhal, Bryan Cantrill and Bill Moore. There are of course many good people still at Oracle, but the people I have been following seem to be on the move.

Best of luck Jeff, I'm sure you will continue to break new ground somewhere else.

Free Your Mind,How a Small, Persistent Team Created a Revolutionary File System
Sun Engineer Jeff Bonwick is New Sun Fellow
Now it can be told (Slab allocator)

New Fishworks release at OpenWorld

Server new products in the S7000 line of the ZFS based OpenStorage products from Oracle was released today. The smallest is the 7120 that comes with configuration from 12TB. The most interesting addition is the 7420 which comes in configurations up over 1PB with optional clustering and caches up to 1TB of RAM and 4TB of Flash. The X4470 or something similar is the Sun server what fits the specification of both the 7420 and the 7720 (which seems to be a in rack cluster version). It features 512GB of memory per controller and can support up to 32 cores just as the 7420. This makes my 7310 feel very even smaller. I guess this is the "Sun T4" I have been referencing earlier.

This is just various hardware configurations running the same OpenStorage software as before, we will see if there is some new features coming in an update to the software.

Oracle Unveils Next-Generation Sun ZFS Storage Appliance Product Line
Sun ZFS Storage 7720 Appliance

Oracle Open World 2010

The first Open World since Oracles acquisition of Sun is here. Oracle likes focus their releases around Open World and we have already seen a new version of Solaris 10, 9/10. I guess that they will release SPARC T3, LDOM 2.0 (Or Oracle VM Server for SPARC) and hardware/software updates to the S7000(FishWorks) the next few days. If we are lucky we might also get some more information on Solaris 11 Express.

I will post as soon as something interesting surfaces.

Good luck Bryan

There are many good engineers working with Solaris, but a few of them get noticed a bit more. Bryan Cantrill is one of those and sadly he is now leaving Sun/Oracle. Bryan is the inventor of DTrace which he created together with Mike Shapiro and Adam Leventhal and he have even kissed a girl. Once upon a time he was requited to Sun Microsystems by Jeff Bonwick, the father of ZFS. The last few years he was a vital part of the FishWorks team used in the S7000 storage series that uses DTrace to produce graphical analytics of performance data.

I wish Bryan the best whatever he decides to do. I will keep an eye out for his next endeavor.

The new home of Bryan's blog which also holds his last entry from the Sun/Oracle blog: dtrace.org/blogs/bmc

Bryan's profile at Sun Labs: Bryan Cantrill - Sun Labs

Various ZFS news

Adam leventhal of the Fishwoks team have tested the DDRDrive X1 as a ZIL in the hybrid storage pool. It is 4GB of NV-RAM backed by flash, which is quite a good match for the ZIL requirements. You can read about his finding on his blog. For a good explanation on ZIL and synchronious writes go and read Constantin Gonzalez latest entry in his blog.

Mirrored root pools been available since the introduction of ZFS boot, but the step for writing the placing a boot block on the mirror to make it usable after a disk failure have remained a manual step, but the automatic fix for this have now mad it into on_145:

6668666 zpool command should put a bootblock on a disk added as a mirror of a root pool vdev.

Slower ZFS scrubs/resilver on the way!

A nice change have made it's way into ON, it fixes a few bugs regarding general user performance when scrubs or resilver are running in a pool. It will delay the scrub thread if there was a non-scrub I/O the last 50 ticks, resilver are considered more importand and are delayed only 2 ticks where scrubs are delayed 4 ticks. This should allow the scrubs/resilver to run at full speed when possible while limiting the impact to other I/O. This will as with many other ZFS features probably make its debut in an update to the S7000 series firmware. Previous to this fix a failed drive or a ordinary scrub could have quite negative impact on other I/O in the pool.

"* We keep track of time-sensitive I/Os so that the scan thread
* can quickly react to certain workloads. In particular, we care
* about non-scrubbing, top-level reads and writes with the following
* characteristics:
* - synchronous writes of user data to non-slog devices
* - any reads of user data
* When these conditions are met, adjust the timestamp of spa_last_io
* which allows the scan thread to adjust its workload accordingly."

hg.genunix.org/onnv-gate.hg/rev/b118bbd65be9
6743992 scrub/resilver causes systemic slowdown

News and two PSARC:s

This is only a relay of some news to keep you occupied while I'm working on a more interesting but far more time consuming posts.

The Fishworks simulator is available for download in a brand new 2010.Q1 version, no upgrade needed and now only for VirtualBox which also had a maintinance release last week (3.1.6).

Two new interesting PSARC:s have emerged the last week:

PSARC/2010/105 zfs diff
Like diff(1) but for ZFS, this will make it possible to calculate filesystem difference between two snapshots of a dataset.

PSARC/2010/106 DTrace TCP and UDP providers
This will add TCP and UDP providers to Solaris making thinks like the following possible (from the PSARC):

# Watch inbound TCP connections by remote address,
dtrace -n 'tcp:::accept-established { trace(args[2]->ip_saddr); }'

# Inbound TCP connections by destination port summary
dtrace -n 'tcp:::accept-established { @port[args[3]->tcp_dport] = count(); }'

# Watch inbound accepted TCP connections by process summary
dtrace -n 'tcp:::accept-established { @cpid[args[1]->cs_pid] = count(); }'

# Watch UDP total number of bytes sent/received by process
dtrace -n 'udp:::send,udp:::receive { @bytes[args[1]->cs_pid] = sum(args[4]->udp_length);}'

Back with some lagging news

I'm back from my vacation and my first post will probably include lots of thinks you already know, but I want to list some interesting things that have happened.

Build 134 of OpenSolaris in now available as images and in the development repository. A new preview release of the text based installer is also available for this build. All the image is as usual available at genunix.org. At least one respin will be done before the final 2010.03 release, build 134a.

The 2010.Q1 release of the Fishworks software is available for download and works for the Storage 7000 appliance as well as the simulator. There are also a few good blog posts from members of the Fishworks team, concerning this release:
Turing the Corner, Multiple pools in 2010.Q1, Replication

A friend also sent me a link to Jonathans new blog outside Sun where he has a interesting entry about the importance of patents for defensive purposes: Good artist copy, great artists steal

S7000 2010Q1 release notes

While the software is yet to be published, the release notes for the new fishworks update is available online.

Besides bug fixes it have several new features such as:

• Deduplication
• Fibre Channel support
• Kerberized NFS
• iSer/SRP
• Multiple pool support
• Replication enhancements

Update: It seems like bug 6924824 (destroying a dedup-enabled dataset bricks system) has been fixed for this release, but it has not made it back to OpenSolaris, or they have failed to mention this in the known issues. Let's hope for the former and that it will be fixed in OpenSolaris real soon, maybe even before this software update is available.

Here are the Release notes.

Fishworks 2010.Q1 release

I was trying to get some more information on the next release of the software for the Sun Storage 7000-series appliance systems, this is what I found:

• It will feature COMSTAR Fiber Channel targets


• It might also include support for multiple zpools, which can be useful if data with very different performance/availability requirements are located in the same box.

The update will probably be released during or before next week since there is a event which will show of new features next Tuesday here in Sweden. Read more and register here.

Here is a short mail conversation with Bryan which mentions this and here are some (translated) references to an internal test version with FC support.

Update: The release date for this update is "in the next few weeks" according to Bryan of the fishworks team, stay tuned.