Tuesday, November 30, 2010

ZFS encryption at ${HOME}

A nice addition to easily use of ZFS encryption for home directories will be available in a future version of Solaris 11. This will allow you to have your password as key for your home dataset which is automatically used at login:

"For users with local ZFS storage we want to provide a very simple and as transparent as possible way of using encrypted ZFS datasets. The target for this is laptops and systems with local ZFS storage for the users home directory.

The goal is to provide as seemless as possible a way to have an encrypted home directory and additional encrypted datasets below the home directory. A new PAM module pam_zfs_key.so will be introduced. This module supports only pam_sm_setcred(3PAM) and pam_sm_chauthtok(3PAM), pam_sm_authenticate(3PAM) is provided but always returns PAM_IGNORE.

It assumes that the users login passphrase is also the passphrase used to protect thier ZFS encrypted home directory and will ensure that when users change their password the passphrase used for deriving the wrapping key for their encrypted ZFS home directory is changed as well."

Bugid: 6983112

Friday, November 26, 2010

Solaris 10 update 10 update

Some additional details on the next Solaris 10 update have surfaced, they compliment the list in my previous post. No huge updates, Solaris 10 should not be seeing any ground breaking changes this late in its lifespan but good additions besides the bug fixes.

  • IPfilter IPv6 NAT support
  • 10Gb Ethernet support for Mellanox ConnectX-2 chipset
  • SSH Support for ChrootDirectory and ConnectTimeout
  • PSARC/2008/256 Native LDAP standalone tools (Duckwater Phase 0)
  • Updated FireFox to 3.6.8+ and Thunderbird 3.1+

The ZFS update should include bits similar to snv_148 which among other things include the following:
  • RAID-Z/mirror hybrid allocator
  • Missing log import (PSARC/2010/292) (CR 6733267)
  • ZIL synchronicity PSARC/2010/108
  • read-only import (PSARC 2010/306)

There might also be an update to the bundled samba which is a bit overdue. This was delayed earlied due to change of licenses but that should now have been solved. This might slip again, as it did previously, but it looks like a 3.5+ version of samba will be included.

I should probably have mentioned that it seems that the name for the release will be Oracle Solaris 10 8/11 which means it is scheduled for a release around August/September 2011. This would be just one or two months before Oracle World so it seems highly likely it will support the expected SPARC T4. This new release cycle with one release per year seems like a better fit for enterprise customers, Solaris 10 9/10 is probably not in many datacenter yet.

Solaris 10 update 10

Monday, November 22, 2010

ZFS Feature/Dist table

For a advanced, low cost, data consistent storage server ZFS still has very little competition. The main challenge today is to choose the correct distribution for your needs. We have S7000/Fishworks and NexentaStor, but both are appliances and cost a fair bit of money, at lest for anything but trivial deployments (>18TB). I put together a table with ZFS features, storage features and cost/support status, included are the larger distributions with Solaris/OpenSolaris kernel today, FreeBSD and OpenSolaris 2009.06 as a reference.

It's not much of a surprise that Solaris 11 Express comes out with the most complete feature set, but it also has a cost of $1000/year for up to 8 cores (1 socket, non Oracle HW) for anything but testing and development. That is cheap for a file server with all the features ZFS provides but for home filers it can be a bit too much.

This table is of course subject to change. Both Nexenta and FreeBSD will surly continue to update ZFS at least to the point were Oracle stopped releasing the source. The available source from Oracle does not include some of the features in the table, for example ZFS crypto and probably a fair amount of bug fixes.

ReleaseS11 ExpNCPFreeBSDOSOL
Free for use in productionNYYY
Commercial support availableY N1 N1 N1
Zpool version31261414

ZFS Diff


RAID-Z/mirror hybrid allocator


Improved 'zfs list' performance




Multiple vdev replacements

zdb w decomp, checksum, raidzYNNN
Low priority resilverYNNN



Tripple-parity raidz


Log device removal


Improved scrub stats


Improved snapshot deleteion performance


zpool recover


resilver prefetch


zpool split


zil synchronicity

1. No support from sponsoring project or company, may be available elsewhere.

Friday, November 19, 2010

Quick Solaris 11 Express walkthrough

This is a quick command/output walkthrough of some example administrative tasks in Solaris 11 Express. You can just skim thought it or run the command yourself on a test installation. It uses zfs-crypto, deduplication, zones, crossbow, ipadm, zonestat and pkg.

Create a alternate boot environment by cloning the current filesystem state for an quick and easy rollback option:

# beadm create initial

Set up static IP and add a persistent route:
# ipadm create-addr -T static -a local= bge0/v4
# ipadm show-addr
lo0/v4 static ok
bge0/v4 static ok
lo0/v6 static ok ::1/128

# route -p add default
add net default: gateway
add persistent net default: gateway

Add mail notifications on hardware/FMA and SMF state changes:
# pkg install smtp-notify
# svccfg setnotify -g from-online,to-maintenance mailto:admin@somehost.com
# svccfg setnotify problem-diagnosed,problem-updated mailto:admin@somehost.com

Install some utilities from the network repository:
# pkg install terminal/screen mercurial nmap

Create an encrypted dataset for the secret project:
# zfs create -o encryption=on -o mountpoint=/vault rpool/vault
Enter passphrase for 'rpool/vault': ********
Enter again: ********

Add a dataset with deduplication:
zfs create -o dedup=on -o mountpoint=/export/vbox rpool/vbox

Add a NFS/SMB share area and ignore the ZIL for asynchronous writes in favor of performance(unsafe):
# zfs create -o mountpoint=/export/share rpool/share
# zfs set sharesmb=on rpool/share
# zfs set sharesmb=name=share rpool/share
# zfs set sharenfs=on rpool/share
# zfs set sync=disabled rpool/share

Create a virtual switch with crossbow and two virtual interfaces for zones with bandwidth management and assigned to different CPUs:
# dladm create-etherstub etherstub01
# dladm create-vnic -l etherstub01 vnic_zone01
# dladm create-vnic -l etherstub01 vnic_zone02
# dladm set-linkprop -p maxbw=10M -p cpus=3 vnic_zone01
# dladm set-linkprop -p maxbw=10M -p cpus=4 vnic_zone02

Configure and install zone capped to 50% of a CPU located on ZFS and using the virtual interface:
# zfs create -o mountpoint=/zones -o compression=on rpool/zones

# zonecfg -z zone01
zonecfg:zone01> create
zonecfg:zone01> set zonepath=/zones/zone01
zonecfg:zone01> set ip-type=exclusive
zonecfg:zone01> ad capped-cpu
zonecfg:zone01:capped-cpu> set ncpus=0.5
zonecfg:zone01> add capped-memory
zonecfg:zone01:capped-memory> set swap=2G
zonecfg:zone01:capped-memory> end
zonecfg:zone01> add net
zonecfg:zone01:net> set physical=vnic_zone01
zonecfg:zone01:net> end
zonecfg:zone01> commit

# zoneadm -z zone01 install
A ZFS file system has been created for this zone.
Publisher: Using solaris (http://pkg.oracle.com/solaris/release/ ).
Image: Preparing at /zones/zone02/root.
Sanity Check: Looking for 'entire' incorporation.
Installing: Core System (output follows)

# zoneadm -z zone01 boot
# zonestat 5
Collecting data for first interval...
Interval: 1, Duration: 0:00:05
SUMMARY Cpus/Online: 4/4 Physical: 8063M Virtual: 11.8G
----------CPU---------- ----PHYSICAL----- -----VIRTUAL-----
[total] 0.76 19.2% - - 946M 11.7% - 1960M 16.2% -
[system] 0.10 2.54% - - 774M 9.60% - 1816M 15.0% -
global 0.66 16.6% - - 133M 1.65% - 113M 0.94% -
zone01 0.00 0.00% 0.03% - 38.1M 0.47% - 30.0M 0.24% 1.46%

Enjoy the new faster scrub/resilver that should have less of impact on other workloads to the pool. It also has a more detailed output:
# zpool scrub rpool
# zpool status rpool
pool: rpool
state: ONLINE
scan: scrub in progress since Sat Nov 20 02:21:51 2010
3.22G scanned out of 19.0G at 35.8M/s, 0h7m to go
0 repaired, 16.95% done

rpool ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0

Check if any updates are available:
# pkg update -n
No updates available for this image.

Oracle blogs on S11 Express features

Lots of Solaris 11 information have been made available after the launch of Solaris 11 Express, including technical blog entries from Solaris engineers. Here are links to the ones I found most interesting:

Darren Moffat on ZFS crypto:
Introducing ZFS crypto in Oracle Solaris 11 Express
Choosing a value for the ZFS encryption property
Assured delete with ZFS dataset encryption
Having my secured cake and Cloning it too (aka Encryption + Dedup with ZFS)
ZFS encryption what is on disk ?

The Observatory:
Upgrading from OpenSolaris 2009.06 to Solaris 11 Express 2010.11

Alan Coopersmith on X11:
X11 changes in the 2010.11 release

Another great resource for Solaris 11 is the Oracle video blog which now hosts the following videos:
  • New Security Features in Oracle Solaris 11 Express
  • ZFS Features in Oracle Solaris 11 Express
  • Solaris 11 Engineering Panel at LISA 10
  • What's Great in Solaris 11 Express for Sysadmins
  • Solaris 11 Packaging & Installation
  • What's Great in Solaris 11 Express for Developers
  • DTrace BoF at LISA10
  • Oracle Solaris Studio and Oracle Solaris 11 Express
  • Solaris 11 Express: Zones

Several of the videos are in multiple parts so I just link to the whole blog:

Monday, November 15, 2010

Oracle Solaris Express 2010.11

A little summary of what you can expect the first release of Solaris 11, Solaris 11 Express 2010.11.

There is nothing drastically different from OpenSolaris, Solaris 11 Express is a rebranded later build with fixed bugs and new features, the build is snv_151a. A result of this is that it is possible to upgrade from OpenSolaris to Solaris 11 Express, there is a chapter about that in the release notes here.

The list of new features is long, personally I most look forward to features in ZFS such as crypto, zones integration with crossbow and the new network administration model with ipadm. For my laptop and desktops the upgraded Xorg and Gnome will be most welcome.

The text based installer is now available for both x86 and SPARC, it is something that has been obviously missing for server installations since the first release of OpenSolaris ( there have been previews ).

A long list I put together of new features, drivers etc between the last publicly available build of OpenSolaris (134) to build 146 it makes a nice overview of all the changes that have happened since the almost released 2010.0[23456].
Lots of enhancements, no build available

As I wrote in my last post Solaris 11 Express is already available for download and it is a Oracle supported release:

"Oracle Solaris 11 Express is fully tested and supported on a variety of SPARC and x86-based systems from Oracle, including the Exadata Database machine and Exalogic Elastic Cloud, and other hardware platforms from 3rd party vendors."

I am missing a way of using Solaris 11 Express for personal, non-profit or even startup companies without buying a license agreement. This is most true for people who want to give the Solaris 11 a try but might want to put anything from a personal web-server to the family albums on the same host. The license agreement says you are allowed to use it for the following:

"only for the purpose of developing, testing, prototyping and demonstrating your applications, and not for any other purpose."

Oracle will host a live Solaris 11 Express webcast on the 7th of December here.

A few earlier posts on Solaris 11 Express:
Solaris 11 Express 2010.11 and ZFS
ZFS crypto integrated

The Oracle Solaris 11 Express Overview with link to Downloads and everything you need to get started: Oracle Solaris 11 Express

OpenSolaris build 134b

A bit ironic but as Oracle released Solaris 11 Express, the build that was once supposed to become Opensolaris 2010.05 was also put out the gates. The opensolaris.org/release repository was updated to be used as a platform for upgrading existing OpenSolaris installations to Solaris 11 Express. You can now upgrade to this version anyway:

                      OpenSolaris 2010.05 snv_134b X86
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 28 May 2010

Solaris 11 Express released!

I will post updates to this meanwhile it's available for download here:


What's new documentation is here:

Tuesday, November 9, 2010

Notes from Solaris Summit, LISA 2010

There was a lot of good recapitulation on features that has been available in OpenSolaris that will be part of Solaris 11 as well as some new things.

Fist of all, Solaris 11 Express is coming any day now, they mentioned weeks and days as the timeframe for the release which I suspect will be build snv_151a (or possible another respin). Given the internal tag of the release 2010.11 they should at least be aiming for a November release.

There was a lot of focus on the integration of zones with different core components of Solaris, IPS was discussed in regard to zones and updates and the synergies of integrating zones with crossbow. In the future (post S11 Express) there will also be possible to control storage inside of zones in a good way, if allowed utilities like format and newfs should be usable on devices dedicated to the zone. The storage focus for also include Zones as NFS servers.

Crossbow will be optimized for NUMA localities, a new framework will be available, NUMA I/O and will at least be used by infiniband besides the networking stack.

Solaris 11 will be able to handle the CPU resources dedicated for a zone even for the network utilization. With crossbow threads can be bound to virtual interfaces and these can be kept in sync with the ones dedicated for the zone making the zone an even more isolated environment.

The rewrite of the virtual memory subsystem (VM2.0) is alive and will be delivered incremental in the Solaris 11 lifetime. This should pave way for thinks like virtualized memory which can be dedicated to zones (Really dedicated RAM and swap to the zone, not the capping we have today) as well as power management of memory. A description of VM 2.0 by Blake A. Jones:

"VM2 is a project to redesign the Solaris virtual memory system around modern computer architectures. The core of the current VM system was designed in 1985-86, when Sun's large computers had 4 megabytes of RAM, one simple CPU with a simple MMU, very few disks, and no NUMA or power management. A couple decades of Moore's Law and many many billions of dollars of hardware development later, things look a bit different. Obviously the software has evolved to deal with these hardware changes, and it's a testament to the original design that it's performed for this long. But the VM system has developed a reputation for being hard to understand, as 20 years' of accreted development will tend to do, and having most of the VM interfaces operate on lists of small fixed-size pages has made it hard to do more significant innovation."

There also was some ZFS news, but that has been covered by this blog before: ZFS crypto, RAID-Z/mirror hybrid allocator and some stuff available in the external source before it closed. One thing new was some numbers on performance impact of crypto and speed enhancement with the new raidz allocator:

"Actual costs for ZFS encryption is 7% for random I/O,
and 3% for sequential I/O

ZFS RAID-Z mirror allocator - preliminary data is that
some workloads are 2 to 4 times faster for things like
directory searching"

There is much more interesting stuff in the videos and presentation slides, both are still available here:

ZFS Crypto integrated
Solaris 11 Express 2010.11 and ZFS
Zones should be able to be NFS servers

Live Solaris 11 streams from Lisa 2010 tonight

There will be a Solaris 11 live stream from LISA 2010 tonight. The topics will include the new packaging system IPS, ZFS, Network virtualization (crossbow, can they still call it that externally?), Solaris zones and security. The selection of speakers looks promising.

Here is the agenda (TZ: PTS, UTC/GMT -8):
  • 9:00 - 9:30 am - Introduction to Oracle Solaris 11 Express, by Markus Flierl
  • 9:30 - 11:00 am - Image Packaging System, by Bart Smaalders
  • 11:00 am - 12:30 pm - Deploying Oracle Solaris 11 in the Enterprise, by Dave Miner
  • 12:30 pm - 1:30 pm - LUNCH BREAK
  • 1:30 pm - 2:30 pm - Advances in Solaris Networking with Crossbow and Beyond, by Nicolas Droux
  • 2:30 pm - 3:00 pm - Oracle Solaris Containers in Oracle Solaris 11 Express, by Dan Price
  • 3:00 pm - 3:15 pm - BREAK
  • 3:15 pm - 4:15 pm - ZFS Features in Oracle Solaris Express, by Cindy Swearingen
  • 4:15 pm - 4:45 pm - New Security Features in Oracle Solaris 11 Express, by Glenn Faden
  • 4:45 pm - 5:30 pm - Deploying Applications Using SMF and Other Solaris 11 Features, by Liane Praza
  • 5:30 pm - 6:30 pm - Beer and Snacks Reception for ALL

Update: Here is a direct link to the stream: www.ustream.tv/channel/solaris-summit-at-lisa10

Oracle Solaris Summit will be streaming live at LISA 2010!
Large Administration System Admministartion (LISA) 2010

Thursday, November 4, 2010

McNealy about Java and Capitalism

Scott McNealy held a keynote at the PostgreSQL database conference. He discussed the Sun acquisition, Java and capitalism. He highlights the points that Sun was trying to do "good" by sharing and working with the community and make money, while Oracle is focuses more on the money part and not so much on sharing.

"Sun was a company of "good capitalists," McNealy said -- implying virtuous as well as effective -- while Ellison is a "great capitalist." The end result is that "he's there and I'm here," meaning Ellison is still running Oracle and McNealy is without a job."


""Sharing's not Larry's middle name," McNealy said. But developers can always take the code for open-source projects and fork it into other projects, something he predicted will happen with both OpenSolaris and Java."

Read the full article at computerworld:
Scott McNealy talks Java, Oracle and Larry Ellison