Solaris 11 / SPARC News

Here is a good summary of a recent online forum, "Solaris 11: What's new since launch?"

Solaris 11 Update 1 (late this year)

• Updated Virtual memory subsystem
This is probably what has been known as vm 2.0 earlier
•  Faster Solaris 11 updates with improved python performance 
•  Already running on the upcoming T5/M4 SPARC(R) chips
• VNIC configuration switch hosts with their zones

There are also hints on what future Solaris 11/hardware updates might bring

• Hotpatching similar to KSplice (Remember DUKS in Solaris 8?)
• Offloading of compression and Oracle arithmetics to CPU besides crypto
• Schedulers for DB or JVM workloads

Summary: What's new with Solaris 11 since the launch?

Fast upgrades, with zones

I've been working with a sub-optimal solution with zones for a few years. The zones are located on SAN and movable between hosts which is good, but they are also located on an a filesystem which is not ZFS and not even UFS. This have made upgrades and patching terribly hard and slow, especially with more than 20 zones per host. All local zones earlier had to be upgraded at the same time as a global zone and all they had to be down for the entire operation (sometimes over 8 hours).

After a testing upgrade-on-attach for a while and combining it with Turbocharged SVr4 packages I now have a solution which brings the downtime to under one hour for an entire upgrade including local zones.

Everything would have been even easier if we had all root filesystems on ZFS. Since migrating several terabytes of data at the same time as the upgrade was not an option this was a good solution:

• Detach all local zones
• Add Live upgrade and Turbocharge patches
  (119254-70,121428-13,121430-40,124630-28 or later, included in S10u8)
• lucreate to a ZFS rpool
  
• luupgrade to S10U10 and add additional patches
  
• luactivate and reboot
  
• $(zoneadm attach -U) all zones in parallel

This solution is dependent on good I/O and perhaps even a separate disk/LUN for every zone root plus sufficient CPU resources on the system.

There you go, Solaris 10 8/11 is ready for the SPARC T4!

Patches for "Turbo-Charging SVR4 Package Install" are now available

Joyent illumos tree

Joyent which builds their cloud service on zones and ZFS just announced a new repository with patches against illumos that provides new features and bug fixes:

"Some highlights of the illumos-joyent tree include:
* ZFS I/O fair-share scheduling for zones
* the Joyent brand, which can be used as a template for other non-SysVR4 or IPS zone brands
* Reintroduction of sparse zone images
* Crossbow vnics on demand for zones & non-unique vnic naming (unique per zone, not per system)
* svcs enhancements ( svcs -Z/-z for interrogating zone services, -L for outputting log files directly (no more ls /var/svc/log | grep... ))
* vfsstat and iostat tweaks and ziostat, iostat(1M) for ZFS I/O
* more per-zone IO kstats
* the zonemon utility for zone kernel state troubleshooting
* DTrace enhancements such as llquantize"

I'll guess many of these changes will find it's way into illumos in a near feature. The whole mail from John Sonnenschein can be read here.

Joyent have always been an interesting company and have recruited very talented people after Sun settled. Both Bryan Cantrill and Brendan Gregg are now employed by Joyent so I would not be surprised if they will become one of the larges contributors to illumos.

illumos-joyent at github

Introduction to Virtual Networks

Project crossbow, the virtualized network stack is a gem of [Solaris11|Nexenta|OpenSolaris]. Jeff Victor has written a good three part introduction to the gains of virtual networks and examples on to how to use Crossbow together with zones, well worth a read:

Virtual networks part 1
Virtual networks part 2
Virtual networks part 3

Quick Solaris 11 Express walkthrough

This is a quick command/output walkthrough of some example administrative tasks in Solaris 11 Express. You can just skim thought it or run the command yourself on a test installation. It uses zfs-crypto, deduplication, zones, crossbow, ipadm, zonestat and pkg.

Create a alternate boot environment by cloning the current filesystem state for an quick and easy rollback option:
# beadm create initial

Set up static IP and add a persistent route:
# ipadm create-addr -T static -a local=10.0.10.242/24 bge0/v4
# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
bge0/v4 static ok 10.0.10.242/24
lo0/v6 static ok ::1/128

# route -p add default 10.0.10.1
add net default: gateway 10.0.10.1
add persistent net default: gateway 10.0.10.1

Add mail notifications on hardware/FMA and SMF state changes:
# pkg install smtp-notify
# svccfg setnotify -g from-online,to-maintenance mailto:admin@somehost.com
# svccfg setnotify problem-diagnosed,problem-updated mailto:admin@somehost.com

Install some utilities from the network repository:
# pkg install terminal/screen mercurial nmap

Create an encrypted dataset for the secret project:
# zfs create -o encryption=on -o mountpoint=/vault rpool/vault
Enter passphrase for 'rpool/vault': ********
Enter again: ********

Add a dataset with deduplication:

zfs create -o dedup=on -o mountpoint=/export/vbox rpool/vbox

Add a NFS/SMB share area and ignore the ZIL for asynchronous writes in favor of performance(unsafe):
# zfs create -o mountpoint=/export/share rpool/share
# zfs set sharesmb=on rpool/share
# zfs set sharesmb=name=share rpool/share
# zfs set sharenfs=on rpool/share
# zfs set sync=disabled rpool/share

Create a virtual switch with crossbow and two virtual interfaces for zones with bandwidth management and assigned to different CPUs:
# dladm create-etherstub etherstub01
# dladm create-vnic -l etherstub01 vnic_zone01
# dladm create-vnic -l etherstub01 vnic_zone02
# dladm set-linkprop -p maxbw=10M -p cpus=3 vnic_zone01
# dladm set-linkprop -p maxbw=10M -p cpus=4 vnic_zone02

Configure and install zone capped to 50% of a CPU located on ZFS and using the virtual interface:
# zfs create -o mountpoint=/zones -o compression=on rpool/zones

# zonecfg -z zone01
zonecfg:zone01> create
zonecfg:zone01> set zonepath=/zones/zone01
zonecfg:zone01> set ip-type=exclusive
zonecfg:zone01> ad capped-cpu
zonecfg:zone01:capped-cpu> set ncpus=0.5
zonecfg:zone01:capped-cpu>end
zonecfg:zone01> add capped-memory
zonecfg:zone01:capped-memory> set swap=2G
zonecfg:zone01:capped-memory> end
zonecfg:zone01> add net
zonecfg:zone01:net> set physical=vnic_zone01
zonecfg:zone01:net> end
zonecfg:zone01> commit

# zoneadm -z zone01 install
A ZFS file system has been created for this zone.
Publisher: Using solaris (http://pkg.oracle.com/solaris/release/ ).
Image: Preparing at /zones/zone02/root.
Sanity Check: Looking for 'entire' incorporation.
Installing: Core System (output follows)

# zoneadm -z zone01 boot
# zonestat 5

Collecting data for first interval...
Interval: 1, Duration: 0:00:05
SUMMARY                    Cpus/Online: 4/4   Physical: 8063M    Virtual: 11.8G
                    ----------CPU---------- ----PHYSICAL----- -----VIRTUAL-----
              ZONE  USED %PART  %CAP %SHRU  USED   PCT  %CAP  USED   PCT  %CAP
           [total]  0.76 19.2%     -     -  946M 11.7%     - 1960M 16.2%     -
          [system]  0.10 2.54%     -     -  774M 9.60%     - 1816M 15.0%     -
            global  0.66 16.6%     -     -  133M 1.65%     -  113M 0.94%     -
           zone01  0.00 0.00% 0.03%     - 38.1M 0.47%     - 30.0M 0.24% 1.46%

Enjoy the new faster scrub/resilver that should have less of impact on other workloads to the pool. It also has a more detailed output:
# zpool scrub rpool
# zpool status rpool
pool: rpool
state: ONLINE
scan: scrub in progress since Sat Nov 20 02:21:51 2010
3.22G scanned out of 19.0G at 35.8M/s, 0h7m to go
0 repaired, 16.95% done
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0

Check if any updates are available:
# pkg update -n
No updates available for this image.

Notes from Solaris Summit, LISA 2010

There was a lot of good recapitulation on features that has been available in OpenSolaris that will be part of Solaris 11 as well as some new things.

Fist of all, Solaris 11 Express is coming any day now, they mentioned weeks and days as the timeframe for the release which I suspect will be build snv_151a (or possible another respin). Given the internal tag of the release 2010.11 they should at least be aiming for a November release.

There was a lot of focus on the integration of zones with different core components of Solaris, IPS was discussed in regard to zones and updates and the synergies of integrating zones with crossbow. In the future (post S11 Express) there will also be possible to control storage inside of zones in a good way, if allowed utilities like format and newfs should be usable on devices dedicated to the zone. The storage focus for also include Zones as NFS servers.

Crossbow will be optimized for NUMA localities, a new framework will be available, NUMA I/O and will at least be used by infiniband besides the networking stack.

Solaris 11 will be able to handle the CPU resources dedicated for a zone even for the network utilization. With crossbow threads can be bound to virtual interfaces and these can be kept in sync with the ones dedicated for the zone making the zone an even more isolated environment.

The rewrite of the virtual memory subsystem (VM2.0) is alive and will be delivered incremental in the Solaris 11 lifetime. This should pave way for thinks like virtualized memory which can be dedicated to zones (Really dedicated RAM and swap to the zone, not the capping we have today) as well as power management of memory. A description of VM 2.0 by Blake A. Jones:

"VM2 is a project to redesign the Solaris virtual memory system around modern computer architectures. The core of the current VM system was designed in 1985-86, when Sun's large computers had 4 megabytes of RAM, one simple CPU with a simple MMU, very few disks, and no NUMA or power management. A couple decades of Moore's Law and many many billions of dollars of hardware development later, things look a bit different. Obviously the software has evolved to deal with these hardware changes, and it's a testament to the original design that it's performed for this long. But the VM system has developed a reputation for being hard to understand, as 20 years' of accreted development will tend to do, and having most of the VM interfaces operate on lists of small fixed-size pages has made it hard to do more significant innovation."

There also was some ZFS news, but that has been covered by this blog before: ZFS crypto, RAID-Z/mirror hybrid allocator and some stuff available in the external source before it closed. One thing new was some numbers on performance impact of crypto and speed enhancement with the new raidz allocator:

"Actual costs for ZFS encryption is 7% for random I/O,
and 3% for sequential I/O

ZFS RAID-Z mirror allocator - preliminary data is that
some workloads are 2 to 4 times faster for things like
directory searching"

There is much more interesting stuff in the videos and presentation slides, both are still available here:
blogs.sun.com/video/entry/join_the_live_video_stream

ZFS Crypto integrated
Solaris 11 Express 2010.11 and ZFS
Zones should be able to be NFS servers

Live Solaris 11 streams from Lisa 2010 tonight

There will be a Solaris 11 live stream from LISA 2010 tonight. The topics will include the new packaging system IPS, ZFS, Network virtualization (crossbow, can they still call it that externally?), Solaris zones and security. The selection of speakers looks promising.

Here is the agenda (TZ: PTS, UTC/GMT -8):

• 9:00 - 9:30 am - Introduction to Oracle Solaris 11 Express, by Markus Flierl
• 9:30 - 11:00 am - Image Packaging System, by Bart Smaalders
• 11:00 am - 12:30 pm - Deploying Oracle Solaris 11 in the Enterprise, by Dave Miner
• 12:30 pm - 1:30 pm - LUNCH BREAK
• 1:30 pm - 2:30 pm - Advances in Solaris Networking with Crossbow and Beyond, by Nicolas Droux
• 2:30 pm - 3:00 pm - Oracle Solaris Containers in Oracle Solaris 11 Express, by Dan Price
• 3:00 pm - 3:15 pm - BREAK
• 3:15 pm - 4:15 pm - ZFS Features in Oracle Solaris Express, by Cindy Swearingen
• 4:15 pm - 4:45 pm - New Security Features in Oracle Solaris 11 Express, by Glenn Faden
• 4:45 pm - 5:30 pm - Deploying Applications Using SMF and Other Solaris 11 Features, by Liane Praza
• 5:30 pm - 6:30 pm - Beer and Snacks Reception for ALL

Update: Here is a direct link to the stream: www.ustream.tv/channel/solaris-summit-at-lisa10

Oracle Solaris Summit will be streaming live at LISA 2010!
Large Administration System Admministartion (LISA) 2010

Zones should be able to be NFS servers

NFS service in local zones have been frequently requested since the introduction of zones in Solaris 10, the change request for this even predates Solaris 10 by two years, it was filled in 2003. I've earlier mentioned that some works was being done to this including PSARC/2010/208. Now it seems that this has been implemented and will be integrated into internal build 154 of Solaris Next. This is hopefully in time for the first release of Solaris 11 Express which is expected next month.

Update: I was a bit quick on this one, build 154 should close in the end of next month, that does not work very well with a November release of Solaris Express, so it might be delayed or more probably it will not be part of the first Express release.

Zones should be able to be NFS servers, bugid 4964859
NFS Instances with NFS shares.
PSARC/2010/208

The evolution of Solaris 10 zones

Solaris zones or Solaris containers dependent on where you work and how you use it have been available in since the initial release of Solaris 10 in 2005. It has worked well for isolating workloads and creating virtual execution environments inside a single kernel with minimal overhead. There have however been a few bumps along the way, especially when it comes to patches and upgrades. With the new Solaris 10 9/10 I feel that most of the problems have been solved so here is a summary of the road behind us.

The upgrade scenario for zones was all but forgotten at the time of the first Solaris 10 release and the first update only supported upgrades when booting from DVD and no live upgrade support.

Sun however realized that this was quite a problem for both them and their customers, so project ZULU was created to solve the problem (Zones Upgrade Live Upgrade, PSARC/2006/167).

ZULU was delivered with Solaris 10 update 4 which was the first release to support real upgrades of zones including live upgrade, but ZFS was still not supported as underlying filesystem for zones and live upgrade did not know how to handle such zones.

ZFS is a key component for easy live upgrade of zones. With zones on ZFS there is no need to copy all OS/Zone instances to a separate filesystem, possible requiring additional disks and overhead of administration. Live upgrade creates a ZFS clone of the needed filesystems and uses them for the upgrade. The support for Zones on ZFS was delivered with Solaris 10 10/08 (Update 6).

Live upgrade was only part of the solution, if you where unable to use live upgrade or if you where forced to have customer applications down even with live upgrade you where still in trouble. The largest problem was the time it took to update a system with many zones, it could take days(!) to update a system with, lets say 30 zones on it. The upgrade/patching process was sequential and on top of that is was very time consuming. This has today been fixed by a couple of enhancements, we now have zones parallel patching that enables several zones to be patched in parallel. The package system have also been updated with "Turbo charged SVR4 packages" making it faster, especially for zone installations and upgrades.

Lastly there is the "update on attach" function making it possible to move zones to a updated machine (or detach the zones and upgrade the current host) then have them updated in parallel when attached to the system with the new OS/patch level. The "update on attach" functionality was delivered in two steps, the first one made sure the zone was in the minimum supported state for running on the host. The second step which came with the latest update of Solaris delivers something more like what most customers would want, all packages are updated which makes it more like an upgrade.

In short, you are implementing zones today, use Solaris 10 9/10, put all zones on ZFS and use Live Upgrade/Update on attach. This will make it easy to upgrade while minimize both the time for the upgrade and the downtime.

Short history of Zone features in Solaris updates:

Solaris 10 11/06 (Update 3)
Zone renaming
Zone move and clone
zone attach/detach
Privileges in zone configuration

Solaris 10 8/07 (Update 4)
Upgrades, Live upgrades (ZULU)
IP Instances (dedicated NIC/separate TCP/IP stack)
Resource setting for memory/CPU

Solaris 10 5/08 (Update 5)
Dry run of zone migration (zoneadm -n)
CPU caps for zones

Solaris 10 10/08 (Update 6)
Update on attach
Default router in shared stack
Full ZFS support for zones

Solaris 10 10/09 (Update 8)
Turbo-Charged SVR4 Packaging
Zones Parallel Patching

Solaris 10 9/10 (Update 9)
Zones P2V (Physical to Virtual)
Host ID Emulation
"Upgrade on attach"

Jerry's blog Zones P2V
Patching zones goes zoom
PSARC/2009/173 Turbo-charging SVr4 package install

zonestat implemented

We are not left out in the dark yet, the gates are still open and we can see changes made to the OpenSolaris source, the source that is going to be used for Solaris 11.

Today PSARC/2010/29 zonestat was implemented, it provides enhanced observability for Solaris zones. prstat -Z can provide a nice overview of all zones on a system but it does not tell you anything about any resource capping of memory or CPU resources. There have been a perl-script with the same name available for some time which is able to extract and present a nice summary of zones and their capping and usage. This is not only a reimplementation in C, it is also architectures the new observability feature into three layers, a collection daemon, an API and a command line utility. This case has requested patch binding and it should not be hard or include any significant risk to backport it to a Solaris 10 update.

Examples from the PSARC:

Example 1: Summary of cpu and memory usage over a 5 second interval:
    # zonestat 5 1
    SUMMARY
                -----CPU------------- ----PHYSICAL--- ----VIRTUAL----
           ZONE USED %PART %CAP %SHRU USED  PCT  %CAP  USED  PCT %CAP
        [total] 9.74   30%    -     - 7576M  23%    - 11.6G  24%    -
       [system] 0.28  0.8%    -     - 6535M  19%    - 10.4G  21%    -
         global 9.10   28%    -     -  272M 0.8%    -  366M 0.7%    -
      kodiak-ab 0.32  1.0%    -     -  256M 0.7%    -  265M 0.5%    -
      kodiak-dp 0.00  0.0%    -     - 77.6M 0.2%    - 71.1M 0.1%    -
kodiak-gjelinek 0.00  0.0%    -     - 58.7M 0.1%    - 59.3M 0.1%    -
     kodiak-edp 0.00  0.0%    -     - 53.0M 0.1%    - 58.9M 0.1%    -
 kodiak-johnlev 0.00  0.0%    -     - 51.9M 0.1%    - 57.4M 0.1%    -
  kodiak-jordan 0.00  0.0%    -     - 51.7M 0.1%    - 56.8M 0.1%    -
   kodiak-steve 0.00  0.0%    -     - 51.5M 0.1%    - 56.2M 0.1%    -
   kodiak-susan 0.00  0.0%    -     - 48.9M 0.1%    - 55.7M 0.1%    -
kodiak-batschul 0.00  0.0%    -     - 48.5M 0.1%    - 49.5M 0.1%    -
 kodiak-garypen 0.00  0.0%    -     - 46.3M 0.1%    - 49.5M 0.1%    -
     kodiak-rie 0.00  0.0%    -     - 22.7M 0.0%    - 49.4M 0.1%    -

PSARC/2010/291 zonestat

Notification enhancements

A common request from smaller shops or private users of Solaris is that they want email notifications when something is wrong with their system. This is most common with ZFS filers, they might not have any surveillance systems implemented but want to be sure to get a notification if something goes wrong with a zpool. A quick fix to this is of course a small script which can dump zpool or FMA errors over email but it is not a standardized or complete solution.

Now there have been a large putback in OpenSolaris that among other things delivers this functionality. FMA faults and SMF events can with these change be configured to deliver notifications over mail or SNMP. It also enables the fault management daemon in local zones so that they can deliver SMF state changes from within the zone. Using this framework you can also configure any of your own SMF services to deliver state change notification over any of these two protocols.

PSARC/2009/617 Software Events Notification Parameters CLI
PSARC/2009/618 snmp-notify: SNMP Notification Daemon for Software Events
PSARC/2009/619 smtp-notify: Email Notification Daemon for Software Events
PSARC/2010/225 fmd for non-global Solaris zones

NFS Instances/Zones with NFS shares

There is a PSARC awaiting approval for providing on of the most long standing enhancements requests for Solaris Zones: NFS Shares from within a local zone. This functionality will not be limited to NFS and will initially also support CIFS. It seems to be planned for Solaris Next only, so no backport to Solaris 10.

From the PSARC:
"Patch binding is requested; however, there are no plans to backport any of
the proposed changes.

The proposed changes are needed to support an NFS server in a non-global
zone; however, the changes are not specific to NFS. They would also be used
when enabling any file sharing protocol (ie. CIFS) server for non-global zones."

PSARC/2010/2080

Zone enhancements

Recently a couple of enhancement for zones have been integrated into the OpenSolaris source.

PSARC/2010/132 Delegated Administration for Zones
This will make it possible to delegate administration of local zones to user with not-superuser privileges. This will add three new privileges: solaris.zone.clonefrom for cloning zones, solaris.zone.login for zlogin and solaris.zone.manage for zone management.

PSARC 2010/144 lofi(7D) in non global zones
Will make it possible to use lofi, the loopback file driver inside a local zone.

Putback and a new build for 2010.05

The DTrace TCP/UDP providers discussed in this post have now been integrated into the OpenSolaris source. Another useful enhancement also made it's way into the source, PSARC/2010/181 PRIV_SYS_RES_BIND privilege. This will make it possible delegate permission to bind processes to specific processor sets from within the zone.

A new build of what is to become the next release of OpenSolaris is also probably finished or at least very close to finished, the second respin of build 134, 134b:

Author: david.comay@oracle.com
Repository: /hg/pkg/gate
Latest revision: 48706bcc893fc2c3ed76528eb4bc4b5dcb940f95
Total changesets: 1
Log message:
16087 resync repository to snv-134b

Solaris 10 and "upgrade on attach"

The next update of Solaris 10, probably due this fall, will include support for the new option for the update on attach function for zones discussed earlier. All packages that where to be installed in a new zone can be updated at attach instead of the bare minimum to get a supported zone running. The option of doing this today is to include all zones in the initial upgrade, this can take a very long time with many zones, even with the recent enhancements to the packaging system in Solaris 10 10/09. Using this feature the global zone can first be upgraded and then the zones can be attached one after another, possibly moved from another global zone to minimize the downtime.

Read my previous posts regarding the other updates to update 9, or Solaris 10 9/10 which is the planned name and release for the update. In short, large ZFS update, JDK 1.6, iSER and Firefox/Thundebird updates.

Solaris 10 9/10 and second ZFS refresh
First hints of Solaris 10 update 9
Desktop update for Solaris 10 update 9

Solaris upgrade on attach, sort of

Over a year ago when "update on attach" became available for zones in Solaris 10 I suggested an enhancement for this feature. Update on attach today only updates packages that _must_ be in sync between the global and local zones, all others are left untouched. A nice use of this feature would be to first upgrade a global zone, then move local zones one by one and have them updated as they attached, but doing this now will leave many packages in an older state, and the global and local zones would be out of sync, if yet in a supported state.

Now for the good news: zone update on attach all suggests something very similar, all attached zones will be in the same state as a newly installed zone, not only updating packages with "SUNW_PKG_ALL_ZONES" set to true in the package. I have no idea when this will be implemented and/or available in S10, but it's something that could make life easier for people with large zone deployments. If it only could be released as a patch for the current Solaris update release, and not a patch as in you must patch your way to a whole new update.

From PSARC/2010/082:

"The current behavior of zones "update on attach" [1] with the "native"
brand using SVr4 packaging is to update the minimal set of packages
needed to make the zone usable.  This is specified with the -u option.
We have heard from many users that this is not meeting their needs
or expectations.  Instead, what they want is to update the maximal
set of SVr4 packages.  That is, they want to update the same set of
packages as would be installed in a newly created zone so that they
can compare an updated zone to a new zone and see that they are
the same [2].  This case adds a new attach option, -U, to the "native"
SVr4 branded zone so that users can use "update on attach" to update
all of the packages as would be installed into a new zone."