illumos distros update

OpenIndiana has finally been updated (to oi_151a8) and with it a few new features and of course many bug fixes. All this should also be true for OmniOS r151006 which has been out for a few month already (and probably SmartOS). Here are a few new features I noticed more than others while scanning the illumos commits since the last update.

3035 LZ4 compression support in ZFS and GRUB
3236 zio nop-write
3137 L2ARC compression
3122 zfs destroy filesystem should prefetch blocks
3805 arc shouldn't cache freed blocks
3561 arc_meta_limit should be exposed via kstats
3537 want pool io stats
3507 Tunable to allow block allocation even on degraded vdevs
749 /usr/bin/kstat" should be rewritten in C

Bug fixes
3422 zpool create/syseventd race yield non-importable pool
3606 zpool status -x shouldn't warn about old on-disk format

Hardware support
3815 AHCI: Support for Marvell 88SE9128
3797 AHCI: Support for ASMedia ASM106x
3500 Support LSI SAS2008 (Falcon) Skinny FW for mr_sas(7D)
3178 Support for LSI 2208 chipset in mr_sas
3408 detect socket type of newer AMD CPUs
3701 Chelsio Terminator 4 NIC driver for illumos

More in depth information on the two larger enhancements to ZFS can be found here:
LZ4 Compression
L2ARC Compression

ZFS Analytics

While woking with ZFS performance I created a dashboard to get a good overview with lots of different statistics. It's powered by Dtrace, python and graphite. There is a high level of detail but still easy to correlate different statistics.

It feels almost like fishworks analytics lite but without advanced features such as drill-down and heat maps. An example from a box running OpenIndiana:

You get a good view of how the layers interact, the latency for reads in ZFS compared to reads in from the physical disks, average latency, maximum latency, average read size and see how much more data ZFS reads form prefetch including hit rate etc.

I based this on the iomon dtrace script with some glue to send it into graphite, I also added ARC statistics and CPU/Network statistics. ( There is a iomon-graphite effort available on the web but that did not give me correct statistics and did not include things like CPU and network utilization ).

iomon.d

SPARC T5 and M5 systems released

Oracle have announced new systems based on the new T5 and M5 processors. The T5 has doubled the number of S3 cores from the T4 and increased the clock frequency to 3.6GHz.

The M5 processor is also based on the S3 core (rebranded M4) clocked at 3.6Ghz but is has 6 cores and 48MB L3$. The M systems supports up to 32 M5 processor so a fully configured systems will have 192 cores and 1536 strands (hardware threads). The M5-32 have 32TB of memory in a single system.

Old and existing processors for reference:
T3 16 cores @ 1.65GHz 6Mb L2$ 1-4 socket systems PCIe 2.0
T4 8 cores @ 3.0GHz 4MBL3$ 1-4 socket systems PCIe 2.0

New processors:
T5 16 cores @ 3.6GHz 8MB L3$1-8 socket systems PCIe 3.0
M5 6 cores @ 3.6Ghz 48MB L3$ 32 socket system PCIe 3.0

Oracle claims 2.3x performance gain compared to the T4 with increased single-thread performance.

 All new systems both T5 and M5 supports LDOM (Oracle VM for SPARC).

Did everyone read that the M5-32 Supports 32TB of memory? No wounder they had to rewrite the virtual memory subsystem in Solaris 11.1

SPARC Servers

T5,M4 and Athena support in S10U11

The kernel patch for Solaris 10 update 11 show support for several new platforms: SPARC T5, SPARC M4 and the Fujitsu Athena (SPARC64-X).

7086173 Solaris support for SPARC M4 platforms
7086179 Solaris support for SPARC T5 platforms

7124696 Solaris support for Athena processor

7142242 Solaris support for Fujitsu's SPARC64-X Athena platforms

The Oracle SPARC T5 and M4 processors are unannounced when I write this but the T5 are expected early 2013.

getupdates.oracle.com/readme/147147-26

LDOM 3.0, T5 and M10

The release notes for LDOM 3.0 confirms what we already knew, logical domains will be supported not only on upcoming SPARC T5 systems but also on the new generation for Fujitsu SPARC Server, M10.

"To enable all the OracleVM Server for SPARC 3.0 features, you must run the required 

system firmware versions on the following platforms:

UltraSPARC T2 server Run at least version 7.4.2 of the system firmware.
UltraSPARC T2 Plus server Run at least version 7.4.2 of the system firmware.
SPARC T3 server Run at least version 8.2.1.b of the system firmware.
SPARC T4 server Run at least version 8.2.1.b of the system firmware.
SPARC T5 server Use any installed version of the system firmware.

This firmware is preinstalled on the SPARC T4 and SPARC T5 servers. The required firmware
for Fujitsu M10 systems is preinstalled on your system. For information about the required
Oracle Solaris OS version, see “Required and Recommended Oracle Solaris OS Versions” on
page 12."

The new SPARC M10 systems can now be seen on Fujitsus site (English translation).
From the Register:  Fujitsu launches 'Athena' Sparc64-X servers in Japan

Solaris 10 1/13 released

The last update of Solaris 10 has now been released, update 11. The final name of the release is Solaris 10 1/13 and not 8/12 as plans earlier indicated. This is probably due to the upcoming SPARC T5 systems, to have the releases close in time. Supports the latest and greatest X86 systems, SPARC M-series, SPARC T4/T5 while still working fine on my 15 year old Ultra 2.

• Live Upgrade and Zone Preflight checkers 
• Install and boot from iSCSI 
• pkgdep command
• SSH, SCP, and SFTP Speed Improvements 
• USB 3 Support 
• Support for new hardware
• Enhanced FMA (AMD,Intel)
• Samba 3.6.8 with SMB2 support

Solaris 10 has full support for another five years and extended support until 2021.

Oracle Solaris 10 Downloads
Oracle Solaris 10 1/13 What's New

LDOM 3.0

Oracle have release Oracle VM for SPARC 3.0 (LDOM).

• Enhances the resource affinity capability to address memory affinity.
  
• Retrieves a service processor (SP) configuration from the SP into the bootset, which is on the control domain.
  
• Enhances the domain shutdown process. This feature enables you to specify whether to perform a full shutdown (default), a quick kernel stop, or force a shutdown.
  
• Adds Oracle Solaris 11 support for Oracle VM Server for SPARC Management Information Base (MIB).
  
• Enables a migration process to be initiated without specifying a password on the target system.
  
• Enables the live migration feature while the source machine, target machine, or both, have the power management (PM) elastic policy in effect.
  
• Enables the dynamic resource management (DRM) feature while the host machine has the PM elastic policy in effect.

I hope this release removes the restriction that prevented dynamic reconfiguration of resources after a live migration.
This release also seems to have been tested on upcoming SPARC processors: "7159011 M4/T5: Migration fails initialization on Logical Domains Manager startup" including Fujitsu Athena: "RFE: Cross CPU Migration support between Athena server and T series".

Oracle VM Server for SPARC Documentation

Solaris 11.1 available for download

Solaris 11.1 is now available for download from Oracle.

The usual SPARC/X86 [text|live|usb|AI] images are available as well as repository images. There are also pre-upgrade repository images available for those of you who are upgrading from 11/11 and have not upgraded to a recent SR or do not have a support contact.

Oracle Solaris 11 Download

Solaris 11.1 announced

Solaris 11.1 have been announced and will be released later this month. It is the first update of Solaris 11 since it release november last year. It contains a few interesting features, I've only list a few, over 200 projects have integrated into this release.

General

• New Virtual memory subsystem (VM2.0 or parts of it)
  Scales beyond 100TB, predicts and adapts to page demand, higher performance
• RSyslog
• USB3 support
• Install on UEFI/4K
• Interactive install on iSCSI
• FedFS

Zones

• Parallel zone update
• Must faster LOFI
• FS statistics per zone
• Physical to virtual Solaris 10 migration
• Better support for shared storage
• Remote Administration Daemon
  Secure zone administration with C, Java, Python API
• VNIC config in zone XML
• Faster install and attach

Security

• ASLR
• OpenSCAP Security Compliance Checking and reporting tool
• Audit remote server

Networking

• Data Center Bridging (DCB) IEEE902.1Qaz
• Link aggregation span across switches
• VNIC migration
• Edge Virtual Bridning (EVB) support
• High performance SSH

X86

• GRUB2
• UEFI support
• Improved hardware support

I asked the Solaris panel on Oracle Openworld about memory set for zones, it is not part of this release but might be implemented now that the initial part of VM2.0 is implemented.

Solaris 11.1 Whats's new

SPARC T5, M4 and SPARC64-X

Short summary of SPARC processor information that was disclosed at Oracle world, in the near future Oracle will release two different SPARC processors and Fujitsu will release a new SPARC64 processor with support for LDOM.

SPARC T5 (early next year)

• 16 Cores 128 threads 
• 28nm 
• 25% increased thread performance to T4 
• 2.5x throughput compared to T4 
• Scales from 1 to 8 processors 
• PCIe Gen 3 
• 8MB L3 cache
• LDOM virtualization (as with all previous T-series)
• Solaris 10 update 11 or Solaris 11.1

SPARC M4 (next year)

• 6 cores 48 threads 
• Scales to 32+ sockets 
• 48MB L2 cache 
• 28nm 
• 3.6GHz 
• 5-6x performance per socket compared to M-series 
• LDOM virtualization
• 32TB+ memory configurations
• Solaris 11 only (but S10 support in LDOM)

SPARC64-X

• LDOM virtualization 
• 16 cores 32 threads 
• 24MB L2 Cache 
• 3 GHz 
• On Chip  DB floating point 
• Crypto acceleration 
• Runs both S10 and S11 in lab

Oaktable world and OpenWorld 2012

I am attending both Oracle OpenWorld an ZFS Day/Oaktable world and will post updates as soon as I get some spare time or at the latest early next week.

It was great to talk and listen to the Joyent/Nexenta/illumos guys at Oaktable world.

Oaktable world
Oracle OpenWorld

Oracle forces wesunsolve to close

In another blow against the community and the people who work with or/and have interest in their products Oracle now has forced wesunsolve.net to close.

In the last two years it has been of tremendous value for administrators who find their support site hard to navigate and the a good overview of patches and updates.

I see nothing Oracle can gain by doing this, no patches where available only metadata with links to their own support site for downloads ( if you have an account with access to the patches ).

Since they already done a much, much worse thing by closing the OpenSolaris source this comes as no surprise. The way companies treat innovation and community efforts are important when you chose a operating system or database engine. Tell your sales representative what you think about these things.

A huge thanks to the people who put time and effort into making wesunsolve.net, it was very useful and will be missed.

wesunsolve.net

Solaris 10 10/12 and T5-8

Oracle keeps quiet about upcoming Solaris release and their features. I've however figured out the name (and planned release month) for the possibly last Solaris 10 update. It seems like s10u11 will be named Solaris 10 10/12 indicating a October release.

Not much is known about this release more than that it will probably feature a ZFS tech refresh, fully integrate OCM and live upgrade enhancements. It could possibly also support the new T5 processors since they might be quite similar to the current T4 processors except for the doubling of cores.

I have also seen fragments of information indicating that Oracle was running Solaris 11 Update 1 on SPARC T5-8 machines as early as March.

Upcoming SPARC CPUs

The upcoming Hot Chips symposiums "Big iron" session will feature two future SPARC processors:

"SPARC64 X; Fujitsu’s new generation 16 core processor for the next generation UNIX servers

16-core SPARC T5 CMT Processor with glueless 1-hop scaling to 8-sockets"

The SPARC T5 is expected to be built using 28nm technology and double the number of cores compared to the current T4 processor. The Sun Oracle server line should also include a 8 processor version, T5-8 which will then be have four times the number of cores (128) compared to the current T4-4 (32).

This session will be held August 29, hopefully more information will surface afterwards. Otherwise it would be a safe bet to say that we will know more about the SPARC T5 after Oracle OpenWorld in October.

The Register has an article about both the T5 and the M4: Drilling into Oracle's performance boasts.

Joyent presentations @ FISL 13

Joyent had a several speakers at the FISL 13 conference and presentations/slides are now available online.

Bryan Cantrill, Corporate Open Source Anti-Patterns: Doing It Wrong
video, slides
Bryan speaks his mind about corporate open source patterns with insights from Sun, the OpenSolaris project and Joyent. He does a bad job hiding what he thinks of Oracle ;)

Brendan Gregg, Performance analysis, the USE method
slides, video
Brendan on performance analysis using the USE method with good examples.

Update: Added Brendans video.

Lots of packages for SmartOS, soon for OpenIndiana

There is now a huge package repository available for illumos-based distributions, initially a dependency prevents it from running on OpenIndiana but that is being fixed:

9000 packages available for SmartOS and illumos

The packages contains a current PostgresSQL (9.1.3), MySQL, Apache, Ruby 1.9.3, Python 3.2.3 both with lots of modules plus many other useful packages.

All should work on SmartOS and when fixed for OpenIndiana this slightly modified procedure (without sudo and install gtar first) should work, as root:

# pkg install gnu-tar
# curl http://pkgsrc.smartos.org/packages/SmartOS/2012Q2/bootstrap.tar.gz | (cd /; gtar -zxpf - )
# pkgin -y update
# pkgin avail | wc -l
    9401
# pkgin search ...
# sudo pkgin -y install

I'll update this entry as soon as it works for OpenIndiana.

Good summary of enhancements in illumos ZFS

I found a good summary of enhancements to the free ZFS implementation in illumos:New features in open source ZFS

Also well worth a read is is Matt Ahrens post about the performance of the new async destroy: Performance of zfs destroy.

OpenIndiana updated (oi_151a5)

A new pre-stable release of OpenIndiana was released a few days ago (oi_151a5), the fifth since the initial illumos-based oi_151a development release in September. Besides bugfixes and minor enhancements the this new release also includes a refresh of the illumos code base which includes a few new noticeable features:

• ZFS feature flags 
• ASynchronous destruction of ZFS file systems 
• ZFS send progress output

There have also been quite a few userland updates, all is documented in the release notes including a list of CVE-fixes:

OI_151a_prestable5 Release Notes

Update or download images here.

LDOM 2.2 released

A new relase of LDOM, currently known as Oracle VM for SPARC has been released. One of the major features of the new release is the ability to do live migration between SPARC T2,T3 and T4 processors. Enabling this features does however have some performance impact:

From ldm(1M):
"cpu-arch=generic|native

Specifies one of the following values:

generic uses common CPU hardware features to enable a guest domain to perform a CPU-type-independent migration.

native uses CPU-specific hardware features to enable a guest domain to migrate only between platforms that have the same CPU type. native is the default value.

Using the generic value might result in reduced performance compared to the native value. This occurs because the guest domain does not use some features that are only present in newer CPU types. By not using these features, the generic setting enables the flexibility of migrating the domain between systems that use newer and older CPU types."

Another major feature is SR-IOV support, which can enable bare metal I/O performance for logical domains, read more here: SR-IOV feature in OVM Server for SPARC 2.2

There a new set of patches available to update the system firmware to 8.2.0 which is needed for the new features.

Announcing Oracle VM Server for SPARC 2.2 Release

illumian available for download

A new illumos based distribution is available for download, illumian. illumian is the based on the APT packaging system and is a successor to Nexenta Core Platform (NCP) which was built by nexenta using the OpenSolaris source and APT packages.

The next version of NexentaStore (4.0) should also be built upon illumian, previous versions was built on NCP.

There is currently one image available for server text install on X86 hosts.

illumian.org