LDOM 3.0

Oracle have release Oracle VM for SPARC 3.0 (LDOM).

• Enhances the resource affinity capability to address memory affinity.
  
• Retrieves a service processor (SP) configuration from the SP into the bootset, which is on the control domain.
  
• Enhances the domain shutdown process. This feature enables you to specify whether to perform a full shutdown (default), a quick kernel stop, or force a shutdown.
  
• Adds Oracle Solaris 11 support for Oracle VM Server for SPARC Management Information Base (MIB).
  
• Enables a migration process to be initiated without specifying a password on the target system.
  
• Enables the live migration feature while the source machine, target machine, or both, have the power management (PM) elastic policy in effect.
  
• Enables the dynamic resource management (DRM) feature while the host machine has the PM elastic policy in effect.

I hope this release removes the restriction that prevented dynamic reconfiguration of resources after a live migration.
This release also seems to have been tested on upcoming SPARC processors: "7159011 M4/T5: Migration fails initialization on Logical Domains Manager startup" including Fujitsu Athena: "RFE: Cross CPU Migration support between Athena server and T series".

Oracle VM Server for SPARC Documentation

LDOM 2.2 released

A new relase of LDOM, currently known as Oracle VM for SPARC has been released. One of the major features of the new release is the ability to do live migration between SPARC T2,T3 and T4 processors. Enabling this features does however have some performance impact:

From ldm(1M):
"cpu-arch=generic|native

Specifies one of the following values:

generic uses common CPU hardware features to enable a guest domain to perform a CPU-type-independent migration.

native uses CPU-specific hardware features to enable a guest domain to migrate only between platforms that have the same CPU type. native is the default value.

Using the generic value might result in reduced performance compared to the native value. This occurs because the guest domain does not use some features that are only present in newer CPU types. By not using these features, the generic setting enables the flexibility of migrating the domain between systems that use newer and older CPU types."

Another major feature is SR-IOV support, which can enable bare metal I/O performance for logical domains, read more here: SR-IOV feature in OVM Server for SPARC 2.2

There a new set of patches available to update the system firmware to 8.2.0 which is needed for the new features.

Announcing Oracle VM Server for SPARC 2.2 Release

S11 and S10 inside LDOM 2.1 on T4

I've finally managed to get some time to play with live migration on a pair of SPARC T4-2. This post is not really adding any new information but is a walk-trough and initial reflections. I am going to continue to write LDOM instead of Oracle VM for SPARC Domains or something like that, even Oracle people still say LDOM and everyone else knows what is.

An interesting note is that I've used Solaris 10 as I/O and Control domain for the T4 servers while the LDOM is installed with Solaris 11 11/11. The disks for the LDOM are on LUNs over FC and MPxIO is used for multipathing from the I/O domain:

t42-01# dskinfo list-long
disk                                   size lun   use           p   spd type lb
c0t5000CBA015B85D98d0                  279G -     rpool         -   -   disk  y
c0t5000CBA015B93B90d0                  279G -     -             -   -   disk  y
c0t50002870000254901593534030832420d0  33G  0x0   -             4   4Gb fc    y
c0t50002870000254901593534030832420d0  33G  0x1   -             4   4Gb fc    y

Examples of migrating and reconfiguring the LDOM while running:

t42-01# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    16    16G      0.1%  12d 6h 37m
ldms11-01        active     -n----  5000    16    8G       0.0%  24m

t42-02# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    16    16G      0.1%  12d 1h 26m

ldms11-01:~$ uptime 
  5:11pm  up 19 min(s),  1 user,  load average: 0.00, 0.00, 0.01
henrikj@ldms11-01:~$ prtconf -v |grep Mem
Memory size: 8192 Megabytes
henrikj@ldms11-01:~$ psrinfo | wc -l
      16

t42-02# ldm set-vcpu 96 ldms11-01
t42-02# ldm set-memory 200G ldms11-01

t42-02# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    16    16G      0.1%  12d 6h 50m
ldms11-01        active     -n----  5000    96    200G     0.1%  24m

ldms11-01:~$ prtconf -v |grep Mem
Memory size: 204800 Megabytes
ldms11-01:~$ psrinfo | wc -l
      96

When performing a live migration between the two hosts, running processes and open network connections are as expected intact, there is only a small delay in the network traffic visible. For my initial tests the delay was about 10 ms.

The live migration seems to work very well and the T4 seems to perform several times faster than the T2/T3 for general workloads. The only thing missing is that LDOM 2.1 is unable to dynamically reconfigure memory and CPU resources for a domain after migration. A reboot is then required, hopefully this will be fixed in the 3.0 release, which people at Oracle Open World said would be focused on removing current limitations (including migration between different types of sun4v processors).

LDOM supported by Oracle

Oracle VM for SPARC is finally supported by Oracle, you can now partition a system with with logical domains and license hardware available to the domain. This is something Oracle customers have been wanting for a while now and I must say I believed it would have been solved the day Oracle acquired Sun. Nevertheless it's now finally supported at the same time the first T4 systems are being delivered to customers, which must be considered good timing.

Server/Hardware Partitioning
Important change regarding OVM/SPARC and Licensing

LDOM 2.1 with Live migration

Oracle VM 2.1 (previously LDOM, Logical domains) was released yesterday, it finally has the ability to perform live migration of domains, much like VMWare vMotion. It is supported on T2 and T3 servers, it should also work on the T4 when it's released this fall (or at least an updated version).

From the press release:

• Live migration: Enables users to migrate an active domain to another host machine while maintaining application services to users. Live migrations are as simple as point and click using Oracle Enterprise Manager Ops Center’s console.
• Secure, encrypted migration included: On-chip cryptographic accelerators deliver secure, wire speed encryption capabilities for live migration – without any additional hardware investments.
• Dynamic Resource Management (DRM) between domains: Ensures that domains running the most important workloads get priority for CPU access over domains with less critical workloads.
• Increased maximum number of virtual networks per domain: Permits a dramatic increase in external access to domains.
• Lower-overhead, higher scalability networking for Oracle Solaris 11 Express: Allows virtual network devices to use shared memory to exchange network packets, enabling improved performance and scalability.
• Support for Virtual Device Service Validation: Immediately validates the name and path for a specified network device or virtual disk, greatly reducing the risk of incorrectly configured I/O.
• Integrated Dynamic Reconfiguration (DR) of Cryptographic units and virtual CPUs: Cryptographic units and CPUs are dynamically reconfigured together to simplify operations and ensure consistent performance.
• Enhanced Management Information Base (MIB): Enables the SNMP MIB to use the latest Logical Domains Manager XML interface, permitting third party management software to access the new features and resource properties.
• P2V tool enhancements: Bring more flexibility to quickly convert an existing SPARC server running Oracle Solaris 8, 9 or 10 into a virtualized Oracle Solaris image to run on SPARC T-series servers."

Oracle Announces Latest Version of Oracle VM Server for SPARC
Oracle VM Server for SPARC 2.1 Release Notes

LDOM 2.0 released at Open World

Oracle today released LDOM 2.0 as Oracle VM Server for SPARC 2.0. This new released features better resource and power management but most importantly it now finally supports dynamic reconfiguration of memory. It should also support more flexible migration of domains in cooperation with Solaris (Cooperative guest migration) but it does not, as expected, seem to be the full live migration that we are eagerly awaiting. Oracle Solaris 11 Express is also mentioned as a possible OS for a guest domain, no news that it will work but interesting that it's mentioned. An the new SPARC T3 processor is of course supported in this new release.

Oracle Unveils Oracle VM Server for SPARC 2.0

Oracle Open World 2010

The first Open World since Oracles acquisition of Sun is here. Oracle likes focus their releases around Open World and we have already seen a new version of Solaris 10, 9/10. I guess that they will release SPARC T3, LDOM 2.0 (Or Oracle VM Server for SPARC) and hardware/software updates to the S7000(FishWorks) the next few days. If we are lucky we might also get some more information on Solaris 11 Express.

I will post as soon as something interesting surfaces.

LDOM 2.0 and next Solaris update

It looks like LDOM 2.0 support will be included in the next Solaris update, update 9 probably planed for release around september. This will in practice mean that it will finally support dynamic reconfiguration of memory. It will also include cooperative domain migration support, which is a foundation for live migration, which at least early this year was scheduled for LDOM 2.1. This will be built on the next LDOM firmware, 7.2.10.

LDOM 2.0 will also support the next Niagara chip, the UltraSPARC T3 with 16 cores. When put together with the announcement that the LDOM team is hiring it sure looks like Oracle is investing in SPARC and LDOM technology.

Solaris Virtualization for Voice of Customer Tour (PDF)
FWARC/2009/452 HV APIs for cooperative guest migration