Cheap, fast and secure storage

This post if about how I protect important data at home. There are a lot of appliances out there with nice front ends but most of them do not store data on ZFS or something with the same level of data protection. Also, if you have anything but trivial space requirements they tend to get expensive fast and don't have the option to easily enhance performance by for example adding an SSD as file cache or adding other hardware. I've built my ZFS storage server based on an quad core AMD CPU, 8GB of ECC memory, a SAS HBA and a bunch of large SATA disks in hot-swap bays.

All my data now lives in this storage server protected against accidental delete, bit rot, disk failures and fire. I use NFS/CIFS for ordinary file data and iSCSI for my Aperture photo libraries and time machine backups.

All data is stored in one large raidz2 pool, so there are two parity disks allowing any two disks to fail without data loss. Since ZFS checksums all data I know it is intact when read and after bi-weekly data scrubs. The most important reason for using raidz2 is that disks have now become so big that there is now a real risk that there will be an unrecoverable read error during a resilver when all data is read, using another parity disk makes this highly unlikely. Snapshots makes it possible to do a quick rollback if any person or software should damage or remove data, this is also very useful when transforming large amounts of data with uncertain outcome. This will keep data safe from most user errors, disks errors and controller errors but fire and major user errors/sabotage (zfs destroy -r) could still make me loose data.

To avoid the later two scenarios I mark my most important datasets with a flag and a script streams them using ZFS send/receive to external disks over eSATA. The disks are then transferred to a second physical location. I can currently fit all critical data on one large SATA disk which makes this cheap and easy. I exclude ISO images and virtual machine disks that I only use for testing. A full backup of critical data takes about 3 hours today, that depends on the backup disks which can write data at about 80-90MB/s. By using incremental ZFS send the time goes down considerably as only the delta between the snapshots need to be transferred.

To be able to recover individual files and recover parts of data even if the disk have errors the streams are sent to a zpool on the backup disk. By using several disks I have at least one at another location and it also gives me multiple backup versions. I was considering placing encrypted ZFS streams on the disks but it is then not possible to recover individual files and if the stream is damaged it becomes useless.

In an ideal world I would have another node set up that receives the incremental ZFS streams over the net, but that is overkill for my current usage and I have no secondary site with good bandwidth (and another storage server).

This setup gives me the following redundancy:

• Integrity of all data is verified every two weeks
• Data has several read-only snapshots from different times
• Data is protected by two disk parity raidz2
• Accessed data is always verified by checksums
• Offsite backups allow disaster recovery
• Backups are also checksummed
• Memory is ECC protected to prevent data corruption

This is all fine, but there is still one single point of failure, if a serious error would creep into the ZFS code it could be replicated to all snapshots and pools, but given the amount of testing ZFS has gone trough it seems unlikely. Here tape backups over NDMP would be of good use but since I do not have any tape hardware all important data is copied with rsync to a disk with an old-school filesystem once every other month.

On top of this I also take advantage of other ZFS features, a cheap SSD is used as L2ARC to accelerate various workloads and compression/de-duplication is as always available with ZFS. It is also possible to add new hardware to the setup without buying a different server or license, 10GbE, Fibre channel, more SSD caches and more RAM for cache/dedup can easily be added, that would probably not be possible with a pre-built NAS appliance or at least not as cheap.

I am evaluating the beta of OpenIndiana 151 on the storage server after upgrading from the now dead OpenSolaris distribution (I would not have tested a beta release without all these backups in place), so far everything works fine. Solaris 11 Express can also be used but that requires a license from Oracle that costs about $1000/year but it will give you ZFS crypto and a few other ZFS features not available in the open ZFS code base.

All the technical features are better than most storage appliances but OpenIndiana/Solaris 11 Express provides no web based administration, there are however add-on software such as nap-it available and commercial ZFS software appliances such as NexentaStor which has a free community edition for up to 18TB of used storage.

I have worked with designing and implementing various similar solutions from small office filers to larger data archives with 96-disk. I work part time as a consultant so I am available to assist in similar projects.

Solaris dskinfo utility

Solaris and it's derivatives are perhaps the best general OS for storage thanks to ZFS, COMSTAR and the native CIFS implementation. It lacks however a single tool for managing disks before they are part of a ZFS pool, disk group or filesystem. With a few disks it's somewhat manageable but when you have tens or hundreds of disks on a system it's hard to get a good overview. This becomes even harder if MPxIO is used or the disks are LUNs from a SAN so that you want to see the logical unit numbers.

You can manually map the MPxIO names against logical unit numbers with luxadm, see the size with format or iostat and find used disks with zpool status, mount or some other. This does require many different commands and you will not get all the information listed in once place.

A while ago I wrote a utility to collect and present disk information in Solaris that I called dskinfo. It relies heavily on existing commands in Solaris and parses their output but have worked very well for me in many different environments.

This initial version is best suited for systems using ZFS since it can only show the usage of disks part of a zpool (exported or imported). It can still be used to get a summary of all disks on the system and later version might add additional filesystems and volume managers as well. I have labeled this version 0.9 and I will use feedback to create a 1.0 version before I look into adding more functionality.

dskinfo displays disk information with four subcommands: list, list-long, list-full or list-parsable. The two first commands will output data on a single line with different number of fields, list-full will display all data on two lines per disk and list-parsable will display all data on one line with fields separated by a colon.

I have tested dskinfo on Solaris 10, Solaris 11, OpenIndiana and Nexenta with both local, iSCSI and fiber channel attached disks.

Example, I have only included the shortest type of list since the formatting here makes the other look terrible, they all look good in 80 columns:

$ dskinfo list                          
disk                 size use              type 
c5t0d0               149G rpool            disk 
c5t2d0               37G  -                disk 
c6t0d0               1.4T zpool01          disk 
c6t1d0               1.4T zpool01          disk 
c6t2d0               1.4T zpool01          disk 

A better formatted text file with additional listing options can be viewed here: dskinfo-example.txt

Please let me know if you find dskinfo useful or have any problems, I will try to fix them when I have time.

Version 1.0 is available for download here: dskinfo

Update: Version 1.0 is now available, it contains bug fixes, warning if SVM is used or VxVM is installed. It can also output driver and instance names and tries harder to acquire the serial number from disks. A quiet options is also available to remove warnings.

ZFS working group

While catching up in zfs-discuss I read an interesting announcement from the ZFS working group. They have created a new versioning method for ZFS allowing different efforts to create new versions of the on-disk format without incompatibility in versioning. This is a thing I myself have been a bit worried about, how differences between different ZFS implementations be handled.

It it good to see efforts to implementing new features for the free ZFS implementation outside of Oracle. Even better is that people like Matthew Ahrens are involved, he was in the ZFS core team at Sun but now works for Delphix. Matthew even worked with implementing BP rewrite which is probably the most requested feature in ZFS.

The ZFS working group are currently working on the future of open ZFS behind closed doors, as described by Garret D'Amore earlier this year:

"There is ZFS development happening outside of Oracle. Many of the
active ZFS developers at a *variety* of organizations are collaborating
within the illumos community using a private e-mail list much like an
standards body Working Group (we even call ourselves the ZFS Working
Group). And not all of the participants here are coming from Solaris
backgrounds -- we have Linux, FreeBSD, and MacOS represented.
(Privacy is important to keep the discussions focused and technical.
And some participants would prefer to keep their participation
non-public.)"

I hope we will see more from this group in a near future, it seems likely the open implementation of ZFS will get new features such as a hybrid block allocator and new compression algorithms. With great minds from Nexenta, Delphix and Joyent among others working with illumos I'm sure great things will come.

In the meantime you can use Solaris 11 Express to try out the latest ZFS feature available or OpenIndiana oi_151 which is the open distribution that has the most recent ZFS implementation from illumos.

[zfs-discuss] ZFS working group and feature flags proposal

Free Solaris patch metadata

I think there are a few people out there frustrated with the availability of Solaris patch information these days. Even if you have a support contract with access to all patches it can be a hassle to get good overview of new Solaris patches in My Oracle Support and even harder if you want to link to that information.

I have just found a fantastic site that provides patch metadata for Solaris in a format easy to view and search. It is also available to the public.

This is their reason for creating the site:
"When the merge of Sun and Oracle has been made, we lost a lot of tools that were present on the famous SunSolve. Like the ability to search efficiently for patches, bugids, sun alerts and so on. This website has the aim to provide part of this informations by indexing all the patches and bugids that we could gather publicly. We want to allow easy search to ease the life of solaris sysadmins."

sunsolve.espix.org

LDOM 2.1 with Live migration

Oracle VM 2.1 (previously LDOM, Logical domains) was released yesterday, it finally has the ability to perform live migration of domains, much like VMWare vMotion. It is supported on T2 and T3 servers, it should also work on the T4 when it's released this fall (or at least an updated version).

From the press release:

• Live migration: Enables users to migrate an active domain to another host machine while maintaining application services to users. Live migrations are as simple as point and click using Oracle Enterprise Manager Ops Center’s console.
• Secure, encrypted migration included: On-chip cryptographic accelerators deliver secure, wire speed encryption capabilities for live migration – without any additional hardware investments.
• Dynamic Resource Management (DRM) between domains: Ensures that domains running the most important workloads get priority for CPU access over domains with less critical workloads.
• Increased maximum number of virtual networks per domain: Permits a dramatic increase in external access to domains.
• Lower-overhead, higher scalability networking for Oracle Solaris 11 Express: Allows virtual network devices to use shared memory to exchange network packets, enabling improved performance and scalability.
• Support for Virtual Device Service Validation: Immediately validates the name and path for a specified network device or virtual disk, greatly reducing the risk of incorrectly configured I/O.
• Integrated Dynamic Reconfiguration (DR) of Cryptographic units and virtual CPUs: Cryptographic units and CPUs are dynamically reconfigured together to simplify operations and ensure consistent performance.
• Enhanced Management Information Base (MIB): Enables the SNMP MIB to use the latest Logical Domains Manager XML interface, permitting third party management software to access the new features and resource properties.
• P2V tool enhancements: Bring more flexibility to quickly convert an existing SPARC server running Oracle Solaris 8, 9 or 10 into a virtualized Oracle Solaris image to run on SPARC T-series servers."

Oracle Announces Latest Version of Oracle VM Server for SPARC
Oracle VM Server for SPARC 2.1 Release Notes

Solaris release dates reminder

As other sources also have now also found out that the next release of Solaris will be named Solaris 10 8/11 which indicates a August release. As written previously we should expect Solaris 11 around Oracle World 2011, a more precise date have been disclosed in Joerg's blog: November 2011, a year after the Solaris 11 Express release. He also speculates in a 11/11/11 release driven by marketing. I will be very interesting to se what the last year of engineering under Oracles guidance have to deliver in the final Solaris 11 release.

Last but not least the OpenIndiana project is just about to release build 151. Update: there are even beta images available here.

Solaris 10 update 10 ZFS refresh

No more public bug information

A tremendously useful site for Solaris/OpenSolaris/Solaris 11 operators, users and developers have been bugs.opensolaris.org. I don't know how many problems I have solved by using this site and I am sad to report that it has now been down for a well over a month. Now even I have given up hope that Oracle will restore the service, all information available is the message on the site: "This service is temporarily unavailable.". It no longer seems like a temporary outage and I would not be surprised if Oracle have cut this source of information too and is trying to do it quietly without any announcement. If that is not the case I love to be proven wrong.

This is another change that will make it harder for me to promote future Solaris releases and write about interesting technical details.