Ramblings from Richard's Ranch

Gotta love the winter

2012-01-02T10:20:00.000-08:00

In this time of great changes, I thought I might share a view from the ranch this morning.

Our cherry trees begin blooming after the rains start in November or December. They will continue to bloom for another month or two. For all of those people who are huddled against the cold, know that soon the changes will bring spring-like weather and the cherry blossoms will bloom.

Interesting new things shown at VMworld in Las Vegas

2011-08-28T17:03:00.000-07:00

If you're traveling to Las Vegas for VMworld 2011, be sure to stop by the Nexenta booth. We've got some interesting new things to show... including something you've never seen before...

NexentaStor 3.1 Released

2011-07-29T18:07:00.000-07:00

Today NexentaStor 3.1 was released to the world. Download a free trial copy today! We've been working hard on this release for some time and it offers significant improvements in stability and performance. Here is a small sample of the changes that I think are cool.

ZFS baseline
zpool version 28 is the default pool version. This version includes a bunch of features that ZFS geeks know and love, such as readonly imports and rollback. Other Nexenta add-ons are also still there, such as WORM.
JBOD enumeration support
One of the most difficult tasks of a general-purpose, x86-based storage appliance is sorting through the plethora of JBODs on the market and being able to reliably map a disk to the JBOD disk slot. This is the first release of this mapping capability in the appliance. Detailed support for a few JBODs is included (LSI DE1600, Xyratex HB 1234 and HB 2435), with generic support for other JBODs. The pipeline is full of JBODs waiting to have detailed support.
iSCSI performance improvements
Several improvements in iSCSI performance, including zero-copy.
Vmware vStorage APIs for Array Integration support
VAAI support has been added to block storage protocols.
Smarter fault triggers
Many of the fault triggers have been improved to include more checks and better condition messages.
HA-Cluster feature enhancements
Many enhancements to HA-Cluster to improve failover time and robustness. Also, you can now have multiple pools (Nexenta volumes) or virtual IP (VIP) addresses per cluster service. My favorite feature is that the requirement for a dedicated heartbeat disk has been removed, we now use shared pool disks for heartbeats, which makes perfect sense.
Auto-sync replication improvements
The auto-sync asynchronous replication service has been redesigned to be more robust and offer better performance. A new one-to-many replication transport has also been added.
More devices supported
Device drivers of note include: LSI 9205 family, Areca 1880. In the pipeline are driver updates from Intel and Emulex.
Better upgrade management
You can now upgrade specific version numbers. In the past you would always be upgraded to the latest version, now you can specify the major version. For example, to upgrade to release 3.1, the NMC command is "setup appliance upgrade -r 3.1"
SMB improvements
The SMB (aka CIFS) share service has improved performance, scalability, and reliability.
Active Directory improvements
AD integration has been significantly improved. AD also handles Domain Controller failover better.
Virtual Machine Data Center improvements
VMDC has better integration with VMware vmotion.

I tried to keep the list to the top-10, but that obviously didn't work. I hope you enjoy the release anyway!

NexentaStor 3.0.4 released

2010-11-16T04:21:00.000-08:00

This week we've launched NexentaStor 3.0.4. In many ways this is a significant milestone, far beyond what may be immediately obvious. For existing Nexenta customers, the feature list will look largely unchanged -- many of the same, great universal storage features available since 3.0.0 earlier this year. But for one who studies how organizations grow and mature, the release represents the best quality, stability, and maturity ever. We have been working hard to earn your trust for protecting your data. I think you will be pleased with the result.

From a ZFS perspective, there have been many bug fixes and stability improvements. This release also deprecates the global ZIL disable feature. If you've ever attended my training courses or research sessions on ZFS, you know that I'm not a big fan of disabling the ZIL or the ease with which disabling the ZIL can be accomplished in NexentaStor. But there are a few, special cases where the performance trade-off is justified. In NexentaStor 3.0.4 you can set the ZIL use policy on a per-dataset basis. In ZFS terms, this is called "ZIL synchronicity" and is three parts really good stuff and one part pun. Special thanks to Robert Milkowski and the ZFS community for the contribution.

In keeping with Nexenta tradition, a free, temporary licensed edition can be downloaded and installed from the main Nexenta site or the European mirror (thanks Jacco!) Give it a try and see why we think it is the best NexentaStor release yet. Of course, if you already have NexentaStor installed, the NMC command is "setup appliance upgrade" When you do the upgrade, a checkpoint will be made of your current system, so you can roll back if you are unhappy -- appliances are such wonderful things, when implemented properly.

Getting arrested at the World Cup celebration

2010-07-18T18:11:00.000-07:00

Shortly after my arrival in Rotterdam last Sunday, the final game of the World Cup began. Though I am a soccer (and football) fan, I must confess that the trip was planned long before the Dutch secured a spot in the final. Arriving in the Netherlands after traveling for nearly 24 hours, I was looking forward to a quick nap before exploring the city. No rest for the weary. The party was already starting and people were collecting in a street near the Stadhuis, a mere block or two away from my hotel. Music was blaring from a rather impressive sound system and cold beer was flowing from the tap. How could I resist? Forget the nap, I'm going in!

The Sea of Orange (Zee van Oranje) was lively and exciting. I wandered through the streets and people seemed to be coming from every direction, flowing towards their favorite spot to watch the game. I grabbed a quick dinner from Dino's, a local restaurant, just before they closed to enjoy the festivities as the pregame broadcast started. Everyone seemed to be in high spirits, demonstrating their national pride.

That is when I got arrested.

The officer picked me right out of the crowd.

Clearly, I must have looked like a criminal?

I immediately wondered if they have racial profiling training, like the 'zonies.

The officer stopped me and asked a question, in Dutch, of course.

Whazzat? I don't speak Dutch, or at least not in any way recognizable by someone who is actually fluent in Dutch.

Egads, I had only been in the country for a few hours and already I'm embroiled in an international incident.

Episodes from Locked up Abroad came flooding into my mind.

I had a vision of a tombstone engraved with "What was he thinking?"

Without missing a beat, the officer switched to English.

"I am from the Orange Police, and I see that you are not wearing Orange."

The horror!

The officer was quite right. I did not have anything Orange. My last Orange T-shirt was added to the rag pile just a few short months ago.

I knew was in big trouble -- time to act like a dumb American.

The setup: "I just arrived in the country..."

The excuse: "... and I have been trying to find a store to buy something Orange."

The saving grace: "Uh, and if you know of a store where I could find such things, I would promptly rehabilitate myself."

I managed a smile, in a hopeful attempt at leniency.

"Oh, well all of the stores are closed at 5 PM..."

Whoa! This might actually be working.

"... you will have to wait until later."

Reprieve?

"Meanwhile, get a beer and enjoy the game."

Free at last! Free at last! I had successfully negotiated my way out of jail in a foreign land, and the beer was very tasty, too!

The result of the game put a damper on the celebrations. Spain scored an impressive goal with just a few minutes before the end of overtime. The streets cleared and the city returned to normal. A national week of celebration was replaced with a warm homecoming for the National Team in Amsterdam. My newfound friends in Rotterdam were a pleasure and I am truly grateful for being a part of the festivities.

As for the Orange Police, I have learned my lesson and will never again transgress.

ZFS Training in Rotterdam next week

2010-07-10T07:26:00.000-07:00

Next week I'll be offering an in-depth ZFS and NexentaStor training session in Rotterdam, the Netherlands. INPROVE is hosting the event and we have a full classroom. Though I did not plan for arriving in Amsterdam at the same time as the World Cup finals, I'm sure it will be a festive occasion. I'm sure my Spanish friends will understand that I will be wearing Orange.

Data protection beyond the pool

2010-05-08T18:43:00.000-07:00

Many people are flocking towards ZFS solutions because of the sound, fundamental science behind ZFS and the features it provides. Two features which complete the hybrid storage pool design are separate logs and cache devices.

Separate log devices allow you to direct synchronous write I/O destined for the ZFS Intent Log (ZIL), away from the main pool and towards devices which are optimized for low-latency writes. One of the best examples of a separate log device, highly optimized for low-latency writes, is the DDRdrive X1.
Cache devices allow you to extend the RAM-based Adaptive Replacement Cache (ARC) by adding cache devices as a second-level cache (L2ARC). If the random read performance of the cache devices is much better than the rand read performance of the pool, then cache devices are more cost-effective than adding RAM to enlarge the ARC.

The data retention policies of the separate log and cache devices are quite different.

The cache device contains data that already exists on the pool devices. If a cache device fails, then a read access is treated as a cache miss and the data is retrieved from the pool. Sure, there might be a slight performance hit, but data is not compromised. Therefore, ZFS does not offer data protection (mirroring) for the cache devices.

Separate log devices contain data that is not yet written to the pool. If the separate log device fails while the pool is imported, then the device is taken offline and writes are sent to the pool. If the entire log device fails when the pool is exported, then ZFS will not allow the pool to be imported because it cannot determine if data has been lost. Hence, for separate log devices, ZFS offers mirroring to protect the data that has not yet been written to the pool.

Bottom line, if you are very concerned about the data retention of the pool and desire the cost-effective and high performance of the separate log and cache devices, then consider mirroring your separate log devices and don't worry about the cache devices.

ZFS data protection comparison

2010-02-17T15:52:00.000-08:00

ZFS now offers triple-parity raidz3. Conceptually, raidz3 is an N+3 parity protection scheme. Today, there are few, if any, other implementations of triple parity protection, so when we say "raidz is similar to RAID-5" and "raidz2 is similar to RAID-6" there is no similar allusion for raidz3. I prefer to say "raidz3 is like raidz2 with one additional level of parity protection. But how much better is raidz3 than raidz2? To help answer that question, I used the simple Mean Time to Data Loss (MTTDL) model to calculate the data retention capabilities of the possible configurations of 12 disks under ZFS. To be fair, the same model applies to other RAID implementations, but I'll use the ZFS terminology here.

In this MTTDL model, the configuration includes N total disks. If the data protection scheme is raidz3, then the minimum N = 1 data disk + 3 parity disks = 4. You can add more data disks to increase the overall available space, so if N=6 then you have 3 data disks + 3 parity disks.

The model uses the Mean Time between Failure (MTBF) as specified in a vendor's datasheet. It also uses a Mean Time to Repair (MTTR) which includes both the logistical repair time and any data reconstruction required. The simple model calculates MTTDL as:

For non-protected schemes (dynamic striping, RAID-0)

MTTDL[1] = MTBF / N

For single parity schemes (2-way mirror, raidz, RAID-1,RAID-5):

MTTDL[1] = MTBF² / (N * (N-1) * MTTR)

For double parity schemes (3-way mirror, raidz2, RAID-6):

MTTDL[1] = MTBF³ / (N * (N-1) * (N-2) * MTTR²)

For triple parity schemes (4-way mirror, raidz3):

MTTDL[1] = MTBF⁴ / (N * (N-1) * (N-2) * (N-3) * MTTR³)

A graph the results for combinations of 12 disks looks like:

The results are consistent with previous MTTDL analysis. The 12-disk Stripe has an MTTDL of 6.7 years, which isn't very good (annualized rate = 15%) whereas the 12 disk 4-way stripe MTTDL is 2.75e+13 years (annualized rate = 3.63e-12%) and the 12 disk raidz3 MTTDL is 1.67e+11 years (annualized rate = 5.99e-10%).

The theory behind raidz3 will allow more parity disks. But at some point, the system design will be dominated by common failures and not the failure of independent disks. I hope this model will be useful for you to evaluate the data retention of your storage system.

ZFS training in Atlanta, March 16-18, 2010

2010-02-04T14:27:00.000-08:00

I will be presenting a 3-day training session for systems and storage administrators on ZFS and NexentaStor in the Atlanta area March 16-18, 2010. The team has put together a fantastic syllabus including in-depth exposure to the latest ZFS and NAS trends.

Attendees can choose to attend the three-day program, or the two-day advanced portion. The course is structured as follows:

Day 1: Introduction to ZFS and Nexenta Systems Storage Technologies
Day 2: De-duplication in a VM World
Day 3: Optimizing NAS Performance

Attendees should have some familiarity with storage concepts and terminology, but does not assume any knowledge of ZFS or familiarity with the NexentaStor storage appliance.

The course will include hands-on exercises with ZFS and NexentaStor.

Best of all, lunch will be provided each day.

To sign up or view the detailed syllabus, visit the nexenta-atlanta.eventbrite.com event registration site.

Community helping the community, ala YouTube

2010-01-31T10:11:00.000-08:00

The video generation is taking hold in the OpenSolaris community. Recently, Michelle Knight, a self-described general lunatic, asked for help on the OpenSolaris ZFS forum. But quite unlike most folks who get help and quietly wander away, or (hopefully) post a summary for posterity, she made a video describing what she learned and posted to YouTube. Very cool. Well done, Michelle!

Magic Mouse and Ring Finger Solution

2010-01-27T10:44:00.000-08:00

I use an Apple Magic Mouse and really do love it. I use Adobe InDesign CS4 for writing technical documents. I also have a ring finger. Each of these things works well by itself. Together, they don't work well. This is a typical systems engineering problem. Each part works as designed, but together they don't work well.
Now you are probably wondering why these three things don't work together - millions of people use mice, thousands of writers use InDesign, and almost all of the people on Earth have ring fingers. Let me explain.

The Magic Mouse is so very, very cool because the entire surface is touch sensitive. It is very easy to use and allows you do things you could never do before with a mouse. For instance, many mice have had a little scroll wheel and OSes are designed to use the scroll wheel movement to scroll up and down inside a window. Some mice have little trackballs that allow you to scroll left or right, too. The Magic Mouse is almost like giving your fingers a trackpad on top of the mouse. Implementing a multiple button click function is simply a matter of the programming that determines where your finger is when you press. Very cool. Very habit forming. In just a few short weeks, my hand is already forgetting how to use older mice.
Adobe InDesign is a very powerful publishing product. I've been using FrameMaker since 1987 and find that InDesign has many of the features I've use in FrameMaker, but is even more powerful and flexible. One of the interesting concepts in InDesign is the pasteboard. Your document sits atop the pasteboard. If you want to move a frame, text, image, or other object out of your document quickly, but without deleting the object, then you can just slide it over to the pasteboard beside the page. Only the objects on the page are printed or exported to PDF, so you can use the pasteboard to keep your miscellaneous collection of stuff very easily. The pasteboard is larger than your page, and by default adds about 8 inches to each side of your document. This means that your pasteboard for a letter sized document is around 24 inches wide. Since my screen is not 24 inches wide (is Santa listening? I'll be a good boy) the windows I use have horizontal scroll bars. For the most common case, the page is in the center of the scroll bar. I've spent a few hours trying to figure out how to make the pasteboard thinner, but none of the tricks work.
My ring finger has a tendency to rest on the right side of the mouse while my index and middle fingers wander about the mouse top and click.

OK, so now you should be able to recognize the problem. My ring finger is interpreted by the Magic Mouse to do a horizontal scroll and InDesign extends the scrolling area by 60%, most of which is area I rarely use. In other words, while I'm working away, I get suddenly scrolled off into the blank area of the pasteboard. Since the document is in the middle, I have to scroll back to the center, which is harder to do than scrolling full left or full right.
The solution I've found is to put a small bit of painter's tape over the area where my ring finger rests. I could have used duct tape, and that would make a good joke, but I prefer the painters tape for now.

So far this is working well. A programmatic way to build dead spots on the Magic Mouse would be a useful feature. InDesign could allow me to control the horizontal size of the pasteboard. All of these programming changes are perhaps not difficult, but will also not be solved soon. For now I can be highly productive without having to horizontally scroll back to center on InDesign.
Now, about those deadlines...

Integrated Systems Engineering Redux

2010-01-27T09:50:00.000-08:00

Today, Oracle is presenting a webcast describing their strategy for the company going forward after the Sun acquisition. In the first 20 minutes there was much discussion about delivering integrated systems: applications + database + OS + hardware. This is a tremendous value proposition. It is such a tremendous value proposition that it could have been taken from the slides we put together 8 years ago in Sun's Integrated Systems Engineering group.
We had difficulties with the business of delivering such integrated solutions. Sure, there were a few technical difficulties, but working together with the different engineering groups at Sun and Oracle, we were able to deliver a good technical solution. However, the business challenges of working across different product groups and companies were insurmountable at the time. In the end, the Integrated Systems Engineering group was disbanded and the products were EOLed.
In my position as Chief Architect for the Integrated Systems Engineering group, I had the pleasure of working with many talented engineers and product marketing teams. But the experience taught me that very good technical solutions may not be successful because of the rest of the business activities needed to ensure the right products offering the right value are delivered at the right time to the right market. And those products include much more than a what a systems engineering team can integrate in a lab. This is why I entered the EMBA program at USC's Marshall School of Business. I already knew how to integrate complex systems and make them simple to install and manage. But I did not know how to take such a product and make it successful in the market. I'm a lot smarter now.
I wish Oracle well in their future endeavors. The value proposition is good. The need exists. The challenges are difficult. If they can overcome the non-technical barriers, the future looks bright.

National recognition for San Diego County CERT

2010-01-25T10:04:00.000-08:00

The AMGEN Tour of California bicycle race rode to the top of Palomar Mountain last year. I mentioned it in my blog prior to the race. This month I am pleased to announce that the CERT National Newsletter (Volume 2, issue 3) features a story about the preparation and nearly flawless execution of the event. This was truly a case where dozens of volunteers came together, at short notice, to pull off a significant event involving thousands of people and a nationwide TV audience.
Bill Leininger and the crew from Palomar Mountain Volunteer Fire Department CERT demonstrated superb leadership and I am proud to have been able to participate. I'd also like to thank all of the volunteers and groups who came together to make this event a success.
On page 2, you can yours truly (in the red shirt, strategically located near the donuts) during the pre-event briefing.

Looking at I/O Performance with Bubbles

2010-01-15T19:02:00.000-08:00

I am helping a client work through some performance problems and thought I might share a view with you. The data was collected for 57 seconds during a production run. The problem we are chasing is the usual performance problem: latency. In some cases the latency is close to 100ms, which would make everyone except a floppy disk user unhappy. The view of the data is intended to shed some light on where problems might exist that we need to further explore. Using summary data from tools like iostat, vmstat, mpstat, prstat, or top won't show you anything like this.

In the bubble chart, the Y axis is the size of the I/Os. Along the X axis, reads are on the left and writes are on the right. The size of the bubbles is the latency in microseconds. Big bubbles mean big performance problems. Press the play button to see the changes over time.

There are two ZFS transaction group (txg) commits: one at 8:49:14 and another at 8:49:44. ZFS will, by default depending on the version, commit the txg every 30 seconds. When the txg commits, you will see a flurry of relatively small (8 KB) write activity. Though this may look really terrible (and it is) remember that txg commits are asynchronous, so you will rarely feel them. But in this sample, some of the txg I/Os take more than 50 milliseconds to complete. In the entire sample, the worst latency was more than 370 milliseconds (more than 1/3 of a second). For a slow HDD, 50 milliseconds might not be so bad. But in this case, the target is an expensive RAID array. More work needed to get to the bottom of this mystery...

If you would like to see this sort of analysis for your system, contact me and we can discuss an engagement.

St. Mac magazine Premier Edition

2010-01-12T14:26:00.000-08:00

I was doing the holiday cleaning recently and came across a blast from the past. Long ago, when I worked for NASA at the Kennedy Space Center, we were doing some work with an Apple Lisa and an Apple ///. Naturally, when the Macintosh first appeared, it created quite a stir. Along with the rush of announcements and press was the creation of a new magazine, St. Mac. I dusted off a copy of the premier edition. Apparently the magazine didn't last long, only 7 issues, but it brought back a lot of memories. My, how computing has changed since then... why kids today can't even remember when pixels weren't square and how the Mac saved mankind from a world of ellipses.

The cover shows the second generation Lisa. You can tell because the first generation used twiggy drives and the second used the new removable storage rage, 3.5" floppy disks.

The table of contents includes a picture of a young Bill Atkinson. Bill made square pixels cool.

I/O Reduction and the ZIL

2010-01-09T10:07:00.000-08:00

I came across an interesting microbenchmark this week. It shows that some workloads can show confusing results, or head fakes, can lead to difficulty in understanding benchmark results. In this case, a method we use for finding the performance envelope for ZFS is not effective.

Before I dive into the microbenchmark, a few words about the ZFS Intent Log (ZIL). ZFS is a transactional file system, which means that it collects I/O into a transaction group (txg) and commits that txg to persistent storage. In later ZFS implementations, that txg commit occurs every 30 seconds. However, if an application needs to ensure that an I/O is written to persistent storage immediately, often called synchronous writes (though that is arguably not the best descriptive term), then waiting for up to 30 seconds is not an option. This is where the ZIL enters the picture. In the synchronous write case, ZFS will write the record to the ZIL and later commit the record with the txg. This ensures the synchronous write agreement between the application and ZFS is honored -- a good thing. Neil Perrin offers a more detailed description in his famous lumberjack blog posting.

Synchronous writes are the bane of high performance. Really. We see this every day. It causes performance guys to gnash their teeth and cuss. When a microbenchmark wanders along and does a lot of synchronous writes, complaints about how " sucks" and "I can't believe those file system developers could be so insensitive" come pouring forth.

To determine the performance envelope of a benchmark, it is relatively easy to disable the ZIL. This is neither a safe nor recommended option for production systems or people who like their data. But for benchmarking, it allows a performance engineer to quickly determine the best possible performance for the given system configuration. The ZIL is then re-enabled and the work can concentrate on how to approach that performance goal. Tools like zilstat are designed to help with this endeavor, and can save you a lot of time when you suspect synchronous write performance might be an issue.

But disabling the ZIL can also hide important behavior. That is why this microbenchmark could be a poster child for benchmarking that doesn't do what you expect. Here it is:

while true; do

echo "blah" > outputfile

done

When run on an Solaris NFS client with a Solaris NFS server using default NFS settings, this will cause the following to occur:

outputfile is LOOKUPed
outputfile is OPENed
ACCESS to outputfile is checked
The data is written to the file with WRITE
The data is COMMITted
outputfile is CLOSEd

This will also, by default, cause the file to be synchronously written, the so-called "sync-on-close" operation.

Argv! This simple microbenchmark actually makes lot of synchronous writes to the file system. zilstat will happily show that the ZIL is working hard when running this microbenchmark. If you run this, then you can experiment with various pool or separate (ZIL) log configurations to your heart's content.

However, if you disable the ZIL, then the number of I/O operations is reduced to just a handfull, every 30 seconds. Why? Because ZFS is clever enough to recognize that the same file is being overlaid and is only concerned with physically commiting the last one in the transaction group. In other words, the amount of I/O traffic to the pool is dramatically reduced. When this happens, you are no longer measuring the affect of ZIL I/O, you are also measuring the main pool I/O. The results look something like this:

ZIL enabled, no separate log = 100 iterations/second
ZIL enabled, separate log on a fast SSD = 1,000 iterations/second
ZIL disabled = 10,000 iterations/second

In other words the affect of eliminating the pool I/O in addition to the ZIL I/O made the system faster! Hurray! But wait just a dog-gone second. That means that the benchmark is basically useless -- it does almost zero physical I/O when the ZIL is disabled. This is kinda like redirecting all of the data to /dev/null -- a fun trick to amuse your friends at parties, but otherwise completely useless.

The moral of this tale is: beware of microbenchmarks and how they can confuse your understanding of the real system behavior.

P.S. Don't disable the ZIL.

P.P.S. I really mean it, don't disable the ZIL. Seriously. I might cut you some slack for benchmark purposes, but other than that, don't disable the ZIL. Period. End of discussion.

My First Macadamia

2010-01-06T09:51:00.000-08:00

A few years ago I planted some macadamia trees in the orchard. It takes a few years before they begin to bear fruit. 2009 was the first year that we had blooms. I was very excited!

This small cluster of blooms produced 6 nuts. Of these, all disappeared over the summer (I blame the ravens) except one. Last week, I harvested the one, special nut. I brought the nut inside to take pictures, but the new kitten, Jellyfish, was convinced that it was her personal playtoy.

Jellyfish attacks the unsuspecting nut!

I had to act fast. Armed with a bit of foam pipe insulation and a hammer, I whacked the nut and extracted the precious seed.

Yummy! And yes, that is a cow floating above my head. This is a ranch, after all.
Nothin' left but the shell. Next year I'm expecting more blooms and nuts, wish me luck.

ZFS Tutorial at the USENIX LISA09 slides posted

2010-01-05T10:53:00.000-08:00

I have posted my slides for the ZFS Tutorial at the USENIX LISA09 conference on slideshare.net. I apologize for the delay, I've been fighting the beast trying to get the PDF uploaded. I finally gave up and uploaded the keynote presentation. As such, I've also disabled the file download. If you want a copy of the PDF, drop me an e-mail and I'll send it to you.

These slides are the full deck. During the presentation, I used a subset of the slides, but made the full deck available to the attendees. I hope you find these useful.

If you'd like to attend one of my ZFS tutorials, then you can also drop me a line. I have several in the planning stage and hopefully we can schedule something nearby.

ZFS Tutorial USENIX LISA09 Conference

View more presentations from Richard Elling.

LISA09 next week

2009-10-26T16:35:00.000-07:00

The USENIX LISA09 conference is next week and I'll be leaving the ranch to travel back east to attend. On Monday, November 2 I'll be giving a tutorial on ZFS. Much has changed since I gave a similar tutorial at the USENIX Technical Conference last June. Somehow I've got to get the content down from 200+ slides into something feasible to deliver as a one-day tutorial. As the LISA audience is more geared towards systems administration, I'll whittle down the technical details and concentrate on more practical applications of the ZFS technology. Afterwards, at the research center (pub) we can cover other details, as needed. Please join us in Baltimore for a week with the talented teams who manage large computer sites!

check integrity of ZFS send streams with zstreamdump

2009-10-21T13:33:00.000-07:00

If you wander through the OpenSolaris ZFS-discuss archives or look at the ZFS Best Practices Guide, then you can encounter references and debates about whether the zfs send and zfs receive commands are suitable for backups. As I've described before, zfs send and zfs receive can be part of a comprehensive backup strategy for high-transaction environments. But people get nervous when we discuss placing a zfs send stream on persistent storage. The reasoning is that if the stream gets corrupted, then it is useless. There is an RFE open to improve the robustness of zfs receive, but that is little consolation for someone who has lost data.

The fundamental design of ZFS is exposed in zfs send -- the send stream contains an object, not files. This is great for replicating objects, and since ZFS file systems and volumes are objects, it is quite handy. This is why zfs send and zfs receive do not replace the functionality of an enterprise backup system that works on files. So, I expect the technologies to remain complementary for a very long time.

But there are some simple things which can improve management of zfs send streams. It is a good idea to check the integrity of the stream when stored on permanent storage before you try a zfs receive, or just to sleep better at night. You can do that by telling zfs to not actually apply the receive using the "-n" option to zfs receive, but this only returns a boolean response. Something more concrete and descriptive would be nice...

OpenSolaris build 125 brings the zstreamdump(1m) command, that allows you to examine the contents of a zfs send stream. To demonstrate, I made a quick (diving) pool called "zdiving," copied some data to it, made a snapshot saved as a file, and ran zstreamdump. Observe:

# ramdiskadm -a rd1 200m

/dev/ramdisk/rd1

# zpool create zdiving /dev/ramdisk/rd1

# cp somefile /zdiving

# zfs snapshot zdiving@today

# zfs send zdiving@today > zdiving.zstream

# zstreamdump < zdiving.zstream

BEGIN record
version = 1
magic = 2f5bacbac
creation_time = 4adf8f3d
type = 2
flags = 0x0
toguid = 582cc98ae8f284cb
fromguid = 0
toname = zdiving@today
END checksum = 705e7f009f/441064477b71ce/1aacbf27d3a3e8a0/119dd4527bd9c6a3
SUMMARY:
Total DRR_BEGIN records = 1
Total DRR_END records = 1
Total DRR_OBJECT records = 5
Total DRR_FREEOBJECTS records = 791
Total DRR_WRITE records = 5
Total DRR_FREE records = 9
Total records = 812
Total write size = 2560 (0xa00)
Total stream length = 256696 (0x3eab8)

So if you have any zfs send streams lying around, give zstreamdump a try. If you find a checksum mismatch, then please let me know... I'm collecting data on real checksum mismatches found in the wild.

Variance and cloud computing

2009-08-20T10:22:00.000-07:00

Recently, I was asked about what I've done for cloud computing. Personally, I think "cloud computing" is just the latest marketing buzzword, and represents a passing fad. But the concepts people use when trying to describe the cloud are a good foundation for providing computing services. Many of these concepts have been in place for 25-30 years, at least in the engineering workstation market, but perhaps not widely applied to other markets. At their core, the goal is to reduce cost and complexity by reducing variance -- a noble goal well served by six sigma-like approaches.

For example, in the conversation the problem of rapid deployment or provisioning arose. 20 years ago this month, when I was the Manager of Network Support for the Auburn University College of Engineering, I had the task of taking a classroom full of boxes containing workstations and deploy them to the departments and classrooms prior to school starting. The only sane way to approach this is to reduce the variance between the systems. I produced a golden image of the OS and started cranking out workstations. But we still had variance problems. The network was not yet implemented in every building. So there were some deployments which were unique. Later, when we were able to connect all of the buildings (and eventually, we even connected to the internet :-) then I quickly resolved the unique systems and brought them into the fold. By the time I left Auburn, we could deploy a new workstation in about 20 minutes, including physically unpacking and installing the hardware, with the workstation being fully usable in the ubiquitous environment and connected to the internet. Sometimes, it would take longer than 20 minutes because we allowed the faculty member or staff to choose their hostname -- a process that could take days. Since the hostname was the only variable, it was no wonder that it cost time to resolve. But it is good to have real names instead of numbers for such things. Today, with a well designed "cloud" you could expect to deploy a New Thing in just a few moments -- after resolving the unique properties such as billing information.

Rapid deployment is a Good Thing, but it isn't a new thing. We've been doing that for many years. What seems to be new is that people who do not have the net in their DNA are starting to figure it out. I can loosely classify these as PC people, those who have been exposed to the software installation and configuration issues in the disconnected, fat client world -- aka PCs.

This is where SaaS comes in. The problem with installing software on each and every fat client is that software changes over time. By the time a software vendor becomes successful in the fat client market, they could have dozens of versions of the software installed in the client base. Keeping track of all of the versions, and the features, bugs, and platform support is a nightmare. At Sun we constantly had customers asking for a compatibility matrix, which I call a sparse matrix, because there were so many products and platforms that were never tested together, it was impossible to make sure everything worked together at any one point in time, let alone as they changed over time. The best way to tackle this variance problem is to not install software on the fat client at all. I know, I know, this is the same mantra we sang about 20 years ago when the network was the computer, but it really is all about decreasing costs and complexity by getting rid of variance.

In a SaaS environment, you can roll out a new, improved version of your software and know that all of your customers will be on the same version. This is a huge reduction of variance in the installed base. This is also a huge win for the customer service group in your company. We did many studies at Sun that found the number of bug reports which had known solutions is large. Many enterprise customers are late adopters, which further compounds the problem of variance in the installed base.

So if it takes a marketing buzzword, like "cloud computing" or "SaaS," to help point people in the direction of removing variance, then I can live with it. Reducing variance is a Good Thing.

Thanks Martha.

Backups for file systems with millions of files

2009-08-19T15:33:00.000-07:00

Recently, there have been a number of discussions about how to backup an active file system with millions of files. This is a challenge because traditional backup tools do a file system walk -- traversing the file system from top to bottom looking at the modification time for each file. This works well for file systems with a modest number of files. For example, one of my OpenSolaris systems has around 62,000 files in the root file system and backups go at media speed. But when you get into the millions of files with deep heirarchy, the time required to manage the file system walk begins to dominate backup time. This can be a bad thing, especially if you backup to tape.

We didn't notice this problem with UFS because, quite simply, UFS can't handle millions of files in a single file system. But with ZFS becoming mainstream, it does not have this limitation and people are taking advantage of the ease of managing large datasets with ZFS.

One successful approach to solving this problem uses ZFS snapshots to backup an active file system to a backup file system. For diversity, the backup file system could be located on another disk, array, or host. For cost efficiency, the backup file system can have different properties than the active file system -- compression is often a good idea. The trick is that the active file system can be optimized for high IOPS and the backup file system can be optimized for low cost per byte.

This might look familiar to you. Many people have performed backups from a replica of the production data. Most of those implementations perform the replication at the block level. ZFS can replicate at the file system level -- allowing policy or configuration changes at the block level, such as different RAID configurations or data compression.

With block-level replicators, such as the Sun StorageTek Availability Suite, the replica has no knowledge of the context of the data. The replica's view is block-for-block identical to the original. With ZFS replicas, the file systems can have different data retention policies. For example, the production site may have a snapshot retention policy of 24 hours and the replica may have a retention policy of 31 days. As long as there is a latest, common snapshot between the production site and replica, any later snapshots can be replicated. Try doing that on a block-level replicator!

There is much more to this story. In particular, retention policies and performance optimization can get very complex. I'm working on a good example which goes into more detail and discusses performance concerns... more on that later...

Justifying new compression algorithms in ZFS?

2009-08-14T09:45:00.000-07:00

Denis Ahrens recently posted a compression comparison of LZO and LZJB to the OpenSolaris ZFS forum. This is interesting work and there are plenty of opportunities for research and development of new and better ways of compressing data. But when does it make sense to actually implement a new compression scheme in ZFS?

The first barrier is the religilous arguments surrounding licensing. I'd rather not begin to go down that rat hole. Suffice to say, if someone really wants to integrate, they will integrate.

The second barrier is patents. Algorithms can be patented, and in the US patents have real value ($). This is another rat hole, so let's assume that monies are exchanged and the lawyers are held at bay.

The third barrier is integration into the OS. Changes to a file system, especially a file system used for boot, take time to integrate with all of the other parts of the OS: installation, backup, upgrades, boot loaders, etc. This isn't especially hard, but it does take time and involves interacting with many different people.

Now we can get down to the nitty-gritty engineering challenges.

Today, disks use a 512 byte sector size. This is the smallest size you can write to the disk. So compressing below 512 bytes gains nothing. Similarly, if the compression on a larger record does not reduce the overall size by 512 bytes, then it isn't worth compressing. Also, compression algorithms can increase the size of a record, depending on the data and algorithm. ZFS implements the policy that if compression does not reduce the record's size by more than 12.5% (1/8), then the record will be written uncompressed. This prevents inflation and provides a minimum limit for evaluating compression effectiveness. The smallest record size of interest is 8 blocks of 512 bytes, or 4 kBytes.

ZFS compresses each record of a file rather than the whole file. If a file contains some compressible bits and some uncompressible bits, then the file can be compressed, depending on the distribution of compressible bits in the file. This seems like an odd thing to say, but it is needed to understand that the maximum record size is 128 kBytes. When evaluating a compression algorithm for ZFS, the record sizes to be tested should range from 4 kBytes to 128 kBytes. In Denis' example, the test data is in the 200 MBytes to 801 MBytes range. Interesting, but it would be better to measure with the same policy that ZFS implements. Also, two of Denis' tests were on a tarball comprised of files. Again, this is interesting, but will not be representative of the compression of the untarred files, especially files smaller than 4 kBytes.

Now we can build a test profile that compares the effectiveness of compression for ZFS. The records should range from 4 kBytes to 128 kBytes. To do this easily with existing files, they could be split, compressed and the results compared for each split after applying ZFS' policies. The results should also be compared at the sector size, not the file length. To demonstrate, I'll use an example. I took the zfs(1m) man page and split it into 4 kByte files. Then I compressed it with compress(1) and gzip(1). For gzip, I used the -6 option, which is the default for ZFS when gzip compression is specified.

	original		compress		gzip -6
file	length	sectors	length	sectors	length	sectors
xaa	4096	8	1688	4	945	2
xab	4096	8	2198	5	1664	4
xac	4096	8	2217	5	1615	4
xad	4096	8	2081	5	1438	3
xae	4096	8	2161	5	1468	3
xaf	4096	8	2170	5	1544	4
xag	4096	8	2174	5	1386	3
xah	4096	8	2163	5	1442	3
xai	4096	8	2154	5	1477	3
xaj	4096	8	2279	5	1747	4
xak	4096	8	2100	5	1299	3
xal	4096	8	2066	5	1335	3
xam	4096	8	2177	5	1481	3
xan	4096	8	2056	5	1326	3
xao	4096	8	2025	4	1206	3
xap	4096	8	2073	5	1388	3
xaq	4096	8	1960	4	1279	3
xar	4096	8	1897	4	1272	3
xas	4096	8	1766	4	1135	3
xat	4096	8	2221	5	1478	3
xau	3721	8	1820	4	1056	3
zfs.1m	85641	168	31464	62	20582	41

It is clear that gzip -6 does a better job compressing for space on these text files. I won't measure the performance costs for this example, but in general gzip -6 uses more CPU resources than compress. The real improvements in space are not readily comparable, though. A bit of spreadsheet summing reveals:

	split files	single file	all files
compress	59%	37%	48%
gzip -6	39%	24%	32%

This shows that the space savings from compression on a single, large file is much better than for smaller files. This also reiterates the issue with compression in general -- you can't accurately predict how well it will work in advance.

In conclusion, the work required to add a compressor to ZFS is largely dominated by non-technical issues. But proper evaluation of the technical issues is also needed to be sure that the engineering results can justify the time and expense to tackle the non-technical issues. This can be done by experiments prior to coding to the ZFS interfaces. Denis and others, are interested in improving ZFS which is very cool. I think you should also help improve ZFS, or at least use it.

Whither Btrfs?

2009-08-13T11:00:00.000-07:00

Is the best technology the pathway to success? Nope. In this post, I'll take a strategic look at the future of the Btrfs file system.

Using B-trees (or modified B-trees) for space allocation has been the rage among file system designers in the past few years. Some of the more notable efforts are ZFS, Btrfs, Reiser4, and NILFS. The availability of open source operating systems, especially BSD and Linux, has enabled explorations of interesting new ways to manage storage and implement file systems. This is a good thing. But being technologically cool and successful does not foretell commercial success. For the purpose of evaluating file systems, I'll define commercially successful as having a large installed base for decades. The list of commercially successful file systems is fairly small: FAT, NTFS, HFS+, UFS, and ext2/3 are perhaps the most commercially successful general-purpose file systems today. The key to commercial success is to provide good value and have a good delivery channel.

Btrfs was announced by Oracle in June 2007 and is being integrated into the Linux kernel. It offers some of the more interesting features of other file systems built on B-tree notions: snapshots, efficient backups, copy-on-write, multiple file systems in a single logical volume (called subvolumes), dynamic inode allocation, multiple device support, internal mirroring, etc. These are all cool features and represent a viable technology direction. But technological feats often run into barriers to adoption which prevent them from becoming commercially successful.

The most important barrier to adoption is the delivery channel. Clearly, Microsoft dominates the industry as it carefully controls the deliver channel of software onto approximately 90% of the computer systems volume. Microsoft owns (is the proprietor of) NTFS and FAT which dominate the market. The next major vendor by volume is Apple, who owns HFS+ that is used as the default file system for OSX. The largest Linux channels, Red Hat and SuSE, use ext2/3 and seem to be planning to use ext4 in the future. Changing the default file system for a popular OS is a very expensive, time-consuming, and disruptive event, which is why OS vendors will spend a lot of time and money to fix and incrementally improve the default file system when possible. The life cycle for a default file system is measured in decades. The development of a new file system takes time, too -- on the order of 5-6 years seems to be typical as measured by having enough stable new features that the value of migration is greater than the inertia of the legacy. The barrier here is time to maturity and time to become the default in the channel. Introduced in 2007, we can expect to see Btrfs be mature in the 2012 timeframe. But what about the prospects of becoming the default in the channel?

Oracle has been trying to reduce their costs by eliminating the OS vendor for many years. Until recently, their efforts were to completely eliminate the OS vendor (raw iron) or to take away Linux from Red Hat (so-called Oracle Enterprise Linux, aka Larry Linux). Neither has been very successful. But Oracle's acquisition of Sun Microsystems changes the industry structure in many ways. Now, Oracle will have an entire solution stack: software, hardware, and services. The solution stack represents a channel for Oracle to deliver innovations, such as a spiffy new file system. Herein lies the problem for Btrfs: Oracle will now own ZFS. This means:

Btrfs is not mature enough to become the default file system for OEL. ZFS is more than 5 years old and stable enough to become the default file system for Solaris 10 and OpenSolaris.
It makes little sense for Oracle to continue funding two, competing file system projects -- one trying to match features with the other. ZFS has approximately 45 associated patents, and patents have real value ($) in the US.
Tossing Btrfs to the open-source winds is not likely to improve its schedule or channel prospects.

There are a couple of scenarios that could still play out -- Oracle could break the GPLv2 barrier that prevents Linux from accepting ZFS in the kernel or Oracle could take a more competitive stance against Red Hat and Novell by leveraging [Open]Solaris. Either way I don't see a good business case for Oracle to continue to invest in Btrfs. What do you think?

Purple Rain

2009-08-03T21:05:00.000-07:00

We were eating dinner this evening out on the deck (hot wings, one of my favorites) when nature created a beautiful sight, purple rain. Some tropical moisture flowed into the San Diego area today and made some beautiful clouds. A few dropped out some rain, though most of it was virga. As the Sun was setting, the colors were just right for a few moments and I was able to snap a picture of the purple rain over Ramona.

new zilstat beta testers wanted

2009-07-07T15:58:00.000-07:00

I've made a change to zilstat which will show how much data is written to the ZIL between txg commits. The old behaviour of time-based reporting is still available.

The way I implemented this was to allow you to specify "txg" as an interval instead of a numerical interval in seconds. However, since there may be multiple pools, zilstat will also require that you specify a pool when you want to look at the txg intervals. I don't know of an easy way to make the output intuitively obvious with multiple pools, but you can run multiple concurrent zilstats to look at multiple pools.

The output looks something like this
zilstat.ksh -tp stuff txg 10
waiting for txg commit...
TIME txg N-Bytes N-Bytes/s N-Max-Rate B-Bytes B-Bytes/s B-Max-Rate ops <=4kB 4-32kB >=32kB
2009 Jul 7 15:51:35 3812765 0 0 0 0 0 0 0 0 0 0
2009 Jul 7 15:52:05 3812766 10216 340 10216 151552 5051 151552 2 0 1 1
2009 Jul 7 15:52:35 3812767 109576 3652 109576 524288 17476 524288 4 0 0 4

I'm looking for a few beta testers with interesting sync workloads who want to try it out before I post it. If you are interested, post a comment or drop me an e-mail at Richard.Elling@RichardElling.com

Oracle Appliances

2009-04-22T10:07:00.000-07:00

Earlier today there was a conversation on the Cloud Computing group where Christopher Steel wrote, "I also [see] a play for Oracle to get into the Database appliance space. With the Sun acquisition, they now have the hardware, OS, and support pieces to delivery out-of-the-box
enterprise Database solution."

I got a tickle out of this because back in 2002, I was Chief Architect for Sun's Enterprise Engineering group where we designed appliances. Ted Persky and I published a Sun BluePrint Online article describing our Oracle - Veritas - Sun (VOS) designed appliance for running Oracle RAC. Technically, this was a challenge because none of the major products used (Solaris, VxVM, Oracle RAC, and required patches) were suited for installation in a factory. So we had to make the glue to put them together for a factory process. Anyone who has ever done this by hand will understand the difficulty of the task, and would have appreciated seeing how well the system worked out of the box -- you could go from out-of-the-box to fully operational RAC cluster in about 30 minutes!

Introducing such a product in 2002 was challenging enough for an appliance built with major components from multiple companies. But the biggest barrier to market acceptance was that nothing was happening in 2002. Nothing. Nada. Nil. Nichts. The dot-com boom had busted and nobody was buying anything. The product never went anywhere and was ultimately shelved.

So what is different today? Oracle is buying Sun and getting into the hardware business. Veritas was bought by Semantic and technologies like ZFS, ASM, and QFS are putting the squeeze on VxVM. The recession means that nothing is happening again. What can I say, but it is just the circle of life...

ZFS Tutorial at the USENIX Technical Conference

2009-04-19T16:28:00.000-07:00

I will be conducting a tutorial on ZFS at the USENIX Technical Conference in San Diego on June 14, 2009. I hope you can attend! Here is the blurb in the conference agenda.

S2 ZFS: A File System for Modern Hardware NEW!

Who should attend: Systems engineers, integrators, and administrators who are interested in deploying ZFS on Solaris, Mac OS X, or FreeBSD. Participants should be familiar with storage devices, RAID systems, logical volume managers, backup, and file system features. Special emphasis will be placed on integration considerations for virtualization, NAS, and databases.

File systems developed in the mid 20th century were severely constrained by the storage hardware available at the time. ZFS was conceived with an eye toward the hardware of the future and how storage would evolve. This perspective on the future presented an opportunity to rethink how file systems use storage hardware. The result of this endeavor is a new way of managing data which can evolve as the hardware changes, while remaining compatible with earlier notions of file system use. Along the way, new concepts such as the Hybrid Storage Pool provide new opportunities for optimization, efficiency, and data protection. In this tutorial, ZFS will be examined from the bottom up, to build a solid understanding of the data-hardware interface, and then from the top down, to provide insight into the best ways to use ZFS for applications.

Take back to work: A solid understanding of the concepts behind ZFS and how to make the best decisions when implementing storage at your site.

Topics include:

Evolution of hardware and file systems
Storage pools

RAID data protection
Import/export and shared storage
Pool parameters and features
On-disk format

Data sets

Volumes
POSIX-compliant file systems
Snapshots
Replication

Practical considerations and best practices

Deployment and migration
Virtualization
Sharing
Performance, observability, and tuning
Data protection
Hybrid storage pools
Backup, restore, and archiving

Its a boy!

2009-03-25T17:22:00.000-07:00

Henrietta gave birth to a shiny new baby boy early this morning. He just missed sharing a birthday with me by a few hours. He looks a lot like his daddy, Lucky, who is a purebred Hereford. We're so proud. Mommy and baby are doing fine.

We're expecting a few more brothers or sisters to arrive soon.

Tour goes to Palomar

2009-02-21T21:51:00.000-08:00

This year the AMGEN Tour of California is (finally) coming to San Diego County to climb Palomar Mountain. Palomar Mountain is not too far from the Ranch, and the Palomar Volunteer Fire Department has put out the call for volunteers to help the department with whatever might happen as thousands of race fans converge on the mountain. The Ramona Outback Amateur Radio Society and Ramona-CERT is providing volunteers to help with communications for the department. I've spent quite a few hours recently with the fine folks on Palomar Mountain helping to coordinate communications for the event.

Today, we held a pre-race communications exercise as part of the 2009 San Diego County CERT Spring Exercise. Dozens of hams and CERT folks surveyed and characterized communications capabilities on the mountain. What is surprising to many people who don't regularly travel to the back country is that cell phone service is likely to be non-existent. And if you have a need for emergency response, you are likely to be an hour, not minutes, from the nearest hospital. There are many dangerous areas, such as high cliffs, sharp curves, and places where you could have an accident and not be found for some time. By providing extra eyes and ears on the mountain while thousands of fans arrive to witness the Tour of California, we're hoping for an accident-free event.

If you happen to be in the crowd coming to the mountain, prepare to arrive early. There are only two roads which are practical for getting to the top and the highway patrol is planning to monitor them closely. Fortunately, the Palomar Mountain Volunteer Fire Department is also sponsoring a bar-b-que, so be sure to stop by and visit.

I look forward to seeing everyone who climbs the mountain, even those arriving in cars. Stop by and visit for a while. And who knows, you might see yourself on Versus!

zilstat improved

2009-02-07T21:02:00.000-08:00

I've improved upon the first release of zilstat by adding a few columns to show the distribution of operations by size. To understand why this might be important for your situation, I refer to the source code comments for zfs_immediate_write_sz (link good until the source changes :-)

    486         /*
    487          * Writes are handled in three different ways:
    488          *
    489          * WR_INDIRECT:
    490          *    In this mode, if we need to commit the write later, then the block
    491          *    is immediately written into the file system (using dmu_sync),
    492          *    and a pointer to the block is put into the log record.
    493          *    When the txg commits the block is linked in.
    494          *    This saves additionally writing the data into the log record.
    495          *    There are a few requirements for this to occur:
    496          *      - write is greater than zfs_immediate_write_sz
    497          *      - not using slogs (as slogs are assumed to always be faster
    498          *        than writing into the main pool)
    499          *      - the write occupies only one block
    500          * WR_COPIED:
    501          *    If we know we'll immediately be committing the
    502          *    transaction (FSYNC or FDSYNC), the we allocate a larger
    503          *    log record here for the data and copy the data in.
    504          * WR_NEED_COPY:
    505          *    Otherwise we don't allocate a buffer, and *if* we need to
    506          *    flush the write later then a buffer is allocated and
    507          *    we retrieve the data using the dmu.
    508          */

zilstat can see the size of the write and compare it to zfs_immediate_write_sz, but it is not so easy to implement the rest of the logic. To get past this difficulty, lets return to the original reason for writing zilstat in the first place: answer the question, “how much ZIL write activity is generated by my workload and will a separate log help?”

To get there from here, we could take a look at the size distribution of the ZIL writes. I've implemented this in an updated version of zilstat as follows:

Itty-bitty writes, those less than 4 kBytes. These might suggest a workload which is updating a lot of small files synchronously or perhaps a lot of metadata writes (though not all metadata writes may fit in small spaces).

Medium-sized writes, 4-32 kBytes. It might be more difficult to pin these down, so they get a bin.

Larger writes, > 32 kBytes. If you do not have a separate log, then these will be written to the pool and not the ZIL.

Using these bins will allow better observability into the work to moved around by adding a separate log. Here is what a sample output looks like:

# ./zilstat.ksh -t 60
TIME N-Bytes N-Bytes/s N-Max-Rate B-Bytes B-Bytes/s B-Max-Rate ops <=4kB 4-32kB >=32kB
2009 Feb 6 14:25:22 0 0 0 0 0 0 0 0 0 0
2009 Feb 6 14:26:22 287368 4789 283064 618496 10308 524288 7 0 2 5
2009 Feb 6 14:27:22 10304 171 10304 94208 1570 94208 2 0 0 2
2009 Feb 6 14:28:22 223120 3718 213152 618496 10308 544768 7 0 2 5
2009 Feb 6 14:29:22 0 0 0 0 0 0 0 0 0 0
2009 Feb 6 14:30:22 3336 55 1264 28672 477 12288 7 7 0 0
2009 Feb 6 14:31:22 0 0 0 0 0 0 0 0 0 0
2009 Feb 6 14:32:22 149768 2496 145480 294912 4915 282624 8 1 2 5
2009 Feb 6 14:33:22 248 4 248 4096 68 4096 1 1 0 0

For this workload, it looks like large synchronous writes are more common than small ones and these will go directly to the pool, since I do not have a separate log on any of my pools. I really can't draw many conclusions from this sample, because the rate is quite low. More experiments with busier workloads may reveal some best practices guidelines. Let me know what you discover.

zilstat

2009-01-30T09:33:00.000-08:00

A few weeks ago I had a workload which I suspected had a fair amount of synchronous writes on a ZFS file system. The general recommendation for this case is to use a separate ZIL log, preferably on a device which has a nonvolatile write cache or a write-optimized SSD. However, once you add a separate log to a pool, you cannot easily remove it. So before you make that move, it is a better idea to look at your workload and answer the question, "how much ZIL write activity is generated by my workload?"

The result of this pondering is a script I call zilstat. You can get it here.

Parallel ZFS send/receive

2009-01-25T18:49:00.000-08:00

I've been doing some work recently centered around how to backup a continuously updated directory structure. The workload generates large numbers of files and directories over time. UFS would have no chance of handling this workload, but ZFS seems to handle it quite well. The current data shows that we can use ZFS send/receive to backup this data efficiently, even when large workloads are present. There are some tricks needed, though.

But first, a quick review of ZFS file systems. ZFS is designed with a different philosophy than many other file systems. In ZFS, physical devices are assigned to a storage pool. File systems (plural) are created in the pool. In traditional file system design, there is a 1:1 relationship between the file system and a physical device or physical device look-alike (which is how RAID systems are traditionally implemented). In all modern file systems, directories (sometimes called folders) are used to manage collections of files. In ZFS, file systems work similarly. In some of the early ZFS documents, you might notice words to the effect of "in ZFS file systems are as easy to manage as directories." To a large extent, this is true: it is as easy to create a ZFS file system as it is to create a directory and for the vast majority of applications, there is no perceptible difference between a heirarchy of ZFS file systems or directories.

However easy it is to create ZFS file systems, they do increase the complexity of systems administration. Many tools for managing files and backups treat the file system boundary differently than directories. To help systems administrators maintain sanity, I recommend using file systems when you need to implement a different policy. For example, you may wish to make a file system read-only instead of read-write. The most common policy seems to be quotas and ZFS today implements quotas only on file systems. A complete list of the possible policies you might consider is available by looking at the parameters you can set on a file system:

# zfs get all my/file/system

The results of my testing on ZFS replication using send/receive adds another reason to use file systems: sends can be done in parallel. This has subtle, but important implications. In my test workload, new files and directories are constantly being created. I used the ZFS auto-snapshot feature to make regularly scheduled snapshots. I then experimented with rsync and zfs send/receive to copy the data to a backup pool. I found:

Both rsync and ZFS send/receive can make incremental backups
Both rsync and ZFS send/receive are I/O bound in performance. rsync also has the ability to manage its bandwidth, which ZFS send does not, yet (this can be managed by an entity placed in the pipeline, though). However, in this workload, the backup was iops bound, not bandwidth bound, so rsync throttling would probably not work well.
Both rsync and ZFS send/receive work on a per-filesystem basis. This is an option for rsync, but an inherent constraint for ZFS. Sends can be recursive, with the -R option, though that will also replicate the file system parameters -- more on that later.
rsync will traverse the directory structure and stat(2) every file. For my workload, this would get slower and slower over time because there would be millions of more files added over time. The performance of these stats will be impacted worse by iop-bound workloads, such as mine. Note: DNLC size is important for my workload and requires tuning.
ZFS send/receive can send differences in the file system dataset, which doesn't really care about files or directories.
ARC size can impact backup performance. The ongoing workload is iop-bound with many application reads and writes occurring continuously while the backup is being made. Recent writes will likely still be in the ARC, as long as the ARC is large enough. For my system, careful monitoring of the ARC size confirmed that I could devote a substantially large amount of RAM for the ARC while meeting application requirements. No L2ARC was used, mostly because operations constraints limited me to using Solaris 10 10/08, which does not have the L2ARC feature. Backups using ZFS send/receive should be even more efficient with a reasonably large L2ARC -- 100+ GBytes.
Over time, I believe rsync will get totally bogged down traversing the directory structure and stat'ing files whereas ZFS send/receive will take nominally the same time -- based on the amount of changes during the time interval.
If something goes amiss, and for some reason the backups are not completed for a long period of time, then it will take a long time to catch up, for either choice. For this workload, the data tends to be added and removed, not reused. For a reuse case, such as you might find with a database, more work will have to be done to fully understand the implications of incremental, real-time replication.

The big trick to get this to work well is the parallelism obtained by backing up multiple file systems in parallel. The bottleneck for the parallel replication is the iops load which will increase for rsync over time, while remain more-or-less constant for ZFS send/receive. Having an iops-bounded workload means that latency is more important than bandwidth, and while one of the parallel sends is waiting, another can be working. In a sense, this is very similar to the way that chip multithreading (CMT) works, but ZFS is waiting on disks, not memory.

There is a lot more work to be done here, but I feel this is a pretty positive start. It may take months or years to really get my workload to a state where the steady state is such that it can be fully characterized. I look forward to seeing how it goes.

Its a conspiracy...

2009-01-25T18:18:00.000-08:00

The other day I blogged about some blown capacitors on my graphics card. Well, around the same time, my wife noticed a strange burning smell near her computer. Sure enough, it smelled like burned electronics. Since there is a bunch of electronics near her desk and everything seemed to be working ok, it took a while to track down the source of the smell.

Argv! Another blown graphics card. This time the fan motor blew and sent the fan tumbling around inside the case. The card seemed to keep working ok, though it is not clear how long that would have been the case. Fortunately, she doesn't stress the 3-D graphics too much, except for the occasional screen blanker.
But now I think there is a conspiracy... how could two different graphics cards from different vendors, in different computers, connected to different power conditioners, both fail at around the same time -- and yet keep working for a while? It must've been related to those strange lights and humming noise we saw moving rapidly over the hilltops... no, those were F/A-18s from Miramar... time to call the ghostbusters...

POP! What was that???

2009-01-23T10:46:00.000-08:00

I was sitting in my office yesterday afternoon, with Rosie, our Australian Shepherd, asleep at my feet, when I heard a loud POP! It sounded like a fire cracker. Rosie didn't budge, which is no longer surprising as she has been losing her hearing lately. I looked around to see if I could find something amiss. Nothing. No smoke. No funny smells. No new holes in the window. Nothing. So I went about my work as usual.
Today, when I went into the office and sat down to read the morning's e-mail, my monitors were all goofed up. Argv! This looks like what I would expect from failed hardware: gobbledegook on the screen. So I pulled the video card and the problem is self-evident.

The tops of three capacitors were blown off! I'm not really surprised the capacitors blew -- that happens on these cheap devices. What surprised me is that I worked well into the evening yesterday with no problems. I presume that when I retired for the evening and the screenblank kicked in, that was the end of proper functioning. Fortunately, I also have plenty of spares hanging around... always be prepared for failures.

Cross over and link back

2009-01-23T10:43:00.000-08:00

I'm continuing my previous blog on this new site.