Monday, October 26, 2009

LISA09 next week

The USENIX LISA09 conference is next week and I'll be leaving the ranch to travel back east to attend. On Monday, November 2 I'll be giving a tutorial on ZFS. Much has changed since I gave a similar tutorial at the USENIX Technical Conference last June. Somehow I've got to get the content down from 200+ slides into something feasible to deliver as a one-day tutorial. As the LISA audience is more geared towards systems administration, I'll whittle down the technical details and concentrate on more practical applications of the ZFS technology. Afterwards, at the research center (pub) we can cover other details, as needed. Please join us in Baltimore for a week with the talented teams who manage large computer sites!

Wednesday, October 21, 2009

check integrity of ZFS send streams with zstreamdump

If you wander through the OpenSolaris ZFS-discuss archives or look at the ZFS Best Practices Guide, then you can encounter references and debates about whether the zfs send and zfs receive commands are suitable for backups. As I've described before, zfs send and zfs receive can be part of a comprehensive backup strategy for high-transaction environments. But people get nervous when we discuss placing a zfs send stream on persistent storage. The reasoning is that if the stream gets corrupted, then it is useless. There is an RFE open to improve the robustness of zfs receive, but that is little consolation for someone who has lost data.

The fundamental design of ZFS is exposed in zfs send -- the send stream contains an object, not files. This is great for replicating objects, and since ZFS file systems and volumes are objects, it is quite handy. This is why zfs send and zfs receive do not replace the functionality of an enterprise backup system that works on files. So, I expect the technologies to remain complementary for a very long time.

But there are some simple things which can improve management of zfs send streams. It is a good idea to check the integrity of the stream when stored on permanent storage before you try a zfs receive, or just to sleep better at night. You can do that by telling zfs to not actually apply the receive using the "-n" option to zfs receive, but this only returns a boolean response. Something more concrete and descriptive would be nice...

OpenSolaris build 125 brings the zstreamdump(1m) command, that allows you to examine the contents of a zfs send stream. To demonstrate, I made a quick (diving) pool called "zdiving," copied some data to it, made a snapshot saved as a file, and ran zstreamdump. Observe:

# ramdiskadm -a rd1 200m
# zpool create zdiving /dev/ramdisk/rd1
# cp somefile /zdiving
# zfs snapshot zdiving@today
# zfs send zdiving@today > zdiving.zstream
# zstreamdump < zdiving.zstream
BEGIN record
version = 1
magic = 2f5bacbac
creation_time = 4adf8f3d
type = 2
flags = 0x0
toguid = 582cc98ae8f284cb
fromguid = 0
toname = zdiving@today
END checksum = 705e7f009f/441064477b71ce/1aacbf27d3a3e8a0/119dd4527bd9c6a3
Total DRR_BEGIN records = 1
Total DRR_END records = 1
Total DRR_OBJECT records = 5
Total DRR_FREEOBJECTS records = 791
Total DRR_WRITE records = 5
Total DRR_FREE records = 9
Total records = 812
Total write size = 2560 (0xa00)
Total stream length = 256696 (0x3eab8)

So if you have any zfs send streams lying around, give zstreamdump a try. If you find a checksum mismatch, then please let me know... I'm collecting data on real checksum mismatches found in the wild.