If you wander through the OpenSolaris ZFS-discuss archives or look at the ZFS Best Practices Guide, then you can encounter references and debates about whether the zfs send and zfs receive commands are suitable for backups. As I've described before, zfs send and zfs receive can be part of a comprehensive backup strategy for high-transaction environments. But people get nervous when we discuss placing a zfs send stream on persistent storage. The reasoning is that if the stream gets corrupted, then it is useless. There is an RFE open to improve the robustness of zfs receive, but that is little consolation for someone who has lost data.
The fundamental design of ZFS is exposed in zfs send -- the send stream contains an object, not files. This is great for replicating objects, and since ZFS file systems and volumes are objects, it is quite handy. This is why zfs send and zfs receive do not replace the functionality of an enterprise backup system that works on files. So, I expect the technologies to remain complementary for a very long time.
But there are some simple things which can improve management of zfs send streams. It is a good idea to check the integrity of the stream when stored on permanent storage before you try a zfs receive, or just to sleep better at night. You can do that by telling zfs to not actually apply the receive using the "-n" option to zfs receive, but this only returns a boolean response. Something more concrete and descriptive would be nice...
OpenSolaris build 125 brings the zstreamdump(1m) command, that allows you to examine the contents of a zfs send stream. To demonstrate, I made a quick (diving) pool called "zdiving," copied some data to it, made a snapshot saved as a file, and ran zstreamdump. Observe:
# ramdiskadm -a rd1 200m
/dev/ramdisk/rd1
# zpool create zdiving /dev/ramdisk/rd1
# cp somefile /zdiving
# zfs snapshot zdiving@today
# zfs send zdiving@today > zdiving.zstream
# zstreamdump < zdiving.zstream
BEGIN record
version = 1
magic = 2f5bacbac
creation_time = 4adf8f3d
type = 2
flags = 0x0
toguid = 582cc98ae8f284cb
fromguid = 0
toname = zdiving@today
END checksum = 705e7f009f/441064477b71ce/1aacbf27d3a3e8a0/119dd4527bd9c6a3
SUMMARY:
Total DRR_BEGIN records = 1
Total DRR_END records = 1
Total DRR_OBJECT records = 5
Total DRR_FREEOBJECTS records = 791
Total DRR_WRITE records = 5
Total DRR_FREE records = 9
Total records = 812
Total write size = 2560 (0xa00)
Total stream length = 256696 (0x3eab8)
So if you have any zfs send streams lying around, give zstreamdump a try. If you find a checksum mismatch, then please let me know... I'm collecting data on real checksum mismatches found in the wild.
I am very interested in this, however, I confirmed one piece of wrong info you have in this blog -
ReplyDeleteIf I store a zfs send, and then I intentionally corrupt it (toggle bits in the middle) and then I pipe it to "zfs receive -n" ... unfortunately I get a 0 exit status regardless of whether the data stream is actually good or bad.
In order to get a "1" exit status, the -n cannot be used. Which means of course, truly doing the restore and writing it to disk.
I will try your zstreamdump and hope for better results. :-)
It depends on the nature of the corruption. Regardless, the answer is zstreamdump.
ReplyDeleteHi, just discovered the inverse of what rahvee said - zstreamdump does not detect a modified stream (simply remove a few trailing bytes at the end of it..) and happily accepts it and exits with status 0, while zfs receive refuses it and exits with 1, so it would be impossible to recover such stream, tested on snv_133, now who/what to believe on checking the stream's integrity ?
ReplyDelete@Jonas, please file a bug and detail the steps you used so that the bug can be recreated. Thanks.
ReplyDeleteRichard, ok, filed a bug in bugster, very simple to reproduce, take any stream and copy the very 1st byte from it into a new file with dd, then pipe it into zstreamdump - it will exit with status code 0.
ReplyDeleteI did a zfs send to an external usb hardrive in order to destroy the pool ( I want to replace a failing drive and add a new drive) Can this tool be applied to a zipped version of the stream dump?
ReplyDeleteYes, but you'll need to unzip the file first.
Delete