The kstat(1m) command rewritten in C was contributed by David Höppner, an active member of the illumos community. It is fast and efficient at filtering and printing kstats. By contrast, the old perl version had to start perl (an interpreter), find and load the kstat-to-perl module, and then filter and print the kstats. Internal to the kernel, kstats are stored as a name-value list (nvlist) containing strongly-typed data. Many of these are 64-bit integers. This poses a problem for the version of perl used (5.12) as the 64-bit support is dependent on the compiled version and illumos can be compiled for both 32 and 64 bit processors. To compensate for this mismatch, the following was added to the man page for kstat(3perl):

Several of the statistics provided by the kstat facility are stored as 64-bit integer values. Perl 5 does not yet internally support 64-bit integers, so these values are approximated in this module. There are two classes of 64-bit value to be dealt with: 64-bit intervals and times

These are the crtime and snaptime fields of all the statistics hashes, and the wtime, wlentime, wlastupdate, rtime, rlentime and rlastupdate fields of the kstat I/O statistics structures. These are measured by the kstat facility in nanoseconds, meaning that a 32-bit value would represent approximately 4 seconds. The alternative is to store the values as floating-point numbers, which offer approximately 53 bits of precision on present hardware. 64-bit intervals and timers as floating point values expressed in seconds, meaning that time-related kstats are being rounded to approximately microsecond resolution.

It is not useful to store these values as 32-bit values. As noted above, floating-point values offer 53 bitsof precision. Accordingly, all 64-bit counters are stored as floating-point values.

For consumers of the kstat(1m) command output, this means that kstat I/O statistics are stored as seconds (floating point) instead of nanoseconds. For example, with formatting adjusted for readability:

Perl-based kstat(1m)

# kstat -p sd:0:sd0

sd:0:sd0:class disk

sd:0:sd0:crtime 1855326.99995062

sd:0:sd0:nread 380919301

sd:0:sd0:nwritten 1984175104

sd:0:sd0:rcnt 0

sd:0:sd0:reads 18455

sd:0:sd0:rlastupdate 2371703.49260763

sd:0:sd0:rlentime 147.154123471

sd:0:sd0:rtime 49.399890683

sd:0:sd0:snaptime 2371828.77138052

sd:0:sd0:wcnt 0

sd:0:sd0:wlastupdate 2371703.49174494

sd:0:sd0:wlentime 2.425675727

sd:0:sd0:writes 103602

sd:0:sd0:wtime 1.43643661

C-based kstat(1m)

# kstat -p sd:0:sd0

sd:0:sd0:class disk

sd:0:sd0:crtime 244.271312204

sd:0:sd0:nread 25549493

sd:0:sd0:nwritten 1698218496

sd:0:sd0:rcnt 0

sd:0:sd0:reads 4043

sd:0:sd0:rlastupdate 104543293563241

sd:0:sd0:rlentime 68750036336

sd:0:sd0:rtime 64365048052

sd:0:sd0:snaptime 104509.092582653

sd:0:sd0:wcnt 0

sd:0:sd0:wlastupdate 104543293482995

sd:0:sd0:wlentime 4569934990

sd:0:sd0:writes 289766

sd:0:sd0:wtime 4551425719

I find kstat(1m) output to be very convenient for historical tracking and use it often. If you do too, then be aware of this conversion.

One of the best features of the new C-based kstat(1m) is the ability to get kstats as JSON. This is even more useful than the "parseable" output shown previously.

# kstat -j sd:0:sd0

[{

"module": "sd",

"instance": 0,

"name": "sd0",

"class": "disk",

"type": 3,

"snaptime": 104547.013504692,

"data": {

"crtime": 244.271312204,

"nread": 25549493,

"nwritten": 1700980224,

"rcnt": 0,

"reads": 4043,

"rlastupdate": 104733296813446,

"rlentime": 68901598866,

"rtime": 64513785819,

"snaptime": 104547.013504692,

"wcnt": 0,

"wlastupdate": 104733296708770,

"wlentime": 4579560895,

"writes": 290404,

"wtime": 4561051625

}

}]

Using JSON has the added advantage of being easy to parse without making assumptions about the data. For example, did you know that some kernel modules use ':' in the kstat module, instance, name, or statistic? This makes using the parseable output a little bit tricky. The JSON output is not affected and is readily and consistently readable or storable in the many tools that support JSON.

Now you can see how to take advantage of the kstat(1m) command and how it has evolved under illumos to be more friendly to building tools and taking quick measurements. Go forth and kstat!