Sidenote: Today we made eight network/iSCSI related tweaks that, in aggregate, have resulted in dramatic performance improvements (some I just hadn't gotten around to yet, others suggested by Sun's Mertol Ozyoney)...
- disabling the Nagle algorithm on the head node
- setting each iSCSI target block size to match the ZFS record size of 128K
- disabling "thin provisioning" on the iSCSI targets
- enabling jumbo frames everywhere (each switch and NIC)- raising ddi_msix_alloc_limit to 8
- raising ip_soft_rings_cnt to 16
- raising tcp_deferred_acks_max to 16
- raising tcp_local_dacks_max to 16
Rerunning the same tests, we now see...
[1GB file size, 1KB record size]
Command: iozone -i o -i 1 -i 2 -r 1k -s 1g -f /data-das/perftest/1gbtest
Rewrite: 183170
Read: 433205
Reread: 435503
Random Read: 90118
Random Write: 19488
[8GB file size, 512KB record size]
Write: 463260
Rewrite: 449280
Read: 1092291
Reread: 881044
Random Read: 442565
Random Write: 565565
[64GB file size, 1MB record size]
Command: iozone -i o -i 1 -i 2 -r 1m -s 64g -f /data-das/perftest/64gbtest
Rewrite: 342788
Read: 609553
Reread: 645618
Random Read: 218874
Random Write: 339624
Thanks so much to everyone for all their great contributions!
-Gray
