Performance before tweaking

In order to measure performance we have to be able to measure performance and understand what we are measuring. I came across this article that explains the benchmarking portion.

Benchmarking explained

iostat -xmhcd 1 10 Linux 3.2.0-35-generic-pae (rick-desktop) 23/12/12 _i686_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.93 0.01 1.48 1.47 0.00 88.12 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.92 28.49 0.14 0.52 0.02 0.11 403.04 0.01 7.82 14.68 5.98 2.84 0.19 sdb 0.05 0.00 1.09 0.00 0.00 0.00 7.99 0.00 0.82 0.82 0.14 0.81 0.09 sdc 0.93 28.53 0.14 0.51 0.02 0.11 410.21 0.01 8.62 15.94 6.60 3.08 0.20 sdd 0.99 28.50 0.14 0.55 0.02 0.11 393.72 0.00 6.73 12.49 5.29 2.60 0.18 sde 2.04 0.00 3.42 0.00 0.26 0.00 155.61 0.01 3.38 3.32 62.51 1.79 0.61 sdf 0.99 4.85 9.87 4.86 0.37 0.55 128.17 0.80 54.08 10.58 142.51 4.21 6.20 md127 0.00 0.00 0.73 4.55 0.07 0.45 202.99 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 8.13 1.33 0.29 0.16 96.72 0.00 0.00 0.00 0.00 0.00 0.00 sdg 0.93 28.53 0.14 0.52 0.02 0.11 406.61 0.01 11.76 15.09 10.86 3.90 0.26 sdh 0.19 0.99 1.37 0.33 0.03 0.16 219.40 0.06 36.58 4.57 169.78 2.67 0.46 sdi 0.95 28.52 0.14 0.53 0.02 0.11 401.79 0.01 10.32 14.03 9.32 3.86 0.26 avg-cpu: %user %nice %system %iowait %steal %idle 37.82 0.00 9.84 16.84 0.00 35.49 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 421.00 5648.00 97.00 99.00 22.18 22.51 466.94 5.64 24.06 33.73 14.59 3.41 66.80 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 220.00 4876.00 54.00 110.00 19.32 20.50 497.27 2.13 12.88 12.89 12.87 3.15 51.60 sdd 465.00 4853.00 50.00 142.00 20.27 20.50 434.83 1.34 7.02 12.00 5.27 2.31 44.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 1.00 0.00 19.00 0.00 0.66 71.16 0.04 2.11 0.00 2.11 0.42 0.80 md127 0.00 0.00 972.00 780.00 96.00 78.00 203.40 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 303.00 4868.00 64.00 119.00 19.68 20.49 449.57 2.23 12.11 16.94 9.51 3.61 66.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 502.00 4911.00 71.00 136.00 20.96 20.70 412.25 1.71 8.31 9.86 7.50 2.69 55.60 avg-cpu: %user %nice %system %iowait %steal %idle 37.34 0.00 9.72 17.39 0.00 35.55 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 284.00 5918.00 50.00 100.00 19.57 24.49 601.55 1.99 14.32 13.04 14.96 3.95 59.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 574.00 5931.00 58.00 74.00 21.22 23.51 693.94 1.52 11.45 11.03 11.78 4.21 55.60 sdd 293.00 5966.00 47.00 107.00 19.08 22.74 556.16 1.50 9.09 10.04 8.67 2.81 43.20 sde 11.00 0.00 11.00 0.00 1.38 0.00 256.00 0.01 1.09 1.09 0.00 0.73 0.80 sdf 0.00 37.00 1.00 78.00 0.00 1.22 31.80 0.22 2.73 28.00 2.41 0.56 4.40 md127 0.00 0.00 972.00 940.00 96.00 94.00 203.51 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 22.00 0.00 1.38 0.00 128.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 534.00 5925.00 72.00 83.00 21.09 22.52 576.15 2.48 15.64 14.50 16.63 4.59 71.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 302.00 5915.00 51.00 104.00 19.63 23.51 570.01 1.73 11.15 12.63 10.42 3.79 58.80 avg-cpu: %user %nice %system %iowait %steal %idle 37.63 0.00 9.02 17.53 0.00 35.82 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 277.00 6228.00 96.00 146.00 19.71 25.01 378.41 3.15 11.64 11.50 11.73 2.66 64.40 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 525.00 6209.00 107.00 178.00 19.73 24.02 314.33 3.56 11.72 18.24 7.80 2.04 58.00 sdd 565.00 6199.00 67.00 189.00 21.69 26.00 381.56 2.52 9.36 12.90 8.11 1.84 47.20 sde 12.00 0.00 12.00 0.00 1.50 0.00 256.00 0.02 2.00 2.00 0.00 1.33 1.60 sdf 0.00 3.00 0.00 7.00 0.00 0.26 76.57 0.01 1.14 0.00 1.14 0.57 0.40 md127 0.00 0.00 972.00 1000.00 96.00 100.00 203.55 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 24.00 0.00 1.50 0.00 128.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 562.00 6194.00 57.00 201.00 20.18 25.01 358.67 3.66 13.95 19.16 12.48 2.85 73.60 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 273.00 6217.00 63.00 161.00 19.55 24.01 398.21 2.38 9.95 12.32 9.02 2.64 59.20 avg-cpu: %user %nice %system %iowait %steal %idle 30.00 0.00 10.26 23.08 0.00 36.67 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 315.00 6128.00 53.00 138.00 19.69 24.50 473.84 2.54 11.58 15.92 9.91 3.12 59.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 263.00 6120.00 67.00 149.00 21.01 24.50 431.56 3.10 13.35 23.70 8.70 3.20 69.20 sdd 419.00 6107.00 58.00 164.00 20.12 24.50 411.57 1.02 4.59 7.66 3.51 1.86 41.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 28.00 0.00 2.00 0.00 0.12 120.00 0.03 14.00 0.00 14.00 14.00 2.80 md127 0.00 0.00 972.00 980.00 96.00 98.00 203.54 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 1.00 0.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 239.00 6142.00 81.00 129.00 20.47 24.50 438.59 2.99 14.34 16.25 13.15 3.52 74.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.00 0.00 0.00 0.00 28.40 sdi 569.00 6126.00 69.00 145.00 21.23 25.00 442.47 2.38 11.21 16.64 8.63 3.14 67.20 avg-cpu: %user %nice %system %iowait %steal %idle 18.13 0.00 9.84 29.02 0.00 43.01 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 298.00 6490.00 61.00 161.00 20.61 26.00 430.02 2.27 10.04 17.51 7.20 2.76 61.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 612.00 6518.00 85.00 120.00 21.97 26.52 484.41 3.39 13.11 14.49 12.13 3.08 63.20 sdd 319.00 6508.00 82.00 127.00 20.79 25.52 453.82 3.16 11.60 17.90 7.53 2.51 52.40 sde 11.00 0.00 11.00 0.00 1.38 0.00 256.00 0.01 0.73 0.73 0.00 0.36 0.40 sdf 0.00 53.00 54.00 41.00 0.22 1.83 44.21 1.35 14.19 7.63 22.83 6.82 64.80 md127 0.00 0.00 1013.00 1043.00 100.00 104.01 203.22 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 56.00 0.00 1.51 0.00 55.14 0.00 0.00 0.00 0.00 0.00 0.00 sdg 431.00 6499.00 78.00 160.00 21.20 26.51 410.52 4.04 17.04 17.74 16.70 3.41 81.20 sdh 0.00 0.00 35.00 0.00 0.14 0.00 8.00 2.25 72.46 72.46 0.00 6.29 22.00 sdi 484.00 6496.00 53.00 160.00 21.33 25.51 450.33 2.21 10.25 14.87 8.72 3.29 70.00 avg-cpu: %user %nice %system %iowait %steal %idle 37.08 0.00 7.57 18.28 0.00 37.08 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 510.00 5403.00 61.00 221.00 19.07 22.01 298.33 2.44 8.50 10.36 7.98 1.99 56.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 560.00 5422.00 101.00 187.00 19.39 22.50 297.86 4.02 11.60 20.44 6.82 1.89 54.40 sdd 598.00 5442.00 87.00 205.00 19.98 22.63 298.85 1.90 6.12 9.66 4.62 1.59 46.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 891.00 882.00 88.00 88.24 203.58 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 496.00 5420.00 59.00 183.00 19.00 21.94 346.45 3.24 12.58 18.31 10.73 2.99 72.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 665.00 5408.00 87.00 212.00 19.75 23.00 292.82 2.44 7.83 12.51 5.91 2.05 61.20 avg-cpu: %user %nice %system %iowait %steal %idle 35.05 0.00 8.51 17.01 0.00 39.43 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 637.00 5176.00 108.00 201.00 21.64 21.10 283.26 3.36 9.37 12.63 7.62 1.88 58.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 417.00 5138.00 80.00 227.00 20.21 21.00 274.89 3.40 9.25 18.05 6.15 1.75 53.60 sdd 630.00 5095.00 106.00 279.00 21.14 21.00 224.19 1.82 4.70 10.34 2.55 1.16 44.80 sde 12.00 0.00 12.00 0.00 1.50 0.00 256.00 0.01 1.00 1.00 0.00 0.67 0.80 sdf 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 972.00 840.00 96.00 84.00 203.44 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 24.00 0.00 1.50 0.00 128.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 352.00 5161.00 75.00 265.00 19.92 21.78 251.18 2.93 8.40 15.63 6.35 2.19 74.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 726.00 5151.00 88.00 192.00 21.95 20.26 308.71 2.72 8.24 11.18 6.90 2.21 62.00 avg-cpu: %user %nice %system %iowait %steal %idle 38.36 0.00 8.95 17.90 0.00 34.78 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 627.00 5603.00 81.00 156.00 19.63 22.50 364.05 2.16 9.05 13.73 6.62 2.43 57.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 545.00 5622.00 49.00 164.00 19.61 22.11 401.13 1.46 6.78 12.57 5.05 2.46 52.40 sdd 499.00 5606.00 77.00 144.00 18.07 22.19 373.07 2.66 10.48 20.05 5.36 2.03 44.80 sde 12.00 0.00 12.00 0.00 1.50 0.00 256.00 0.02 2.00 2.00 0.00 1.33 1.60 sdf 0.00 9.00 2.00 16.00 0.01 0.84 96.89 0.08 4.67 26.00 2.00 3.33 6.00 md127 0.00 0.00 891.00 900.00 88.00 90.00 203.54 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 24.00 0.00 1.50 0.00 128.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 502.00 5612.00 70.00 135.00 19.04 22.00 410.07 3.03 14.30 17.09 12.86 3.63 74.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 500.00 5597.00 62.00 162.00 19.00 22.93 383.39 1.89 8.71 13.74 6.79 2.45 54.80 avg-cpu: %user %nice %system %iowait %steal %idle 37.78 0.00 9.82 16.62 0.00 35.77 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 349.00 5913.00 51.00 74.00 19.73 22.50 691.84 2.37 14.78 17.65 12.81 4.74 59.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 334.00 5906.00 52.00 90.00 19.77 23.91 629.97 1.70 12.08 12.46 11.87 3.80 54.00 sdd 382.00 5901.00 51.00 95.00 21.41 23.73 633.15 1.15 8.11 10.04 7.07 2.77 40.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 972.00 938.00 96.00 93.76 203.47 0.00 0.00 0.00 0.00 0.00 0.00 md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 260.00 5903.00 50.00 93.00 19.94 23.91 628.03 2.68 18.85 22.32 16.99 4.95 70.80 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 316.00 5952.00 50.00 91.00 19.68 23.91 633.19 1.75 12.48 13.60 11.87 4.20 59.20


bonnie++ output before any tweaking is done. I use the 8Gb size during the tests because the RAM size on the machine is 4Gb.

bonnie++ -u rick -s 8192 -d /TEMP/ Using uid:1000, gid:1000. Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rick-desktop 8G 570 96 149337 16 93168 13 2014 93 440459 32 456.1 11 Latency 70374us 105ms 239ms 13849us 135ms 1618ms Version 1.96 ------Sequential Create------ --------Random Create-------- rick-desktop -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15454 17 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 578us 2880us 1489us 1646us 41us 804us 1.96,1.96,rick-desktop,1,1356253972,8G,,570,96,149337,16,93168,13,2014,93,440459,32,456.1,11,16,,,,,15454,17,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,70374us,105ms,239ms,13849us,135ms,1618ms,578us,2880us,1489us,1646us,41us,804us

Whoaa!!! ~50MB/s average transfer.. Great improvement over the previous ~10MB/s. This is not a good test because the transfer rate is dependent on the source drive speed which in my case is the slowest drive (/dev/sdf if you have been following)
rsync --progress /home/rick/rhel-server-6.0-x86_64-dvd.iso /TEMP rhel-server-6.0-x86_64-dvd.iso 3431618560 100% 48.52MB/s 0:01:07 (xfer#1, to-check=0/1) sent 3432037552 bytes received 31 bytes 50845001.23 bytes/sec total size is 3431618560 speedup is 1.00

dd gives us write speed but uses buffering
sudo dd if=/dev/zero of=/TEMP/testfile bs=800M count=10 10+0 records in 10+0 records out 8388608000 bytes (8.4 GB) copied, 56.6043 s, 148 MB/s

and finally hdparm

sudo hdparm -tT /dev/sd[abcdefghi] /dev/md126 /dev/md127 /dev/sda: Timing cached reads: 8872 MB in 2.00 seconds = 4438.75 MB/sec Timing buffered disk reads: 342 MB in 3.01 seconds = 113.80 MB/sec /dev/sdb: Timing cached reads: 8850 MB in 2.00 seconds = 4427.25 MB/sec Timing buffered disk reads: 306 MB in 3.01 seconds = 101.60 MB/sec /dev/sdc: Timing cached reads: 8492 MB in 2.00 seconds = 4248.25 MB/sec Timing buffered disk reads: 358 MB in 3.01 seconds = 119.12 MB/sec /dev/sdd: Timing cached reads: 8560 MB in 2.00 seconds = 4282.43 MB/sec Timing buffered disk reads: 338 MB in 3.00 seconds = 112.49 MB/sec /dev/sde: Timing cached reads: 8948 MB in 2.00 seconds = 4476.95 MB/sec Timing buffered disk reads: 442 MB in 3.01 seconds = 146.75 MB/sec /dev/sdf: Timing cached reads: 8468 MB in 2.00 seconds = 4236.59 MB/sec Timing buffered disk reads: 226 MB in 3.02 seconds = 74.93 MB/sec /dev/sdg: Timing cached reads: 8338 MB in 2.00 seconds = 4171.41 MB/sec Timing buffered disk reads: 332 MB in 3.01 seconds = 110.44 MB/sec /dev/sdh: Timing cached reads: 8598 MB in 2.00 seconds = 4301.59 MB/sec Timing buffered disk reads: 358 MB in 3.00 seconds = 119.23 MB/sec /dev/sdi: Timing cached reads: 8464 MB in 2.00 seconds = 4234.00 MB/sec Timing buffered disk reads: 322 MB in 3.00 seconds = 107.25 MB/sec /dev/md126: Timing cached reads: 9650 MB in 2.00 seconds = 4828.44 MB/sec Timing buffered disk reads: 946 MB in 3.02 seconds = 313.69 MB/sec /dev/md127: Timing cached reads: 9910 MB in 2.00 seconds = 4958.40 MB/sec Timing buffered disk reads: 314 MB in 3.01 seconds = 104.26 MB/sec