Find the bottleneck with iostat

Current performance before I start rebuilding:

While running bonnie++ I run iostat in an attempt to pinpoint the bottleneck:iostat -xmhcd 1 10 Linux 3.2.0-34-generic-pae (rick-desktop) 18/12/12 _i686_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 10.56 0.02 3.32 2.22 0.00 83.89 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 39.36 3.98 18.52 0.29 3.83 0.02 418.70 0.23 12.15 11.76 37.45 3.71 6.97 sdb 0.02 0.40 0.42 0.22 0.00 0.06 199.06 0.02 28.42 0.95 82.06 1.45 0.09 sdc 39.19 4.00 18.51 0.29 3.83 0.02 418.82 0.20 10.46 10.12 32.16 3.30 6.19 sdd 39.04 4.05 18.49 0.27 3.83 0.02 419.67 0.34 18.16 10.99 513.49 3.67 6.89 sde 0.05 1.14 0.78 0.14 0.00 0.00 18.21 0.01 6.05 1.73 29.75 1.97 0.18 sdf 0.35 0.91 1.97 0.97 0.02 0.01 23.79 0.08 28.21 20.15 44.63 4.55 1.33 md0 0.00 0.00 280.99 15.76 19.11 0.06 132.29 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 2.37 83.17 0.01 9.63 230.89 0.00 0.00 0.00 0.00 0.00 0.00 sdg 39.20 4.06 18.48 0.33 3.83 0.02 418.71 0.19 10.31 10.35 7.86 3.50 6.57 sdh 0.01 60.48 1.13 20.92 0.00 9.57 889.31 5.62 254.71 11.03 267.82 6.36 14.02 sdi 39.17 4.04 18.51 0.26 3.83 0.02 419.49 0.38 20.40 12.34 587.26 4.21 7.91 avg-cpu: %user %nice %system %iowait %steal %idle 14.43 0.00 2.53 22.03 0.00 61.01 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 237.00 301.00 48.00 49.00 1.09 1.14 47.09 3.25 30.35 29.67 31.02 3.26 31.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 280.00 518.00 21.00 45.00 1.28 2.88 128.97 0.34 13.21 14.29 12.71 2.42 16.00 sdd 257.00 479.00 14.00 46.00 1.06 2.26 113.33 8.37 144.73 52.57 172.78 12.87 77.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 1499.00 0.00 5.86 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 164.00 370.00 40.00 23.00 0.80 1.62 78.60 1.13 14.79 18.30 8.70 2.41 15.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 231.00 352.00 24.00 45.00 1.00 1.64 78.38 7.51 106.26 37.67 142.84 11.07 76.40 avg-cpu: %user %nice %system %iowait %steal %idle 14.03 0.00 1.79 20.15 0.00 64.03 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 269.00 518.00 43.00 84.00 1.24 2.58 61.67 2.36 20.94 23.44 19.67 2.61 33.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 394.00 600.00 22.00 57.00 1.61 2.21 98.94 2.52 7.85 21.27 2.67 2.63 20.80 sdd 485.00 591.00 43.00 38.00 2.06 2.39 112.59 3.84 41.38 42.05 40.63 11.26 91.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 1988.00 0.00 7.77 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 522.00 584.00 42.00 38.00 2.22 2.43 119.00 0.77 9.65 9.52 9.79 1.65 13.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 228.00 564.00 9.00 20.00 0.82 2.21 214.07 6.63 158.21 62.22 201.40 30.90 89.60 avg-cpu: %user %nice %system %iowait %steal %idle 14.83 0.00 1.53 20.46 0.00 63.17 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 187.00 389.00 32.00 60.00 0.86 1.75 58.09 0.87 9.48 21.62 3.00 1.65 15.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 220.00 213.00 29.00 62.00 0.99 1.43 54.51 3.27 56.84 27.31 70.65 2.73 24.80 sdd 239.00 361.00 41.00 36.00 1.10 1.73 75.22 9.09 91.53 76.10 109.11 10.70 82.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 1166.00 0.00 4.55 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 113.00 210.00 28.00 18.00 0.55 0.89 64.17 0.10 2.09 2.86 0.89 0.70 3.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 229.00 178.00 27.00 34.00 1.09 0.78 62.69 9.32 160.92 86.96 219.65 15.15 92.40 avg-cpu: %user %nice %system %iowait %steal %idle 14.50 0.00 2.29 20.36 0.00 62.85 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 576.00 871.00 52.00 46.00 2.45 3.63 127.10 3.71 26.41 36.92 14.52 3.02 29.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 500.00 1013.00 43.00 54.00 2.12 4.18 133.03 1.61 16.45 23.07 11.19 3.59 34.80 sdd 311.00 841.00 30.00 60.00 1.34 3.02 99.02 7.99 40.36 18.80 51.13 7.56 68.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 3153.00 0.00 12.32 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 686.00 978.00 41.00 91.00 2.84 4.18 108.85 1.22 9.21 11.12 8.35 1.97 26.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 391.00 872.00 38.00 46.00 1.69 3.15 117.90 13.41 101.29 63.47 132.52 11.62 97.60 avg-cpu: %user %nice %system %iowait %steal %idle 13.96 0.00 2.28 36.29 0.00 47.46 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 28.00 546.00 2.00 32.00 0.12 2.26 143.06 0.38 11.18 16.00 10.88 1.76 6.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 16.00 506.00 3.00 28.00 0.07 2.09 142.97 0.34 10.97 28.00 9.14 2.97 9.20 sdd 319.00 287.00 20.00 27.00 1.32 1.42 119.66 11.57 303.83 105.00 451.11 21.28 100.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 6.00 0.00 3.00 0.00 0.04 24.00 0.04 13.33 0.00 13.33 13.33 4.00 md0 0.00 0.00 0.00 2007.00 0.00 7.84 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 84.00 518.00 14.00 30.00 0.38 2.14 117.45 0.36 8.27 11.71 6.67 1.45 6.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 157.00 449.00 11.00 48.00 0.66 2.63 114.31 1.66 118.03 24.00 139.58 3.86 22.80 avg-cpu: %user %nice %system %iowait %steal %idle 14.82 0.00 1.51 40.20 0.00 43.47 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 112.00 162.00 6.00 13.00 0.46 0.68 122.95 0.24 12.63 22.67 8.00 4.21 8.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 95.00 162.00 2.00 12.00 0.38 0.68 154.29 0.35 24.86 26.00 24.67 4.00 5.60 sdd 0.00 161.00 0.00 16.00 0.00 0.54 69.00 8.16 290.75 0.00 290.75 62.50 100.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 514.00 0.00 2.01 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 37.00 124.00 9.00 5.00 0.18 0.50 99.43 0.10 7.43 8.00 6.40 1.43 2.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 21.00 161.00 0.00 12.00 0.00 0.68 115.33 1.63 1.00 0.00 1.00 51.67 62.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.83 0.00 0.51 40.66 0.00 43.99 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 21.00 0.00 1.00 0.00 0.09 176.00 0.07 68.00 0.00 68.00 68.00 6.80 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.00 0.00 0.00 0.00 0.00 100.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 0.00 21.00 1.00 3.00 0.09 0.09 90.00 2.40 1003.00 1312.00 900.00 200.00 80.00 avg-cpu: %user %nice %system %iowait %steal %idle 15.00 0.00 2.50 37.75 0.00 44.75 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 28.00 392.00 5.00 22.00 0.13 1.62 132.44 0.59 21.93 24.80 21.27 7.26 19.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 32.00 403.00 2.00 22.00 0.13 1.66 153.00 0.47 19.50 20.00 19.45 4.67 11.20 sdd 241.00 371.00 20.00 19.00 1.02 2.02 159.79 6.16 448.62 50.80 867.37 21.64 84.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 1624.00 0.00 6.34 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 279.00 316.00 8.00 20.00 1.12 1.31 178.29 0.34 12.29 11.00 12.80 2.14 6.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 117.00 353.00 4.00 6.00 0.16 0.88 213.60 1.24 8.00 8.00 8.00 12.40 12.40 avg-cpu: %user %nice %system %iowait %steal %idle 14.54 0.00 3.57 22.19 0.00 59.69 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 546.00 1175.00 31.00 54.00 3.70 4.80 204.80 1.56 18.02 28.39 12.07 4.47 38.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 629.00 1140.00 55.00 52.00 4.48 4.68 175.33 3.35 25.83 25.24 26.46 5.31 56.80 sdd 511.00 1017.00 53.00 76.00 3.64 4.27 125.58 6.09 47.22 60.60 37.89 4.74 61.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 6.00 0.00 0.03 9.33 0.02 3.33 0.00 3.33 0.67 0.40 md0 0.00 0.00 84.00 4392.00 8.01 17.16 11.52 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 413.00 988.00 38.00 66.00 3.21 4.12 144.23 0.78 7.46 10.32 5.82 2.23 23.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 723.00 1285.00 54.00 57.00 4.26 5.82 185.95 7.02 63.10 72.59 54.11 7.10 78.80


From this it is quite clear that sdd and sdi are reaching 100% utilisation and in this case the 'await' time is quite high on the same two drives. These drives are thus the weak links in the RAID and the other drives will have to wait for the slow ones to catch up. A RAID array is only as fast as it's slowest drive.

Here is the bonnie++ output

bonnie++ -u rick -s 8192 -d /Storage Using uid:1000, gid:1000. Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rick-desktop 8G 512 90 11907 2 12226 2 1897 85 272555 22 215.8 5 Latency 315ms 2983ms 2979ms 56084us 180ms 2293ms Version 1.96 ------Sequential Create------ --------Random Create-------- rick-desktop -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7964 8 +++++ +++ 6197 4 11867 11 +++++ +++ 20016 14 Latency 18497us 591us 834us 41422us 30us 830us 1.96,1.96,rick-desktop,1,1355829672,8G,,512,90,11907,2,12226,2,1897,85,272555,22,215.8,5,16,,,,,7964,8,+++++,+++,6197,4,11867,11,+++++,+++,20016,14,315ms,2983ms,2979ms,56084us,180ms,2293ms,18497us,591us,834us,41422us,30us,830us

And for reference, the same performance measures for the 500Gb WD Green OS drive:

iostat

iostat -xmhcd 1 10 Linux 3.2.0-34-generic-pae (rick-desktop) 19/12/12 _i686_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.06 0.00 3.91 20.05 0.00 61.98 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 7.00 0.00 2.00 0.00 0.94 0.00 964.00 0.01 4.00 4.00 0.00 2.00 0.40 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 8.00 0.00 3.00 0.00 1.00 0.00 682.67 0.01 2.67 2.67 0.00 1.33 0.40 sdd 4.00 0.00 2.00 0.00 0.56 0.00 576.00 0.01 4.00 4.00 0.00 4.00 0.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 1.00 165.00 0.00 81.00 999.37 86.98 573.90 188.00 576.24 6.02 100.00 md0 0.00 0.00 41.00 0.00 4.00 0.00 200.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 8.00 0.00 2.00 0.00 1.00 0.00 1024.00 0.05 24.00 24.00 0.00 14.00 2.80 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 4.00 0.00 1.00 0.00 0.50 0.00 1024.00 0.00 4.00 4.00 0.00 4.00 0.40

As we would expect, the utilisation is pushed to 100% or close to that for the entire test.

bonnie++

Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rick-desktop 8G 502 98 76789 10 34234 7 2242 86 81158 9 199.3 5 Latency 29422us 479ms 474ms 68689us 337ms 437ms Version 1.96 ------Sequential Create------ --------Random Create-------- rick-desktop -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 27403 34 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 28878us 1344us 962us 66420us 612us 1395us 1.96,1.96,rick-desktop,1,1355859677,8G,,502,98,76789,10,34234,7,2242,86,81158,9,199.3,5,16,,,,,27403,34,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,29422us,479ms,474ms,68689us,337ms,437ms,28878us,1344us,962us,66420us,612us,1395us

The throughput is comparable but there is a massive difference between the latency for the RAID Array and the OS drive.


Another test is to copy a large file to the array, check iostat again to confirm consistency with the bonnie++ test:

iostat -xmhcd 1 10 Linux 3.2.0-34-generic-pae (rick-desktop) 19/12/12 _i686_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.19 0.03 3.34 2.80 0.00 85.64 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 38.17 8.73 16.48 0.59 3.42 0.04 414.81 0.20 11.82 11.18 29.97 3.84 6.56 sdb 1.36 1.03 0.23 0.45 0.01 0.16 505.41 0.05 68.42 1.73 102.16 2.45 0.16 sdc 38.03 8.78 16.46 0.60 3.42 0.04 414.95 0.18 10.28 9.76 24.78 3.46 5.91 sdd 37.84 8.74 16.45 0.56 3.42 0.04 416.12 0.29 16.59 10.74 189.65 4.02 6.83 sde 1.49 1.89 0.47 0.18 0.01 0.01 49.52 0.00 5.73 2.22 15.16 2.63 0.17 sdf 1.37 1.48 3.75 1.85 0.26 0.27 193.68 0.55 97.85 11.73 271.99 3.82 2.14 md0 0.00 0.00 250.11 34.49 1.41 0.13 11.11 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 21.96 101.55 1.26 11.70 214.86 0.00 0.00 0.00 0.00 0.00 0.00 sdg 37.98 8.81 16.44 0.67 3.42 0.04 413.76 0.17 10.00 10.03 9.30 3.64 6.23 sdh 11.24 72.85 11.41 25.32 1.26 11.53 713.32 6.07 165.24 3.92 237.91 5.27 19.36 sdi 37.91 8.72 16.46 0.54 3.42 0.04 416.19 0.32 18.65 12.29 212.49 4.67 7.94 avg-cpu: %user %nice %system %iowait %steal %idle 12.60 0.00 11.31 7.97 0.00 68.12 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 377.00 2811.00 51.00 154.00 1.67 11.59 132.45 3.68 17.97 35.84 12.05 1.54 31.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 202.00 2813.00 45.00 143.00 0.96 11.58 136.68 1.23 6.15 15.64 3.16 1.13 21.20 sdd 205.00 2826.00 12.00 125.00 0.73 11.49 182.77 4.32 15.27 76.67 9.38 3.15 43.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 1.00 0.00 248.00 14.00 31.25 0.05 244.70 0.92 3.63 2.03 32.00 1.24 32.40 md0 0.00 0.00 0.00 11765.00 0.00 45.96 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 227.00 2810.00 51.00 146.00 1.09 11.58 131.70 1.71 8.32 13.57 6.49 1.44 28.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 289.00 2837.00 48.00 105.00 1.51 11.77 177.78 9.11 42.98 104.67 14.78 4.29 65.60 avg-cpu: %user %nice %system %iowait %steal %idle 6.30 0.00 7.09 13.39 0.00 73.23 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 591.00 1458.00 35.00 62.00 2.45 5.73 172.54 2.09 21.53 33.60 14.71 4.29 41.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 541.00 1500.00 32.00 63.00 2.24 6.11 179.87 1.76 18.53 35.62 9.84 4.00 38.00 sdd 553.00 1630.00 46.00 51.00 2.52 6.35 187.30 4.43 58.19 70.96 46.67 7.71 74.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 172.00 0.00 21.50 0.00 256.00 0.27 1.56 1.56 0.00 0.95 16.40 md0 0.00 0.00 3.00 5316.00 0.01 20.77 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 768.00 1376.00 38.00 60.00 3.15 5.40 178.61 1.07 10.94 16.42 7.47 2.86 28.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 505.00 1388.00 27.00 57.00 2.08 5.43 183.14 2.75 32.71 49.78 24.63 7.38 62.00 avg-cpu: %user %nice %system %iowait %steal %idle 5.63 0.00 4.86 37.85 0.00 51.66 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 167.00 546.00 2.00 17.00 0.66 2.46 336.00 0.40 21.26 32.00 20.00 8.63 16.40 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 187.00 631.00 5.00 40.00 0.75 2.62 153.60 0.68 15.11 35.20 12.60 3.82 17.20 sdd 102.00 598.00 23.00 13.00 0.49 2.45 167.11 4.36 22.22 15.65 33.85 26.33 94.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 150.00 0.00 18.75 0.00 256.00 0.21 1.44 1.44 0.00 1.07 16.00 md0 0.00 0.00 3.00 2348.00 0.01 9.17 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 111.00 601.00 34.00 34.00 0.57 2.71 98.59 0.56 8.18 6.71 9.65 1.35 9.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 178.00 421.00 3.00 33.00 0.71 1.51 126.00 3.20 7.89 14.67 7.27 22.78 82.00 avg-cpu: %user %nice %system %iowait %steal %idle 1.00 0.00 0.25 49.38 0.00 49.38 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.02 0.00 0.00 0.00 0.00 100.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 2.00 0.00 0.25 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 0.00 0.00 0.00 4.00 0.00 0.50 254.00 2.02 1233.00 0.00 1233.00 127.00 50.80 avg-cpu: %user %nice %system %iowait %steal %idle 5.54 0.00 7.81 21.16 0.00 65.49 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 697.00 1385.00 22.00 57.00 2.81 5.63 218.84 1.16 14.73 26.00 10.39 4.51 35.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 532.00 1306.00 21.00 49.00 2.16 5.29 218.06 1.54 22.00 26.10 20.24 5.54 38.80 sdd 444.00 1052.00 13.00 49.00 1.79 4.78 216.90 2.74 183.35 189.23 181.80 10.06 62.40 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 152.00 0.00 19.00 0.00 256.00 0.22 1.47 1.47 0.00 0.92 14.00 md0 0.00 0.00 2.00 4776.00 0.01 18.66 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 387.00 1320.00 21.00 52.00 1.59 5.36 195.18 0.59 8.11 13.52 5.92 3.45 25.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 558.00 1131.00 21.00 45.00 1.91 4.39 195.27 1.50 14.61 27.62 8.53 7.70 50.80 avg-cpu: %user %nice %system %iowait %steal %idle 4.57 0.00 5.08 18.27 0.00 72.08 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 705.00 772.00 38.00 48.00 2.91 3.21 145.58 1.72 19.49 34.00 8.00 6.00 51.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 698.00 854.00 36.00 56.00 2.68 3.13 129.39 1.23 11.91 22.67 5.00 3.04 28.00 sdd 436.00 913.00 40.00 63.00 1.73 3.80 109.98 1.99 17.67 19.40 16.57 4.08 42.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 2.00 1.00 102.00 3.00 13.00 0.02 253.87 0.28 2.67 2.27 16.00 1.64 17.20 md0 0.00 0.00 1.00 3209.00 0.00 12.54 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 200.00 941.00 20.00 79.00 0.86 3.98 100.20 0.22 2.22 3.20 1.97 1.21 12.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 377.00 989.00 31.00 67.00 1.95 4.33 131.18 3.90 42.33 40.00 43.40 8.53 83.60 avg-cpu: %user %nice %system %iowait %steal %idle 2.25 0.00 3.50 21.00 0.00 73.25 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 268.00 592.00 15.00 47.00 1.11 2.50 118.97 0.33 5.35 15.47 2.13 2.19 13.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 240.00 441.00 30.00 31.00 1.10 2.12 108.33 1.81 24.66 35.07 14.58 5.44 33.20 sdd 281.00 639.00 17.00 36.00 1.29 2.66 152.60 2.40 46.34 50.12 44.56 11.17 59.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 48.00 0.00 6.00 0.00 256.00 0.07 1.50 1.50 0.00 1.00 4.80 md0 0.00 0.00 0.00 2210.00 0.00 8.63 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 121.00 540.00 21.00 42.00 0.55 2.27 91.94 0.51 8.06 17.52 3.33 2.10 13.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 402.00 711.00 20.00 28.00 1.54 2.63 177.83 5.89 104.83 76.80 124.86 20.83 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.76 0.00 1.53 23.41 0.00 74.30 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 388.00 586.00 36.00 44.00 1.20 2.25 88.20 5.08 61.05 63.56 59.00 9.20 73.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 333.00 515.00 35.00 39.00 1.50 2.31 105.62 1.57 26.54 31.09 22.46 4.70 34.80 sdd 394.00 497.00 34.00 38.00 1.67 2.10 107.33 1.84 25.11 41.18 10.74 6.56 47.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 16.00 2.00 2.00 0.01 228.44 0.06 3.33 3.75 0.00 2.00 3.60 md0 0.00 0.00 0.00 1697.00 0.00 6.63 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 325.00 426.00 34.00 42.00 1.40 1.83 87.16 0.38 4.95 7.18 3.14 1.68 12.80 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 331.00 590.00 32.00 31.00 1.31 2.74 131.81 14.19 166.54 163.12 170.06 15.87 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 1.51 0.00 1.76 23.37 0.00 73.37 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 396.00 700.00 34.00 40.00 1.91 2.66 126.49 1.44 20.00 23.41 17.10 4.54 33.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 191.00 613.00 27.00 45.00 0.93 2.40 94.89 1.37 18.22 28.44 12.09 4.78 34.40 sdd 342.00 612.00 19.00 38.00 1.41 2.57 143.16 5.50 93.40 21.05 129.58 15.02 85.60 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 22.00 0.00 2.75 0.00 256.00 0.02 1.09 1.09 0.00 0.73 1.60 md0 0.00 0.00 1.00 1847.00 0.00 7.21 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 332.00 565.00 24.00 51.00 1.39 2.41 103.79 0.36 4.80 6.67 3.92 1.55 11.60 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 394.00 404.00 26.00 24.00 1.84 1.66 143.20 6.89 206.32 41.85 384.50 19.28 96.40

During the rsync, the transfer rate fluctuates greatly. The range is from ~5MB/s to ~40MB/s and is therefore very inconsistent and just downright slow:

rsync --progress rhel-server-6.0-x86_64-dvd.iso /Storage/ rhel-server-6.0-x86_64-dvd.iso 3431618560 100% 10.61MB/s 0:05:08 (xfer#1, to-check=0/1) sent 3432037552 bytes received 31 bytes 10809567.19 bytes/sec total size is 3431618560 speedup is 1.00

One last test with dd:

iostat -xmhcd 1 10 Linux 3.2.0-35-generic-pae (rick-desktop) 19/12/12 _i686_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 18.79 0.08 2.15 15.16 0.00 63.81 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 7.31 5.95 6.68 2.13 0.25 0.03 64.75 0.14 15.56 13.89 20.76 5.69 5.01 sdb 0.74 0.00 0.50 0.00 0.00 0.00 7.99 0.00 0.51 0.51 0.00 0.51 0.03 sdc 6.34 5.51 6.98 1.95 0.24 0.03 62.20 0.13 14.59 14.00 16.70 5.89 5.26 sdd 8.02 7.56 6.97 1.96 0.25 0.04 66.01 3.18 354.42 45.62 1453.80 14.42 12.88 sde 0.00 0.00 0.29 0.00 0.00 0.00 7.98 0.00 3.76 3.76 0.00 3.76 0.11 sdf 8.37 6.03 40.82 4.26 0.65 0.06 32.39 5.06 112.25 21.04 985.82 4.37 19.69 md0 0.00 0.00 32.22 22.24 1.09 0.09 44.08 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 1.04 0.00 0.00 0.00 1.76 0.00 0.00 0.00 0.00 0.00 0.00 sdg 8.07 7.13 7.18 2.25 0.25 0.04 62.72 0.07 7.07 7.86 4.57 3.82 3.60 sdh 0.16 0.00 0.31 0.00 0.00 0.00 7.98 0.00 2.40 2.40 0.00 2.40 0.07 sdi 6.86 5.76 7.16 1.83 0.25 0.03 62.82 0.70 76.14 25.97 272.34 9.67 8.70 avg-cpu: %user %nice %system %iowait %steal %idle 15.29 0.00 2.01 21.55 0.00 61.15 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 74.00 74.00 14.00 14.00 0.34 0.34 50.29 0.14 5.14 8.29 2.00 2.00 5.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 140.00 140.00 27.00 27.00 0.65 0.65 49.48 0.40 7.48 13.33 1.63 1.48 8.00 sdd 0.00 0.00 0.00 10.00 0.00 0.27 55.20 1.79 381.20 0.00 381.20 30.80 30.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 432.00 0.00 1.69 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 202.00 214.00 39.00 41.00 0.94 1.00 49.60 0.22 2.80 3.69 1.95 0.75 6.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 0.00 0.00 0.00 19.00 0.00 0.29 30.74 18.98 851.37 0.00 851.37 50.53 96.00 avg-cpu: %user %nice %system %iowait %steal %idle 15.62 0.00 2.02 20.65 0.00 61.71 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 101.00 104.00 27.00 25.00 0.50 0.50 39.54 0.88 16.92 29.78 3.04 2.62 13.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 198.00 204.00 44.00 43.00 0.95 0.96 44.97 2.69 30.90 25.36 36.56 2.07 18.00 sdd 113.00 125.00 31.00 29.00 0.56 0.61 39.87 1.67 27.87 48.39 5.93 3.60 21.60 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 448.00 0.00 1.75 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 253.00 255.00 56.00 59.00 1.21 1.23 43.34 0.82 7.10 11.36 3.05 1.25 14.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 180.00 186.00 41.00 21.00 0.86 0.53 45.94 24.12 409.42 172.20 872.57 16.13 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.97 0.00 1.27 20.81 0.00 62.94 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 138.00 77.00 31.00 31.00 0.42 0.42 27.87 1.18 17.61 29.68 5.55 1.81 11.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 165.00 77.00 22.00 23.00 0.39 0.38 34.84 0.69 10.49 18.18 3.13 1.87 8.40 sdd 78.00 76.00 17.00 21.00 0.38 0.38 40.84 0.87 22.84 40.47 8.57 7.58 28.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 293.00 0.00 1.14 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 55.00 71.00 28.00 33.00 0.32 0.41 24.52 0.29 4.72 8.14 1.82 0.79 4.80 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 106.00 81.00 18.00 19.00 0.38 0.45 45.62 21.58 631.89 364.44 885.26 27.03 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.57 0.00 2.51 18.59 0.00 64.32 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 13.00 74.00 14.00 18.00 0.34 0.36 45.00 0.45 16.75 32.00 4.89 3.62 11.60 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 131.00 239.00 43.00 55.00 1.02 1.16 45.63 1.24 14.86 26.51 5.75 1.63 16.00 sdd 35.00 23.00 9.00 9.00 0.17 0.12 33.78 0.33 18.44 29.78 7.11 6.22 11.20 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 1.00 403.00 0.00 1.57 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 77.00 81.00 17.00 23.00 0.37 0.41 39.60 0.08 1.90 3.06 1.04 1.50 6.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 60.00 81.00 17.00 21.00 0.38 0.54 49.05 25.58 549.05 61.88 943.43 26.32 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 16.04 0.00 1.50 19.55 0.00 62.91 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 60.00 32.00 9.00 8.00 0.14 0.16 35.29 0.50 8.71 14.22 2.50 2.59 4.40 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 51.00 48.00 18.00 9.00 0.27 0.18 34.07 0.38 6.22 9.33 0.00 1.33 3.60 sdd 20.00 11.00 3.00 3.00 0.05 0.05 37.33 1.23 30.00 60.00 0.00 19.33 11.60 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 9.00 0.00 0.04 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 158.00 0.00 0.62 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 101.00 102.00 32.00 24.00 0.50 0.46 34.86 0.90 6.36 10.00 1.50 1.29 7.20 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 73.00 32.00 12.00 22.00 0.20 0.45 38.82 14.53 723.76 149.00 1037.27 29.41 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.76 0.00 2.04 21.63 0.00 61.58 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 47.00 84.00 39.00 61.00 0.47 0.57 21.20 1.60 19.52 37.33 8.13 1.12 11.20 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 81.00 116.00 24.00 55.00 0.41 0.71 29.06 0.76 12.30 33.33 3.13 2.13 16.80 sdd 162.00 172.00 77.00 62.00 0.97 0.64 23.77 36.69 121.61 101.14 147.03 6.82 94.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 1.00 0.00 2.00 0.00 0.01 12.00 0.03 14.00 0.00 14.00 14.00 2.80 md0 0.00 0.00 0.00 332.00 0.00 1.30 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 59.00 67.00 54.00 71.00 0.46 0.57 17.02 0.92 11.68 23.04 3.04 0.64 8.00 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 162.00 231.00 86.00 22.00 1.13 0.38 28.67 55.95 242.22 163.44 550.18 9.26 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.39 0.00 2.27 19.95 0.00 63.38 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 24.00 20.00 5.00 4.00 0.59 0.09 156.44 0.02 1.78 3.20 0.00 0.44 0.40 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 26.00 23.00 6.00 5.00 0.56 0.11 124.36 0.06 5.45 7.33 3.20 1.09 1.20 sdd 5.00 0.00 0.00 25.00 0.00 0.23 19.20 18.70 1253.44 0.00 1253.44 40.00 100.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 13.00 79.00 1.25 0.31 34.70 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 6.00 0.00 1.00 0.00 0.03 0.00 56.00 0.02 16.00 16.00 0.00 16.00 1.60 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 14.00 3.00 0.00 25.00 0.00 0.22 17.92 47.22 1217.92 0.00 1217.92 40.00 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.61 0.00 2.02 29.47 0.00 53.90 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 42.00 34.00 13.00 10.00 1.23 0.17 124.87 0.48 20.70 26.46 13.20 8.00 18.40 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 62.00 50.00 16.00 12.00 1.74 0.24 145.14 0.25 9.00 9.75 8.00 3.43 9.60 sdd 12.00 5.00 6.00 9.00 1.53 0.19 234.67 1.47 645.07 142.00 980.44 25.87 38.80 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 69.00 67.00 6.76 0.26 105.71 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 27.00 21.00 6.00 4.00 1.57 0.10 341.60 0.10 10.00 14.00 4.00 4.40 4.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 59.00 68.00 19.00 19.00 1.80 0.27 111.16 34.57 1142.11 146.11 2138.11 26.32 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 14.21 0.00 1.78 20.05 0.00 63.96 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 11.00 5.00 2.00 2.00 0.03 0.03 28.00 0.09 8.00 16.00 0.00 22.00 8.80 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 75.00 56.00 15.00 12.00 0.34 0.27 45.93 0.46 13.93 25.07 0.00 3.85 10.40 sdd 6.00 0.00 1.00 0.00 0.00 0.00 8.00 0.11 52.00 52.00 0.00 56.00 5.60 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 209.00 0.00 0.82 8.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdg 62.00 53.00 13.00 11.00 0.29 0.25 46.33 0.07 3.00 3.08 2.91 1.00 2.40 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 118.00 118.00 22.00 24.00 0.51 0.59 49.22 34.02 1156.43 110.55 2115.17 21.74 100.00 </testarea>

and again the results form the dd test are appalling:

sudo dd if=/dev/zero of=/Storage/testfile bs=800M count=10 [sudo] password for rick: 10+0 records in 10+0 records out 8388608000 bytes (8.4 GB) copied, 665.696 s, 12.6 MB/s