In order to measure performance we have to be able to measure performance and understand what we are measuring. I came across this article that explains the benchmarking portion.
Benchmarking explained
bonnie++ output before any tweaking is done. I use the 8Gb size during the tests because the RAM size on the machine is 4Gb.
Whoaa!!! ~50MB/s average transfer.. Great improvement over the previous ~10MB/s. This is not a good test because the transfer rate is dependent on the source drive speed which in my case is the slowest drive (/dev/sdf if you have been following)
dd gives us write speed but uses buffering
and finally hdparm
Tuesday 25 December 2012
Sunday 23 December 2012
mdadm RAID creation
Here comes the fun part!! Let's create the RAID...
A good read on partition alignment.
chunk size = stripe size = minimum size that can be written to the disk
I wil use parted for the for the partitioning and alignment. Do the following for each drive.
And here we go:
The RAID array starts in degraded mode with a spare drive. The spare drive is added to the array at the end and the array can then perform at full speed
and then we create the new partition on the newly created device
'stride' is the amount of data written to each disk and 'stripe width' is the stride length * number of active drives. 4 in my case as the fifth drive is a parity drive. There are many calculators available, just google for on.
Remember to add the new device in fstab if you would like it to be mounted automatically.
add this line to /etc/fstab in my case:
A good read on partition alignment.
chunk size = stripe size = minimum size that can be written to the disk
I wil use parted for the for the partitioning and alignment. Do the following for each drive.
And here we go:
The RAID array starts in degraded mode with a spare drive. The spare drive is added to the array at the end and the array can then perform at full speed
and then we create the new partition on the newly created device
'stride' is the amount of data written to each disk and 'stripe width' is the stride length * number of active drives. 4 in my case as the fifth drive is a parity drive. There are many calculators available, just google for on.
Remember to add the new device in fstab if you would like it to be mounted automatically.
add this line to /etc/fstab in my case:
Friday 21 December 2012
That old chestnut: ALIGNMENT
Misaligned of partitions can cause severe performance issues. It is important to make sure that the alignment is correct before even touching mdadm. Remember, it can't be changed afterwards
Here is a good source that explains alignment with modern drives.
Linux on 4KB-sector disks: Practical advice
Another good explanation can be found here:
disk-performance-part-2-raid-layouts-and-stripe-sizing
Here is an example of a misaligned partition. The partition starts at sector 63.
Using parted we can get confirmation of the issue:
We are looking to align on multiples of the stripe size. If we select a stripe size of 512KB or in mdadm terms a chunk size of 512KB.
Note: This is not to be confused with stripe width which is derived from the stripe size, the number of load bearing disks, and the block size on the disk (4KB) We will set the stripe width when we create the partition.
Here is a good source that explains alignment with modern drives.
Linux on 4KB-sector disks: Practical advice
Another good explanation can be found here:
disk-performance-part-2-raid-layouts-and-stripe-sizing
Here is an example of a misaligned partition. The partition starts at sector 63.
Using parted we can get confirmation of the issue:
We are looking to align on multiples of the stripe size. If we select a stripe size of 512KB or in mdadm terms a chunk size of 512KB.
Note: This is not to be confused with stripe width which is derived from the stripe size, the number of load bearing disks, and the block size on the disk (4KB) We will set the stripe width when we create the partition.
Wednesday 19 December 2012
hdparm settings
hdparm is used to get and set drive parameters
hdparm -tT on all the drives gives the following output
From the previous test we recall that sdd and sdi were the bottlenecks, but in these tests they run just fine which brings me to the conclusion that the hdparm settings have little bearing on performance. This is because UDMA is being used.
After hooooooours of tinkering and tweaking I saw little consistent improvement, so I will spend no more time on it :) The default settings are pretty good on the modern drives.
hdparm -tT on all the drives gives the following output
From the previous test we recall that sdd and sdi were the bottlenecks, but in these tests they run just fine which brings me to the conclusion that the hdparm settings have little bearing on performance. This is because UDMA is being used.
After hooooooours of tinkering and tweaking I saw little consistent improvement, so I will spend no more time on it :) The default settings are pretty good on the modern drives.
Tuesday 18 December 2012
Find the bottleneck with iostat
Current performance before I start rebuilding:
While running bonnie++ I run iostat in an attempt to pinpoint the bottleneck:
From this it is quite clear that sdd and sdi are reaching 100% utilisation and in this case the 'await' time is quite high on the same two drives. These drives are thus the weak links in the RAID and the other drives will have to wait for the slow ones to catch up. A RAID array is only as fast as it's slowest drive.
Here is the bonnie++ output
And for reference, the same performance measures for the 500Gb WD Green OS drive:
iostat
As we would expect, the utilisation is pushed to 100% or close to that for the entire test.
bonnie++
The throughput is comparable but there is a massive difference between the latency for the RAID Array and the OS drive.
Another test is to copy a large file to the array, check iostat again to confirm consistency with the bonnie++ test:
During the rsync, the transfer rate fluctuates greatly. The range is from ~5MB/s to ~40MB/s and is therefore very inconsistent and just downright slow:
One last test with dd:
and again the results form the dd test are appalling:
While running bonnie++ I run iostat in an attempt to pinpoint the bottleneck:
From this it is quite clear that sdd and sdi are reaching 100% utilisation and in this case the 'await' time is quite high on the same two drives. These drives are thus the weak links in the RAID and the other drives will have to wait for the slow ones to catch up. A RAID array is only as fast as it's slowest drive.
Here is the bonnie++ output
And for reference, the same performance measures for the 500Gb WD Green OS drive:
iostat
As we would expect, the utilisation is pushed to 100% or close to that for the entire test.
bonnie++
The throughput is comparable but there is a massive difference between the latency for the RAID Array and the OS drive.
Another test is to copy a large file to the array, check iostat again to confirm consistency with the bonnie++ test:
During the rsync, the transfer rate fluctuates greatly. The range is from ~5MB/s to ~40MB/s and is therefore very inconsistent and just downright slow:
One last test with dd:
and again the results form the dd test are appalling:
Monday 17 December 2012
Intro
I am rebuilding a mdadm RAID5 array that has been a performance headache for both reading and writing to it.
The array is used primarily to store media information on. Specifically large files of a few hundred MB to 10s of GB. I also use it as general storage in the house so I store pictures, music, documents, previous backups etc on it.
The reason for the decision to go with RAID5 is that I like the idea of having some degree of redundancy. With this I can have up to 1 drive fail without any data loss. RAID10 which will give much better redundancy is just not practical for me for private use as it will be too expensive to implement.
The current setup looks like this:
- 5x2Tb WD Green Drives in a RAID5 configuration with mdadm
- Ubuntu 12.04 Lucid Lynx
- Intel i3
- Gigabyte GA-H55M-USB3
- 4 GB 1333 MHz RAM
The hardware is pretty average, nothing spectacular there.
cat /proc/mdstat
md0 : active raid5 sdg1[2] sdi1[0] sdd1[1] sdc1[3] sda1[4]
7814047744 blocks level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
The array is used primarily to store media information on. Specifically large files of a few hundred MB to 10s of GB. I also use it as general storage in the house so I store pictures, music, documents, previous backups etc on it.
The reason for the decision to go with RAID5 is that I like the idea of having some degree of redundancy. With this I can have up to 1 drive fail without any data loss. RAID10 which will give much better redundancy is just not practical for me for private use as it will be too expensive to implement.
The current setup looks like this:
- 5x2Tb WD Green Drives in a RAID5 configuration with mdadm
- Ubuntu 12.04 Lucid Lynx
- Intel i3
- Gigabyte GA-H55M-USB3
- 4 GB 1333 MHz RAM
The hardware is pretty average, nothing spectacular there.
cat /proc/mdstat
md0 : active raid5 sdg1[2] sdi1[0] sdd1[1] sdc1[3] sda1[4]
7814047744 blocks level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
Subscribe to:
Posts (Atom)