Using a SVM submirror on a ramdisk to increase read performance

by Bernd Schemmer, November 2008

14.07.2012/bs: Links checked and corrected

Homepage: http://www.bnsmb.de/


Table of Contents

  1. Using a SVM submirror on a ramdisk to increase read performance
  2. Introduction
    1. Purpose
    2. Scope
    3. Prerequisites
    4. Used Environment
    5. References
  3. Installation
    1. What about write performance?
    2. Recreate the mirror after booting the machine
  4. Performance tests
    1. In a virtual QEMU machine:
      1. Configuring SVM Mirror
      2. Using a mirror with two disk based submirrors
      3. Using a mirror with two disk based submirrors and one ramdisk based submirror with the default read policy
      4. Using a mirror with two disk based submirrors and one ramdisk based submirror and the changed read-policy
    2. On a physical machine with SCSI disks:
      1. Configuring SVM Mirror
      2. Using a mirror with two disk based submirrors
      3. Using a mirror with two disk based submirrors and one ramdisk based submirror with the default read policy
      4. Using a mirror with two disk based submirrors and one ramdisk based submirror and the changed read-policy
  5. Conclusion


Introduction

Purpose

This article describes how to use a ramdisk as a 3rd submirror for an SVM mirror to improve the read performance of the mirror.

Scope

This article only describes how to configure a SVM submirror on a ramdisk - it does not discuss the concept of a volume manager like SVM in general.  It also does not talk about a practical usage for this configuration -- in fact, it may be that there is no practical usage for it. 

It's only meant as a proof of concept.

Prerequisites

A SPARC or x86 machine running Solaris 10 with enough memory to create the ramdisk.

Used Environment

The environment for the example below was Solaris 10 Update 6 running in a QEMU virtual machine with 1 GB memory and three virtual disks.

All commands must be executed by the user root or a user with the approbate rights.

References

Talking about RAM disks in the Solaris OS http://www.bnsmb.de/http://wiki.qemu.org/Index.html
QEMU Homepage http://wiki.qemu.org/Index.html
QEMU Project at OpenSolaris http://www.opensolaris.org/os/project/qemu/

Installation

First create the SVM mirror on the disks in the normal way, e.g.:

bash-3.00# metainit d11 1 1 c0d1s0
d11: Concat/Stripe is setup

bash-3.00# metainit d12 1 1 c1d0s0
d12: Concat/Stripe is setup

bash-3.00# metainit d10 -m d11
d10: Mirror is setup

bash-3.00# metattach d10 d12
d10: submirror d12 is attached

bash-3.00# metastat -p d10
d10 -m d11 d12 1
d11 1 1 c0d1s0
d12 1 1 c1d0s0

bash-3.00# newfs /dev/md/rdsk/d10
/dev/md/rdsk/d10: Unable to find Media type. Proceeding with system determined parameters.
newfs: construct a new file system /dev/md/rdsk/d10: (y/n)? y
Warning: inode blocks/cyl group (306) >= data blocks (59) in last
    cylinder group. This implies 944 sector(s) cannot be allocated.
/dev/md/rdsk/d10:       529200 sectors in 35 cylinders of 240 tracks, 63 sectors
        258.4MB in 7 cyl groups (5 c/g, 36.91MB/g, 17536 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 75696, 151360, 227024, 302688, 378352, 454016,

bash-3.00# echo "/dev/md/dsk/d10 /dev/md/rdsk/d10        /mnt    ufs     2       yes     logging" >>/etc/vfstab

bash-3.00# mount /mnt

bash-3.00# df -k /mnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d10       249127    1041  223174     1%    /mnt

The next step is to create the  ramdisk for the 3rd submirror using ramdiskadm:

bash-3.00# ramdiskadm -a testdisk 230m
/dev/ramdisk/testdisk

Note: The ramdisk must be as big as the other submirrors.

Due to a restriction in the SVM commands (metainit, metastat, etc) you must create symbolic links for the ramdisk in the /dev/dsk and /dev/rdsk directories:

bash-3.00# ln -s /dev/ramdisk/testdisk /dev/dsk/c5t0d0s0

bash-3.00# ln -s /dev/rramdisk/testdisk /dev/rdsk/c5t0d0s0

Note: Use a control number (c5 in this example) which is not already in use.

Now create a metadevice on the ramdisk

bash-3.00# metainit d13 1 1 c5t0d0s0
d13: Concat/Stripe is setup

bash-3.00# metastat d13
d13: Concat/Stripe
    Size: 471000 blocks (229 MB)
    Stripe 0:
        Device     Start Block  Dbase   Reloc
        c5t0d0s0          0     No      No

Device Relocation Information:
Device   Reloc  Device ID
c5t0d0   No     -

And attach the new metadevice to the existing mirror:

bash-3.00# metattach d10 d13
d10: submirror d13 is attached

bash-3.00# metastat -p d10
d10 -m d11 d12 d13 1
d11 1 1 c1d0s0
d12 1 1 c0d1s0
d13 1 1 c5t0d0s0

That's it -- now  we have a SVM mirror with one submirror on a ramdisk.

But as can be seen in the iostat output below this configuration not really improves the read performance of the mirror. For this we must  change the read policy for the mirror.

The default read policy for SVM mirror devices is round robin:

bash-3.00# metaparam d10
d10: Mirror current parameters are:
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)

We want to change it , so that reads are done from our ramdisk based submirror. Unfortunately there is no option to select a specific submirror as primary submirror for read requests:

bash-3.00# metaparam
usage:  metaparam [-s setname] [options] concat/stripe | RAID
        metaparam [-s setname] [options] mirror

Concat/Stripe or RAID options:
-h      hotspare_pool | "none"

Mirror options:
-r      roundrobin | geometric | first
-w      parallel | serial
-p      0-9

Therefore we must first destroy and recreate the mirror (without loosing the data, of course):

Stop all applications accessing the mirror and ensure that all three submirrors are in sync:

bash-3.00#  metastat | grep sync

Now umount the mirror and clear the SVM metadevice for the mirror

bash-3.00#  umount /mnt

bash-3.00#  metaclear d10

and recreate the mirror using the ramdisk as first submirror:

bash-3.00#  metainit d10 -m d13

bash-3.00#  metattach d10 d12

bash-3.00#  metattach d10 d11

bash-3.00#  bash-3.00# metastat -p d10
d10 -m d13 d12 d11 1
d13 1 1 c5t0d0s0
d11 1 1 c1d0s0
d12 1 1 c0d1s0

So now we can change the read policy for the mirror to "first":

bash-3.00# metaparam -r first d10

bash-3.00# metaparam d10
d10: Mirror current parameters are:
    Pass: 1
    Read option: first (-r)
    Write option: parallel (default)

This configuration now improves the read performance for the mirror a lot  (see the iostat output below).

What about write performance?

There is a small performance impact for writing to the mirror but that should not be a problem.

Examples:

Writing to a mirror with two submirrors on disk

bash-3.00# time dd if=/dev/zero of=/mnt/testfile bs=512
write: No space left on device
429777+0 records in
429777+0 records out

real    0m28.010s
user    0m0.249s
sys     0m23.197s

Writing to a mirror with two submirrors on disk and one submirror on a ramdisk:

bash-3.00# time dd if=/dev/zero of=/mnt/testfile bs=512
write: No space left on device
429777+0 records in
429777+0 records out

real    0m33.227s
user    0m0.247s
sys     0m26.602s

Recreate the mirror after booting the machine

One open issue for this configuration is the behaviour of the mirror after a reboot:

After the machine is rebooted, the ramdisk does not exist and therefore the submirror on the ramdisk also does not exist and the mirror needs maintenance:

# bash
bash-3.00# metastat
d10: Mirror
    Submirror 0: d13
      State: Needs maintenance
    Submirror 1: d11
      State: Okay         
    Submirror 2: d12
      State: Okay         
    Pass: 1
    Read option: first (-r)
    Write option: parallel (default)
    Size: 460800 blocks (225 MB)

d13: Submirror of d10
    State: Needs maintenance
    Invoke: metareplace d10 c5t0d0s0 <new device>
    Size: 460800 blocks (225 MB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c5t0d0s0          0     No     Maintenance   No  


d11: Submirror of d10
    State: Okay         
    Size: 465885 blocks (227 MB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c0d1s0          0     No            Okay   Yes


d12: Submirror of d10
    State: Okay         
    Size: 465885 blocks (227 MB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c1d0s0          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c1d0     Yes    id1,cmdk@AQEMU_HARDDISK=QM00003
c5t0d0   No     -
c0d1     Yes    id1,cmdk@AQEMU_HARDDISK=QM00002

This is not a big problem because the data still exists on the existing submirrors and the mirror also works without the 3rd submirror - but without the read performance improvement, of course.

To repair the mirror recreate the ramdisk

bash-3.00# ramdiskadm -a testdisk 230m

and reattach the submirror on the ramdisk

bash-3.00# metareplace -e d10 c5t0d0s0
d10: device c5t0d0s0 is enabled

bash-3.00# metastat d10
d10: Mirror
    Submirror 0: d13
      State: Okay         
    Submirror 1: d11
      State: Okay         
    Submirror 2: d12
      State: Okay         
    Pass: 1
    Read option: first (-r)
    Write option: parallel (default)
    Size: 460800 blocks (225 MB)

d13: Submirror of d10
    State: Okay         
    Size: 460800 blocks (225 MB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c5t0d0s0          0     No            Okay   No  


d11: Submirror of d10
    State: Okay         
    Size: 465885 blocks (227 MB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c0d1s0          0     No            Okay   Yes


d12: Submirror of d10
    State: Okay         
    Size: 465885 blocks (227 MB)
    Stripe 0:
        Device   Start Block  Dbase        State Reloc Hot Spare
        c1d0s0          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c5t0d0   No     -
c0d1     Yes    id1,cmdk@AQEMU_HARDDISK=QM00002
c1d0     Yes    id1,cmdk@AQEMU_HARDDISK=QM00003

That's it - now the 3rd submirror on the ramdisk is again in place and after the synchronisation is done we have a mirror with faster read performance again.

Performance tests

In a virtual QEMU machine:

Configuring SVM Mirror

bash-3.00# metastat -p d10
d10 -m d11 d12 d13 1
d11 1 1 c1d0s0
d12 1 1 c0d1s0
d13 1 1 c5t0d0s0

 d10 is the mirror, d11 and d12 are the sub mirrors on two virtual harddisks, and d13 is the submirror on the ramdisk.

Using a mirror with two disk based submirrors

bash-3.00# dd if=/dev/md/rdsk/d10 of=/dev/null bs=512
    
bash-3.00#  iostat -xn 1 5555
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0d0
 1905.7    0.0  952.8    0.0  0.0  0.4    0.0    0.2   0  41 c0d1
 1904.7    0.0  952.3    0.0  0.0  0.4    0.0    0.2   0  42 c1d0
 3810.3    0.0 1905.2    0.0  0.0  0.8    0.0    0.2   0  85 md/d10
 1905.7    0.0  952.8    0.0  0.0  0.4    0.0    0.2   0  42 md/d11
 1904.7    0.0  952.3    0.0  0.0  0.4    0.0    0.2   0  42 md/d12

Using a mirror with two disk based submirrors and one ramdisk based submirror with the default read policy

bash-3.00# dd if=/dev/md/rdsk/d10 of=/dev/null bs=512

bash-3.00#  iostat -xn 1 5555
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0d0
 1643.0    0.0  821.5    0.0  0.0  0.4    0.0    0.2   0  36 c0d1
 1644.0    0.0  822.0    0.0  0.0  0.4    0.0    0.2   0  36 c1d0
 4930.1    0.0 2465.1    0.0  0.0  0.8    0.0    0.2   0  82 md/d10
 1644.0    0.0  822.0    0.0  0.0  0.4    0.0    0.2   0  37 md/d11
 1643.0    0.0  821.5    0.0  0.0  0.4    0.0    0.2   0  37 md/d12
 1643.0    0.0  821.5    0.0  0.0  0.1    0.0    0.0   0   8 md/d13
 1643.0    0.0  821.5    0.0  0.0  0.1    0.0    0.0   0   6 ramdisk1

Using a mirror with two disk based submirrors and one ramdisk based submirror and the changed read-policy

bash-3.00# dd if=/dev/md/rdsk/d10 of=/dev/null bs=512

bash-3.00#  iostat -xn 1 5555
                    extended device statistics             
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0d1
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1d0
 16022.1    0.0 8011.0    0.0  0.0  0.6    0.0    0.0   1  62 md/d10
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 md/d11
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 md/d12
 16022.1    0.0 8011.0    0.0  0.0  0.6    0.0    0.0   0  58 md/d13
 16022.1    0.0 8011.0    0.0  0.0  0.4    0.0    0.0   0  41 ramdisk1

On a physical machine with SCSI disks:

Configuring SVM Mirror

noname:/# metastat -p d80
d80 -m d81 d82 d83 1
d81 1 1 c3t2d0s0
d82 1 1 c3t3d0s0
d83 1 1 c7t0d0s0

d80 is the mirror, d81 and d82 are the submirrors on two harddisks, and d83 is the submirror on the ramdisk

Using a mirror with two disk based submirrors

noame:/var# time dd if=/dev/md/rdsk/d80 of=/dev/null bs=512
522000+0 records in
522000+0 records out

real    1m13.890s
user    0m1.084s
sys     0m13.530s

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
 7170.6    0.0 3585.3    0.0  0.0  0.9    0.0    0.1   1  90 md/d80
 3585.8    0.0 1792.9    0.0  0.0  0.4    0.0    0.1   0  43 md/d81
 3584.8    0.0 1792.4    0.0  0.0  0.4    0.0    0.1   0  43 md/d82
 3585.8    0.0 1792.9    0.0  0.0  0.4    0.0    0.1   3  36 c3t2d0
 3584.8    0.0 1792.4    0.0  0.0  0.4    0.0    0.1   3  36 c3t3d0

Using a mirror with two disk based submirrors and one ramdisk based submirror with the default read policy

noname:/var# time dd if=/dev/md/rdsk/d80 of=/dev/null bs=512
522000+0 records in
522000+0 records out

real    0m54.764s
user    0m1.098s
sys     0m13.353s

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
 9765.7    0.0 4882.8    0.0  0.0  0.9    0.0    0.1   1  88 md/d80
 3254.9    0.0 1627.4    0.0  0.0  0.4    0.0    0.1   0  40 md/d81
 3255.9    0.0 1627.9    0.0  0.0  0.4    0.0    0.1   0  40 md/d82
 3255.9    0.0 1627.9    0.0  0.0  0.0    0.0    0.0   0   4 md/d83
 3255.9    0.0 1627.9    0.0  0.0  0.0    0.0    0.0   0   2 ramdisk1
 3254.9    0.0 1627.5    0.0  0.0  0.3    0.0    0.1   2  34 c3t2d0
 3255.9    0.0 1628.0    0.0  0.0  0.3    0.0    0.1   2  34 c3t3d0

Using a mirror with two disk based submirrors and one ramdisk based submirror and the changed read-policy

noname:/var# time dd if=/dev/md/rdsk/d80 of=/dev/null bs=512
522000+0 records in
522000+0 records out

real    0m9.799s
user    0m1.004s
sys     0m8.773s

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
 54466.5  149.3 36656.3 9497.7  0.0  1.6    0.0    0.0   3 100 md/d80
 54467.5    0.0 36656.8    0.0  0.0  0.5    0.0    0.0   2  47 md/d83
 54467.5    0.0 36656.8    0.0  0.0  0.3    0.0    0.0   0  24 ramdisk1

Conclusion

On one hand using a ramdisk as 3rd submirror for a SVM mirror significantly improves the read access. But on the other hand it's something complicated to configure and the SVM mirror must be repaired after each reboot. And you need a lot of memory for it.

So it's possible to use this approach but it's only useful if an application really needs it and it's not possible to get the necessary read performance with other methods (like tuning the cache, for example).