RAID 5 with svm on Storage 3000

[Pavlos Parissis]

Hello all,
I have a very strange problem with SVM on 3000 storage.
I am using Solaris 9 12/02 with latest patches and I have one StorEdge^(TM)
PCI Dual Ultra3 SCSI Host Adapter connected to the storage using 2 paths.

For the local disk,v280r server, I have created mirror raid like this
d1: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d12
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 10247232 blocks (4.9 GB)

d11: Submirror of d1
State: Okay
Size: 10247232 blocks (4.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes


d12: Submirror of d1
State: Okay
Size: 10247232 blocks (4.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V81439
c1t1d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V82476

d2: Mirror
Submirror 0: d21
State: Okay
Submirror 1: d22
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4100928 blocks (2.0 GB)

d21: Submirror of d2
State: Okay
Size: 4100928 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s1 0 No Okay Yes


d22: Submirror of d2
State: Okay
Size: 4100928 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s1 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V81439
c1t1d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V82476
lab-s1uxjps-001@/:# metastat d3
d3: Mirror
Submirror 0: d31
State: Okay
Submirror 1: d32
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 20484288 blocks (9.8 GB)

d31: Submirror of d3
State: Okay
Size: 20484288 blocks (9.8 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s3 0 No Okay Yes


d32: Submirror of d3
State: Okay
Size: 20484288 blocks (9.8 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s3 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V81439
c1t1d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V82476

lab-s1uxjps-001@/:# metastat d4
d4: Mirror
Submirror 0: d41
State: Okay
Submirror 1: d42
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 10247232 blocks (4.9 GB)

d41: Submirror of d4
State: Okay
Size: 10247232 blocks (4.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s4 0 No Okay Yes


d42: Submirror of d4
State: Okay
Size: 10247232 blocks (4.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s4 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V81439
c1t1d0 Yes id1,ssd@SFUJITSU_MAN3735F_SUN72G_000303V82476




The d1 is used by / and d2 by swap, d3 by /opt, d4 /var.
I have 8 replicas in the first disk and other 8 replicas on the second
internal disk.
ab-s1uxjps-001@/:# metadb
flags first blk block count
a m p luo 16 8192 /dev/dsk/c1t0d0s5
a p luo 8208 8192 /dev/dsk/c1t0d0s5
a p luo 16400 8192 /dev/dsk/c1t0d0s5
a p luo 24592 8192 /dev/dsk/c1t0d0s5
a p luo 16 8192 /dev/dsk/c1t1d0s5
a p luo 8208 8192 /dev/dsk/c1t1d0s5
a p luo 16400 8192 /dev/dsk/c1t1d0s5
a p luo 24592 8192 /dev/dsk/c1t1d0s5
a p luo 32784 8192 /dev/dsk/c1t1d0s5
a p luo 40976 8192 /dev/dsk/c1t1d0s5
a p luo 49168 8192 /dev/dsk/c1t1d0s5
a p luo 57360 8192 /dev/dsk/c1t1d0s5
a p luo 32784 8192 /dev/dsk/c1t0d0s5
a p luo 40976 8192 /dev/dsk/c1t0d0s5
a p luo 49168 8192 /dev/dsk/c1t0d0s5
a p luo 57360 8192 /dev/dsk/c1t0d0s5







I created a RAID 1 using 2 disk from the storage
5: Mirror
Submirror 0: d51
State: Okay
Submirror 1: d52
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 71023176 blocks (33 GB)

d51: Submirror of d5
State: Okay
Hot spare pool: hsp001
Size: 71023176 blocks (33 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c3t8d0s0 0 No Okay Yes


d52: Submirror of d5
State: Okay
Hot spare pool: hsp001
Size: 71023176 blocks (33 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c4t8d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c3t8d0 Yes id1,sd@SSEAGATE_ST336607LSUN36G_3JA0KHQ700007324R2 KP
c4t8d0 Yes id1,sd@SSEAGATE_ST336607LSUN36G_3JA0KGR900007324QM GK

I have created RAID 5 using 8 disk from the storage
d6: RAID
State: Okay
Hot spare pool: hsp001
Interlace: 32 blocks
Size: 497139120 blocks (237 GB)
Original device:
Size: 497139552 blocks (237 GB)
Device Start Block Dbase State Reloc Hot Spare
c3t9d0s0 3219 No Okay Yes
c3t10d0s0 3219 No Okay Yes
c3t11d0s0 3219 No Okay Yes
c3t12d0s0 3219 No Okay Yes
c4t9d0s0 3219 No Okay Yes
c4t10d0s0 3219 No Okay Yes
c4t11d0s0 3219 No Okay Yes
c4t12d0s0 3219 No Okay Yes

All the disk in the storage have the same slices.
I also created 8 replicas in two different slices on different disk on
the storage.

When I reboot the server I got this
checking ufs filesystems
/dev/md/rdsk/d5: is clean.
/dev/md/rdsk/d3: is clean.
wait: No child processes

WARNING - Unable to repair one or more filesystems.
Run fsck manually (fsck filesystem...).
Exit the shell when done to continue the boot process.


Type control-d to proceed with normal startup,
(or give root password for system maintenance):

If I press control-d the system boots without problems.
If I run fsck and the reboot I get the same


I have also noticed tha the replicas on storage marked as M
M p 16 unknown /dev/dsk/c3t8d0s7
M p 8208 unknown /dev/dsk/c3t8d0s7
M p 16400 unknown /dev/dsk/c3t8d0s7
M p 24592 unknown /dev/dsk/c3t8d0s7
M p 16 unknown /dev/dsk/c4t8d0s7
M p 8208 unknown /dev/dsk/c4t8d0s7
M p 16400 unknown /dev/dsk/c4t8d0s7
M p 24592 unknown /dev/dsk/c4t8d0s7


When I remove the line from vfstab about d6,RAID 5, the system boots up
without problem.

Any ideas why I am having this?

Regards,
Pavlos

[Anthony Mandic]

Pavlos Parissis wrote:

> I have a very strange problem with SVM on 3000 storage.
> I am using Solaris 9 12/02 with latest patches and I have one StorEdge^(TM)
> PCI Dual Ultra3 SCSI Host Adapter connected to the storage using 2 paths.

....

> I have 8 replicas in the first disk and other 8 replicas on the second
> internal disk.

Personally, I think this is more than you need. 4 on each should be
more than adequate.

> All the disk in the storage have the same slices.
> I also created 8 replicas in two different slices on different disk on
> the storage.

That's sounds like far too many.

> When I reboot the server I got this
> checking ufs filesystems
> /dev/md/rdsk/d5: is clean.
> /dev/md/rdsk/d3: is clean.

Have you enabled logging for your filesystems?

> wait: No child processes
>
> WARNING - Unable to repair one or more filesystems.
> Run fsck manually (fsck filesystem...).

....

> If I run fsck and the reboot I get the same
>
> I have also noticed tha the replicas on storage marked as M
> M p 16 unknown /dev/dsk/c3t8d0s7

....

> When I remove the line from vfstab about d6,RAID 5, the system boots up
> without problem.
>
> Any ideas why I am having this?

Hard to say. I would enable logging on the filesystems first and
reduce the number of metadbs. It could be a hardware issue but its
too soon to tell.

-am © 2003