Interesting SunFire V65x configuration issue - Sun Solaris

[Dennis]

Hello fellow Solaris for Intel folks, I have a good one here for you to think
about. I have to come up with an interesting solution to what seems to be a
simple problem. Seems to be. I have an E450 running Solaris 7 and it has two
external disk arrays attached. One of them is an A1000 so therefore I need to
run Raid Manager. The other array is an Integrix unit and it just works as an
normal external differential SCSI device should. The performance of the A1000
is really bad. I don't think that I ever get more than 12Mb/sec to it. The
unit reports a clean healthcheck but the I/O is pathetic. The Integrix unit
is only a little better. This server holds the email for about 12,000 users.
It is time for a better solution. The V65x looks like a real winner as it is
three or four times faster than the E450 ( 3 x 300MHz + 2.5Gb RAM ) and it
will use Ultra320 LVD SCSI disks internally. The idea is to setup a V65x and
attach some sort of external storage that will work with either the E450 or on
the V65x. You may think that is strange but read on for further info.

The current E450 config has the standard internal SCSI as well as two PCI
adapters X6541A differential SCSI with the high density SCSI ports on them.
That allows for the two arrays to be on separate controllers. A good idea.
They are currently plugged into old slow array technology. A bad idea. Any
decent SCSI disk will move along at 25Mb/sec without question. We should get
more with more spindles in parallel. What I would like to do is upgrade the
E450 to Solaris 9. I will install another CPU and beef the RAM up to 4Gb.
That will make for a fairly maxed out E450. There are faster CPU options but
that is a waste of money on an E450 when IO is the issue, not CPU. I can then
take a pile of fast SCSI disks and put them into a 6-bay UltraSCSI multipack
and use SVM to stripe them. If I use differential disks then I can plug the
entire multipack into the X6541A controller. A striped set of SCSI disks
should allow for about 50Mb/sec of IO on a differential controller. If I
create another identical multipack and plug it onto the other X6541A
controller then I can mirror the stripes and I have good redundency without
any controller contention. Sounds easy for the E450 and it is simple to do.
I may get better performance if I use Ultra320 disks in the multipacks with
the LVD X6758 PCI controller.

Now here comes the tough part.

The V65x is new technology and while it has the Sun logo on it there is still
the issue of trust. The E450 can be trusted to run and run and run. The V65x
needs to be tested for a while. So here is the idea. I don't want the V65x
to become a replacement for the E450 but I do want to see if it can do the job
better. I want to be able to pull the SCSI cables from off the back of the
E450 and plug them into the V65x. If the V65x is running Solaris 9 with SVM
setup for striped mirrors in the precisely same config as the E450 then it
would be an issue of metainit and run with the data intact. I have tested
this with the Sparc platform and moved entire stripe sets from one machine to
another without dropping a bit. Does that still apply for a move from Sparc
to Intel architecture? I don't know.

There is a bigger problem still.

What are the PCI adapters going to be on the V65x? Let's take a look at the
Sun System Handbook for PCI SCSI Host Adapters :

http://sunsolve.sun.com/handbook_pub/Devices/SCSI/SCSI_TOC.html

The only two PCI cards that I am interested in is the X6758 which is a LOW
Voltage Differential and the X6541 which is just differential. But look close
and you will see that neither are for the V65x. In fact, have a look at the
options for the V65x and there are none. Not one PCI adapter for differential
SCSI. Zero. Nada. Zilch.

Do I believe that? Do I simply plug the X6541A into the V65x and see if the
53C876 chip is well supported? Do I take the same risk with the X6758 and its
QLogic chips? That is too much of a risk with a new V65x.

I want a pure Sun solution. Is that too much to ask or should I simply
install an adaptec HBA? If I do, will the data that is laid down by the
X6541A within the E450 be readable by the adaptec? I doubt it.

The Solaris x86 HCL has 12 disk controllers that are "reported to work". One
of them is a PCI-X type. The term "reported to work" is not good enough to
house the email for 12,000 people. Not good enough at all.

Is the V65x even worth looking at or should I stick with UltraSparc for triple
the price and half the performance? I can't.

Maybe I should scrap the idea of trying to make the external disk packs work
with the E450 and the V65x at the same time. Maybe I should just work with
the V65x and use an external array like the StorEdge S1 which seems to work
well with LVD controllers. But wait! What is that SCSI port on the back of
the V65x? Is that a LVD SCSI port? Who knows as its not doented much of
anywhere.

sigh ....

Dennis

[Dennis]

On Wed, 2 Oct 2003, Kim Sebo wrote: 
>
>This wont work. SPARC and intel have a different byte order, the
>filesystem metadata are written in native byte order so it is
>garbled when you mount them on a machine with the opposite byte order.[/ref]

Are you certain of this? Have you pulled a disk from a Sparc system and
then run fsck on an intel system? I do remember that ufsdumps done with an
intel system are worthless to a sparc system which can not read them.
 

Well, the filesystem would be ufs but on a metadevice as opposed to a device.
What scares me is that the metadevice may be different between the two
architectures.
 
>
>The sun system handbook says its a "Ultra320 68-pin, high-density SCSI
>connector (VHDCI)."[/ref]

Yep, looks good but way more expensive than a multipack with six SCSI disks.
 

The handbook says that is works with the X6758 controller so it looks good.

Dennis

[Logan]

Dennis Clarke wrote: 
:
: 

I have never run a mail server for 12,000 people[1], but are you sure
stripes are the best way to go about it?

It would seem with 12,000 users accessing mailboxes via POP and similar,
and with a steady stream of incoming messages, what you'd really want
would be good random-access performance. Striping is going to give you
good sequential performance for one-request-at-a-time kind of loads,
which is not what you have.

Also, there is a cost to striping: when joeuser logs in via POP
(or IMAP or whatever), he'll typically download his whole mailbox.
Unless he has a very fast network connection, this will not happen
in one burst but will be a few I/O requests scattered over the
course of a few seconds (or sometimes minutes). If you use
striping, his mailbox is likely to be close to contiguous
and will thus span multiple spindles in the stripe. So, it would
seem that every time he needs to download his 250k mailbox, if you
have 5 disks in the stripe, every single disk head will have to
move to the same general area, where his mailbox is. This seems
kind of silly to me. Fetching his data as fast as possible
isn't what's desired; what you need is the ability to fetch his
data with minimal impairment to your ability to fetch others'
data at the same time.

Imagine you concatenate instead. Then, most likely (although
not definitely), joeuser's mailbox lies all on one disk.
If you have 5 concatenated disks, maybe you can get lucky and
have 5 people logged in at once, with each one fetching stuff
from their mailbox off one spindle and not disturbing others.

In fact, were it not for the extra administration headache,
I'd be really tempted to suggest making a separate filesystem
for each disk and splitting up /var/spool/mail so that different
users' mailboxes are hashed onto different disks.

So, it seems to me that perhaps a good compromise would be to
create a series of two-way mirrors and then concatenate the
mirrors. This would allow joeuser's mailbox activity to affect
only one spindle, and every mailbox would be accessible through
one of two spindles, whichever is most convenient.

Of course, this is just a suggestion and should be taken as such.
There may be some magic reason I'm not understanding why mail
servers work really well with stripes. (If so, I'd be interested
to know what it is.)
 

There are two really obvious problems you're going to run into here.
The first is that the fdisk partitioning arrangement doesn't exist on
SPARC and does on Intel.

If you can get around that, then you still have a problem. As I
understand it, the byte order of some of the data within the filesystem
is native byte order. So, some data is actually written to disk as
big-endian on SPARC and little-endian on Intel. If that's true, it
would seem to be a showstopper.
 

Is it a strict requirement to have HVD? Why wouldn't LVD be sufficient?
As far as I can tell from http://www.sun.com/servers/entry/v65x/specs.html ,
the v65x has "rear dual channel Ultra320 SCSI", which would imply LVD.
Do you need more than two channels? Ultra320 is pretty fast, and I
would think you could support at least, say, 6 or 8 disks on each
channel.

The same page also says that it's an Adaptec 7902 controller.
I wonder if this is the same chip as on Adaptec's PCI-X cards;
if so, presumably you should be able to go get an Adaptec 39320D-R
or something and it should work. (Their web site has drivers
for Solaris 8 x86 but not Solaris 9 x86, but if it's the same
chip as what comes with the system, the driver should work,
at least one would expect...)

- Logan

[1] although I did once have a long conversation at a seminar (given by
a sendmail consultant) with the people who ran the UC Santa Cruz
mail server at the time -- they did some interesting things to
optimize it.

[Greg]

Logan Shaw <rr.com> writes: 
[filesystem holding 12,000 mailboxes] 

If UFS is the filesystem, wouldn't concatenation tend to put
all the metadata onto only one or two spindles?

-Greg
--
Do NOT reply via e-mail.
Reply in the newsgroup.

[Dennis]

On Thu, 2 Oct 2003, Logan Shaw wrote: 
> :
> : 
>
>I have never run a mail server for 12,000 people[1], but are you sure
>stripes are the best way to go about it?[/ref]

Nope. I am beginning to think ( a real plus I assure you ) that the V210 or
V240 is the only way to go here. That way I can use FC-AL and an external
A5200 box with a whack of 9Gb disks. I can mirror pairs of disks together to
form a small 9Gb filesystem. I would then mount that filesystem under
/var/mail/{username first letter}/username or some similar deal. That way I
have 24 filesystems maximum ( I can double up on the letter x, y and q ) and
get disks running independantly of each other.
 

right .. I agree.
 

What is worse is that I expect to see 20,000 users in a year!
 

I agree and the admin overhead is worth the silence of the users.
 

Can't be done with SVM. You can mirror, you can concatonate, you can concat
and then mirror but you can not mirror and then concat. :(
 

me too ..
 
>
>There are two really obvious problems you're going to run into here.
>The first is that the fdisk partitioning arrangement doesn't exist on
>SPARC and does on Intel.[/ref]

Yep .. that is a bugger. I thought that the metadevice would get me past that
but then again metadevices are composed of stripes. I may give up and go with
a big fat NFS server for the storage.
 

Yep. It kills ufsdumps and ufsrestores between architectures also.
 
>
>Is it a strict requirement to have HVD? Why wouldn't LVD be sufficient?[/ref]

I'll work with whatever works. I want FC-AL but x86 will not support it yet.
I want Ultra320 which is LVD and that is the X6758A, not the X6541A. Neither
work in the V65x which has only a single port on the back. I want redundency
and distributed IO. I have 12,000 users to host here.
 

One hopes.
 

I think that it is a black art of some sort.

Thanks for the input, I am certain that I will go V240 now.

Dennis

[Philip]

On Wed, 1 Oct 2003 21:45:08 -0700, org wrote: 

you remember incorrectly... I just tested ufsdump compatibility.
From my x86 desktop, to blaster, funnily enough.



--
http://www.blastwave.org/ for solaris pre-packaged binaries with pkg-get
Organized by the author of pkg-get
[Trim the no-bots from my address to reply to me by email!]
S.1618 http://thomas.loc.gov/cgi-bin/bdquery/z?d105:SN01618:D
http://www.spamlaws.com/state/ca1.html

[Philip]

On Thu, 02 Oct 2003 05:07:19 GMT, rr.com wrote: 

depends on the stripe width.

If you set the width so that maxphys covers all the disks at once,
you'll get single-read high throughput (suposedly)

However, if you set the width so that maxphys covers a SINGLE STRIPE,
then you get in effect a round-robin load distribution for I/O across
disks.

eg: set your stripe width to 1 meg per disk, if you can, for this purpose


--
http://www.blastwave.org/ for solaris pre-packaged binaries with pkg-get
Organized by the author of pkg-get
[Trim the no-bots from my address to reply to me by email!]
S.1618 http://thomas.loc.gov/cgi-bin/bdquery/z?d105:SN01618:D
http://www.spamlaws.com/state/ca1.html

[Logan]

Greg Andrews wrote: 

I wondered about that. I thought I understood that i-nodes
spread across the cylinder groups, and thus across the entire
space, but I'm not really sure that that's the case.

- Logan

[Logan]

Philip Brown wrote:
 
 [/ref]
 

Interesting. I don't guess I've ever actually seen someone set up
a stripe width that large, but of course I can't see a reason why
you couldn't do it.

Doing that would have one other advantage over concatenation: if
you concatenate 3 disks and then only fill up your filesystem to
33%, the I/O is not exactly going to be balanced evenly among the
disks. Or at least such is the case with filesystems that tend
to fill up the first part first. (I assume ufs falls into that
category.)

- Logan

[Chris]

In article <R_Peb.17039$austin.rr.com>,
Logan Shaw <rr.com> wrote: 
>
>I wondered about that. I thought I understood that i-nodes
>spread across the cylinder groups, and thus across the entire
>space, but I'm not really sure that that's the case.[/ref]

It's true: every cylinder group contains the same number of inodes.
When a UFS filing system is growfs'd, new cylinder groups are created
and each gets the same allocation of inodes as the existing ones.

Chris Thompson
Email: cet1 [at] cam.ac.uk

[Scott]

In comp.unix.solaris Dennis Clarke <org> wrote: 
>
> Are you certain of this? Have you pulled a disk from a Sparc system and
> then run fsck on an intel system? I do remember that ufsdumps done with an[/ref]

Yes, they are incompatible. If you don't believe me, try mounting the
UFS slice of a Solaris 9 CD onto a box of a different architecture. (ie,
mount a SPARC CD on an Intel box or vica versa)

Scott

[Dennis]

 [/ref]
 

CDROM == hsfs <> ufs

My intel Sol9 cd mount just fine on my Sparc system.

UFS is another story.

Dennis

[Dennis]

 

Sorry, I delved too deep into the memory, that was my Sol 2.5.1 for x86 stuff.

dc

[Joerg]

In article <slrnbnnhra.vo4.phil+com>,
Philip Brown <phil+no-bots.com> wrote: 
>
>you remember incorrectly... I just tested ufsdump compatibility.
>From my x86 desktop, to blaster, funnily enough.[/ref]

Are you 100% sure?

UFSdump uses a binary format but it includes a structured byte swapper.

--
EMail:isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
tu-berlin.de (uni) If you don't have iso-8859-1
fraunhofer.de (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

[Goran]

In article <Pine.GSO.4.58.0310020638300.22128blastwave>,
Dennis Clarke <org> wrote:
 [/ref]
 

This is not true. A CDROM does not have to be hsfs. A Solaris
distribution CD contains one slice with an ISO 9660 filesystem
(can be mounted by different architectures) and several slices
with architecture dependant UFS filesystems (can only be used
by the intended architecture).

Taken from < http://www.sun.com/solutions/blueprints/0301/BuildBoot.pdf >

| * Slice 0 contains the Solaris OE packages to be installed and is on
| the HSFS (High Sierra File System) partition of the CD.
| * Slice 1 contains the generic kernel and what will be the systems /
| (root) directory after boot.
| * Slice 2 contains the boot block for the sun4c architecture.
| * Slice 3 contains the boot block for the sun4m architecture.
| * Slice 4 contains the boot block for the sun4d architecture.
| * Slice 5 contains the boot block for the sun4u architecture.

| As noted earlier, slice 0 is HSFS, with all other slices being UFS
| partitions.
 

The slice with the ISO 9660 filesystem will mount, but the
slices with UFS will not mount.

--
Göran Larsson http://www.mitt-eget.com/

[John]

In article <Pine.GSO.4.58.0310011948460.2059blastwave>,
Dennis Clarke <org> wrote: 

What did your sales rep tell you when you asked him when those
options will be certified for the V65?

John
org

[WilliamREMOVEWyattTHIS]

> [...] ... E450 when IO is the issue, not CPU. I can then 

The Sun multipack are strictly SE, not LVD, not HVD.

[...] 

This is dual Ultra-160, not 320. But the point remains...

For maximum spindles, get 3 of the X6758's , put in the 3 66 MHz slots
on the
E450, attach up to (gulp) 60 lvd disks, possibly as JBOD with software
striping.


Bill Wyatt (harvard.edu) "remove this" for email
Smithsonian Astrophysical Observatory (Cambridge, MA, USA)

[Fredrik]

In article <Pine.GSO.4.58.0310011948460.2059blastwave>,
Dennis Clarke <org> wrote:
 

 

Well get a V240 instead and get better performance from day 1.
 

This will not work as the multipack is single ended with termination.
 

You want a raid-controller with plenty of cache. This will make all
typical writes more than 10 times faster than any super duper scsi 4 unit
you can dream of.
 

No, the IO/s is the bottleneck, the CPU:s in the E450 is another.
Get a V240 connected to a SE3310. This will fly in your setup.
Need more IO? add a JBOD 3310. Need more CPU!? Consider the V440.
 

Look at V240/440, not triple the price. The performance your users will notice
is not just a factor of how fast the SCSI-bus is.
A busy mailserver needs a lot of RAM to be happy! How can a 32 bit
system efficiently handle more than a puny 4GB?

/wfr
Fredrik


--
Fredrik Lundholm
dol ce.chalmers.se

[Philip]

On 2 Oct 2003 14:48:54 GMT, tu-berlin.de wrote: 
>
>Are you 100% sure?
>
>UFSdump uses a binary format but it includes a structured byte swapper.[/ref]

"I just tested [it]"

I only did it for ascii files, but they restored just fine on the sparc,
after creating the dump file on x86.



--
http://www.blastwave.org/ for solaris pre-packaged binaries with pkg-get
Organized by the author of pkg-get
[Trim the no-bots from my address to reply to me by email!]
S.1618 http://thomas.loc.gov/cgi-bin/bdquery/z?d105:SN01618:D
http://www.spamlaws.com/state/ca1.html

[Chris]

chalmers.se (Fredrik Lundholm) writes:
 

Intel PAE allows for three level page tables and hence up to 64GB
RAM can be used in total on one system.

It's ugly, but it works (I think Solaris x86 can even make use of it -
man xmemfs)

Chris
--
Chris Morgan
"Post posting of policy changes by the boss will result in
real rule revisions that are irreversible"

- anonymous correspondent