Professional Web Applications Themes

Interesting SunFire V65x configuration issue - Sun Solaris

Hello fellow Solaris for Intel folks, I have a good one here for you to think about. I have to come up with an interesting solution to what seems to be a simple problem. Seems to be. I have an E450 running Solaris 7 and it has two external disk arrays attached. One of them is an A1000 so therefore I need to run Raid Manager. The other array is an Integrix unit and it just works as an normal external differential SCSI device should. The performance of the A1000 is really bad. I don't think that I ever get ...

  1. #1

    Default Interesting SunFire V65x configuration issue


    Hello fellow Solaris for Intel folks, I have a good one here for you to think
    about. I have to come up with an interesting solution to what seems to be a
    simple problem. Seems to be. I have an E450 running Solaris 7 and it has two
    external disk arrays attached. One of them is an A1000 so therefore I need to
    run Raid Manager. The other array is an Integrix unit and it just works as an
    normal external differential SCSI device should. The performance of the A1000
    is really bad. I don't think that I ever get more than 12Mb/sec to it. The
    unit reports a clean healthcheck but the I/O is pathetic. The Integrix unit
    is only a little better. This server holds the email for about 12,000 users.
    It is time for a better solution. The V65x looks like a real winner as it is
    three or four times faster than the E450 ( 3 x 300MHz + 2.5Gb RAM ) and it
    will use Ultra320 LVD SCSI disks internally. The idea is to setup a V65x and
    attach some sort of external storage that will work with either the E450 or on
    the V65x. You may think that is strange but read on for further info.

    The current E450 config has the standard internal SCSI as well as two PCI
    adapters X6541A differential SCSI with the high density SCSI ports on them.
    That allows for the two arrays to be on separate controllers. A good idea.
    They are currently plugged into old slow array technology. A bad idea. Any
    decent SCSI disk will move along at 25Mb/sec without question. We should get
    more with more spindles in parallel. What I would like to do is upgrade the
    E450 to Solaris 9. I will install another CPU and beef the RAM up to 4Gb.
    That will make for a fairly maxed out E450. There are faster CPU options but
    that is a waste of money on an E450 when IO is the issue, not CPU. I can then
    take a pile of fast SCSI disks and put them into a 6-bay UltraSCSI multipack
    and use SVM to stripe them. If I use differential disks then I can plug the
    entire multipack into the X6541A controller. A striped set of SCSI disks
    should allow for about 50Mb/sec of IO on a differential controller. If I
    create another identical multipack and plug it onto the other X6541A
    controller then I can mirror the stripes and I have good redundency without
    any controller contention. Sounds easy for the E450 and it is simple to do.
    I may get better performance if I use Ultra320 disks in the multipacks with
    the LVD X6758 PCI controller.

    Now here comes the tough part.

    The V65x is new technology and while it has the Sun logo on it there is still
    the issue of trust. The E450 can be trusted to run and run and run. The V65x
    needs to be tested for a while. So here is the idea. I don't want the V65x
    to become a replacement for the E450 but I do want to see if it can do the job
    better. I want to be able to pull the SCSI cables from off the back of the
    E450 and plug them into the V65x. If the V65x is running Solaris 9 with SVM
    setup for striped mirrors in the precisely same config as the E450 then it
    would be an issue of metainit and run with the data intact. I have tested
    this with the Sparc platform and moved entire stripe sets from one machine to
    another without dropping a bit. Does that still apply for a move from Sparc
    to Intel architecture? I don't know.

    There is a bigger problem still.

    What are the PCI adapters going to be on the V65x? Let's take a look at the
    Sun System Handbook for PCI SCSI Host Adapters :

    http://sunsolve.sun.com/handbook_pub/Devices/SCSI/SCSI_TOC.html

    The only two PCI cards that I am interested in is the X6758 which is a LOW
    Voltage Differential and the X6541 which is just differential. But look close
    and you will see that neither are for the V65x. In fact, have a look at the
    options for the V65x and there are none. Not one PCI adapter for differential
    SCSI. Zero. Nada. Zilch.

    Do I believe that? Do I simply plug the X6541A into the V65x and see if the
    53C876 chip is well supported? Do I take the same risk with the X6758 and its
    QLogic chips? That is too much of a risk with a new V65x.

    I want a pure Sun solution. Is that too much to ask or should I simply
    install an adaptec HBA? If I do, will the data that is laid down by the
    X6541A within the E450 be readable by the adaptec? I doubt it.

    The Solaris x86 HCL has 12 disk controllers that are "reported to work". One
    of them is a PCI-X type. The term "reported to work" is not good enough to
    house the email for 12,000 people. Not good enough at all.

    Is the V65x even worth looking at or should I stick with UltraSparc for triple
    the price and half the performance? I can't.

    Maybe I should scrap the idea of trying to make the external disk packs work
    with the E450 and the V65x at the same time. Maybe I should just work with
    the V65x and use an external array like the StorEdge S1 which seems to work
    well with LVD controllers. But wait! What is that SCSI port on the back of
    the V65x? Is that a LVD SCSI port? Who knows as its not doented much of
    anywhere.

    sigh ....

    Dennis


    Dennis Guest

  2. #2

    Default Re: Interesting SunFire V65x configuration issue


    On Wed, 2 Oct 2003, Kim Sebo wrote: 
    >
    >This wont work. SPARC and intel have a different byte order, the
    >filesystem metadata are written in native byte order so it is
    >garbled when you mount them on a machine with the opposite byte order.[/ref]

    Are you certain of this? Have you pulled a disk from a Sparc system and
    then run fsck on an intel system? I do remember that ufsdumps done with an
    intel system are worthless to a sparc system which can not read them.
     

    Well, the filesystem would be ufs but on a metadevice as opposed to a device.
    What scares me is that the metadevice may be different between the two
    architectures.
     
    >
    >The sun system handbook says its a "Ultra320 68-pin, high-density SCSI
    >connector (VHDCI)."[/ref]

    Yep, looks good but way more expensive than a multipack with six SCSI disks.
     

    The handbook says that is works with the X6758 controller so it looks good.

    Dennis

    Dennis Guest

  3. #3

    Default Re: Interesting SunFire V65x configuration issue

    Dennis Clarke wrote: 
    :


    I have never run a mail server for 12,000 people[1], but are you sure
    stripes are the best way to go about it?

    It would seem with 12,000 users accessing mailboxes via POP and similar,
    and with a steady stream of incoming messages, what you'd really want
    would be good random-access performance. Striping is going to give you
    good sequential performance for one-request-at-a-time kind of loads,
    which is not what you have.

    Also, there is a cost to striping: when joeuser logs in via POP
    (or IMAP or whatever), he'll typically download his whole mailbox.
    Unless he has a very fast network connection, this will not happen
    in one burst but will be a few I/O requests scattered over the
    course of a few seconds (or sometimes minutes). If you use
    striping, his mailbox is likely to be close to contiguous
    and will thus span multiple spindles in the stripe. So, it would
    seem that every time he needs to download his 250k mailbox, if you
    have 5 disks in the stripe, every single disk head will have to
    move to the same general area, where his mailbox is. This seems
    kind of silly to me. Fetching his data as fast as possible
    isn't what's desired; what you need is the ability to fetch his
    data with minimal impairment to your ability to fetch others'
    data at the same time.

    Imagine you concatenate instead. Then, most likely (although
    not definitely), joeuser's mailbox lies all on one disk.
    If you have 5 concatenated disks, maybe you can get lucky and
    have 5 people logged in at once, with each one fetching stuff
    from their mailbox off one spindle and not disturbing others.

    In fact, were it not for the extra administration headache,
    I'd be really tempted to suggest making a separate filesystem
    for each disk and splitting up /var/spool/mail so that different
    users' mailboxes are hashed onto different disks.

    So, it seems to me that perhaps a good compromise would be to
    create a series of two-way mirrors and then concatenate the
    mirrors. This would allow joeuser's mailbox activity to affect
    only one spindle, and every mailbox would be accessible through
    one of two spindles, whichever is most convenient.

    Of course, this is just a suggestion and should be taken as such.
    There may be some magic reason I'm not understanding why mail
    servers work really well with stripes. (If so, I'd be interested
    to know what it is.)
     

    There are two really obvious problems you're going to run into here.
    The first is that the fdisk partitioning arrangement doesn't exist on
    SPARC and does on Intel.

    If you can get around that, then you still have a problem. As I
    understand it, the byte order of some of the data within the filesystem
    is native byte order. So, some data is actually written to disk as
    big-endian on SPARC and little-endian on Intel. If that's true, it
    would seem to be a showstopper.
     

    Is it a strict requirement to have HVD? Why wouldn't LVD be sufficient?
    As far as I can tell from http://www.sun.com/servers/entry/v65x/specs.html ,
    the v65x has "rear dual channel Ultra320 SCSI", which would imply LVD.
    Do you need more than two channels? Ultra320 is pretty fast, and I
    would think you could support at least, say, 6 or 8 disks on each
    channel.

    The same page also says that it's an Adaptec 7902 controller.
    I wonder if this is the same chip as on Adaptec's PCI-X cards;
    if so, presumably you should be able to go get an Adaptec 39320D-R
    or something and it should work. (Their web site has drivers
    for Solaris 8 x86 but not Solaris 9 x86, but if it's the same
    chip as what comes with the system, the driver should work,
    at least one would expect...)

    - Logan

    [1] although I did once have a long conversation at a seminar (given by
    a sendmail consultant) with the people who ran the UC Santa Cruz
    mail server at the time -- they did some interesting things to
    optimize it.

    Logan Guest

  4. #4

    Default Re: Interesting SunFire V65x configuration issue

    Logan Shaw <rr.com> writes: 
    [filesystem holding 12,000 mailboxes] 

    If UFS is the filesystem, wouldn't concatenation tend to put
    all the metadata onto only one or two spindles?

    -Greg
    --
    Do NOT reply via e-mail.
    Reply in the newsgroup.
    Greg Guest

  5. #5

    Default Re: Interesting SunFire V65x configuration issue

    On Thu, 2 Oct 2003, Logan Shaw wrote: 
    > :
    > : 
    >
    >I have never run a mail server for 12,000 people[1], but are you sure
    >stripes are the best way to go about it?[/ref]

    Nope. I am beginning to think ( a real plus I assure you ) that the V210 or
    V240 is the only way to go here. That way I can use FC-AL and an external
    A5200 box with a whack of 9Gb disks. I can mirror pairs of disks together to
    form a small 9Gb filesystem. I would then mount that filesystem under
    /var/mail/{username first letter}/username or some similar deal. That way I
    have 24 filesystems maximum ( I can double up on the letter x, y and q ) and
    get disks running independantly of each other.
     

    right .. I agree.
     

    What is worse is that I expect to see 20,000 users in a year!
     

    I agree and the admin overhead is worth the silence of the users.
     

    Can't be done with SVM. You can mirror, you can concatonate, you can concat
    and then mirror but you can not mirror and then concat. :(
     

    me too ..
     
    >
    >There are two really obvious problems you're going to run into here.
    >The first is that the fdisk partitioning arrangement doesn't exist on
    >SPARC and does on Intel.[/ref]

    Yep .. that is a bugger. I thought that the metadevice would get me past that
    but then again metadevices are composed of stripes. I may give up and go with
    a big fat NFS server for the storage.
     

    Yep. It kills ufsdumps and ufsrestores between architectures also.
     
    >
    >Is it a strict requirement to have HVD? Why wouldn't LVD be sufficient?[/ref]

    I'll work with whatever works. I want FC-AL but x86 will not support it yet.
    I want Ultra320 which is LVD and that is the X6758A, not the X6541A. Neither
    work in the V65x which has only a single port on the back. I want redundency
    and distributed IO. I have 12,000 users to host here.
     

    One hopes.
     

    I think that it is a black art of some sort.

    Thanks for the input, I am certain that I will go V240 now.

    Dennis


    Dennis Guest

  6. #6

    Default Re: Interesting SunFire V65x configuration issue

    On Wed, 1 Oct 2003 21:45:08 -0700, org wrote: 

    you remember incorrectly... I just tested ufsdump compatibility.
    From my x86 desktop, to blaster, funnily enough.



    --
    http://www.blastwave.org/ for solaris pre-packaged binaries with pkg-get
    Organized by the author of pkg-get
    [Trim the no-bots from my address to reply to me by email!]
    S.1618 http://thomas.loc.gov/cgi-bin/bdquery/z?d105:SN01618:D
    http://www.spamlaws.com/state/ca1.html
    Philip Guest

  7. #7

    Default Re: Interesting SunFire V65x configuration issue

    On Thu, 02 Oct 2003 05:07:19 GMT, rr.com wrote: 

    depends on the stripe width.

    If you set the width so that maxphys covers all the disks at once,
    you'll get single-read high throughput (suposedly)

    However, if you set the width so that maxphys covers a SINGLE STRIPE,
    then you get in effect a round-robin load distribution for I/O across
    disks.

    eg: set your stripe width to 1 meg per disk, if you can, for this purpose


    --
    http://www.blastwave.org/ for solaris pre-packaged binaries with pkg-get
    Organized by the author of pkg-get
    [Trim the no-bots from my address to reply to me by email!]
    S.1618 http://thomas.loc.gov/cgi-bin/bdquery/z?d105:SN01618:D
    http://www.spamlaws.com/state/ca1.html
    Philip Guest

  8. #8

    Default Re: Interesting SunFire V65x configuration issue

    Greg Andrews wrote: 

    I wondered about that. I thought I understood that i-nodes
    spread across the cylinder groups, and thus across the entire
    space, but I'm not really sure that that's the case.

    - Logan

    Logan Guest

  9. #9

    Default Re: Interesting SunFire V65x configuration issue

    Philip Brown wrote:
     
     [/ref]
     

    Interesting. I don't guess I've ever actually seen someone set up
    a stripe width that large, but of course I can't see a reason why
    you couldn't do it.

    Doing that would have one other advantage over concatenation: if
    you concatenate 3 disks and then only fill up your filesystem to
    33%, the I/O is not exactly going to be balanced evenly among the
    disks. Or at least such is the case with filesystems that tend
    to fill up the first part first. (I assume ufs falls into that
    category.)

    - Logan

    Logan Guest

  10. #10

    Default Re: Interesting SunFire V65x configuration issue

    In article <R_Peb.17039$austin.rr.com>,
    Logan Shaw <rr.com> wrote: 
    >
    >I wondered about that. I thought I understood that i-nodes
    >spread across the cylinder groups, and thus across the entire
    >space, but I'm not really sure that that's the case.[/ref]

    It's true: every cylinder group contains the same number of inodes.
    When a UFS filing system is growfs'd, new cylinder groups are created
    and each gets the same allocation of inodes as the existing ones.

    Chris Thompson
    Email: cet1 [at] cam.ac.uk
    Chris Guest

  11. #11

    Default Re: Interesting SunFire V65x configuration issue

    In comp.unix.solaris Dennis Clarke <org> wrote: 
    >
    > Are you certain of this? Have you pulled a disk from a Sparc system and
    > then run fsck on an intel system? I do remember that ufsdumps done with an[/ref]

    Yes, they are incompatible. If you don't believe me, try mounting the
    UFS slice of a Solaris 9 CD onto a box of a different architecture. (ie,
    mount a SPARC CD on an Intel box or vica versa)

    Scott
    Scott Guest

  12. #12

    Default Re: Interesting SunFire V65x configuration issue

     [/ref]
     

    CDROM == hsfs <> ufs

    My intel Sol9 cd mount just fine on my Sparc system.

    UFS is another story.

    Dennis


    Dennis Guest

  13. #13

    Default Re: Interesting SunFire V65x configuration issue


     

    Sorry, I delved too deep into the memory, that was my Sol 2.5.1 for x86 stuff.

    dc


    Dennis Guest

  14. #14

    Default Re: Interesting SunFire V65x configuration issue

    In article <slrnbnnhra.vo4.phil+com>,
    Philip Brown <phil+no-bots.com> wrote: 
    >
    >you remember incorrectly... I just tested ufsdump compatibility.
    >From my x86 desktop, to blaster, funnily enough.[/ref]

    Are you 100% sure?

    UFSdump uses a binary format but it includes a structured byte swapper.

    --
    EMail:isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
    tu-berlin.de (uni) If you don't have iso-8859-1
    fraunhofer.de (work) chars I am J"org Schilling
    URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily
    Joerg Guest

  15. #15

    Default Re: Interesting SunFire V65x configuration issue

    In article <Pine.GSO.4.58.0310020638300.22128blastwave>,
    Dennis Clarke <org> wrote:
     [/ref]
     

    This is not true. A CDROM does not have to be hsfs. A Solaris
    distribution CD contains one slice with an ISO 9660 filesystem
    (can be mounted by different architectures) and several slices
    with architecture dependant UFS filesystems (can only be used
    by the intended architecture).

    Taken from < http://www.sun.com/solutions/blueprints/0301/BuildBoot.pdf >

    | * Slice 0 contains the Solaris OE packages to be installed and is on
    | the HSFS (High Sierra File System) partition of the CD.
    | * Slice 1 contains the generic kernel and what will be the systems /
    | (root) directory after boot.
    | * Slice 2 contains the boot block for the sun4c architecture.
    | * Slice 3 contains the boot block for the sun4m architecture.
    | * Slice 4 contains the boot block for the sun4d architecture.
    | * Slice 5 contains the boot block for the sun4u architecture.

    | As noted earlier, slice 0 is HSFS, with all other slices being UFS
    | partitions.
     

    The slice with the ISO 9660 filesystem will mount, but the
    slices with UFS will not mount.

    --
    Göran Larsson http://www.mitt-eget.com/
    Goran Guest

  16. #16

    Default Re: Interesting SunFire V65x configuration issue

    In article <Pine.GSO.4.58.0310011948460.2059blastwave>,
    Dennis Clarke <org> wrote: 

    What did your sales rep tell you when you asked him when those
    options will be certified for the V65?

    John
    org
    John Guest

  17. #17

    Default Re: Interesting SunFire V65x configuration issue

    > [...] ... E450 when IO is the issue, not CPU. I can then 

    The Sun multipack are strictly SE, not LVD, not HVD.

    [...] 

    This is dual Ultra-160, not 320. But the point remains...

    For maximum spindles, get 3 of the X6758's , put in the 3 66 MHz slots
    on the
    E450, attach up to (gulp) 60 lvd disks, possibly as JBOD with software
    striping.


    Bill Wyatt (harvard.edu) "remove this" for email
    Smithsonian Astrophysical Observatory (Cambridge, MA, USA)

    WilliamREMOVEWyattTHIS Guest

  18. #18

    Default Re: Interesting SunFire V65x configuration issue

    In article <Pine.GSO.4.58.0310011948460.2059blastwave>,
    Dennis Clarke <org> wrote:
     

     

    Well get a V240 instead and get better performance from day 1.
     

    This will not work as the multipack is single ended with termination.
     

    You want a raid-controller with plenty of cache. This will make all
    typical writes more than 10 times faster than any super duper scsi 4 unit
    you can dream of.
     

    No, the IO/s is the bottleneck, the CPU:s in the E450 is another.
    Get a V240 connected to a SE3310. This will fly in your setup.
    Need more IO? add a JBOD 3310. Need more CPU!? Consider the V440.
     

    Look at V240/440, not triple the price. The performance your users will notice
    is not just a factor of how fast the SCSI-bus is.
    A busy mailserver needs a lot of RAM to be happy! How can a 32 bit
    system efficiently handle more than a puny 4GB?

    /wfr
    Fredrik


    --
    Fredrik Lundholm
    dol ce.chalmers.se

    Fredrik Guest

  19. #19

    Default Re: Interesting SunFire V65x configuration issue

    On 2 Oct 2003 14:48:54 GMT, tu-berlin.de wrote: 
    >
    >Are you 100% sure?
    >
    >UFSdump uses a binary format but it includes a structured byte swapper.[/ref]

    "I just tested [it]"

    I only did it for ascii files, but they restored just fine on the sparc,
    after creating the dump file on x86.



    --
    http://www.blastwave.org/ for solaris pre-packaged binaries with pkg-get
    Organized by the author of pkg-get
    [Trim the no-bots from my address to reply to me by email!]
    S.1618 http://thomas.loc.gov/cgi-bin/bdquery/z?d105:SN01618:D
    http://www.spamlaws.com/state/ca1.html
    Philip Guest

  20. #20

    Default Re: Interesting SunFire V65x configuration issue

    chalmers.se (Fredrik Lundholm) writes:
     

    Intel PAE allows for three level page tables and hence up to 64GB
    RAM can be used in total on one system.

    It's ugly, but it works (I think Solaris x86 can even make use of it -
    man xmemfs)

    Chris
    --
    Chris Morgan
    "Post posting of policy changes by the boss will result in
    real rule revisions that are irreversible"

    - anonymous correspondent
    Chris Guest

Page 1 of 2 12 LastLast

Similar Threads

  1. configuration issue....maybe?
    By whgill in forum Coldfusion Server Administration
    Replies: 0
    Last Post: August 9th, 11:42 PM
  2. Interesting Issue
    By eonyron webforumsuser@macromedia.com in forum Macromedia Flash Actionscript
    Replies: 1
    Last Post: February 6th, 03:03 PM
  3. Possible interesting QT/hardware issue for multimedia developers
    By Ziggi in forum Macromedia Director Lingo
    Replies: 10
    Last Post: November 30th, 08:25 AM
  4. Configuration issue on AIX 4.3.3
    By dbf in forum AIX
    Replies: 1
    Last Post: August 3rd, 04:46 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139