Professional Web Applications Themes

NFS/SMB storage solution - Linux / Unix Administration

My apologies if this seems a somewhat lengthy post. I'm considering/planning a new storage system. We're currently ~40 people in a development business, expecting continued growth. All users' homes are NFS and/or SMB mounted. ATM moving ~10GB/day on a 500GB system based on FreeBSD for NFS/SMB, this will increase a good deal within the next half to one year. Our experiences with Linux and FreeBSD (some before my time as sysadm, so I cannot fully evaluate those problems), have been NFS timeouts, usually some 2-3 minutes a time, with mounts magically coming back, no log entries et al... From above, ...

  1. #1

    Default NFS/SMB storage solution

    My apologies if this seems a somewhat lengthy post.


    I'm considering/planning a new storage system.
    We're currently ~40 people in a development business, expecting
    continued growth. All users' homes are NFS and/or SMB mounted.

    ATM moving ~10GB/day on a 500GB system based on FreeBSD for NFS/SMB,
    this will increase a good deal within the next half to one year.

    Our experiences with Linux and FreeBSD (some before my time as sysadm,
    so I cannot fully evaluate those problems), have been NFS timeouts,
    usually some 2-3 minutes a time, with mounts magically coming back, no
    log entries et al...
    From above, and from talks with others, I kind of see this picture:

    Linux: Less good at NFS, in terms of performance and stability.
    Good support for journalized filesystems.
    Good mySQL performance.

    FreeBSD: Good NFS stability, decent performance.
    No journalized filesystem, though has background check
    and snapshot.
    Less good mySQL performance.

    Sun HW +Solaris: Excelent NFS stability and performance.
    Journalized filesystem, plus Veritas filesystem.
    mySQL performance? (I know Oracle performs well)

    I'm looking at a combination of two Sun Fire X2100 server's, hooked up
    to a Sun StorEdge 3511 SATA Array with 5x250GB SATA drives in raid5 for
    homes and 2x250GB mirrored for database.

    This will hold our NFS/SMB mounted homes and run internal databases.

    A Dell blade will run other parts of our infrastructure, including ldap,
    dhcp, dns, http, svn, bugzilla, plus a not yet determined groupware.
    I may later decide to run some of those services on the Sun setup.

    NFS clients are mostly Linux (RHEL, SLES, Debian, Gentoo), with a few
    OSX, Irix, Sun, True64 thingies. Most can be setup/upgraded to use NFS4,
    if need be.

    At this stage, I'm unsure of:
    Choise of technology:
    .. Go with the Sun-setup
    .. Continue with either Linux or FreeBSD
    .. Expect to be able to solve Linux/FreeBSD NFS timeouts/stability
    .. Continue with NFS, or look at AFS, CODA et al...

    Sun solution:
    .. How good performance I can expect from the StorEdge SATA-based part
    .. Which disk controller combination I should go with
    .. Which filesystem I should choose, Sun's (zfs?) or Veritas
    .. Hardware vs. software raid WRT scaling, performance, admin

    Non-Sun solution:
    .. Whether or not Linux or FreeBSD works well with NFS
    .. If Linux or FreeBSD NFS stability is a tuneable thingy
    .. If Linux or FreeBSD NFS stability depends on which kernel versions


    I'm interested in any [install/adm/stability/performance] experiences
    with mentioned products, and various other ways of semilar setups,
    including experiences with other methods than NFS: AFS, CODA et al...


    --
    Kind regards,
    Mogens V.

    Mogens Guest

  2. #2

    Default Re: NFS/SMB storage solution

    Mogens V. wrote: 

    No log entries on the server, you mean?

    That's possibly fairly easy to diagnose (which isn't the same thing
    as fixing). Run tcpdump on the client and server, filter to see
    only NFS-related stuff between those two hosts, and see what who is
    not responding to whom. On very common cause of NFS timeouts used
    to be a lack of threads on the NFS server: each thread answers a
    request (basically a packet in the old days of UDP-based NFS), and
    there is a fixed number of threads, so if they are all busy, the
    packet is simply dropped, and the client is expected to retransmit
    it. The client can't tell the difference between this and a
    crashed/missing/unreachable NFS server, so you see timeouts.

    As I recall, there are two solutions to this: the first is to
    increase the number of threads. This ensures that there will
    always be one available when a request comes in. However, it's
    not a perfect solution, because for a client to believe the NFS
    server is OK, the server has to not only assign the request to
    a thread but also do the physical I/O (read or write from disk)
    and return a packet. That means that the more I/O load you have
    on your server, the longer clients are going to have to wait, and
    the more danger of timeouts. As far as I know, the only solution
    to this (other than increasing disk I/O capacity of the server)
    is to increase the timeouts on the clients and increase the time
    before they retransmit requests (which further bog down the
    server).

    Anyway, the point is that timeouts aren't necessarily an indication
    of a bug or anything like that. Sometimes they can just be an
    indicator that performance tuning is needed.
     

    I suspect that as far as choosing Linux, FreeBSD, or Sun goes, you
    are going to have advantages and disadvantages with any of those
    in the area of stability and performance, because NFS compatibility
    between client and server just varies with the different combinations.

    As far as I know, Coda isn't really ready for prime time, so I would
    personally avoid that. AFS isn't a bad option, but I'm not really
    sure if it offers any huge advantages in your situation (which,
    as far as you've described, is entirely confined to a small LAN).

    On the other hand, if the NFS interoperability problems turn out to
    be unsolvable, then any other filesystem might be a better choice. :-)
    AFS probably has an advantage here since as far as I know there aren't
    multiple independent implementations of AFS, and interoperability is
    usually better between ports of the same implementation of something
    than it is between independent implementations.
     
     

    I don't think ZFS is ready for prime time either, yet, although it
    promises to be ready pretty soon (maybe 6 months). Because it uses
    copy on write for all writes, thus turning essentially all writes
    into sequential I/O, it should be a pretty good combination with NFS,
    since NFS really benefits from a filesystem that can complete write
    requests quickly.

    Personally, I wouldn't go with Veritas, because I just don't see the
    benefit over plain UFS, now that UFS has support for large filesystems,
    journaling, etc. Especially since UFS is built in, which means
    administration is simpler.

    - Logan
    Logan Guest

  3. #3

    Default Re: NFS/SMB storage solution

    Logan Shaw wrote: 
    >
    >
    > No log entries on the server, you mean?[/ref]

    On both sides...
     

    Your above comments made me think a Bit. This FreeBSD server use a
    Promise TX4 4ch SATA card. I know the TX has had driver issues for a
    long time (maybe still has) on Linux. Could be the same on FreeBSD.
    Disk I/O problems due to driver/firmware issues, combined with no tuning
    so far, might cause those timeout problems.
    I only got the job 2 month ago, so I haven't been into tuning yet.
     
    >
    >
    > I suspect that as far as choosing Linux, FreeBSD, or Sun goes, you
    > are going to have advantages and disadvantages with any of those
    > in the area of stability and performance, because NFS compatibility
    > between client and server just varies with the different combinations.
    >
    > As far as I know, Coda isn't really ready for prime time, so I would
    > personally avoid that. AFS isn't a bad option, but I'm not really
    > sure if it offers any huge advantages in your situation (which,
    > as far as you've described, is entirely confined to a small LAN).
    >
    > On the other hand, if the NFS interoperability problems turn out to
    > be unsolvable, then any other filesystem might be a better choice. :-)
    > AFS probably has an advantage here since as far as I know there aren't
    > multiple independent implementations of AFS, and interoperability is
    > usually better between ports of the same implementation of something
    > than it is between independent implementations.[/ref]

    Agreed. I have bee thinking about how best to make different *nix's
    interoperate over NFS.
    We're using RHEL/Centos/FC4, SLES, Debian, Gentoo, OSX as both Ws's and
    build/test servers, with FreeBSD only as server. Plus a lonesome Irix
    and Solaris for occational builds. Our True64 may never run again ;)
    Getting so many different NFS implementations interoperate may prove
    difficult. It's curreently NFS3, and maybe a wonder problems aren't
    bigger than the 2-3 mins outage, plus some annoying SSH timeouts.
    Otherwise it's working. I'll start by give the tuning a go, while trying
    to work out if the Promise driver/firmware could be another source.
     
    >

    >
    >
    > I don't think ZFS is ready for prime time either, yet, although it
    > promises to be ready pretty soon (maybe 6 months). Because it uses
    > copy on write for all writes, thus turning essentially all writes
    > into sequential I/O, it should be a pretty good combination with NFS,
    > since NFS really benefits from a filesystem that can complete write
    > requests quickly.[/ref]

    AFAIK, ZFS should be deemed ready with the upcoming june/july Solaris
    release. And yes, I was thinking about those features, combined with
    what others have described as quite good disk admin features.
     

    Thanks for all pointers, very helpful, I'm sure.

    --
    Kind regards,
    Mogens V.



    "During ongoing internal quality testing of the HP ProLiant DL585
    server, a potential vulnerability has been identified, where a
    remote unauthorized user may gain access to the server controls,
    but only when the server is powered down."

    Mogens Guest

Similar Threads

  1. Local storage
    By Big Mike7211 in forum Macromedia Flash Player
    Replies: 1
    Last Post: May 3rd, 08:29 PM
  2. USB Mass Storage
    By Jano in forum Windows Server
    Replies: 3
    Last Post: July 10th, 02:36 AM
  3. How Energy Storage can help us ?
    By Flywheel in forum PHP Development
    Replies: 0
    Last Post: September 1st, 11:13 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139