nfs client any better on linux?

[jasonedgar@digitalnPAM.com]

Background:

I run a big cluster of servers that handle all the
video processing for some media files. We used to
run on high end SGI systems but migrated over to
Linux (high end proliants, have a good relationship
with Compaq/HP so it made sense to use these guys).

Saved alot of money going with Intel but now we're
hitting some problems.

We used ot be able to use local disk to stage the
files but as we grow this is becoming unwieldly.

We bought 3 Sun V480's (maxxed out with RAM & CPU)
and run NFSD on them. The NFS servers held all
the media files and the linux machines would access
them over NFS and do their magic there.

Problem:

Bad idea. The Suns held their own (the least you can
expect when you put down that much cash) but the
NFS client support on Linux wasn't very good under
load. Apparently some bugs in the 2.4.18 kern for
udp out of order messages.

Also had to something really odd, turn off locking in
the fstab to stabilize the machines.

Is the NFS client any better under Linux now?

If not, we'll probably have to convert over to Sol
x86. Which I want to avoid :-(

-Jason

[jasonedgar@avoidingspam.com]

In comp.os.linux.networking Kjetil Torgrim Homme <kjetilho@yksi.ifi.uio.no> wrote:

> I don't think you will see much difference between 2.4.18 and 2.4.21.
> we're using the stock Red Hat kernels, and haven't really had much
> problems with that. even the automounter seems to behave now. are
> you using NFS version 2 or 3? (we're running the default: nfs3,udp)
> locking is a bit flaky, though. what are the symptoms you're seeing?

We tried default settings and then experimented (went with nfs v3
tcp, ran it thru auto-mounter which amazingly made it better even
though we were not using any automounter features).

I agree with your observation on locking. One bizarre symptom we
saw was that we had to disable lockign (i believe it is the
option 'nolock' in fstab) on RH7.3 to get it to go.

Basic problem was the udp out-of-order bug in the 2.4.18 kern.
Stranger problem was one rev of the kern actually caused the
system to hang unless we disabled the locking in fstab.

> Solaris/x86 has a solid NFS implementation. the reason we went for
> Linux instead was lack of commercial software (Matlab etc.), this
> probably doesn't apply to you.

No, it doesn't. Should just be a matter of runnign 'make' on the
source again.

Thank you both for the follow-up.

[Joerg Schilling]

In article <pan.2003.08.03.12.39.13.790455@yahoo.com>,
Dave Uhring <daveuhring@yahoo.com> wrote:

>On Sun, 03 Aug 2003 11:33:02 +0000, Joerg Schilling wrote:
>

>> In article <pan.2003.08.02.22.25.39.912886@yahoo.com>,
>> Dave Uhring <daveuhring@yahoo.com> wrote:
>>

>>>> Is the NFS client any better under Linux now?
>>>
>>>The Linux NFS client has been working well for my light usage. Presently
>>>using 2.4.21-xfs on desktop boxes with Solaris SPARC and x86 servers.

>>
>> On my 2.4.21 (IBM-STB-405-xxx) I still get no large file support with the
>> NFS client :-(

>
>Does your glibc support that? You might consider using XFS, also.
>
> [url]http://oss.sgi.com/projects/xfs/[/url]

Large files work fine on a _local disk_, but this is a SetTopBox and it would
be nice to be able to work with large files for video mounted via NFS.


Via NFS, I can only access all files < 2 GB.

XFS is a local disk filesystem, so I don't see why it should help for NFS to use
XFS.

--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
[email]js@cs.tu-berlin.de[/email] (uni) If you don't have iso-8859-1
[email]schilling@fokus.fraunhofer.de[/email] (work) chars I am J"org Schilling
URL: [url]http://www.fokus.fraunhofer.de/usr/schilling[/url] [url]ftp://ftp.berlios.de/pub/schily[/url]

[Rod Smith]

In article <bghcv6$gvm$1@bob.news.rcn.net>,
[email]jasonedgar@digitalnPAM.com[/email] writes:

>
> Problem:
>
> Bad idea. The Suns held their own (the least you can
> expect when you put down that much cash) but the
> NFS client support on Linux wasn't very good under
> load. Apparently some bugs in the 2.4.18 kern for
> udp out of order messages.

....

> Is the NFS client any better under Linux now?
>
> If not, we'll probably have to convert over to Sol
> x86. Which I want to avoid :-(

I have no specific advice on improving Linux's NFS client support, but
you might want to consider other file-sharing options as an alternative
to switching to Solaris for x86. The Linux kernel network filesystems
configuration menu shows that it supports Coda, InterMezzo, NFS, SMB, and
NCP. Of these, I'm most familiar with SMB (aka CIFS, and served by Samba
on Unix-like platforms). It doesn't support some Unix features (ownership
and some permissions, for instance), but depending upon your needs, that
might not be important. I've seen people claim that Samba and the Linux
kernel's SMB/CIFS support is better than Linux's NFS support. I tend to
agree, although for my own personal needs, the differences are minor.

I don't know offhand if Solaris can run servers for any of these except
NFS and SMB/CIFS (Samba definitely does run on Solaris). Anyhow, if you've
got more than a couple of clients, it'd probably be easier to replace your
server software with one that uses a different protocol than to replace
the OS on all your clients.

--
Rod Smith, [email]rodsmith@rodsbooks.com[/email]
[url]http://www.rodsbooks.com[/url]
Author of books on Linux, FreeBSD, and networking

[Dave Uhring]

On Sun, 03 Aug 2003 13:14:12 +0000, Joerg Schilling wrote:

> Large files work fine on a _local disk_, but this is a SetTopBox and it would
> be nice to be able to work with large files for video mounted via NFS.
>
>
> Via NFS, I can only access all files < 2 GB.
>
> XFS is a local disk filesystem, so I don't see why it should help for NFS to use
> XFS.

Well, I just created an empty file 2.5GB on a Solaris x86 machine which is
an NFS server, then copied it to a Linux machine.

[Les Mikesell]

"Joerg Schilling" <js@cs.tu-berlin.de> wrote in message
news:bgj1r4$ms4$1@news.cs.tu-berlin.de...

> Large files work fine on a _local disk_, but this is a SetTopBox and it

would

> be nice to be able to work with large files for video mounted via NFS.
>
>
> Via NFS, I can only access all files < 2 GB.
>

You need to use nfsv3 for large files, but that should be the default in
most distributions. Are you sure the server is using v3? You also need
to be sure the applications were compiled with large file support but this
should also be the default for current distribution.

---
Les Mikesell
[email]lesmikesell@comcast.net[/email]

[Akop Pogosian]

If this problem is easily reproducible, perhaps you could buy a
support contract, at least for one of the machines and submit a
problem report to your Linux vendor? (They also have bug tracking
systems that don't require a support contract but the response time
and patch availability with those really depends on how much the
developers themselves care about the problem..)

-akop


In comp.unix.solaris [email]jasonedgar@digitalnpam.com[/email] wrote:

> Background:

> I run a big cluster of servers that handle all the
> video processing for some media files. We used to
> run on high end SGI systems but migrated over to
> Linux (high end proliants, have a good relationship
> with Compaq/HP so it made sense to use these guys).

> Saved alot of money going with Intel but now we're
> hitting some problems.

> We used ot be able to use local disk to stage the
> files but as we grow this is becoming unwieldly.

> We bought 3 Sun V480's (maxxed out with RAM & CPU)
> and run NFSD on them. The NFS servers held all
> the media files and the linux machines would access
> them over NFS and do their magic there.

> Problem:

> Bad idea. The Suns held their own (the least you can
> expect when you put down that much cash) but the
> NFS client support on Linux wasn't very good under
> load. Apparently some bugs in the 2.4.18 kern for
> udp out of order messages.

> Also had to something really odd, turn off locking in
> the fstab to stabilize the machines.

> Is the NFS client any better under Linux now?

> If not, we'll probably have to convert over to Sol
> x86. Which I want to avoid :-(

> -Jason

--
Akop Pogosian

This space has been accidentally left blank.

[Bernd Schubert]

> Large files work fine on a _local disk_, but this is a SetTopBox and it

> would be nice to be able to work with large files for video mounted via
> NFS.
>
>
> Via NFS, I can only access all files < 2 GB.

As already pointed out by Les only valid for nfs < version3!

>
> XFS is a local disk filesystem, so I don't see why it should help for NFS
> to use XFS.
>

Well, of course you need a local filesystem that supports files >2GB (e.g.
reiserfs-3.5 doesn't whereas 3.6 *does*).

Bernd

[Bernd Schubert]

[email]jasonedgar@digitalnPAM.com[/email] wrote:

> Background:
>
> I run a big cluster of servers that handle all the
> video processing for some media files. We used to
> run on high end SGI systems but migrated over to
> Linux (high end proliants, have a good relationship
> with Compaq/HP so it made sense to use these guys).
>
> Saved alot of money going with Intel but now we're
> hitting some problems.
>
> We used ot be able to use local disk to stage the
> files but as we grow this is becoming unwieldly.
>
> We bought 3 Sun V480's (maxxed out with RAM & CPU)
> and run NFSD on them. The NFS servers held all
> the media files and the linux machines would access
> them over NFS and do their magic there.
>
> Problem:
>
> Bad idea. The Suns held their own (the least you can
> expect when you put down that much cash) but the
> NFS client support on Linux wasn't very good under
> load. Apparently some bugs in the 2.4.18 kern for
> udp out of order messages.
>
> Also had to something really odd, turn off locking in
> the fstab to stabilize the machines.
>
> Is the NFS client any better under Linux now?
>
> If not, we'll probably have to convert over to Sol
> x86. Which I want to avoid :-(
>
> -Jason

Hmm, our linux-based nfs-server and clients run rock solid (we just rebooted
our clients and the server after an uptime of > 250 days; and only due to
some security updates and a distribution update).

I don't know how much you depend on your Sun-servers, but if I were you, I
would try to run the nfsd also on a linux-server. Of course running the
latest kernel versions is usually a good idea (2.4.21 has lots of
bug-fixes).

Also, there is an nfs-ML: <nfs@lists.sourceforge.net>
where you should describe your problems.

Regards,
Bernd