Silly network question -- 100BT -> 1GB

[Will Hartung]

We have some machines that we've plopped a couple of 1GB ethernet cards.

They're Sun 420s running Solaris 8.

Two other machines, V100's w/100BT, are able to FTP files at ~8.5MB per
second.

Now, while I'd be all giggly if we actually got "10x" performance moving
from 100BT->1000BT, I was realistic enough to not completely expect it.

However, running "iperf", and running through a Cisco switch with CAT 6
wires, we're only clocking ~325Mbps, or ~3 times faster than the V100's.

Now, I'm a network dunderhead, and this is second hand information, but
doesn't that seem low? I can see FTP limited by things like drive speed and
what not. But this "iperf" thing shouldn't be constrained by something like
that.

While there are vast array of things that could be tweaked to affect this, I
was curious if anyone had some things I could look at to maybe speed this up
a wee bit.

Thanx!

Regards,

Will Hartung
(willh@msoft.com)

[Lon Stowell]

Will Hartung wrote:

> We have some machines that we've plopped a couple of 1GB ethernet cards.
>
> They're Sun 420s running Solaris 8.
>
> Two other machines, V100's w/100BT, are able to FTP files at ~8.5MB per
> second.

That seems a bit low, but perhaps it is being limited by disk, etc.
If you move disk I/O out of the way and there is nothing much
going on in the client, say you are using a server faster than
the client and the server isn't diskbound, and you use something
like mkfile, you should see closer to 12 Mbytes/second.

>
> Now, while I'd be all giggly if we actually got "10x" performance moving
> from 100BT->1000BT, I was realistic enough to not completely expect it.

A gig card can saturate a PCI bus if there is nothing else on the
bus AND the cpu and operating system stuffing data at the NIC are
fast enough and the source of the data is is either a really fast
array or not a disk. I've measured near these speeds on a fast
server running a realtime engine for the protocols... but changing
the gig NIC from one brand to another dropped this by roughly 40%.

And to run a gig at anywhere near continuous speed, you need some
serious CPU power plus an operating system that doesn't annoy it.
Some systems benefit from bumping the tcp and udp watermarks to
roughly 64K.

Have also seen Gig speeds drop by about 10 Mbytes/second simply
going thru a switch. Some of those switches were bigger than the
server... the difference being that they can maintain a LOT of
gig connections.

[Dave Uhring]

On Thu, 24 Jul 2003 21:33:49 +0000, Rick Jones wrote:

> I could see perhaps unidirectional Gig traffic saturating a PCI-1X bus
> (33 MHz, 32 bit), but not anything faster. What sort of PCI slots are
> in a 420R?
>
> Didn't Sun at one point have some "1MHz per Mbit" rule of thumb? What
> frequency of CPUs are in a 420R?

Sun publishes their specs ;-)

[url]http://sunsolve.sun.com/handbook_pub/Systems/E420R/spec.html[/url]

[Sam N]

"Will Hartung" <willh@msoft.com> wrote in message
news:bfp9sc$h6l7d$1@ID-197644.news.uni-berlin.de...

> We have some machines that we've plopped a couple of 1GB ethernet cards.
>
> They're Sun 420s running Solaris 8.
>

<snip>

Hi Will

Check there's nothing untoward going on. Get hold of:

[url]http://www.unix.ms/cegrep[/url]
[url]http://www.unix.ms/netgrep[/url]
[url]http://www.unix.ms/nddgrep[/url]


and remember: gbit *must* be autonegged, and don't stick anything in
/etc/system - instead create a ce/ge.conf file for any edits.

cheers

Sam N

[Rick Jones]

Dave Uhring <daveuhring@yahoo.com> wrote:

> On Thu, 24 Jul 2003 21:33:49 +0000, Rick Jones wrote:

>> I could see perhaps unidirectional Gig traffic saturating a PCI-1X bus
>> (33 MHz, 32 bit), but not anything faster. What sort of PCI slots are
>> in a 420R?
>>
>> Didn't Sun at one point have some "1MHz per Mbit" rule of thumb? What
>> frequency of CPUs are in a 420R?

> Sun publishes their specs ;-)

> [url]http://sunsolve.sun.com/handbook_pub/Systems/E420R/spec.html[/url]

So, there is one PCI-1X slot, a pair of PCI-2X slots and a PCI-4X
slot, and 450 MHz of CPU. 300 and change Mbit/s doesn't sound all
that unreasonable then.

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

[Duncan Baillie]

Will, here are the latest tips I got from Sun:

I've double-checked the link speed/mode parameters, and basically you
ONLY need to set the
desired value to 1.

ie. to Enable auto-negotiation (automatically talk to the link partner
to establish either a
1000/100/10 full/half-duplex connection):

ndd -set /dev/ce adv_autoneg_cap 1
ndd -set /dev/ce adv_1000fdx_cap 0
ndd -set /dev/ce adv_1000hdx_cap 0
ndd -set /dev/ce adv_100fdx_cap 0
ndd -set /dev/ce adv_100hdx_cap 0
ndd -set /dev/ce adv_10fdx_cap 0
ndd -set /dev/ce adv_10hdx_cap 0

Or, to force 1000Mbps full-duplex:

ndd -set /dev/ce adv_autoneg_cap 0
ndd -set /dev/ce adv_1000fdx_cap 1
ndd -set /dev/ce adv_1000hdx_cap 0
ndd -set /dev/ce adv_100fdx_cap 0
ndd -set /dev/ce adv_100hdx_cap 0
ndd -set /dev/ce adv_10fdx_cap 0
ndd -set /dev/ce adv_10hdx_cap 0

Or, to force 100Mbps half-duplex:

ndd -set /dev/ce adv_autoneg_cap 0
ndd -set /dev/ce adv_1000fdx_cap 0
ndd -set /dev/ce adv_1000hdx_cap 0
ndd -set /dev/ce adv_100fdx_cap 0
ndd -set /dev/ce adv_100hdx_cap 1
ndd -set /dev/ce adv_10fdx_cap 0
ndd -set /dev/ce adv_10hdx_cap 0

All of the above settings are mutually-exclusive - you should set only
one of them to 1.


1. Expected performance of the 'ce' Giga-swift interface?

Card in a 33MHz pci slot 500-650Mbps
Card in a 66MHz pci slot 600-800Mbps

Note that these figures are from my colleague's observation, and not
from any published
documentation. It's possible that TCP tuning will also slightly
improve these figures.

Note also that as a rule of thumb, to drive a link at its maximum
speed requires a
similar amount of CPU MHz, to the Mbps rating of the link. We used
to say that a
4x100MHz qfe card needed 1x400MHz USII CPU to keep it busy, for example.


2. Correct method for enabling/disabling ce interface auto-negotiation?

There is currently an issue with the ce interface which means that we
do not recommend
forcing the link to 1000Mbps Full-Duplex mode - the card may fail to
correctly establish
link with a switch. It is best if you keep auto-negotiation switched on.

Auto-negotiation can be turned on by adding this to
/platform/sun4u/kernel/drv/ce.conf:

adv-autoneg-cap=1
adv-1000fdx-cap=0
adv-1000hdx-cap=0

Or, it can be turned on by creating an /etc/rc2.d/S68net-tune script
to include:

ndd -set /dev/ce instance 0
ndd -set /dev/ce adv_autoneg_cap 1

Both are correct. The support documents I'm reading advise creating
S68net-tune, but in
theory ce.conf should be preferable, as the changes take place
earlier in the boot
sequence, while the networking hardware is still configuring itself.


3. Best means for testing throughput of the link?

Use ttcp, rather than ftp. ttcp can open multiple streams to the
link, whereas ftp uses
just one. Hence ttcp is able to drive the link closer to its limit.

I've attached usage instructions, and the package includes a man page
and README.


4. Latest ce driver patch?

The latest version of the driver patch is currently 111883-17. It
would be worth making
initial tests with your current driver revision, then (if applicable)
upgrading the
driver patch and repeating the tests.

Regards

Duncan Baillie

Will Hartung wrote:

>We have some machines that we've plopped a couple of 1GB ethernet cards.
>
>They're Sun 420s running Solaris 8.
>
>Two other machines, V100's w/100BT, are able to FTP files at ~8.5MB per
>second.
>
>Now, while I'd be all giggly if we actually got "10x" performance moving
>from 100BT->1000BT, I was realistic enough to not completely expect it.
>
>However, running "iperf", and running through a Cisco switch with CAT 6
>wires, we're only clocking ~325Mbps, or ~3 times faster than the V100's.
>
>Now, I'm a network dunderhead, and this is second hand information, but
>doesn't that seem low? I can see FTP limited by things like drive speed and
>what not. But this "iperf" thing shouldn't be constrained by something like
>that.
>
>While there are vast array of things that could be tweaked to affect this, I
>was curious if anyone had some things I could look at to maybe speed this up
>a wee bit.
>
>Thanx!
>
>Regards,
>
>Will Hartung
>(willh@msoft.com)
>
>
>
>
>

[Darren Dunham]

Duncan Baillie <duncan@duncanthuy.com> wrote:

> Will, here are the latest tips I got from Sun:

> I've double-checked the link speed/mode parameters, and basically you
> ONLY need to set the
> desired value to 1.

> Or, to force 100Mbps half-duplex:

> ndd -set /dev/ce adv_autoneg_cap 0
> ndd -set /dev/ce adv_1000fdx_cap 0
> ndd -set /dev/ce adv_1000hdx_cap 0
> ndd -set /dev/ce adv_100fdx_cap 0
> ndd -set /dev/ce adv_100hdx_cap 1
> ndd -set /dev/ce adv_10fdx_cap 0
> ndd -set /dev/ce adv_10hdx_cap 0

If the 'ce' device acts like the 'hme' device, then you should run the
'adv_autoneg_cap' command last. When it is set, it will reconfigure the
device. If you run them in your order, then all the later commands are
ignored until the link is reinitialized.

--
Darren Dunham [email]ddunham@taos.com[/email]
Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

[Mark Stoltzfus]

In article <C2cWa.11518$Mg7.11373@newssvr22.news.prodigy.com> ,
Darren Dunham <ddunham@redwood.taos.com> wrote:

>Duncan Baillie <duncan@duncanthuy.com> wrote:

>> Will, here are the latest tips I got from Sun:

>

>> I've double-checked the link speed/mode parameters, and basically you
>> ONLY need to set the
>> desired value to 1.

>

>> Or, to force 100Mbps half-duplex:

>

>> ndd -set /dev/ce adv_autoneg_cap 0
>> ndd -set /dev/ce adv_1000fdx_cap 0
>> ndd -set /dev/ce adv_1000hdx_cap 0
>> ndd -set /dev/ce adv_100fdx_cap 0
>> ndd -set /dev/ce adv_100hdx_cap 1
>> ndd -set /dev/ce adv_10fdx_cap 0
>> ndd -set /dev/ce adv_10hdx_cap 0

>
>If the 'ce' device acts like the 'hme' device, then you should run the
>'adv_autoneg_cap' command last. When it is set, it will reconfigure the
>device. If you run them in your order, then all the later commands are
>ignored until the link is reinitialized.

When I spoke to Sun about this, they also recommended the following:

/usr/sbin/ndd -set /dev/ce link_master 1

FYI,

Mark