Gigabit LAN PCI speeds

[ratdude747]

Question: My Supermicro has two PCI-X channels, split as such:

#1: 66MHz PCI-X slot, twin Silicon image FakeRaid (Disabled)
#2: 100MHz PCI-X slot, Twin Intel Copper 10/100/1000 (Max 133MHz)




I had been running it with my 66MHZ 3ware card in slot #2 and no card in slot #1, allowing my gigabit LAN to run at 133MHZ.

However, thanks to a new GPU on the way, I had to move a 33MHZ PCI sound card to one of the slots.

Option 1 is to move the RAID card to #1 and the Sound card to #2, Stepping down the Gigabit LAN to 66MHz

Option 2 is to move the Sound card to #1, stepping down the Gigabit LAN to 33MHz

Option 2 has the benefit of my HDD LED lead not being wedged between two large cards, While Option 1 is the least harmful to my LAN.

Here's the question. I currently only use one of the gLan ports, and it's at 10/100. However, since I plan to keep the rig awhile, It may go to 1000. I doubt I'll use both ports.

So, How big of a performance loss is 33MHz vs 66MHz?

[Topcat]

Re: Gigabit LAN PCI speeds

Here's the question. I currently only use one of the gLan ports, and it's at 10/100. However, since I plan to keep the rig awhile, It may go to 1000. I doubt I'll use both ports.

So, How big of a performance loss is 33MHz vs 66MHz?

Over the LAN, you won't notice any difference.

Glad you're keeping that old rig for a while....

[ratdude747]

Re: Gigabit LAN PCI speeds

Over the LAN, you won't notice any difference.

Glad you're keeping that old rig for a while....

After the $ and effort that went into it, I better.

So I wouldn't get nailed if I ran the LAN at 33MHZ?

[Topcat]

Re: Gigabit LAN PCI speeds

After the $ and effort that went into it, I better.

So I wouldn't get nailed if I ran the LAN at 33MHZ?

No.

[ratdude747]

Re: Gigabit LAN PCI speeds

No.

Sweet.

[japlytic]

Re: Gigabit LAN PCI speeds

With only one PCI slot used for a Gigabit LAN card, it would be getting close to saturation, but if you have other PCI devices, saturation can occur.
So for Gigabit PCI LAN cards, I would stick with PCI Express.

[ratdude747]

Re: Gigabit LAN PCI speeds

With only one PCI slot used for a Gigabit LAN card, it would be getting close to saturation, but if you have other PCI devices, saturation can occur.
So for Gigabit PCI LAN cards, I would stick with PCI Express.

My board doesn't have PCI express. It's the last intel generation to use AGP (and also the last x86 only generation).

[Mad_Professor]

Re: Gigabit LAN PCI speeds

Well you would be halving your bandwidth pool for the gigabit, but sound doesn't take up that much, depending on how often you use it, is there a reason why you need a sound card? I ran my old mythtv box with two PCI tuners, PCI sound card, PCI RAID card and a PCI network card, it worked fine, I had some performance issues, but pci latency to 64 fixed that, but I never intended to file transfers, only streaming myth to other frontends in the house.

Anyways,
Why not set it up the way you want it? then test and see if there is any issues with your setup using iperf between two gigabit computers across a gigabit switch, if you hit 80% of gigabit, great! if not, well some thing will have to be done unless you're happy getting less then 80%.

I was happy with 600Mbit/s for awhile until I overhauled my fileserver. now I'm hitting 950Mbit/s in iperf and done file transfers around 120MB/s, which is awesome! On average I only do about 75-90MB/s, it really depends on hardware and type of files you're moving. For example, I slow to a crawl when I hit a bunch small files, but if I'm doing ISO or drive images, I can do the full 125MB/s or close to it.

Hopefully this will give you some insight, since PCI and PCI-X are old friends of mine and I hate them both dearly.

[mariushm]

Re: Gigabit LAN PCI speeds

Well, on pci 33 mhz if i remember correctly you have 125 MB/s .. 1 gbps is 120 MB/s or around that value.
You also have the onboard sound, maybe an usb controller on pci, everything shares the pci bus.

But on the other hand, you're not blocking all those 120 MB/s permanently, just by placing the card on pci slot, and you're not transferring at 1gbps 24/7 so it's not really an issue.
It's more of an issue that on pci, you'd have a lot of interrupt requests and the cpu will be bogged down by large speed transfers.
Cards that are on pci typically don't have interrupt moderation, hardware offloading and so on so cpu usage would be high.

That twin port intel card is certainly fast and has good chip on it so that's not the case.

Now... as for your options. I'd leave the network card on the pci-x 100 mhz and play around with the other cards.
But it's really about WHAT you use it for. Is that a computer for regular use, you connect this card to a router and that's it? Or is it a NAS/media sort of server?

If it's just for your own use connected to a router or in your small home network, it doesn't really matter where it's connected, it will work fine.

If it's a NAS/storage server for your house and you want bandwidth/throughput... leave it on 100mhz and think of doing port trunking (to get 2gbps) out of the server.

[Topcat]

Re: Gigabit LAN PCI speeds

All good points....but as I said, you won't even feel it.... I've been there done that. You're pushing the bus close to its peak, but it'll be ok. I dont think you realize the amount of data it takes to peg a 1gbit LAN. if the bus is clogged up with other crap (say a massive USB transfer for example, assuming its sharing the bus) and you're web browsing, you still won't feel it. Audio takes almost no bandwidth.

[Mad_Professor]

Re: Gigabit LAN PCI speeds

I agree with top cat, it takes a lot of effort to saturate 1Gbit link by yourself, not so much when you have 100 computers hitting it at the same time.

@mariushm: Gigabit Ethernet (1000BASE-T) 1 Gbit/s 125 MB/s
source: http://en.wikipedia.org/wiki/List_of..._area_networks

[mariushm]

Re: Gigabit LAN PCI speeds

Thanks for bringing it up, made me review my post and spotted an error.
I don't know where I got that 125 MB/s, it's actually 133 MB/s for standard PCI. That's MB with 1000 bytes in a byte

33.33 MHz x 32 bits ÷ 8 bits/byte = 133 MB/s
33,330,000 x 32 / 8 = 133,320,000 / 1024 = 130,195.3125 KB/s / 1024 = 127.14 MB/s (with MB using multiples of 1024)

but I was sort of right about the gigabit.

Gigabit is 1.000.000.000 bits per second ÷ 8 bits/byte = 125.000.000 bytes/s ÷ 1024 bytes in a KB = 122070.3125 KB/s ÷ 1024 KB in a MB = 119.2 MB/s

I guess wikipedia uses the SI kilobyte which is 1000 bytes while I use the KiB which is 1024 bytes/s

So
PCI : 133 MB/s or ~127 MiB/s
Gbps : 125 MB/s or ~119 MiB/s

[ratdude747]

Re: Gigabit LAN PCI speeds

FYI:

I need a sound card because the Supermicro X5DAL-TG2 has no onboard sound.

Here's my slot layout:


1: No connector (Bottom of CPU #2 socket)
2: AGP Pro 8X
3: PCI 33MHz
4: PCI 33MHz
5: PCI-X 100MHz
6: PCI-X 66MHz
7: No connector (Fake raid chips and SATA connectors)

My GPU will take up Slots 2 and 3. Slot 4 is a 1394 card; its' also a spacer (so the two other large cards don't block the GPU fan intake). Slots 5 and 6 are the ones I'm having to pick from; both cards are quite large.

[mariushm]

Re: Gigabit LAN PCI speeds

Dump the 1394 card, think really hard how often do you really use that nowadays.

Move the sound card in the second PCI slot.

Problem solved.

[eccerr0r]

Re: Gigabit LAN PCI speeds

Has anyone ever gotten close to linefill on a PCI ethernet?

So far the only computers that I could get well over 95MB/sec through Gbit are ones with onboard ethernets (PCIe, PCIX). The PCI33 boards I've not gotten anywhere close yet, highest I've seen is around 36MB/sec which is far short of theory. This also could be related to the computer however, as well as the quality of the board (RTL8169) could be implicated. But some of the onboard pcie are Realteks as well and can do 90MB/sec+ ...

[mariushm]

Re: Gigabit LAN PCI speeds

When I was network administrator in a campus, I had a Barton based PC with 2 hard drives connected to the building's switches on a gigabit port.

With DC++, I was able to get sustained 47-55 MB/s to my computer. It was a pci based onboard network card, some Marvell gigabit controller.

The server doing the routing had a Tyan motherboard with an onboard Intel gigabit card (probably on pci-x 66 mhz) and I used a second PCI Intel gigabit card because there was no budget for an expensive pci-x card.

So the whole building was using 2x48+1x24 switches interconnected with (i think it was 17 gbps) backplane, then out came a gigabit link to the server, out went a gigabit link to a gigabit port.
A few of the buildings in the campus were connected with FTP cat5e cable, the longest distance was about 112 meters (over the maximum length permitted).
No packet loss in case of my building (other building had about 0.3-0.6% packet loss with that long distance), in tests got to about 85-90 MB/s between campus buildings.

[ratdude747]

Re: Gigabit LAN PCI speeds

Dump the 1394 card, think really hard how often do you really use that nowadays.

Move the sound card in the second PCI slot.

Problem solved.

Quite often, Actually. Mainly for my DV camera.

Also, my sound card is quite massive and I thought it might interfere with my GPU's fan intake:





However, it might clear.

If it clears, I could send the the 1394 card to the bottom, the RAID above it, and the Sound at the top of the 3.

I won't know until the GPU comes in.

[Uranium-235]

Re: Gigabit LAN PCI speeds

are you switched at 1gbps? does this thing access/transfer any local data that needs to be past 12.5 MB/s? (100 mbit)? if not, you don't need it

I have a poweredge 1800 with a gigabit switch in the next room, I transfer files off it at ~95MBps (past the hard drives transfer speed, cause its using EXT compression)

if you're not using a setup like that, and just using it for regular surfing, you dont' need gigabit crap

[ratdude747]

Re: Gigabit LAN PCI speeds

right now I'm on 10/100... the gigabit is onboard (intel copper controller).

I ended up putting the sound card to the bottom since the cards did indeed block the fan intake. I fixed the LED header by bending the pins slightly so the header connects at an angle, thus allowing plenty of clearance.