Ratdude's 3U Server Shenanigans

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Made some more purchases:

http://www.ebay.com/itm/Top-Motor-DF...72.m2749.l2649

That should quiet up the coolers while still providing excellent air flow when needed

http://www.ebay.com/itm/Kingston-2GB...72.m2749.l2649

Bought 6 2GB PC3200 sticks for $5 a pop shipped. 4 for the server, 2 for spares (since this will be system #4 for me that is running such RAM). Say what you want about kingston, but the four that didn't get nuked in storage run great in my wife's rig (which is also socket 940 with Opteron X2 285's). I'll happily buy more of those for peanuts.

Now I'm waiting on PCBs to come in...

[Topcat]

Re: Ratdude's 3U Server Shenanigans

I just bought some of those very same fans for the big black server.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Did more debian configuration today... that is, moving the SAMBA share to the RAID array and resetting network interfaces. Tried to bond (mode 0, round robin) my two gigabit ports but the switch didn't like it (or there is something I screwed up. No bonding (aka teaming in windows) for me right now

Also setup weekly backups to the former network drive since the RAID array is handling the actual shares.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Did more looking, yup, the stupid switch isn't up to snuff for bonding. I'd need a managed switch to do that? Oh well, I tried.

Edit- called topcat, nope. He has the exact same switch and teams just fine...

[Topcat]

Re: Ratdude's 3U Server Shenanigans

I have 2 switches on my LAN. One is a Linksys SR2024, gigabit unmanaged. This is the switch that is directly connected to the NAS and the WAN router. My office (addition to the building) is on the 3com switch, which is basically stacked on the Linksys (yes I know, creates a bottleneck, but I'm working on changing that). There's a teamed system connected on each switch with no issues.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Posted this on debian forums:

Question:

I'm trying to get bonding working on my new (to me) server. It uses a pair of broadcom netextreme gigabit ports (two different MAC adresses) for the bond (eth1 and eth2, eth0 is an intel 10/100 for 3 NICs on the mainboard).

What happens is, after ifdown'ing eth0/1/2 and stopping /etc/init.d/networking, I change /etc/network/interfaces to have bonding. ifup'ing bond0 results in good results... but then starting /etc/init.d/networking eth1 and eth2 shut down (eth1 goes to 10/100 WOL, eth2 goes dead).

Here's my /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#allow-hotplug eth0

#iface eth0 inet dhcp

auto bond0

iface bond0 inet static
    address 192.168.1.3
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    bond-mode balance-rr
    bond-miimon 100
    bond-slaves eth1 eth2
    #dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 192.168.1.1

auto eth1
iface eth1 inet manual
    bond-master bond0

auto eth2
iface eth2 inet manual
    bond-master bond0

and my /etc/udev/rules.d/70-persistent-net.rules

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.


# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:48:9a:64", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x14e4:0x1648 (tg3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:48:9a:28", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x14e4:0x1648 (tg3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:48:9a:29", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

I do have ifenslave and non-free firmware installed. The bonded NICs shut down on startup.

Is there something with /etc/init.d/networking that needs configured in addition to interfaces?

[diif]

Re: Ratdude's 3U Server Shenanigans

What switch are you using ?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

I think I fixed it. Apparently Ubuntu changed things and the "correct" way to run bonding on debian based systems (in balance-rr mode) is a bit different:

Here's my revised /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#allow-hotplug eth0

#iface eth0 inet dhcp

auto bond0

iface bond0 inet static
    address 192.168.1.3
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    bond-mode balance-rr
    bond-miimon 100
    bond-slaves none
    #dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 192.168.1.1


auto eth1
iface eth1 inet manual
    bond-master bond0
    bond-primary eth1 eth2

auto eth2
iface eth2 inet manual
    bond-master bond0
    bond-primary eth1 eth2

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

What switch are you using ?

https://www.badcaps.net/forum/showthread.php?t=63357

3Com Baseline 2824. 24 ports, unmanaged, full gigabit.

-----

Did more testing last night, works great, 2.0gbs reported. Note to self: Instead of all interfaces being slave, all are primary with no slaves. Yay obsolete information persisting on the web...


If I was dumber and greedier, I'd nab a gigabit PCI-x card or two (I probably have one somewhere actually) and up the ante... nice thing about linux is that bonding doesn't care about brand of chip (although my card IIRC is also broadcom netextreme). But with only 24 ports, I'm probably asking for trouble by doing more bonds (or not, depends on how many cables to each room?). TC is running 4 on his twin servers, so I'm behind... but this isn't a competition? Or is is...

Ok, I've already lost... but who said I'm not competitive???

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

If you first don't succeed, call it revision 1.


Here's revision 2, which acutally is NPN as called for in the intel PWM spec:

https://oshpark.com/shared_projects/C1SdRzD4





I also added a 5V full speed force jumper (JP2). It forces channel 2 to full, and also channel 1 if they're linked.

Better?

Bummer... Today I remembered something about KICAD: the TO220 pinout is wrong and pins 2 and 3 are backwards!

For testing at least, I'll probably be able to cross-leg the transistors, I hope.

In the mean time, I'll redo the board design and re-order them.

Doh!

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Ok, Here's Rev 3:

https://oshpark.com/shared_projects/V9E8DxQL





Changes:

• TO220 footprint fixed (Damn you KiCAD... and Damn me since I've already been burned on that before )
• Copper fill is ground, not 12V (no more warning needed)
• Swapped JP2 for a DPDT switch, like used on my Supermicro breakout board

I'll sleep on it before I order more... Did I mention I'm a dumbass?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Another update:

https://oshpark.com/shared_projects/MdnDZqET





Changes:

• Added Screw holes in a 1.5" x 2.5" pattern (6-32 HDD/case screws and standoffs) as my old idea of using foam tape is probably doomed to fail
• Addeda bottom silkscreen layer as it's no longer getting taped (if you have a better message for the bottom, I'm open to funnier suggestions)
• Rechecked a print against real heatsinks to verify clearances (this was an issue with versions not posted)

Look good? As much as I hate it, I won't order until I have one of the V1 PCBs hacked together to verify the schematic.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Did more looking... the Darlingtons use too much base current.

Looking at mosfets instead... found this:

https://electronics.stackexchange.co...ns-with-mosfet

Apparently when reading the intel spec I had things backwards? If it is indeed open collector, then that's the right circuit. I'll make a mosfet version based on that circuit.

Edit- Yup, confirmed. Mosfets and pullups it is.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

https://oshpark.com/shared_projects/zH7ELRHW





Changes:

• Conversion to a pulled up zener-based MOSFET Driver
• Removal of Force switch, as the pull up achieves the normal PWM fan function of no input= full blast
• Added a Tachometer passthrough to J8 and J11 via JP2 and JP3, as apparently some systems need the feedback to set base fan speed
• Took a tommy gun to it; due thin spots in the ground plane I decided to via stitch critical areas to increase current carrying capacity

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

As for why/how I ended up barking up the wrong tree design-wise, I'm blaming the cesspool of bad information online and me missing a few critical design points.

A careful read of the Intel's spec for PWM (see attached file), especially page 9, cleared up a lot of the nonsense. Yup RTFM yet again.

Once I saw the sentence by itself saying "pull up to" in the Intel tech sheet, I saw what was really going on. Hence why I saw a lot of schematics with TIP120's and resistor bridges...

...as for the move to MOSFET, that's because a tip122 only has a minimal gain of 1000. The intel spec limits the PWM input to a fan to 5mA... which for a spec'd load of 10A (a rough number I've used in the design), on both channels linked to one input we'd have a 10mA load. Since we have a pull-up circuit, that 10mA is over 7.3V (4.7V zener on the other side), which would be a 730 ohm ideal resistance. Since the input has to sink that, assuming a perfect 0V CE drop (worst case), that would be 16.4mA! Over 3x the current limit! However, the schematic I found online, when the output sinks, only has at most 4.44mA, which is close but not above the limit (what we want when designing a transistor/MOSFET driver).

Look good?


EDIT- FAIL. I'm going to make some changes.... As the driver will draw too much current to run another PWM signal off of it, I'm going to axe the PWM out ports and the jumpers.... and also move the tach ports and respace so PWM fan connectors will clear. Also seeing if I can half the resistance on the pull up resistors. Ugh. Here comes Rev 5!

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Did a crap ton of math to find out that one driver per PWM is the max, and 2.4K is ideal (due to the relatively high 25kHz waveform used).

So, I'll have to put in a 5A rated slide switch instead, as one of the Power MOSFETS can handle 10A in theory.

Fun fun fun...

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

... and here we go:

https://oshpark.com/shared_projects/Zx7VlDjY





No more tommy gun via stitching... instead we have hobo patches (fill zones)...

Look Good?

[Topcat]

Re: Ratdude's 3U Server Shenanigans

Looks good in a picture....the tell-tale will be if it works.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Looks good in a picture....the tell-tale will be if it works.

We shall see.

---------------------



Found my old trusty (and never actually used for anything) Broadcom NetXtreme BCM5703X Gigabit PCI-X 133MHz card in the pile (read: sitting in a dormant Netfinity 7000) and decided to have some fun.

That said, at least with this, the bonding setup is actually different than above... seems all my troubleshooting for 2 ports quit working, as the old file layout actually works:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#allow-hotplug eth0

#iface eth0 inet dhcp


auto bond0

iface bond0 inet static
    address 192.168.1.3
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    bond-mode balance-rr
    bond-miimon 100
    bond-slaves eth1 eth2 eth3
    #dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 192.168.1.1


auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

What was the real problem? Apparently ifup/ifsdown and the networking service do not agree and one shouldn't use use the former in debian. Either that, or the latter does the former when starting/stopping. The "new" file format was giving a bunch of "option primary not supported in balance-rr" errors when the service started, which hinted me towards reverting to the old format and skipping ifup (which did the trick).

Something I wasn't sure about but seems to be doing OK is the usage of two 133MHz cards in both 133MHz slots... as the manual wasn't super clear on on if that would work... they "recommend using slot 1). lspci -vv is reporting 133mhz on all 3 gigabit NICs and also the 3ware 9550 too. So far, so good...

Yay 1/8th of my switch is now used by my server... maybe I should have gone for a 48 port switch?

[Topcat]

Re: Ratdude's 3U Server Shenanigans

You need one more...you're one gigabit NIC away from a tie.