Ratdude's 3U Server Shenanigans

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

What switch are you using ?

https://www.badcaps.net/forum/showthread.php?t=63357

3Com Baseline 2824. 24 ports, unmanaged, full gigabit.

-----

Did more testing last night, works great, 2.0gbs reported. Note to self: Instead of all interfaces being slave, all are primary with no slaves. Yay obsolete information persisting on the web...


If I was dumber and greedier, I'd nab a gigabit PCI-x card or two (I probably have one somewhere actually) and up the ante... nice thing about linux is that bonding doesn't care about brand of chip (although my card IIRC is also broadcom netextreme). But with only 24 ports, I'm probably asking for trouble by doing more bonds (or not, depends on how many cables to each room?). TC is running 4 on his twin servers, so I'm behind... but this isn't a competition? Or is is...

Ok, I've already lost... but who said I'm not competitive???

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

I think I fixed it. Apparently Ubuntu changed things and the "correct" way to run bonding on debian based systems (in balance-rr mode) is a bit different:

Here's my revised /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#allow-hotplug eth0

#iface eth0 inet dhcp

auto bond0

iface bond0 inet static
    address 192.168.1.3
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    bond-mode balance-rr
    bond-miimon 100
    bond-slaves none
    #dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 192.168.1.1


auto eth1
iface eth1 inet manual
    bond-master bond0
    bond-primary eth1 eth2

auto eth2
iface eth2 inet manual
    bond-master bond0
    bond-primary eth1 eth2

[diif]

Re: Ratdude's 3U Server Shenanigans

What switch are you using ?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Posted this on debian forums:

Question:

I'm trying to get bonding working on my new (to me) server. It uses a pair of broadcom netextreme gigabit ports (two different MAC adresses) for the bond (eth1 and eth2, eth0 is an intel 10/100 for 3 NICs on the mainboard).

What happens is, after ifdown'ing eth0/1/2 and stopping /etc/init.d/networking, I change /etc/network/interfaces to have bonding. ifup'ing bond0 results in good results... but then starting /etc/init.d/networking eth1 and eth2 shut down (eth1 goes to 10/100 WOL, eth2 goes dead).

Here's my /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#allow-hotplug eth0

#iface eth0 inet dhcp

auto bond0

iface bond0 inet static
    address 192.168.1.3
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    bond-mode balance-rr
    bond-miimon 100
    bond-slaves eth1 eth2
    #dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 192.168.1.1

auto eth1
iface eth1 inet manual
    bond-master bond0

auto eth2
iface eth2 inet manual
    bond-master bond0

and my /etc/udev/rules.d/70-persistent-net.rules

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.


# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:48:9a:64", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x14e4:0x1648 (tg3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:48:9a:28", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x14e4:0x1648 (tg3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:81:48:9a:29", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

I do have ifenslave and non-free firmware installed. The bonded NICs shut down on startup.

Is there something with /etc/init.d/networking that needs configured in addition to interfaces?

[Topcat]

Re: Ratdude's 3U Server Shenanigans

I have 2 switches on my LAN. One is a Linksys SR2024, gigabit unmanaged. This is the switch that is directly connected to the NAS and the WAN router. My office (addition to the building) is on the 3com switch, which is basically stacked on the Linksys (yes I know, creates a bottleneck, but I'm working on changing that). There's a teamed system connected on each switch with no issues.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Did more looking, yup, the stupid switch isn't up to snuff for bonding. I'd need a managed switch to do that? Oh well, I tried.

Edit- called topcat, nope. He has the exact same switch and teams just fine...

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Did more debian configuration today... that is, moving the SAMBA share to the RAID array and resetting network interfaces. Tried to bond (mode 0, round robin) my two gigabit ports but the switch didn't like it (or there is something I screwed up. No bonding (aka teaming in windows) for me right now

Also setup weekly backups to the former network drive since the RAID array is handling the actual shares.

[Topcat]

Re: Ratdude's 3U Server Shenanigans

I just bought some of those very same fans for the big black server.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Made some more purchases:

http://www.ebay.com/itm/Top-Motor-DF...72.m2749.l2649

That should quiet up the coolers while still providing excellent air flow when needed

http://www.ebay.com/itm/Kingston-2GB...72.m2749.l2649

Bought 6 2GB PC3200 sticks for $5 a pop shipped. 4 for the server, 2 for spares (since this will be system #4 for me that is running such RAM). Say what you want about kingston, but the four that didn't get nuked in storage run great in my wife's rig (which is also socket 940 with Opteron X2 285's). I'll happily buy more of those for peanuts.

Now I'm waiting on PCBs to come in...

[Topcat]

Re: Ratdude's 3U Server Shenanigans

^
I told you... Sorry about the screws....they're sitting here in a little medicine cup. Makes up for the 2 dead CMOS batteries in the SM blades.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

UPDATE:

Bad RAM is confirmed. Ran it off 8gb pulled from Main Rig V2.5, rock solid.

Time to get more RAM I guess.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Stuff has come in! Pictures:

First attempt at SATA cables:



The metal clips on the plugs didn't agree, so I axed them, which helped a lot:



Cables routed:



Topcat, you forgot the HDD screws... Turns out Ace hardware had competitive prices on them (in stainless too!) so I bought 200 just to be safe (since pretty much all caddies use these, and I hate torx screws):



While I was near my parents house, I swung by and nabbed some goodies:

• an old 3ware 9500 with a good battery (unlike the two I've tried and reported as "weak" and wouldn't charge)
• another Molex to SATA adapter
• some NEC USB 2.0 cards
• Floppy disks

So I installed the drives, 14 of the 16 good (one of the lot of 12 failed SMART, one of the singles wouldn't spin and is being returned). I also pulled the battery from the 9500 (which I don't care for when I can drop in a 9550 and get a 10mb/s speed boost and also get SMART checking).

Threw in an NEC USB 2.0 card, as this board only has 1.1 oddly (AMD 3100 chipset limitation)

As a test, I moved over the drives from my crappy server, and while it did run, I got some ECC errors which eventually ended in a Kernel Panic. Memtest 86+ (Booted from USB floppy) doesn't report any errors but will randomly reboot. Bad RAM?

Final state tonight:



Progress?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Also, to save money and trouble, since I have a spare port on each channel, I'll probably run the CPU fans off of this as well.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Made a slight change (fixed fan input labels). Ordered boards and components.

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

Yes and yes.

[Topcat]

Re: Ratdude's 3U Server Shenanigans

^
I take it this is an attempt to quiet this monstrosity down?

[Topcat]

Re: Ratdude's 3U Server Shenanigans

^
I take it this is an attempt to quiet this monstrosity down?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

If you first don't succeed, call it revision 1.


Here's revision 2, which acutally is NPN as called for in the intel PWM spec:

https://oshpark.com/shared_projects/C1SdRzD4





I also added a 5V full speed force jumper (JP2). It forces channel 2 to full, and also channel 1 if they're linked.

Better?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

A few hours on KiCAD and...

https://oshpark.com/shared_projects/mWDGPILr





All this does is take one or two PWM signals and amplifies them to 3 or 6 outputs though a TIP122. It's set up as two separate channels, with a jumper to link them if only one PWM signal is used. The idea being that since I have PWM control on the CPU coolers, I can tap into that (or another PWM controller, they do exist on the net but are not high output). See attached schematic.

It uses a molex plug for power (like the old breakout board), standard 4 pin PWM fan headers for the PWM source (that way all I need to do is run a cable to the port, and another to the Fan unless the fan reaches by itself)... I'm sure some cheap fan extenders (if one can buy such) and some splicing/crimping would do the trick.

To keep the TIP122's cool, I designed it to use one of these:

http://www.digikey.com/scripts/DkSea...mSeq=235228169

Hence the layout I chose.

Something I didn't do but though about was adding a switch to short the PWM in to the 5V on the molex to force full blast... but if I need to do that, I'll swap back to the old breakout.

Look like a winner? Edit- Maybe not, it looks like I tried to make it PNP, not NPN. That's what I get for using schematics online. Boards were not ordered. Yay Respin time?

[ratdude747]

Re: Ratdude's 3U Server Shenanigans

For the fans, would something like this make sense?: (only beefier, I have 4 1.53A fans in the system)

https://www.newegg.com/Product/Produ...9157-_-Product

Edit- No luck, I'm out luck, unless I bought one of those and modded it. But if I'm going there, I'd be tempted to design my own PCB and make one that used one of the CPU PWM signals for fan control (via darlington of MOSFET). Thoughts?

And the Dynatrons I could probably swap for PWM fans, as the CPU fan connectors do support PWM.