Obligatory backup server question

[Hondaman]

Thus far, I have backup files on half a dozen hard disks - mostly they contain lots of copies of the same files. Up until now, I have been trying to get everything centralized on one SATA hard disk in a USB enclosure, getting rid of duplicates and stuff I don't need anymore (like the bios upgrade from "Mr. BIOS" for my K5-166 MHz machine).

Should I build a dedicated file server from spare parts? If this server is always on, I can get a list of files on this server anytime I want. This would be easier than turning on my USB enclosure drive each time I want to "clean up this mess". An always-on server could be combined with awk pattern matching or dedicated de-duping software to get rid of duplicate files.

Perhaps I could use an automated script that only backs up recently-altered files from certain directories on my Windows machine at a time of day when I am never online?

Does anyone have any experience with a "dedicated file server" type of linux distribution?

Should I use multiple hard drives in a RAID config? SSD? Upgrade to a fanless PSU later on?

The essential files I have do not take up much space - spreadsheets, docs, PDFs of mortgage statements. Space requirements explode, however, when you include photos, DVD rips, Wikileaks "global warming" leaks and Scientology crap.

Any ideas?

EDIT: Does anyone have experience with FreeNAS or the forked project NAS4Free? Would they be good to use?

[shovenose]

Re: Obligatory backup server question

zfs

[Hondaman]

Re: Obligatory backup server question

ZFS is a file system. After some work, I got NAS4Free to run on an old PC that I put together. I'm using the UFS filesystem. ZFS might offer much more capabilities, but it really needs ECC memory, they say, so I'm sticking with UFS. My storage is on two 320 GB SATA disks in Raid-1. For a "starter" system, it's pretty good backup. A fanless CPU cooler (Prolimatech Samuel 17, for socket 755) is already on its way, and I may grab a passive-cooled PSU later this year or next year.

(It sure beats paying $300 for EIDE Travan-4 tape equipment 20 years ago! Tape sucked! And it cost me $300 to find that out!)

Thanks shovenose.

[stj]

Re: Obligatory backup server question

how are you actually backing up between systems? stupid bloatware, or something like Rsync?

[diif]

Re: Obligatory backup server question

Don't leave the backup permanently connected. Cryptoware encrypts any connected drive/s.

[stj]

Re: Obligatory backup server question

that's what i was getting at, there should not be any type of network sharing service running, only ftp / rsync / ssh type links

[TechGeek]

Re: Obligatory backup server question

Guys, GUYS, GUYS!!!!!!!!!!!!!! HE USES ZFS!!! ON NAS4FREE!!!! NOT NTFS AND WINDOWS!!!!!! I'm posting something in the rant thread...

[Uniballer]

Re: Obligatory backup server question

If he was running, for example, Samba to provide Windows with network file services then it doesn't matter what operating system the server is running: the data is vulnerable to cryptoware.

It doesn't matter if you are running BSD, Linux, OS X, or Windows. If the disk is plugged in to the grid when lightning strikes then you could burn it up. If that matters then you need offline backups. If the building burns down or gets flooded then you could lose your offline backups, too. If that matters then you need offsite backups.

[TechGeek]

Re: Obligatory backup server question

...And can current cryptoware even WRITE TO ZFS filesystems?

[Uniballer]

Re: Obligatory backup server question

...And can current cryptoware even WRITE TO ZFS filesystems?

Probably not directly, but it can if the server is exporting filesystems based on the ZFS volumes (e.g. Samba based, or NFS, or whatever).

[TechGeek]

Re: Obligatory backup server question

It just simply runs the ZFS filesystem.

[Per Hansson]

Re: Obligatory backup server question

TechGeek: Perhaps before bashing other people for not reading the thread try doing the same yourself?
Hondaman said in his second post that he went for UFS over ZFS due to lack of ECC RAM in his system.
And it's really does not matter because as diif correctly pointed out a compromised client that has write access to the ZFS share can easily delete/encrypt/corrupt all the data on the server anyway.

[stj]

Re: Obligatory backup server question

It just simply runs the ZFS filesystem.

which brings us back to my question - how does the system get backed up?

[ratdude747]

Re: Obligatory backup server question

I run a couple file servers, one at my apartment, one at my parents house. The hardware is a bit different (and in a while I'll be changing the one at my apartment) but the software is the same (mostly)

(I've talked about this before, and I know I will get some shit, but I'm going to answer the original question)

Both run debian x64, CLI only. Mine has jessie, theirs is still on wheezy, as I didn't feel like dicking with an update, and given how poorly the smb.conf I copied ran on jessie (I had to start from scratch), I probably won't for a while (if it aint broke and stillg ets updates, don't fix it). They are samba and WINS servers, and my parents' server also serves a HP Laserjet 1012 through samba. Theirs was also setup to install security updates manually, while mine is manually updated. I have ssh setup to allow access via ssh in linux and PuTTY in windows. While I coudl have just USB'd a HDD to the router in both cases, even on DD-WRT (and other firmwares like that) I've had meh luck with that, especially with using it as a WINS server.

Here's where I'll get some bullshit: I do leave them permanently connected and I don't run ZFS. When I originally setup my parents' server, I didn't know about ZFS, and swapping it over now would be a PAIN. Likewise, my server's main storage is a 2TB WD green with a copypasta of what was on the other server. I was in a rush and just needed to get the files copied onto something that I could use for storage; Putting the drive in a server wasn't the original plan.

That being said I may look at ZFS when I work on the new server, as it will use 100% fresh hardware.

Speaking of hardware, my parents' server is a pentium 4 (775) based dell poweredge with a hacked-in 4 caddy SAS backplane and PERC 5/i card. It has 3 500GB HDDs in a RAID5 setup (and yes there is a BBU). Eventually it will get a 4th drive for a hotspare.

My current server is one of two I received from a buddy who had to dispose of two rack servers (with the condition that I wipe the drives, which I did 3 times at a low level). It's a Dell Poweredge R200 1U, with a 80GB OS drive and a 2TB storage drive.

My new server will be an IBM system X3650 (2U), which was the other server I was given. It has a 6 caddy SAS backplane, which I plan to populate with large capacity 7200RPM SATA drives (1TB+) in a RAID 10 setup, possibly with hotspares.

That's what I do for my file server.

[Hondaman]

Re: Obligatory backup server question

How am I backing up? I'm still figuring that out. I set up Samba on the NAS - maybe I'll set up FTP later. I was thinking about using Microsoft Backup (Win 7 Pro). First, copy everything from the USB enclosure drive, then set up MS Backup to back up E:\My Documents, E:\My Spreadsheets, E:\My MP3s, etc to the server each week (my files are never on the same partition as Windows, going back to when I dual-booted Windows 3.1 / Caldera Linux).

After a while, I'll boot my main machine to Linux instead of Win7 (I just upgraded from Mandriva 2010.2 to Mageia 5). Then the fun begins. Using awk, I can construct a command to back up only files that have a modification date within the last week, and only in certain directories - so only a few files will need to be copied to the server each backup. I can issue commands like "ls -og -R *" to get a recursive directory listing of every file and the file's size and pipe it to a text file. Get lists of files from different hard drives and use Notepad++ inside Windows to look for duplicates.

Cryptoware? Viruses? I live behind a router. I'm the only user on any machine in the house. I don't click on un-trusted links or icons. I have run Windows as admin and with no anti-virus for years. I even fixed a friend's rootkit Szrizbi infection some years back. But yes, I promise to keep practicing SAFE COMPUTING.

I see lots of people out there with fancy new servers that would work better than what I have. I want one, but I don't want to spend hundreds of dollars. For now, I'll stick with 2 drives maximum, RAID-1 (mirror), 2 TB BIOS limitation (the IDE chip may have died during this installation). My unit probably consumes more power than others, there is no hard drive spindown, and my tower case is ordinary looking. But I have put a lot of effort into computers over the years, and this time, my efforts are paying off (it actually works). So I'm happy.

[stj]

Re: Obligatory backup server question

your taking a big risk with that windows networking,
crypto-crap hooks into dynamic add's on legit websites.

btw, i dont know about Mageia5, but next laptop i setup, i'll try couldron - i need cutting-edge versions of some stuff.

[mariushm]

Re: Obligatory backup server question

My first advice would be to invest some money in a quality hard drive.

Buy a new NAS or Enterprise hard drive, put it in a machine and leave it running 24/7. After about 3 months, you can start using it and trust that the chances of the drive failing are minimal.
My current preference is HGST Deskstar NAS

You mentioned a 320 GB drive, which I guess would be rather old - studies like the Google one show that a drive is more susceptible to die in the first few months or after about 3 years of continuous operation :



If you're doing only weekly backups or monthly backups, I'd say don't go with any kind of RAID solution, just go with two independent hard drives. Copy contents from first drive to second drive periodically.

For incremental backups, I've found it's just easier to compress data using WinRAR configured with about 10% error recovery data and 1 GB volumes. It also has good encryption, which makes you worry less if you want for example to upload your backups to a remote storage like a shared hosting account or something like Amazon Glacier .

When I want to perform an incremental backup I just create a copy of the old archive (to make sure it won't be changed) and then choose to compress the folder with the fresh files and choose "add to existing archive" - Winrar will parse the old archive to rebuild the dictionary and compression data it used for the original archive and it will basically compress just the changes in the new files.

This can be done just the same with 7zip (open source and works on Linux too) but in my experience I found it lacking in recovery of files after errors.

For example if you have a scratched disc with a 7zip archive and you try to decompress the archive but a few bits are screwed up or a disc sector couldn't be read by the drive, you can lose the contents of the whole archive because 7zip by default uses "solid mode" compression, where it treats all files as one huge file... basically you can't extract the 2nd file in an archive without 7zip having to decompress in memory the first file... winrar let's you choose to enable or disable this "solid mode" compression.

The error correction data added to rar archives is on par with PAR2 correction data (excuse the pun) and it actually saved my ass a couple of times.. if you're lucky winrar can recover/correct a few bits and by default, it's also configured to allow you to extract individual files from archives without having to extract the previous content.

Saving to optical media is why I said 1GB volumes.. a DVD+R single layer holds 4.37 GB and a DVD-R holds 4.38 GB so you'd have 4 x 1 GB volumes with about 300 MB to spare which if you're extra paranoid could be filled by 5% PAR2 recovery data besides the recovery data you've built inside the actual archive volumes.
I've thought about using double layer DVDs for archiving stuff, but then I'd have to use 900 MB volumes because DVD+R DL is about 7.96 GB in size, and also it's not worth the extra cost - here, a double layer disc costs about as much as 3 single layer discs, it's even cheaper if I go and buy 25-50pcs packs of single layer discs.

So yeah, backup as often as you want, you can leave the server create the archive in the background and have it done the next day... once a month you could burn a few discs with the latest archive (set of volumes) and optionally also connect a usb drive and copy the archives once a month
It would make sense to store the discs in another location if you're worried about theft or burning or water damage.

If you're really paranoid, you could also upload the archives to cloud or a shared hosting account or a service like Amazon Glacier (you pay $0.007 per GB / month and it costs nothing to upload or download, pay just for storage space for most regions). This would work since your archive volumes would be encrypted - rar encryption in new versions is very hard to break.

And PS.

And seriously, if you're serious about your data, just spend a few dollars and build a decent server, instead of using iffy parts - when I read something about 2 tb limitation and faulty chips and stuff about ide it just makes me sad and angry.
A MSI AM1I board is 35$ people ... 30$ after rebate card: http://www.newegg.com/Product/Produc...59&ignorebbr=1 - all solid polymer, 2 SATA 6 gbps ports, lots of usb headers on the board,
Add a $12 2 GB DDR3 stick to it and just slap a $30 Sempron 2650 cpu on it (with built in video card) , with a tdp of 25w the included heatsink and fan will be more than adequate. You can probably find AM1 processors cheaper on eBay or Amazon
There you go, about 80$ for a solid system which will use less than 50 watts of electricity.

You could even splurge for a $50 Asrock AM1H-ITX that has 4 SATA ports and also has a USB 3 header on motherboard (useful for front panel usb 3.0) and (will allow you to reuse a laptop adapter power supply ( standard 19v minimum 41w with standard 2.5/5.5mm connector ) so you can make your system even more silent by not having a noisy ATX power supply and you save money by not buying an atx power supply if you already have a laptop charger around. Of course, it also has the old atx connector so you have choices.

[Hondaman]

Re: Obligatory backup server question

Thank you.

1. One IDE burner and one IDE hard drive died (but there is still the 2 TB limitation on my current hardware). I have two 320 GB SATA drives from 2008 (low hours) and two 320 GB SATA drives with zero hours (New Old Stock). I misplaced one of those zero hour drives though. New, small drives are good for testing operating systems and hardware, and I can still buy them.

2. Will NAS4Free (FreeBSD-based) be able to handle the newer chips on those mini-ITX boards? Ubuntu Server would work with more hardware, and it could do a Raid-1 setup. Then I could install Ubuntu Desktop and Samba. I could install Kodi on this new machine and have a dual-purpose computer - file server and playing DVD rips. (I'm not sure why I want RAID, but for some reason I do want it.)

I got Kodibuntu to work on my old i3-3245 computer, and it only uses 120 volts, 30 watts. If I install Kodi on my new server, then my i3-3245 machine could go back to its old purpose, recording from my HD Homerun Dual.)

My 320 GB drives will let me prove the software works. Then I will get some better, newer drives and hardware. I have heard that HGST drives are very solid performers. When I buy the "big drives", that is the brand I will buy.

[ChaosLegionnaire]

Re: Obligatory backup server question

yupp, the best 2tb legacy format (512 byte sector) hitachi drive u can get is the 7k3000 2tb hds723020bla642 drive. take a look at the sustained transfer rate benchmark from mine. faster than even the wd caviar black 2tb or re4 2tb.

[Topcat]

Re: Obligatory backup server question

Well, I'll throw my 2 cents in as far as local backup.... I keep it simple and not worry about it. The master file/media server is running S2K8, HTPC's feed off it through mapped network drives with read-only access. Its a very K*I*S*S method, but rock solid....simple enough that my non-tech-savvy wife can use it without issue. The HTPC's have no connection to the outside world, not that it would be a problem if they did...I have them configured correctly. I manually back thing up any time theres changes (additions, deletions, modifications) to a specific group of files (music, movies, programs, utilities, etc). To back things up, I connect a removable SAS/SATA HDD, copy the updated files/directories over, and then remove the drive. I then place the drive in a static shielded bag and lock it in a fireproof safe. Easy peasy. It seems so many people make things a lot harder than they need to be...