well I lost everything on my server

[Curious.George]

Re: well I lost everything on my server

My only, and ONLY, point is that it is usually not that difficult to transplant a software RAID, if you happen to set one up. Help me out here folks, am I so unclear?

As a friend who operates a large data center for a multinational bank is so fond of pointing out:

"There are two types of RAID (or any similar technology) users: those who HAVE experienced data loss -- and those who WILL"

Hint: the ONLY time I came close to losing data was when an OS upgrade introduced a bug into the SCSI disk driver that caused the disks attached to the controller to be scrambled. So much for "free" software...

[sofTest]

Re: well I lost everything on my server

[Snipp] Bare drives and shelfs are the only approach that work reliably for really large data stores managed without an IT budget or staff.]

Until your building burns down, hit by a meteor, or similar. Or your entire city get destroyed by some other freak of nature. You are talking about far fetched scenarios, don't you?

Your don't get it do you? I don't care about your backup strategy, or whatever you call it.

I don't even talk about any backup strategy, not even mine. I'm not talking about long time offline data storage either. A NAS alone will never be a backup strategy, but might be part of a backup strategy.

Not important for this discussion, but for your info. The order of the disks are not important when moving ZFS. You can always import a pool, whatever order of the disks. ZFS uses vdevs and pools, not arrays. You really do need to update.

My only, and ONLY, point is that it is usually not that difficult to transplant a software RAID, if you happen to set one up. Help me out here folks, am I so unclear?

[Curious.George]

Re: well I lost everything on my server

You haven't researched this have you? If I wanted to, I could put my drives in USB docks, use simple USB-SATA cables or USB-enclosures (mixed any way I want) and hook them up to a laptop, to use your "strategy". I could even have some on SATA on a MB and some on USB. If I wanted to, I could have run my NAS entirely on external USB-drives. With todays SS-USB, it probably wouldn't be much of a speed penalty either.

Now you've got a bunch of docks/enclosures to keep on a shelf -- because you need to be able to mount all drives in the array at a given time -- just to give you a compatible hardware interface. Then, have to ensure the drives are probed in the correct order (so the RAID software doesn't think you've mixed them up). And, you're just dealing with 5 drives!

Now, manage ~100T of data like that -- which set of drives go together? How were they configured? What's on them? Which physical case holds the first drive in the array? The second?? etc.

By contrast, I can take a drive to a neighbor, the local library or any other computer and read its contents -- without having to boot from a thumb drive/CD/DVD and hope the hardware in that computer is "compatible" with that software.

If your juggling and keeping track of disks, WORM's, tapes or whatever suits you, fine. I certainly have no problem with it. My argument about it's not difficult moving a software RAID, is still valid.

Yet people always seem to have problems when their feet are in the fire.

Throw one of your drives away. Today (cuz you presumably are READY for a catastrophic failure TODAY!) Wait a week so you know it's in a landfill, somewhere. Now, try to recover (if you're confident that you CAN do this, then this shouldn't be an issue for you and you shouldn't even break into a sweat over it!)

[I invited a colleague to "borrow" one of my drives on a recent visit to prove that I could recreate it's contents (though not its sector-wise image) while he had it in his possession. If I couldn't, the drive (and its contents) would be his to keep!]

Repeat the exercise with a new version of the RAID software (or, try FreeNAS instead). I.e., imagine having built an array -- perhaps a BACKUP of your array -- today. Placed it on the shelf. Then, in 2020, dragging it off the shelf (because your main array was vandalized or stolen) and try to resurrect it. Do you remember what version of the software you used? Are you sure the 2020 version's changes are 100% compatible?

"FOSS" doesn't mean "bug free" any more than COTS means "buggy"!

With an archive as large as mine, 98% of it sits dormant on a shelf for very long periods of time. When I start a new project, I build a "guest disk" with whatever I need, culled from the archive. I have media that were written almost a decade ago by various different OS's. All I care about is the format of the BASIC filesystem(s) used, not additional structures layered atop that (by the RAID software). The 720K 3" floppies (FAT12) for my Unisite are handled just as easily as the 4TB (NTFS) SATA disks and the 146GB SCA (FFS) drives.

[I've owned -- and retired -- perhaps a dozen RAID/NAS appliances over the years. Each with its own set of headaches as each vendor/model chose to implement things slightly differently from model to model or vendor to vendor. Bare drives and shelfs are the only approach that work reliably for really large data stores managed without an IT budget or staff.]

[sofTest]

Re: well I lost everything on my server

Now, your HP Microserver dies -- power supply catches fire.

You pull your 5 drives. And, discover that the only other box you have on hand AT THIS MOMENT will only support 4 drives. Or, 2. Ooops!

To truly protect your data investment, you need a spare HP Microserver lying around, idle. (or, the ability to MAKE it idle while you access your files).

Everyone should have a strategy on how to get access to their data according to urgency. The most important thing is to make sure that important data is likely retrievable and intact at all, even if it takes a day or a week.

If your array uses an advanced RAID controller (so-called "hardware RAID"), then you also need to have one of those controllers on hand (and whatever box it runs inside).

If you buy a RAID appliance, then you're faced with having to sort out how that vendor chose to implement the technology in that device. (you're not going to pull the drives from a Synology box (that has died) and throw them into a QNAP box.

Yes, that's why I went for a software RAID, and explicitly talked about software RAID...

If you only have one array, keeping a spare isn't that hard (though it represents a big cost penalty unless you can put it into use -- and live with NO spare :> ). Once you end up with many arrays, you either have to tie yourself down to one make/model/implementation or deal with having lots of different spares on hand (as you can't know which will crap out).

[I can have upwards of 70 disks spinning at any given time. And, I can "recover" from ALL of that hosting hardware failing, simultaneously, by dragging out a laptop and an external, single-drive dock. No magic OS to install cuz the volumes aren't "special" in any way -- I can even read them under Windows]

You haven't researched this have you? If I wanted to, I could put my drives in USB docks, use simple USB-SATA cables or USB-enclosures (mixed any way I want) and hook them up to a laptop, to use your "strategy". I could even have some on SATA on a MB and some on USB. If I wanted to, I could have run my NAS entirely on external USB-drives. With todays SS-USB, it probably wouldn't be much of a speed penalty either.

If your juggling and keeping track of disks, WORM's, tapes or whatever suits you, fine. I certainly have no problem with it. My argument about it's not difficult moving a software RAID, is still valid.

[Curious.George]

Re: well I lost everything on my server

Actually not difficult if we are talking about HD's and software RAID. My NAS is a HP Microserver with 5 WD Red's running software ZFS-raid (RAIDZ) under Nas4free.

Now, your HP Microserver dies -- power supply catches fire.

You pull your 5 drives. And, discover that the only other box you have on hand AT THIS MOMENT will only support 4 drives. Or, 2. Ooops!

To truly protect your data investment, you need a spare HP Microserver lying around, idle. (or, the ability to MAKE it idle while you access your files).

If your array uses an advanced RAID controller (so-called "hardware RAID"), then you also need to have one of those controllers on hand (and whatever box it runs inside).

If you buy a RAID appliance, then you're faced with having to sort out how that vendor chose to implement the technology in that device. (you're not going to pull the drives from a Synology box (that has died) and throw them into a QNAP box.

If you only have one array, keeping a spare isn't that hard (though it represents a big cost penalty unless you can put it into use -- and live with NO spare :> ). Once you end up with many arrays, you either have to tie yourself down to one make/model/implementation or deal with having lots of different spares on hand (as you can't know which will crap out).

[I can have upwards of 70 disks spinning at any given time. And, I can "recover" from ALL of that hosting hardware failing, simultaneously, by dragging out a laptop and an external, single-drive dock. No magic OS to install cuz the volumes aren't "special" in any way -- I can even read them under Windows]

[sofTest]

Re: well I lost everything on my server

But you then need ZFS to access the medium. What do you do if the box(es) that support it are down?

The same problem applies to the various RAID technologies. When your RAID hardware (or system) dies, how do you access (or recover) the contents of those volumes?

Actually not difficult if we are talking about HD's and software RAID. My NAS is a HP Microserver with 5 WD Red's running software ZFS-raid (RAIDZ) under Nas4free. The whole Nas4free is installed on a bootable USB-pen. If I need to move my NAS to new hardware, I pull the drives and the USB-pen, connect it up to new hardware and boot. Should the USB-pen be bad, I keep a copy of the config file elsewhere.

Due to space restrictions and drive sizes at the time, I elected to have only RAID-Z1 to achieve my storage needs. RAID-Z2 is possible, but not recommended on a 5 disk setup. It's possible to use up to RAID-Z3 with ZFS, were up to 3 drives can go bad, and still save the pool.

[Curious.George]

Re: well I lost everything on my server

I fear that a collision in the future, can result in failure to detect corruption.

Even though it's much better than CRC, with files. I saw MD5 do a GJ with optical drive mis-reads, IIRC.

Realistically, the sorts of errors that will manifest will be:

• file not found
• unrecoverable read errors
• drive failure

Keep in mind that, unlike ZFS, RAID, etc. my scheme tolerates the volumes being accessed "unsupervised". E.g., I can take an external USB drive and manually change "something" -- or someTHINGS -- without the system ever seeing me make those changes. So, I can delete a file -- or rename it, or move it, etc. -- and the system will not see me doing those things (so that it can update its notion of the file's new name, location, etc.).

I can likewise make changes to the file's content while "out of sight" and it won't know to update the checksum (signature) stored in the database to reflect those changes.

Removable media (esp CD/DVD) can fail while offline and throw UREs. Again, something that doesn't happen with RAID/ZFS/etc. (volumes are never really "offline" while the rest of those system is running).

And, of course, a drive can always have a catastrophic failure (fail to spin up).

Note that most archive formats (ZIP, ARC, RAR, etc.) rely on simple checksums to vouch for the integrity of their contents. How often have you encountered one that fails to self-verify after it had previously done so?

Compute the MD5 of this message. Then, alter it in such a way that its length and MD5 remain unchanged. Then, try to convince me that your alterations are representative of a likely hardware/media failure!

[RJARRRPCGP]

Re: well I lost everything on my server

MD5 is obsolete due to its vulnerability to HACKING/cracking. It is still robust enough to produce a unique signature of any nontrivial file contents without concern for collisions. I.e., ask yourself what type of data corruption would cause a file's contents to change in such a way that there would be an undetectable collision in the MD5 wrt the "correct" contents.

All I use the signature for is to verify that the contents of the file appear to be unaltered -- WITHOUT having to do a bytewise compare to another copy of the file (which may not be "online" at the moment).

MD5 is, on average, faster to compute than any of the SHA variants when the host platform -- as well as file size -- is variable. My goal, of course, is to process as many files as quickly as I can so the user doesn't have to "wait" while the system runs around checking things.

(You want to be able to mount a volume to access something of interest to YOU, not to cater to the system's need to check files. The system, OTOH, wants to exploit every opportunity it has to access the files on that volume so it can vouch for their integrity, NOW.)

I fear that a collision in the future, can result in failure to detect corruption.

Even though it's much better than CRC, with files. I saw MD5 do a GJ with optical drive mis-reads, IIRC.

[Curious.George]

Re: well I lost everything on my server

MD5 is obsolete, FFS! For example, for ISOs, SHA is regularly used now.

MD5 is obsolete due to its vulnerability to HACKING/cracking. It is still robust enough to produce a unique signature of any nontrivial file contents without concern for collisions. I.e., ask yourself what type of data corruption would cause a file's contents to change in such a way that there would be an undetectable collision in the MD5 wrt the "correct" contents.

All I use the signature for is to verify that the contents of the file appear to be unaltered -- WITHOUT having to do a bytewise compare to another copy of the file (which may not be "online" at the moment).

MD5 is, on average, faster to compute than any of the SHA variants when the host platform -- as well as file size -- is variable. My goal, of course, is to process as many files as quickly as I can so the user doesn't have to "wait" while the system runs around checking things.

(You want to be able to mount a volume to access something of interest to YOU, not to cater to the system's need to check files. The system, OTOH, wants to exploit every opportunity it has to access the files on that volume so it can vouch for their integrity, NOW.)

[RJARRRPCGP]

Re: well I lost everything on my server

But you then need ZFS to access the medium. What do you do if the box(es) that support it are down? How do you implement it on already written WORM media? etc.

The same problem applies to the various RAID technologies. When your RAID hardware (or system) dies, how do you access (or recover) the contents of those volumes?

I perform the checksums in-band and deliberately store them ON ANOTHER MACHINE (which can be replicated). There's nothing magical about the volumes that I'm checking -- no reliance on particular hardware (can you pull a drive from a Synology RAID array and install it in a "software RAID" box and expect to access its contents?) There's nothing magical about the filesystems being used -- I can check FAT12 floppies just as easily as EXTFS2 or NTFS or...

And, I can mount a volume on any machine (with compatible hardware -- SCSI drives obviously need a SCSI HBA for access) and still gain access to the data.

The cost of keeping a spare HBA or SCSI enclosure is trivial compared to keeping a spare RAID *box* (that claims to be compatible with the other boxes you might have).

[You learn these lessons when you discover the hardware to access various types of media that you've used over the decades are suddenly not obtainable. Or, the support for them (OS drivers) has disappeared. Do you scurry to move all of that data forward onto new media? (how do you know i is intact when you do so?) Or, do you try to maintain legacy hardware to make it accessible in its original form? (What will you do when you can't buy CD/DVD drives anymore?)]

And, because I have the checksums (MD5s) for all of these files available, I can find likely duplicates just by querying the database: two files (which might have different names and reside in different folder -- on different volumes OR ON THE SAME VOLUME) that share a checksum value are likely the same -- or, I can be prompted to make both available to the system so it can make that determination (and record it!).

This has already been helpful in identifying duplicate copies of files that I did not care to maintain (e.g., "805-1709-12.pdf" and "Sun Ultra 60 Service Manual.pdf" are identical documents differing only in the name that I assigned to them and the folders I stuffed them into!)

MD5 is obsolete, FFS! For example, for ISOs, SHA is regularly used now.

[Curious.George]

Re: well I lost everything on my server

this is a good reason to try to use ZFS for backups.
everything gets checksumed.

But you then need ZFS to access the medium. What do you do if the box(es) that support it are down? How do you implement it on already written WORM media? etc.

The same problem applies to the various RAID technologies. When your RAID hardware (or system) dies, how do you access (or recover) the contents of those volumes?

I perform the checksums in-band and deliberately store them ON ANOTHER MACHINE (which can be replicated). There's nothing magical about the volumes that I'm checking -- no reliance on particular hardware (can you pull a drive from a Synology RAID array and install it in a "software RAID" box and expect to access its contents?) There's nothing magical about the filesystems being used -- I can check FAT12 floppies just as easily as EXTFS2 or NTFS or...

And, I can mount a volume on any machine (with compatible hardware -- SCSI drives obviously need a SCSI HBA for access) and still gain access to the data.

The cost of keeping a spare HBA or SCSI enclosure is trivial compared to keeping a spare RAID *box* (that claims to be compatible with the other boxes you might have).

[You learn these lessons when you discover the hardware to access various types of media that you've used over the decades are suddenly not obtainable. Or, the support for them (OS drivers) has disappeared. Do you scurry to move all of that data forward onto new media? (how do you know i is intact when you do so?) Or, do you try to maintain legacy hardware to make it accessible in its original form? (What will you do when you can't buy CD/DVD drives anymore?)]

And, because I have the checksums (MD5s) for all of these files available, I can find likely duplicates just by querying the database: two files (which might have different names and reside in different folder -- on different volumes OR ON THE SAME VOLUME) that share a checksum value are likely the same -- or, I can be prompted to make both available to the system so it can make that determination (and record it!).

This has already been helpful in identifying duplicate copies of files that I did not care to maintain (e.g., "805-1709-12.pdf" and "Sun Ultra 60 Service Manual.pdf" are identical documents differing only in the name that I assigned to them and the folders I stuffed them into!)

[Topcat]

Re: well I lost everything on my server

I keep 2 on 2 different locations just because I hate loosing things.

As do I. I keep 2 external drives in my safe on the property, and I've got another in a safe deposit box. RAID's are nice, but that's a common misconception that so many make....RAID's provide redundancy, not backup!

[Curious.George]

Re: well I lost everything on my server

A backup isn't a backup until it's been verified.

A backup can be successfully verified and still found to be defective when it is eventually needed! E.g., both of my "cold backups" were actually intact -- it was the OS that had been compromised which rendered them inaccessible at a much LATER date.

My "system" tracks the last time it "examined" each volume in the database. When a volume is encountered, it determines which files on that volume have not been re-verified in a particular number of days and starts a task running to read the file in its entirety and verify the checksum computed matches the checksum stored for that file in that folder on that volume in the database. If so, it records the timestamp for that "verification" and then moves on to process the next such file.

If the database indicates files that haven't been checked in some "verification interval", it emails me to mount those volumes so they can be examined.

This happens regardless of where (which network node) the devices are mounted and regardless of the media involved. E.g., my CDs, DVDs, MOs, thumb drives, drives in sleds, external USB drives, etc. are all managed with the same mechanism (though I can schedule different "verification intervals" for each volume to manage the amount of manual intervention that are required of me). Do you know that the pile of optical media you've written over the years are still intact? If you don't care, then why hold onto them??

Because of this, any time I happen to mount a volume for "some other reason", the contents of the volume can be checked "for free".

When a discrepancy is encountered (file can't be read, file not found, checksum mismatch), the database tells me where I can find a "copy" of that particular file so that the defective copy can be repaired.

Does your RAID array tell you if ALL the files you are NOT accessing, now, are intact? Do you have to verify its entire contents in order to reassure yourself that it is intact? Are ALL of the files on that medium equally important to you? Do you really want to verify the ISO images of the installation CDs -- which you happen to have squirreled away in a desk drawer -- just because they happen to reside on THAT array? Or, would you be equally confident verifying them every month or three -- KNOWING that the masters also exist on non-rust?

[petehall347]

Re: well I lost everything on my server

not sure if running nautilus will help here but thought i would mention it anyway ..

[Uranium-235]

Re: well I lost everything on my server

Well this linux raid is being a pain. I created new partitions on the disk to join, and even though it's the same number of blocks, says it's too small. Ugh. I have s perc 5 card I managed to get out of an old customers workstation cause he had a crashed 5 array and I convinced him to use intel on board raid 1 (that and the card started to have an odd pcie conflict with another device)

Not sure the impi will enumerate it for control on a server this old though

[stj]

Re: well I lost everything on my server

this is a good reason to try to use ZFS for backups.
everything gets checksumed.

[ratdude747]

Re: well I lost everything on my server

This is why I setup my server to back up it's RAID10 array to a separate drive every week and then to clear out old backups so it doesn't overflow. It's still in-server, but it's on a separate drive controller, so I consider that to be good enough.

[diif]

Re: well I lost everything on my server

A backup isn't a backup until it's been verified.

[Curious.George]

Re: well I lost everything on my server

I keep 2 on 2 different locations just because I hate loosing things.

Yup, though even that may not be sufficient!

I keep "cold" backups of everything -- two on rust and at least one on another medium (tape, MO, CD/DVD, etc.).

Some years ago, I had a "disk crash" (or, so i thought!). So, I shrugged and pulled the (first!) cold backup out and mounted it in an external case (SCSI-based system). This disk proved unreadable (WTF?). Maybe bad luck or a problem with the enclosure/cabling?

Set it aside and pulled (SECOND!) backup out and repeated the exercise. And, should NOT have been surprised to see the exact same results!

Now we KNOW something is seriously hosed!!

So, I pulled out the MO backup, cabled an MO drive to the system and restored the drives from the MO (largely read-only) backup.

Turns out the OS I had upgraded, previously, had a bug in the quirks table for the drives that I happened to be using as my cold backups. Mounting any of them (without manually installing the "read-only" jumper on the drive itself) would result in the superblock being trashed.

So, roll back to an earlier OS release, copy the MO image onto both cold backups (which, not surprisingly, actually DO work!) and only lose a day of my time (and a few years off my life from the stress).

Now, I keep multiple copies of "stuff" and in varied places. I have a database that tells me which files (and their MD5's) are located on which media so if I lose any copy of a file, I can quickly locate a backup copy of it, regardless of where it may be stored.

[Uranium-235]

Re: well I lost everything on my server

Yeah, you need to watch that. Running it from the wrong path, or with unset variables can cause chaos. Take the Steam Client for an example: https://www.theregister.co.uk/2015/0...ans_linux_pcs/ I got bit by that bug. Took out the drives being backed up and the drives that were taking the backup... Both my "production" data and backup data gone like that.

seems something you could sue valve for