"Solid State" media failures

[tom66]

Originally Posted by Curious.George

So, in case of a disaster on your end, and you want to retrieve your backup, you do so through the same 20Mb pipe, over the course of another few months? Hoping, all the while, that the company maintaining it hasn't changed their terms of service (or gone belly up or been hacked offline)?

Well, I have 80Mbit downstream, but I don't even have to use that.

The company will send you a 4TB HDD for ~$200 but if you send that back to them, the service is free of charge - you effectively just pay the postage to send it back and they give you a full refund, or you can decide to keep the drive for the $200 deposit. So maybe $20 to get the data back in a few days.

Other companies have similar policies, but Backblaze seemed to be one of the most attractive in terms of customer service and software. 100TB is a huge amount of data, I am no where near that level! FWIW they are unlimited in terms of backup (I am guessing that MOST of their customers don't use 100TB of space) but obviously if you had to recover the full 100TB that would take some time and a lot of drives.

[Curious.George]

Originally Posted by tom66

The company will send you a 4TB HDD for ~$200 but if you send that back to them, the service is free of charge - you effectively just pay the postage to send it back and they give you a full refund, or you can decide to keep the drive for the $200 deposit. So maybe $20 to get the data back in a few days.

That makes sense. But, you'd have to prepare a list of what objects you needed to recover -- I assume you create a "virtual drive" and then tell them to reify it?

100TB is a huge amount of data, I am no where near that level! FWIW they are unlimited in terms of backup (I am guessing that MOST of their customers don't use 100TB of space) but obviously if you had to recover the full 100TB that would take some time and a lot of drives.

It's relatively easy to end up with a huge archive -- if you assume storage is cheap/free!

For example, I store images of all of my COTS optical media IN the magnetic portion of the archive (giving me an optical copy and a magnetic copy). The magnetic copy is much more convenient to access/use -- no need to physically sort through stacks of optical media for the disc in question!

Likewise, I image every computer that I build, several times during the build process, so I can restore a computer to one of those states in a matter of minutes (instead of having to reinstall the OS, drivers, updates, apps and configure same). And, build VM's of each "work environment" so I can return to one of those at a future date.

And, because "storage is free", I think nothing of adding an object to the archive -- without bothering to check if its already present, in some form, there.

But, one of the best consequences of having a big archive is that it forces you to come up with ways of tracking its contents. And, lets you find duplicates of objects that you may not have realized you had!

("Geee, foo.pdf is the same object as 1986Taxes.pdf!")

[tom66]

Originally Posted by Curious.George

That makes sense. But, you'd have to prepare a list of what objects you needed to recover -- I assume you create a "virtual drive" and then tell them to reify it?

Yes, that would be how it works.

You having 100 TB would certainly make that harder! I have at most 8 TB of data, about 2 TB of which is backed up (the rest is easily recoverable stuff, or junk like Downloads folder.)

[Curious.George]

Originally Posted by tom66

Yes, that would be how it works.

You having 100 TB would certainly make that harder! I have at most 8 TB of data, about 2 TB of which is backed up (the rest is easily recoverable stuff, or junk like Downloads folder.)

I have 2T just in "music"! :< Like I said, when you treat storage as free, it's amazing how quickly you can consume it (e.g., instead of transcoding, on-the-fly, that music for the different file formats that I require, I just do the conversions once and store the results).

It would be "harder" if I lost a significant portion of it. But, as mentioned up-thread, there's a point where I'd just throw up my hands and say "F*ck it!".

Currently, I am provided with a list of "object instances" that need to be "restored" accompanied by a volume identifier (disk number) that will allow those objects to be recreated from other instances that have the same contents (but may have different names).

E.g., If <23>/foo/bar/taxes.2018 is corrupt (can't be read, struggled to be read, fails its signature check), there may be instances of the same content in <4>/la/dee/dah/2018 and <19>/woop/de/do/IRS. Mounting either volume <4> or <19> (coincident with <23>) will cause taxes.2018 to be "refreshed" with the correct contents (and its original "name" restored).

All potential "sources" for a refresh image are logged and a job runs through and tries to minimize the number of different volume combinations that must be mounted to cause the desired updates to occur -- because I often am the physical means of bringing about those mounts!

So, if <19> will be needed for some other purpose, it will be suggested as the source for the "taxes.2018" image -- instead of <4>.

[The process isn't as verbose as my description suggests. I'm just emailed a report of which volumes I should mount along with a reason (some volumes may just be due for a patrol scan). Figuring out what objects to copy is handled automatically and the repairs noted.]

In practice, the messages are all prompts for me to mount volumes that haven't been scanned in their prescribed intervals. E.g., a "store bought" optical medium needs to be "verified" very infrequently -- while a writeable optical disc needs more frequent reassurances. Both are tedious to verify (due to the effort required to load them into a drive and their relatively low storage capacity) so should see less mounts than spinning rust might.

But, whenever any medium is mounted, the "jobs" that have been queued for it are automatically executed -- whether they be verification tasks or "refresh" tasks. Given that a medium tends to be mounted for some amount of time longer than the barest minimum required to read the data (or write) that you want to copy from (or add to) the archive, that extra time where the drive would otherwise be idle can be used to process any outstanding jobs -- or, get ahead on verification tasks for object instances on the medium that may not yet be "due" for verification.

Trying to keep such a local copy of an archive (or "Backup") in sync with a copy stored on a remote server would be problematic -- you'd have to summarize the remote data (hashes) and, thereafter, assume the data is never corrupted and only pay the price for retrieving it when you determine that a local instance has been corrupted.

[ChaosLegionnaire]

lol i think curious george doesnt need anyone or any stinkin' web based cloud management system to store his stuff for him. he can more than handle everything himself. i dont think there is a cloud based storage solution out there that is as good as or if not, better than his current implementation.

what he should do though, is to sell his locally based data storage solution and archival system as some kind of a software kit. im sure such a product differentiation would stand out in the market and offer a breath of fresh air compared to all the web based cloud storage solutions out there.

so its basically, locally based data storage vs cloud based data storage. so if u dont trust someone else to store your private data for you, just buy the locally based data storage solution software. im sure u can trust yourself not to be pebkac when setting it up, right? hehe...

[Curious.George]

Originally Posted by ChaosLegionnaire

what he should do though, is to sell his locally based data storage solution and archival system as some kind of a software kit. im sure such a product differentiation would stand out in the market and offer a breath of fresh air compared to all the web based cloud storage solutions out there.

The advantage of "local" (besides bandwidth) is convenience of access. For any remote service, you have to leave a (one or more) door open to the outside world. And, it sure seems like EVERY major IT organization gets compromised, sooner or later. What makes you think your defenses will be any better? Why leave an opening for adversaries into your organization (any machine that may want access to YOUR remote data needs a way "out" to it)?

so its basically, locally based data storage vs cloud based data storage. so if u dont trust someone else to store your private data for you, just buy the locally based data storage solution software. im sure u can trust yourself not to be pebkac when setting it up, right? hehe...

"Setting it up" is a lead-pipe cinch! The problem lies in day-to-day utilization, in its current form. It relies on self-discipline to keep everything working. All the system does is tell you when you've screwed up -- possibly irrecoverably! The tools are all layered on the filesystem instead of integrated into it. So, I can conceivably delete every copy of a file and only find out about it LATER, when I'm "told" that those files are no longer accessible and need to be restored (but can't because their backups have also disappeared).

OTOH, you WANT to be able to delete every instance of a file.

A RAID/ZFS configuration sits between the user and the store so it has some say over what happens -- and, a means of providing "real-time" reporting to the user as he acts on the store. As well as taking remedial actions as soon as needed instead of "when you get around to it". The user has no direct responsibility in its performance.

But, that requires all of the media to be mounted and spinning at all times. I can't afford the thermal load that would represent nor the waste of electricity. And, I imagine it would adversely affect the reliability of the store, especially when it is accessed at such a low utilization rate (contrast with a regular file server that gets hammered on for daily use).

OTOH, some of the mechanisms I've put in place can be applied to filestores, in general. E.g., being able to locate duplicate files "for free".

[goontron]

Originally Posted by Topcat

Yup, dead in that machine too. Connector/cabling is fine.

Anyway you could ship that dead SSD out here? After my research on the Sandforce controller that crapped out on my Chronos SSD, ive learned a bit about SSDs and their failure modes, and this one is very odd.

[Curious.George]

Originally Posted by goontron

Anyway you could ship that dead SSD out here? After my research on the Sandforce controller that crapped out on my Chronos SSD, ive learned a bit about SSDs and their failure modes, and this one is very odd.

Any insights into recovering/accessing data in the over-provisioned portions of the drive?

[goontron]

Originally Posted by Curious.George

Any insights into recovering/accessing data in the over-provisioned portions of the drive?

I haven't been able to work out the mapping mechanism yet, no. Mainly because the controller encrypt the data. But, there is certain commands that can be sent to some controllers (Marvell and others) to rebuild said map (wiping all data in the process)

With the sandforce controller i do know the mapping can be worked out via way of a known file (EG, BOOTMGR) from the individual flash chips and the (weak) private key used by the controller for encryption.