well I lost everything on my server

[Uranium-235]

Re: well I lost everything on my server

the cap db is hosted on a 1und1 account (I say und because it's a german company :P)

I might get another TB drive in there to do monthly backups to. I've been reading on commands to spindown/spinup drives, so I might make a script to spinup (which I think can be done just by mounting), backup, mark spindown time as short, and unmount it

or get an Esata -> Sata bracket, and put the drive in an enclosure with an esata port and hotswap it, power off, so it's not exposed to possible power fluxuations, and I can keep the backup schedule on my time

[momaka]

Re: well I lost everything on my server

So back to the original topic in hand...

Uranium, does that mean the entire electrolytic capacitor data sheet database is gone too? :\
I hope not, and it would be a shame if it is. But I guess these things happen.

Good thing I kind of keep my own "database" of cap data sheets.

Similar to Curious.George's, I forgo the whole RAID/ZFS stuff and just keep copies onto random PCs, spare HDDs, Flash drivers, and DVDs/CDs (but much MUCH less elaborate and more manual/not automated compared to his, if it can even be compared ).

I'm actually pretty messy about my backups. Some files I have backed up in over 6 places (again, different PCs with different HDDs/interfaces and OSes, CDs, and flash drives), and some files I have only in one or two backup HDDs. It all depends how important the files are to me. Things like my pictures collection is well backed up: it exists on my main PC, a backup (portable) HDD, my laptop (which is also portable, sort of ), and another PC in my uncle's house in an entirely different continent. I also have partial backups of that collection in various PCs I've used over the years before that (partial, because at some point these PCs were my main PCs, and that means at some point they were the "original" copy/backup before I retired them).

So all in all, even a major wipe out will still likely leave me with *most* (if not all) of my data intact. Because of the messy way I've backed up things, of course, retrieving some files back will be a pain in the butt, if I happen to need them again. But nothing lasts forever, so whatever!

[sofTest]

Re: well I lost everything on my server

^Yup, incompetent.

(amazing how when forced to stand by your words, you run and hide... I guess even YOU don't believe what you're saying!)

Bye, bye, little person!

Forced to stand by my words? My only claim from the start has been that it is not usually not hard to move a software RAID. I have not solicited people, not even you, to use any kind of storage solution. You on the other hand has repeatedly and boringly droned on about your magnificent shelf stores, under ways making several incompetent claims about ZFS. Then you are trying to draw me out on "designing" a storage system, despite I clearly has stated before that your data, your problem, not interested. Then you want information about my system, despite I clearly said before, none of your business. You sir, are simply a repetitive boring troll, and I'm simply to old to play.

[Curious.George]

Re: well I lost everything on my server

Yupp, trolling.^

^Yup, incompetent.

(amazing how when forced to stand by your words, you run and hide... I guess even YOU don't believe what you're saying!)

Bye, bye, little person!

[sofTest]

Re: well I lost everything on my server

(sigh) Sad little person. You've obviously not considered the consequences of your decisions. Fear not, life will teach you these lessons! You'll remember this discussion and be even angrier at me when you realize the prophecy!

Ask yourself how you're going to increase your storage capacity: copy everything off the ZFS onto ???; tear down your vdevs, add another drive and then rebuild everything and copy everything back? Or, replace drives, one at a time in each vdev with larger drives, resilver and then then let the pool expand? Discard the old (now too small) drives? Add a second vdev (3 more drives) -- if the hardware can support it? Add a second box with a different configuration (components of which will be useless as spares for the first box; and vice versa).

You may not have 100T now. But, once you start getting used to having "unlimited storage" you'll discover that you use it, fast. With ZFS/RAID, all of the data is spinning at any given time. So, it's all AT RISK. A careless keystroke, a catastrophic hardware failure, a bug in the software, etc.

(So, just keep duplicates of EVERY box that you have as insurance! )

I add new drives just by plugging them in (unplugging something that isn't CURRENTLY being used). I keep 1, 2 or 30 copies of files as I see fit. They can reside on the same volumes or on N different ones -- in any combination. They can have different names and reside in different directories across volumes -- beyond even the capacity of symlinks to map equivalences!

I can mail a medium to a client and he can read/write it at his leisure. When it comes back to me, I can see what (if anything) has been changed. I can store any portion of my archive offsite on as many volumes as I see fit (including just one). A client can mail me a volume and I can import it to my archive "as is" -- so, when he refers to "/projects/2010/summer/proposals/blah" I'll be able to directly and unambiguously access it.

I can access it via Solaris, Windows (XP, Vista, 7, 8, 10 -- even 95/98 with USB support), Linux, MacOSX -- or even a thin client with just 100MB of RAM (and no display!). I can upgrade any of these OS's and never put more than one volume at risk (to prove the integrity of the OS upgrade).

I can browse my entire repository's namespace without mounting ANY volumes. I can create a new volume drawing files from any number of volumes. I can create volumes on R/O media. I can do this on a laptop without AC mains being available.

I can FREELY incorporate volumes of different types, technologies and interfaces (SCA/FC-AL/SCSI/SATA/PATA/floppy/memory card/etc.). I can export an NFS/SMB share and treat it as part of the archive -- verifying its integrity or automatically "reinitializing it" (mark it as corrupt) with the "backup originals" from the rest of the archive.

I don't need special tools to do any of them (other than some SQL scripts and the ability to mount volumes in whatever way the host OS desires). So, I don't have to worry about updates to those tools "breaking things" or putting the entire archive at risk.

You can do all of those things with ZFS, right?

In your professional capacity , address the "hypothetical" client with 100T of storage doubling every 3-4 years. How should he ensure he can access old tax records (in the UNLIKELY event of an audit), old sales histories (when proposing a new product), old development costs (when budgeting the development effort for that new product), etc. Be sure to tell him what sort of equipment he should expect to buy and what sort of staff (and labor costs) he should be prepared to encounter in order to maintain that data. Prepare a budget of the hardware maintenance costs for the foreseeable future (how long before those drives fail? How much money and time to replace?

I'll gladly present your recommendations to MY clients -- as EXPERT ADVICE. So, be sure to reflect all the costs/benefits in your proposal.

Or, are you a one-trick pony? Would you have advocated building an addition on the corporate office to hold more FILE CABINETS to keep ALL of the companies records "accessible" -- despite the age or value thereof -- instead of moving old stuff to "secondary storage"? Sure sounds like it!

To mimic your comments, based on how evasive you have been over real issues (you haven't even disclosed the size of your array, number of objects it contains, the hardware its running on, your costs to implement/maintain, etc.) and other wild claims, I'm sorry to say that I'm leaning more toward you being a simple ZFS fanboi rather than credible.

[or, having sorely limited experience "in the wild"... ever work for a company with 10 employees? 100? 1000? 100000? If you have, you'll know that the way problems are solved varies greatly, as do the resources available to throw at problems! Hint: I've worked at or for all of the above and had to come up with solutions that "made sense" to each of them]

I'll respond when you've proposed a solution that I'll gladly pass by real clients for their unbiased opinions. I'm sure the big companies with systems in place will be EAGER (not!) to adopt your solution. As will the companies with no IT departments.

Yupp, trolling.^

[Curious.George]

Re: well I lost everything on my server

Nope, one do not design storage systems based on information on need for storage capacity alone. But you wouldn't know that. But I'm not going to waste more time on this. Based on what you have proven not knowing above, and other wild claims, I'm sorry to say that I'm leaning more toward you trolling than credible.

(sigh) Sad little person. You've obviously not considered the consequences of your decisions. Fear not, life will teach you these lessons! You'll remember this discussion and be even angrier at me when you realize the prophecy!

Ask yourself how you're going to increase your storage capacity: copy everything off the ZFS onto ???; tear down your vdevs, add another drive and then rebuild everything and copy everything back? Or, replace drives, one at a time in each vdev with larger drives, resilver and then then let the pool expand? Discard the old (now too small) drives? Add a second vdev (3 more drives) -- if the hardware can support it? Add a second box with a different configuration (components of which will be useless as spares for the first box; and vice versa).

You may not have 100T now. But, once you start getting used to having "unlimited storage" you'll discover that you use it, fast. With ZFS/RAID, all of the data is spinning at any given time. So, it's all AT RISK. A careless keystroke, a catastrophic hardware failure, a bug in the software, etc.

(So, just keep duplicates of EVERY box that you have as insurance! )

I add new drives just by plugging them in (unplugging something that isn't CURRENTLY being used). I keep 1, 2 or 30 copies of files as I see fit. They can reside on the same volumes or on N different ones -- in any combination. They can have different names and reside in different directories across volumes -- beyond even the capacity of symlinks to map equivalences!

I can mail a medium to a client and he can read/write it at his leisure. When it comes back to me, I can see what (if anything) has been changed. I can store any portion of my archive offsite on as many volumes as I see fit (including just one). A client can mail me a volume and I can import it to my archive "as is" -- so, when he refers to "/projects/2010/summer/proposals/blah" I'll be able to directly and unambiguously access it.

I can access it via Solaris, Windows (XP, Vista, 7, 8, 10 -- even 95/98 with USB support), Linux, MacOSX -- or even a thin client with just 100MB of RAM (and no display!). I can upgrade any of these OS's and never put more than one volume at risk (to prove the integrity of the OS upgrade).

I can browse my entire repository's namespace without mounting ANY volumes. I can create a new volume drawing files from any number of volumes. I can create volumes on R/O media. I can do this on a laptop without AC mains being available.

I can FREELY incorporate volumes of different types, technologies and interfaces (SCA/FC-AL/SCSI/SATA/PATA/floppy/memory card/etc.). I can export an NFS/SMB share and treat it as part of the archive -- verifying its integrity or automatically "reinitializing it" (mark it as corrupt) with the "backup originals" from the rest of the archive.

I don't need special tools to do any of them (other than some SQL scripts and the ability to mount volumes in whatever way the host OS desires). So, I don't have to worry about updates to those tools "breaking things" or putting the entire archive at risk.

You can do all of those things with ZFS, right?

In your professional capacity , address the "hypothetical" client with 100T of storage doubling every 3-4 years. How should he ensure he can access old tax records (in the UNLIKELY event of an audit), old sales histories (when proposing a new product), old development costs (when budgeting the development effort for that new product), etc. Be sure to tell him what sort of equipment he should expect to buy and what sort of staff (and labor costs) he should be prepared to encounter in order to maintain that data. Prepare a budget of the hardware maintenance costs for the foreseeable future (how long before those drives fail? How much money and time to replace?

I'll gladly present your recommendations to MY clients -- as EXPERT ADVICE. So, be sure to reflect all the costs/benefits in your proposal.

Or, are you a one-trick pony? Would you have advocated building an addition on the corporate office to hold more FILE CABINETS to keep ALL of the companies records "accessible" -- despite the age or value thereof -- instead of moving old stuff to "secondary storage"? Sure sounds like it!

To mimic your comments, based on how evasive you have been over real issues (you haven't even disclosed the size of your array, number of objects it contains, the hardware its running on, your costs to implement/maintain, etc.) and other wild claims, I'm sorry to say that I'm leaning more toward you being a simple ZFS fanboi rather than credible.

[or, having sorely limited experience "in the wild"... ever work for a company with 10 employees? 100? 1000? 100000? If you have, you'll know that the way problems are solved varies greatly, as do the resources available to throw at problems! Hint: I've worked at or for all of the above and had to come up with solutions that "made sense" to each of them]

I'll respond when you've proposed a solution that I'll gladly pass by real clients for their unbiased opinions. I'm sure the big companies with systems in place will be EAGER (not!) to adopt your solution. As will the companies with no IT departments.

[sofTest]

Re: well I lost everything on my server

It's not intended for work in a professional setting where folks have staff dedicated to IT! It's intended for folks with HUGE archives (do you have at least 20T in your pool?) and who want not to spend time rebuilding arrays and wondering if the latest update will break something and render their data inaccessible.

I'll be delighted to hear how YOU solve this problem when your dataset approaches 100T. Heck, you can even fantasize and SUGGEST your solution, today! In your "professional capacity", that is... [snip]

Nope, one do not design storage systems based on information on need for storage capacity alone. But you wouldn't know that. But I'm not going to waste more time on this. Based on what you have proven not knowing above, and other wild claims, I'm sorry to say that I'm leaning more toward you trolling than credible.

[Curious.George]

Re: well I lost everything on my server

Jokes on you. It already has! The computer locked up and the engine went WOT until it blew. I hope you never worked on the computers for the '06 Prius.

EDIT: And before you say "throttle issue", it didn't go into neutral, the ODO was unresponsive, and i was able to pull the wiring loom off on the throttle. Still ran away.

Then you've seen the consequences of a NOT great rider! I've never had a design come back to me after it was released. And, as my contracts include a "bugs fixed free" clause -- with no limits on time since original release -- the client always has a strong incentive to throw any such problems back to me.

[Curious.George]

Re: well I lost everything on my server

Really? However many times you keep repeating yourself, I am not interested in how you organize your storage needs. Your data (hopefully), your time, your problem, not interested. I will however say as much that your scheme is not for everyone, and if I proposed it in any sort of professional capacity, I wouldn't keep that capacity long.

It's not intended for work in a professional setting where folks have staff dedicated to IT! It's intended for folks with HUGE archives (do you have at least 20T in your pool?) and who want not to spend time rebuilding arrays and wondering if the latest update will break something and render their data inaccessible.

I'll be delighted to hear how YOU solve this problem when your dataset approaches 100T. Heck, you can even fantasize and SUGGEST your solution, today! In your "professional capacity", that is...

I wouldn't keep my "professional capacity" long if I couldn't reproduce the entire development environment for any of the products that I've created years after their initial release. Does the latest version of Cadence recognize PCB layouts created from OrCAD 9? 7?? Can I modify the design of the FPGA that I designed under the original Xilinx tools with the latest Vivado suite? Will the latest version of gcc handle the extensions that the Whitesmith compiler supported?

If not, those products can't be maintained or evolved ("We'd like to do a version 2 of that product you designed for us several years ago"). Of course, if you've got a full-time staff on hand, they can keep upgrading your tools and porting old designs forward -- all reflected in your "overhead".

Perhaps you've never been in a position where your client is genuinely concerned about the consequences -- to HIS organization -- of your being hit by a bus. You're likely pretty easily replaced. My files/designs/data aren't. Being able to prove that to clients goes a long way to reassuring them -- and justifying the rates that I charge!

[sofTest]

Re: well I lost everything on my server

I ran ZFS under Slowaris more than a decade ago when it was still a part of Sun's offerings. It simply will not address the size of my archive (without me NEEDLESSLY having all that rust spinning).

ZFS from more than a decade ago is not relevant. A lot has happened with it since then, and features are still being implemented. From what you have proven above, your level of knowledge of ZFS is likely being able to spell ZFS. No one have suggested that you should use any other storage solution than you have today. Your data, your time, your problem.

I really want to keep the source code, schematics, board layouts, component datasheets/manuals, toolchains, etc. for every project I've ever designed. And my tax records. And my music archive. And...

Do I want them to reside on media that is accessible any time of day or night? Nope. So, why have them spinning, needlessly? Should I buy 30 NASs and leave them all powered off until needed (configured for whatever level of redundancy I deem appropriate for those particular files)?

Do you have all your CD/DVD media in individual drives so you can access them at a moments notice? (I doubt it) OTOH, I suspect you want to be able to access the data stored on those media WHEN you decide you have a need for it. What measures have you taken to ensure that it remains accessible?

I've come up with a solution that lets me:

• know what I have
• where its located
• verify its integrity
• locate redundant copies in case of corruption

And, it costs me only -- and exactly -- the price of the media required to store those copies. I have no special hardware or software requirements to recover (read) any of that data. Or, alter it. And, use zero power to run that "system".

Really? However many times you keep repeating yourself, I am not interested in how you organize your storage needs. Your data (hopefully), your time, your problem, not interested. I will however say as much that your scheme is not for everyone, and if I proposed it in any sort of professional capacity, I wouldn't keep that capacity long.

As for what personal data I keep online and offline; my data, my time, my problem, none of your business.

[goontron]

Re: well I lost everything on my server

You'd best hope I'm a great rider because many of the devices I've designed have a significant impact on your health and life! ... or, have your car stop in the middle of the highway; or, ...

Jokes on you. It already has! The computer locked up and the engine went WOT until it blew. I hope you never worked on the computers for the '06 Prius.

EDIT: And before you say "throttle issue", it didn't go into neutral, the ODO was unresponsive, and i was able to pull the wiring loom off on the throttle. Still ran away.

[Topcat]

Re: well I lost everything on my server

kids kids kids.....lets all be friends!

[Curious.George]

Re: well I lost everything on my server

Well, considering i am employed by one of the quoted companies I AM a support personnel.... And that companies name doesn't start with a "W".

So, the size of your IT department is exactly 1. And the time you spend on "support" comes at the expense of the other activities in which you'd probably prefer to be engaged.

I would much rather be designing a new piece of medical equipment or a slot machine or a process control system than attending to a "storage mechanism". OTOH, I rely on that storage mechanism to enable me to design new pieces of medical equipment, slot machines, process control systems, etc. So, it becomes an issue of how to maximize the effectiveness of ANY solution while minimizing its ongoing costs of maintenance.

BTW, you might want to hop off that horse before you fall, i got a feeling you don't know how to ride.

You'd best hope I'm a great rider because many of the devices I've designed have a significant impact on your health and life! :> Wouldn't want you to get misdiagnosed as HIV positive; or, have your car stop in the middle of the highway; or, ...

[petehall347]

Re: well I lost everything on my server

is the data not recoverable ?

[Curious.George]

Re: well I lost everything on my server

As for FreeBSD, Nas4free is based on that. From above about ZFS, you have obviously not staid current/updated. ZFS has been part of FreeBSD since v7 and NetBSD since 2009.

I ran ZFS under Slowaris more than a decade ago when it was still a part of Sun's offerings. It simply will not address the size of my archive (without me NEEDLESSLY having all that rust spinning).

I really want to keep the source code, schematics, board layouts, component datasheets/manuals, toolchains, etc. for every project I've ever designed. And my tax records. And my music archive. And...

Do I want them to reside on media that is accessible any time of day or night? Nope. So, why have them spinning, needlessly? Should I buy 30 NASs and leave them all powered off until needed (configured for whatever level of redundancy I deem appropriate for those particular files)?

Do you have all your CD/DVD media in individual drives so you can access them at a moments notice? (I doubt it) OTOH, I suspect you want to be able to access the data stored on those media WHEN you decide you have a need for it. What measures have you taken to ensure that it remains accessible?

I've come up with a solution that lets me:

• know what I have
• where its located
• verify its integrity
• locate redundant copies in case of corruption

And, it costs me only -- and exactly -- the price of the media required to store those copies. I have no special hardware or software requirements to recover (read) any of that data. Or, alter it. And, use zero power to run that "system".

[goontron]

Re: well I lost everything on my server

And they all have paid staff working to keep that software running properly. How many guys in your (personal) IT department? :>

[I suspect I'm one of the few folks, here, who has actually done any kernel hacking or contributed to any FOSS projects -- having run NetBSD since 0.8 and FreeBSD since 0.9 (with the 5" floppies to prove it!)]

Well, considering i am employed by one of the quoted companies I AM a support personnel.... And that companies name doesn't start with a "W".

BTW, you might want to hop off that horse before you fall, i got a feeling you don't know how to ride.

[sofTest]

Re: well I lost everything on my server

And they all have paid staff working to keep that software running properly. How many guys in your (personal) IT department? :>

[I suspect I'm one of the few folks, here, who has actually done any kernel hacking or contributed to any FOSS projects -- having run NetBSD since 0.8 and FreeBSD since 0.9 (with the 5" floppies to prove it!)]

Sure, if you don't feel competent, or are unable or unwilling to acquire the required competence, you should stay away. The knowledge is out there for the grab for those interested. Today you can also pay for free software training, as for paid software. It's not like paid software doesn't require competence. Some even have paid and free versions of their software, often with main difference access to support.

As for FreeBSD, Nas4free is based on that. From above about ZFS, you have obviously not staid current/updated. ZFS has been part of FreeBSD since v7 and NetBSD since 2009.

[Curious.George]

Re: well I lost everything on my server

^ As does the supercomputer that the NWS uses. And Greenhouse Data Centers. And the Echostar uplink center. And Walmart. And even Microsoft in the Azure DC over here.

And they all have paid staff working to keep that software running properly. How many guys in your (personal) IT department? :>

[I suspect I'm one of the few folks, here, who has actually done any kernel hacking or contributed to any FOSS projects -- having run NetBSD since 0.8 and FreeBSD since 0.9 (with the 5" floppies to prove it!)]

[goontron]

Re: well I lost everything on my server

^ As does the supercomputer that the NWS uses. And Greenhouse Data Centers. And the Echostar uplink center. And Walmart. And even Microsoft in the Azure DC over here.

[sofTest]

Re: well I lost everything on my server

As a friend who operates a large data center for a multinational bank is so fond of pointing out:

"There are two types of RAID (or any similar technology) users: those who HAVE experienced data loss -- and those who WILL"

Sure, right. Have I said that RAID is a warranty against data loss? Again, RAID is no substitute for good backup routines, of which I have said nothing. RAID only reduces the risk for data loss. But I know one thing, any data centres for multinational banks does not operate shelves with bare drives.

Hint: the ONLY time I came close to losing data was when an OS upgrade introduced a bug into the SCSI disk driver that caused the disks attached to the controller to be scrambled. So much for "free" software...

Hint: No software, paid or free, gives warranties against data loss. I you know any, please list them, and I'll be sure to stay away from them. They would be as dishonest as their warranties.

BTW, many of the biggest data centres in the world run "free" software.