Looking for a NAS

[Dannyx]

Re: Looking for a NAS

Unless you're running a speedy RAID array or SSDs it's pointless as your disc speed is the bottleneck.

Correct, that's why I stopped looking into it and decided to focus on improving "disk" speed...somehow

[diif]

Re: Looking for a NAS

Correct, that's why I stopped looking into it and decided to focus on improving "disk" speed...somehow

An SSD, even on SATA 2 is about 250MB/s with the bonus of low latency and fast random access.

[Curious.George]

Re: Looking for a NAS

Correct, that's why I stopped looking into it and decided to focus on improving "disk" speed...somehow

MEASURE before you start optimizing. Premature Optimization is the bane of the naive.

And, measure based on YOUR expected usage patterns, not some canned benchmark (unless all you plan on using the shared data for is running benchmarks!).

There's a reason I transfer files with FTP and not SMB -- performance (for the usage model that I've adopted)!

For an eyeopener, try copying a huge disk (TB+) with millions of files via SMB (i.e., as if you were making a backup of a volume that was remotely mounted). Then, try making a local copy. Note the difference. Same disk. Same OS. Different protocol stack. (Gee, why is it so much SLOWER??!)

[If you really want to get a feel for how underperforming these protocols are, try mounting a datastore remotely and doing queries on the database that uses it!]

Replace the physical disk with a memory disk and see how the numbers change, in each case.

THEN, decide where you want to spend your time/money.

[Dannyx]

Re: Looking for a NAS

Will have to get my hands on the hardware first...don't quite have an SSD on hand readily available, plus a machine to install it in.

Actually I could do some rough tests this way: laptop's got an SSD in it. Share a folder on it, drop some files in there (music), then access them from a wired station, the theory behind all this BS being that it would be the same scenario but in reverse (client is wired and "NAS" is wireless). Pretty flawed, isn't it ? Storage controller speed is probably another (critical) factor here, so there is that...

[Curious.George]

Re: Looking for a NAS

Will have to get my hands on the hardware first...don't quite have an SSD on hand readily available, plus a machine to install it in.

You don't need an SSD to do the testing. Pick a disk, ANY disk. Characterize its performance under various usage scenarios: copying to/from the bit bucket (gives you an idea of the disk's max performance with the least OS involvement); to/from another LOCAL disk; to from another remote disk -- under a variety of different protocols and usage patterns.

In my case, I want to copy entire "objects" across the wire. So, a protocol that supports random access is useless overhead (and software complexity): "give me block #1, then block #2, then block #3, ..."

With the "per block" protocol minimized, then you look at the setup time for each transfer (i.e., specifying the object of interest). The client/server have to sort out permissions issues, locking, etc. BEFORE they can start moving data.

This can be a horrendous performance hit when you're dealing with gobs of tiny objects!

My first optimization was to implement server-side compression. So, I could say "get foldername.tar.gz" and the SERVER would build a tarball of the foldername's contents (accessing its LOCAL disk at high speed), compress them (so fewer bytes to transfer over the wire) and, finally, tranfer them to the client.

The downside of this is there aren't as many tools with intuitive interfaces that can understand gzip'd tarballs. And, the archive format has high startup costs if you're just wanting to take a peek inside.

So, now I build .iso's on the server and mount them on the client to give me finer-grained access to their contents.

Actually I could do some rough tests this way: laptop's got an SSD in it. Share a folder on it, drop some files in there (music), then access them from a wired station, the theory behind all this BS being that it would be the same scenario but in reverse (client is wired and "NAS" is wireless). Pretty flawed, isn't it ? Storage controller speed is probably another (critical) factor here, so there is that...

If you want to understand the source(s) of performance hits, you need to look at your "system" (client+server) in small pieces -- so you understand what the effects/costs/contributions of each of those pieces are on your overall performance (e.g., before buying a faster disk, newer NAS, etc.). There are lots of micro decisions that your "system" is making of which you are completely unaware.

[I dislike using Windows machines as test platforms cuz there's too much extra cruft running under the hood that can impact performance.]

For example, the actual dialect of SMB that is being used will vary based on the version of the OS (assuming you're just dealing with Windows) that each (client + server) is running. E.g., Vista or 2K8s to/from any newer Windows version will cause both client and server to use dialect 2.0. For 7even/2K8R2 it's 2.1; W8/2K12s it's 3.0; 8.1/2K12R2 it's 3.02 and W10/server10 it's 3.1. The actual performance you'll see in each combination (client+server) will thusly vary (newer dilects try to add functionality, improve performance, and reduce "anomalous behaviors".

Reading from a particular physical disk will give different performance than writing to it.

Reading from a protocol stack gives different performance than writing to it.

Small files give different "steady state" performance than larger ones (read and write caches fill up!)

You want to be able to figure out WHY you're seeing the performance that you are seeing -- is it because of protocol, hardware, software, fabric, access mode, etc.?

UNLESS (of course) you are happy with the performance you are getting in your "typical usage scenario" (in which case, don't look any further!)

[I wanted to get the most out of my "transfer entire files/folders" usage pattern, hence the effort I put into optimizing that path. The SAN I'm building for my VMs has an entirely different goal so a different optimization pattern.]

[Curious.George]

Re: Looking for a NAS

I just have a hard time trusting critical data to a SSD for some reason.

Agreed. Too many horror stories of nasty (i.e., data-eating) bugs FROM REPUTABLE MANUFACTURERS (e.g., Samsung, Intel).

I think much of the problem comes from the newness of the technology. Folks keep discovering new ways that the media can falter (e.g., read disturb and write disturb in addition to write wear). And, these change with the choice of underlying flash technology (SLC/MLC/TLC/QLC/etc.). Couple this with shrinking geometries and operating voltages and you're counting individual electrons -- a few dozen to differentiate between adjacent bit states (SSDs are becoming ANALOG devices as the controllers attempt to infer more from the actual "charge" seen at each cell to adjust write/refresh algorithms).

This complicates the design of the FTL. Add to that the incentive to push performance limits causes the higher abstraction levels of the device to be merged into the lower levels of the FTL (to exploit efficiency hacks that span the layer boundaries).

Add to that, legacy usage patterns that may be predicated on the assumption that you're dealing with spinning rust which don't hold true for the newer media type (e.g., I can OVERwrite the SAME DATA countless times to a HDD with little cost to the medium; not true for SSDs!)

Bottom line, the firmware gets more complicated and more prone to errors creeping in. Benign errors just cost you some performance. Other errors cost you *data*!

I keep good backups of mine....but unless you dump it with every change (which for me would be daily), there's far greater risk for data loss. I usually do a full dump once a month and depend on the RAID (probably more than I should) in the mean time.

I'm setting up a SAN to support snapshotting in the hope that I can just worry about archiving the snapshots, as needed.

[Dannyx]

Re: Looking for a NAS

In my case, I want to copy entire "objects" across the wire. So, a protocol that supports random access is useless

Quite the opposite in my case: random access IS what I'm doing here with the music and the photos and all that...
Of course, your method could apply here as well: pull an entire file locally THEN play it, but that would probably take the same amount of time as waiting for the thing to buffer to where I want it to - obviously this time gets longer the longer the file is. A 6h track easily takes 1 minute to buffer towards the end....over WiFi that is, sine a wired connection probably cuts down the time significantly. Sadly, WiFi is the way I roll around the house, so I'll have to deal with it and probably any improvements made to the STORAGE side would be hampered by the WiFi bottleneck anyway.

Have you got any WiFi clients in your setup ?

[Curious.George]

Re: Looking for a NAS

Quite the opposite in my case: random access IS what I'm doing here with the music and the photos and all that...

Actually, that's no different from copying entire objects -- you just want to be able to access LOTS of different objects. E.g., drag a "Music" (or "Photos") folder over to local storage.

The sort of "random" access that SMB affords (over FTP) is being able to access (read or write) any part of a file without also accessing the whole file. Like tacking a "P.S." onto the end of a file without having to drag the whole file over to do so.

Of course, your method could apply here as well: pull an entire file locally THEN play it, but that would probably take the same amount of time as waiting for the thing to buffer to where I want it to - obviously this time gets longer the longer the file is. A 6h track easily takes 1 minute to buffer towards the end....over WiFi that is, sine a wired connection probably cuts down the time significantly.

Our usage pattern is to create local copies of the things that we need, pulling them off the archive only when our needs "change". E.g., loading a new set of music onto a phone; or new apps onto a computer; etc.. So, the archive lies dormant (powered off) for 95% of the time.

Sadly, WiFi is the way I roll around the house, so I'll have to deal with it and probably any improvements made to the STORAGE side would be hampered by the WiFi bottleneck anyway.

... unless you can accommodate local copies of those things. E.g., I've got ~3-4T spinning in each workstation so I can just pull entire (archived) projects onto them and poke around to see what I want/need. As I know there's an "original" residing in the archive, there's no problem with me erasing all or part of what I've copied, locally. That way, I don't have to leave the archive (NAS) running any longer than necessary.

Hence the big appeal of using "archives" (.RAR, .ZIP, .TGZ, .ISO, etc.) in the Archive! Just grab it all and toss what you don't need!

Have you got any WiFi clients in your setup ?

I have several (7?) hard-wired AP's, here, but leave them off. There are 25 uncommitted "drops" around the house so you're never more than a ~10' patch cord from a network connection.

[Dannyx]

Re: Looking for a NAS

I was going to go wired whenever possible, such as when the laptop is on the desk, near a readily available switch, but its network card kicked the bucket a while back and I never got around to fixing it (some broken traces on its board as a result of me using a metal screwdriver to lever off the heatsink which)...so for now I'll have to stick to WiFi...

BTW, 7 APs ? Are we talking about your home here or your business ? That's a lot, unless you've got like a 2 acre household

[Curious.George]

Re: Looking for a NAS

BTW, 7 APs ? Are we talking about your home here or your business ? That's a lot, unless you've got like a 2 acre household

I use them to track the physical location of "moving nodes". E.g., turn the WiFi on your phone and I can tell if you're in the living room vs. dining room -- and whether you're seated on the sofa, loveseat or one of the chairs.

[I will be gutting 5 of them and replacing them with BT radios so I can track earpieces instead of "phones"]

[Dannyx]

Re: Looking for a NAS

Isn't that a bit overkill for your average household ? Not gonna judge, it's your space after all Have you got that many visitors around that they constantly require tracking ? I just find it funny actually...in a geeky positive sort of way

[Curious.George]

Re: Looking for a NAS

Isn't that a bit overkill for your average household ? Not gonna judge, it's your space after all Have you got that many visitors around that they constantly require tracking ? I just find it funny actually...in a geeky positive sort of way

It's so the House knows that it should turn OFF the TV in the Living Room when I move into the kitchen -- and, turn ON the TV in the kitchen (tuned to the same source, of course). Or, serve up MY music tastes in the Office when I enter it -- or my other half's when SHE does so!

Etc.

Why should I have to TELL the House what to do?? That;s so 1970's...

[Dannyx]

Re: Looking for a NAS

So you've got some IoT going on...sounds interesting and tempting to do, though for my apartment of 2 rooms (not counting the kitchen and bathroom) that is slightly superfluous. It may sound like I'm living in some sort of coffin-sized dump, but where I live this represents your typical flat in an apartment block and it's fine for 3 people. 3 and 4 room apartments also exist of course, but the prices are way up there, especially if it's located in a "premium" area close to the city center. I was lucky to inherit this one which was in a pretty central area to begin with, but has since evolved into one of the trendiest places in the city thanks to all the malls built around here....I digress

[Topcat]

Re: Looking for a NAS

Why should I have to TELL the House what to do?? That;s so 1970's...

Ohh come on man, nothing a "Clapper" wouldn't do!!

[Curious.George]

Re: Looking for a NAS

Ohh come on man, nothing a "Clapper" wouldn't do!!

Clapper won't know how to ask the living room TV what content its displaying. Or, whether that "program" is over and there's no need to turn on the kitchen TV to watch it's conclusion.

The problem I see with all of the Alexa/IoT offerings is that they are little more than modern version of "remote control" technologies dating back to the 70's (e.g., BSR/X-10). The "user" still ends up having to do all the thinking.

As most of us have highly "regular" (read: repetitive) lifestyles, why can't the devices learn what we'll want and anticipate this, for us? If I've gone to bed, why can't the garage door close itself (because I forgot?). If I've driven off and left the stove on, why can't it turn itself off? If I'm taking a shower, why can't the washing machine pause its rinse cycle (to not cause an uncomfortable drop in water pressure/temperature)? If its going to rain TOMORROW, why can't the irrigation system adjust its cycle so that it only waters those plants ("zones") that can't wait another day for water?

These are all simple decisions that don't need a human intellect to resolve...
("Oh, I'd better go pause the wash cycle before I step into the shower -- and, hopefully, remember to resume it when I'm done...")

Even coffee pots make a token attempt to address these sorts of issues (letting you set a time at which it will start brewing next morning).

[Curious.George]

Re: Looking for a NAS

So you've got some IoT going on...sounds interesting and tempting to do, though for my apartment of 2 rooms (not counting the kitchen and bathroom) that is slightly superfluous.

Size of house isn't a defining issue.

I suspect you have a different sleep/wake schedule on weekends than weekdays? (or, whenever your "off" days are) So, the wake time set on your alarm clock should change to match those needs, right? I.e., alarm clock, at the very least, needs to know what day of week it is to compensate.

Ah, but what about holidays? Should it know the legal holidays that are observed in your region? Any other holidays that your employer might observe?

And, what if you have to deal with inclement weather? I.e., it snowed last night while you were asleep and you'll need an extra 10 minutes to prep the car for travel (or, an hour to clear the driveway)? Should the alarm clock take this into consideration? Or, just snicker at you when you wake to discover you're going to be "late"?

In addition to waking at a different time, do your breakfast habits change based on your schedule for the day (e.g., perhaps skip breakfast when you'd rather have an extra half hour of sleep before heading off to work)? Maybe just a quick cup of coffee on those days?

Do you leave the heat/cooling set at the same temperature regardless of whether the house is occupied? Should it be turned down on the days you'll be heading off to work? Should it be turned UP a few minutes before you are expected to get out of bed on weekdays? Weekends?

Do you, perhaps, try to catch the morning news on workdays, while attending to your toilet? And, maybe, instead, listen to music as you're waking up on those days off?

Do you always tune to the Evening News on TV at a particular scheduled time? Or, have any favorite shows on TV/cable in the evenings?

Do you have to remind yourself when you're running low on toilet paper? Or "milk"? Or beer?

If you start looking at all of the little things you've been coerced into doing to be able to live your life, you'll be amused at just how many things you are keeping track of on a daily basis.

Do you know, this morning, BEFORE heading out to the car, whether or not you'll have to plan on stopping at the gas station before work? Did you REMEMBER, last night, to make a mental note of the state of the tank and your expected needs for fuel, today?

Now, imagine you're older and possibly more forgetful. And/or living alone (and wanting to avoid having to be "cared for" at hundreds of dollars per day). How do the consequences of forgetting any of these little things fare? Are there new things that you've never had to address before (e.g., daily medication needs)?

Machines (computers) are great for doing repetitive things. Yet we insist (have no choice!) on doing them ourselves...

[Dannyx]

Re: Looking for a NAS

Speaking of WiFi: how do you manage security ? WiFi always seemed like fairly vulnerable thing to me, as just about any security feature a (home) router/AP can offer can be breached in some way: hidden SSID and WiFi passwords mean virtually nothing, then you've got the slightly more advanced MAC filtering which of course can also be breached somehow....stuff like that. Sure, it's more than plenty to keep your average idiot out (not that they'd find anything there other than free WiFi) but still, it's nice to know your network is safe from more than just your average Joes. I myself don't know how to crack a WiFI network, mainly because I haven't looked into it deep enough, but I'm sure it's one of the easiest things to do

[diif]

Re: Looking for a NAS

Wpa2-psk aes

[Curious.George]

Re: Looking for a NAS

Speaking of WiFi: how do you manage security ? WiFi always seemed like fairly vulnerable thing to me, as just about any security feature a (home) router/AP can offer can be breached in some way: hidden SSID and WiFi passwords mean virtually nothing, then you've got the slightly more advanced MAC filtering which of course can also be breached somehow....stuff like that.

Encrypting traffic is the way to secure your comms. There have been several different standards, over the years. Each eventually falling to some exploit or other. (and, rarely does SOHO kit support the sorts of upgrades that are needed to move to more elaborate encryption schemes).

And, the AP/wireless router itself may have flaws that can be exploited "behind" the encryption -- not to mention "defective users". Google "wireless router vulnerabilities" and you'll quickly see that even encryption with a OTP is laughable!

[With any security mechanism, you have to first identify the assumptions upon which you're relying -- then look at how those can be subverted (cuz the attacker will take a close look at them)]

The more insidious (and usually neglected) problem is that there is no way to secure the (wireless) network from DoS attacks. Or, from leaking "activity" and location information.

[Dannyx]

Re: Looking for a NAS

Wpa2-psk aes

That's how I have it set up, but I was thinking about the more advanced topics of WiFi security