Question : Problem: Need LOTS-O-STORAGE space.  Can you help design it?

We are putting together a business plan for a new venture and there is one part that I need to get some clarity on.  Online storage.  This project might take up 500-800 TB of space.  I am a LAN network guy and none of my networks has anything over 2 TB so my questions is, what is the best way to create an environment that can handle that amount of data? Most of the files will be 500-600MB.  I have 5Gbps bandwidth that we can easily increase and plenty or physical storage space and power.


Direct questions:
1) I am most familiar with Windows but the 2TB volume size limit is a problem. Or is it?
2) I am not afraid of LINUX but what limitations would I run into with that?
3) What would the hardware setup on a large storage system like this look like?  I.E. what components would I need to put together?
4) I have a pretty strong feeling that in the end, we will have to hire a company to design/build something like this like IBM or someone but I want some info so I am not just accepting the recommendation blindly.

Answer : Problem: Need LOTS-O-STORAGE space.  Can you help design it?

There are way too many questions in designing something like this to be realistically answered in a forum like this. I used to work for EMC and agree that they have the expertise, as do vendor companies like IBM, HP, Hitachi, etc. as well as consulting houses like Deloitte, Pomeroy, MTI, and many others.

A few general questions to consider:
1) How many servers will need access (just one, or many)?
2) What type of I/O throughput will be expected? Read-heavy, Write-heavy, or balanced?
3) Do you really need to see up to 800 TB in one volume? I've never seen anything like that, although maybe it's been done. Usually SAN Array volumes are maybe 500GB at most, then larger volumes are built on the OS by a logical volume manager.
4) How much money is able to be spent on this - are they willing to spend enough to do it *right*??

With a small number of servers, you may not need fibre channel switches (go direct-attached) but you need to do more than just allocate gobs of storage on the array, you need to profile your I/O patterns, do the RAID layouts based on acceptable risk, performance and cost, and discuss issues like HBA (host bus adapter) redundancy, failover, and load balancing. Lots of planning and decisions will be involved.