Question : Problem: Need expert advise.  Large backup on limited resources.

First, let me start with an overview of our operations.

We are backing up about 85 servers through two media servers-One has backup exec 8.6 while the other has backup exec 9.1.  Each one of these media servers has a a LTO-1 autoloader with two drives

The backup exec 9.1 server backups up about 55 servers and will have be backing up about 1.2 TB of data (900 GB daily, 1.5 TB on weekends)

.5 TB of this is SQl databases that gives us about 2 GB a minute (has a 2gbit ethernet connection to a SAN)   FULL BACKUPS each night
.1 of this is microsoft exchange mailboxes (100 Mb connection to these servers)                                          DIFF on weekdays, full on weekends      
.5 of this is operating system backups (mixed network speed, mostly 100 Mb)                                              DIFF on weekdays, full on weekends
.2 of this is flat volume files (about 13,000,000 files all in one directory! less than 1K each, 100 Mb)               DIFF on weekdays, full on weekends


The backup exec 8.6 server does the same procedures as the 9.1, but at a smaller scale (.6 TB of so)

You can see that our backups are mixed and matched, not to mention that we have unix, freebsd, and windows NT AND 2000 machines to backup.  Also, with the hardware we have (and software), it is not up to finishing in the time window (due to hardware and software issues)  We have finally fixed our current backup operations, and removed as many bottlenecks as possible, but it still will not finish...


To alleviate the problem, we are given a $20,000 budget to make our backups finish consistently within the time window.  I need some advice on the most proactical solutions to fix this issue.  So far, I have come up with a few ways to spend 20K, and not sure which will be the most effective.  The reason that the backups are not finishing are as follows

Reason 1-Not enough network speed

Reason 2-Too many files and fragmentation on the local hard drive (i.e. the servers will millions of flat files)

Reason 3-Backup exec only copies these files one at a time to the tape, so the operation will wait and run at the slowest variable in the equation (network, hard drives not able to index/read fast enough,)

Option 1
Buy 2 LTO-1 drives to go into the backup exec 9.0 server.  This will give us more active jobs that can be chugging away.

Option 2
Buy a new SDLT 600 OR LTO-2 library.  We can get this for a little over 20K, but we'll be OK.  This will allow more throughput, active jobs, capacity, and reliability.  Plus, each drive has its advantages (SDLT 600 verifies as it writes and LTO-2 has increased media life)

Option 3
Add LTO-2 drives to the library and "mutlipath" it (IBM 3583 library).  This will give us increased tape tranfer speed and capacity, if backup exec can send it fast enough.  This is mixing the LTO-1 and LTO-2 drives in the same library and giving rules to the drives (ie LTO-2 drive us LTO2 media if present, otherwise use LTO1; LTO1 drive only load LTO1 tapes)

Option 4
Buy another backup software that can multiplex the data to the tape drive.  This has its inherent advantages and disadvantages.  First, the advantage is that the tape will be writing close to its theoretical maximum most of the time.  The disadvantage is longer and slower restores and a bigger emphasis on the integetrity of data and tape organization.  Such programs as arcserve and legato networker (not sure on price of these, some easily are over 20K for our application)  We would have to worry about mixing of agents on the remote servers and keeping the appropriate tapes in the appropriate systems.  We would upgrade the 9.1 server to new backup software to do this.  

Option 5
Buy a disk storage system.  We could buy a 2TB iscsi storage system (a little under 20K for redundant array)  We already have the 2 storage routers for this (Cisco 5428)  Then perform a D2D2T operation, with all jobs starting at the same time, and then when finished being archived to tape.  I am unsure if this could be done with backup exec, as its D2D2T options are virtually not there....  We could make it work however, and perhaps upgrade to a 4TB storage system so we could have much faster and accessible restores.  This would be a balancing act on what to keep locally and what to archive without running out of disk space.


So all in all, theres even more ways then this to spend our budget.  In our situation however, functionality and protection is more important then cost, and it could be pushed a reasonable amount.  If anyone has any suggestions, then it would be greatly appreciated.  I know that this is limited information for a broad question.  Thanks!

Answer : Problem: Need expert advise.  Large backup on limited resources.

I used to manage a network about half the size.  We used 4 SDLT heads, 2x110, 2x160.  

Backup time was always a problem.  But we often found our bottleneck was not tape throughput (SCSI throughput), but rather we were backing up bricklevel exchange (slow) and user files - MANY small files which just take forever en masse to backup.

Also keep in mind your network at 100 Mbit has a max throughput of about 10MBytes per second or 600 MB per minute (AT BEST, assuming no one is using the network during the backup times)

Here's a couple more ideas for you to ponder:
1.  Install a backup network - a private gigabit ethernet network that  only the backup servers acccess using a reservered IP.  Buy an additional Gigabit NIC for all servers and get a few gig ethernet switches.

2.  For a D2D2T idea, try an Aberdeen server (we bought a few and were pleased with them).    The 6TB "TeraStorus" is $15K.  Fully decked out with 4GB RAM and Dual 3.2 Xeon Cpus.

I see your biggest problem being your 13 million files at less than 1K each.  What's your throughput on this part of the backup job?  Actually, it might help everyone if you could review your throughput for a few jobs and post the averages for each of the 4 areas you are backing up.  

In many respects your question isn't so much "how should we spend $20 to improve backup" as it is a question of "how can we improve our backup, if necessary spending under $20K."