Offsite Backup Strategy for Video

Has anyone managed to successfully implement an offsite backup strategy for their growing video library.

I now have over 7 Tb of video event library (FCP X) stored on a 9 Tb Raid 5 external SATA drive and backed up to second identical Raid 5 drive. And I have over 2 Tb of finished video on a 3 Tb internal drive, backed up to another internal drive in my second MacPro. So they are getting close to full. I predict I'll be getting close to 90% full by end of this year. I am thinking that my backup Raid drive will have to become my second active drive so I can keep growing up to 18 Tb. But, unless I spend a lot on a whole new Raid system, I won't have any backup.

And, I am concerned that, even with my current backup approach of two Raid 5 drives, the 2 systems sit side by side. I've been worried about that proximity risk. A fire or thief could take out both.

Have any of you tackled this issue? Best solution?

1. Several cheap portable 3Tb drives? (but then how to spread one library over several separate drives?)

2. Or a tape backup system? Which one works on MacPro? Reliability? Does not need to be fast but needs longevity.

3. Or dozens of RW Blu-ray disks? This is what I do for my image library but it is only 1.5 Tb in total. How to spread one video library over many separate disks?

Your thoughts please.

Regards

Peter

Hey Peter,

Not many folk have the resources for the 2 RAID 5's that you have. RAID 5, IMHO, is the best RAID configuration. I would strongly urge you not to fill any drive, RAID or otherwise, more than 80-85%, it can cause a slowing down of the drive and an eventual crash. If you to copy over a specific event's footage library to a smaller external drive there is nothing wrong with that as long as you run the drive every once in awhile. I use an Accusys RAID 5 and a Drobo Pro, which is RAID like, which I use to back up everything. I use Event Manager X to stipulate which drives I want showing in the X browser. I would spread specific events and their projects to designated drives and label the drive so there is no mix-up and that should serve as a good backup. You also might want to duplicate and rename by date every project that you are working on as you work on them.

However, I also back up all footage to other smaller external drives. The clips are separated into species folders on these smaller drives and they are backed up as well. Personally, I have no experience using Blu Ray DVDs for back up storage and while I have an abundance of stuff originally shot on tape, since what I wanted off of them is already digitized, I just properly stack them vertically and run them back and forth once a year. Tape often gets degraded over time but, in truth, I have never really needed to go back to the master tapes so I don't really worry about it.

You do have a lot of footage; are all the clips footage that you really want? The thing I liked about log and capturing in legacy FCP was that I could set in and out points prior to downloading to my projects and avoid capturing footage that I thought was bad, thus saving tons of hard drive space. In X, you can do the same thing off a card but only one clip at a time. I look forward to Apple changing this feature. If you have footage that is unusable, trash it and save some space for yourself. You might also want to consider not capturing your audio. Really, who needs 60 hours of bubbles? Capture the audio that you really need and forget the rest.

Hope this helps,

Steve

PS. One of my editing students has just moved to Singapore and appears to be doing well there. I still owe her a couple of hours of paid for lessons. Waiting for her to completely unpack and then she will start learning X via iChat with me.

Hey Peter,

.

However, I also back up all footage to other smaller external drives. The clips are separated into species folders on these smaller drives and they are backed up as well. Personally, I have no experience using Blu Ray DVDs for back up storage and while I have an abundance of stuff originally shot on tape, since what I wanted off of them is already digitized, I just properly stack them vertically and run them back and forth once a year. Tape often gets degraded over time but, in truth, I have never really needed to go back to the master tapes so I don't really worry about it.

 

Hope this helps,

Steve

 

PS. One of my editing students has just moved to Singapore and appears to be doing well there. I still owe her a couple of hours of paid for lessons. Waiting for her to completely unpack and then she will start learning X via iChat with me.

Thanks for the advice.

I think that I'll buy some 3Tb external drives and use Retrospect 9 to back up the Raid5 drives using its multiple target drive spanning feature. Then I can leave them offsite to protect against a major disaster.

I know that I should be more frugal with what I save but I've never had the patience. I' also suffer from the delusion that, sometime in the future, i might like to dig through my library to see if there are any gems that I missed when I did my initial edit.

Regards

Peter

PS. One of my editing students has just moved to Singapore and appears to be doing well there. I still owe her a couple of hours of paid for lessons. Waiting for her to completely unpack and then she will start learning X via iChat with me.

If she needs some Singapore advice, e.g. Where to buy stuff, she can contact me.

. I' also suffer from the delusion that, sometime in the future, i might like to dig through my library to see if there are any gems that I missed when I did my initial edit.

I think most of us suffer from this

Never fall in love with your own footage. It's a cardinal rule. ;-)

Peter, she is not a member of Wetpixel I don't think. Could you pm me a # so I can pass it on to her.

Steve

Thanks for the advice.

 

Then I can leave them offsite to protect against a major disaster.

 

Mother in law apartment would be perfect

If you have older computers, turn them into servers so they maintain the server by backing up to spare disks etc without issue. I have 2 off site servers in 2 different countries should my 2 servers at home fail. The older ones are run by my old desktops or even laptop (eSATA PM into a array of 2x10 drives in RAID 100), all powered by a APS system that will shutdown and restart, all Apple remote callable for monitoring.RAID 5 is ok but slow for working. RAID 10 and 100 is faster and actually more reliable. One thing is that if you go the RAID card route, don't buy from companies like Highway, which drop support for OS versions. I just had to rebuild my RAID 100 via JBOD and software because their RAID cards don't have firmware supporting OS.7. ANd you're pretty much stuck with a bunch of dead drives unless you buy another RAID card that reads the volumes it builds. I've switched to software RAID now with old computers running the servers for ease of operation and connect either via 10gbether or SAS.SAS is a nice alternative as it's fast and relatively cheap to build and can act as JBOD.For longevity, buy the server quality drives. The WD Green enterprise drives have been awesome for me. 6-7W running per drive vs 11-15 for 7200rpm normal drives. My RAID100 can playback uncompressed 200MB/s files without a hitch.

I would firstly try and reduce the footage that you have, do you keep all the render files etc, if so delete them as they will take up a stack of space. Also as tedious as it is do you cut your clips down or keep everything, cutting them down and exporting the usable sections could save a fair amount of space?

For an offsite backup I purchased a 3.5" HDD dual cradle and buy 2TB drives and dump projects onto that and store them in another office. It is a pretty cheap way of doing it.

While Drew's contention that RAID 10 is faster than a RAID 5 is correct this doesn't mean that a RAID 5 is slow. My RAID 5 has read and write speeds of 754 and 685 which is plenty fast. However, as any drive, RAID or otherwise, fills up a slow down in speed performance will naturally occur; another reason not to fill your drives more than 80-85%. Often listed as a RAID 0+1, the RAID 10 needs a minimum of 4 disks. The upper layer mirrors the two striped sets below it, however, if one drive of each striped set fails, the data would be lost. Thus, one has to take into consideration the possibility of two drives failing at the same time. It doesn't seem likely, but when you figure that the drives might well have been bought and installed at the same time, should one go down, there may be a good chance that another, being the same age and having the same usage, might well go down as well. It provides excellent redundancy, as the blocks of data are mirrored, and still have good performance. Remember that when striping is involved you lose 50% of the storage capacity. For critical applications, a RAID 10 may be the best option as it offers faster data read and writes and does not need to manage parity however, I do not like the loss of half my hard drive space and so chose a RAID 5 configuration.

Did some shopping over the weekend and am now executing my new backup strategy.

Decided to stick with Raid5 and bought 4 x 3Tb Seagate Barracuda drives so now I have some spare capacity in my my primary Raid5 drive.

Bought a license for Retrospect 9 as it seems to be the most competent backup tool, especially at spanning multiple backup drives.

Bought 5 x 3Tb WD MyBook Studio drives and have started backing up my primary Raid5 drive to 3 of them (will take 24 hours over FW800 @ 400 Mb/min).

For now, I will keep my primary and backup Raid5 drives in sync. But, when my primary Raid5 reaches 80% full, I will convert the backup Raid5 drive to be a second primary drive and rely only on the 5 x 3Tb WD MyBook Studio drives for my backup. I will run a weekly backup and keep the WD backup drives at another location. I might get a second set of the 5 x 3Tb WD MyBook Studio drives, so that I can cycle the backups more safely.

Quite an expensive solution ($1200 for the 5 WD drives and $130 for Retrospect) but, given the time and money that I have put into capturing this video over many years, the peace of mind seems worth it.

Thanks for everyone's advice.

Regards

Peter

I've never felt comfortable using RAID.

My preferred approach is to ruthlessly cull everything asap...even on site at the end of the day's diving. This gets rid of mediocrity from the outset...and ends up saving drive space and time in the long run.

Next I divide into two categories...Useable & Maybe Useable.

"Useable" goes on to external USB3 drives (or eSATA) in duplicate for redundancy. "Maybe Useable" goes on to whatever old USB2 drives are available.

When it comes time for editing I transfer only the selected scenes to an internal SSD on the NLE workstation. This makes a huge difference over even the fastest spinning platter HDD for editing.

I've never felt comfortable using RAID.

 

My preferred approach is to ruthlessly cull everything asap...even on site at the end of the day's diving. This gets rid of mediocrity from the outset...and ends up saving drive space and time in the long run.

 

When it comes time for editing I transfer only the selected scenes to an internal SSD on the NLE workstation. This makes a huge difference over even the fastest spinning platter HDD for editing.

For me, it depends on the project. If it is my own project with self-managed deadline, then I take a cull approach, by selecting quality footage into a "first pass" project timeline. Anything else gets zapped. But if it is a client project, I'll usually do a "select what the client wants" pass and building a timeline from that. I always say I'll go back and take another "first pass" approach but time slips away. It is those projects where I end up keeping more raw footage.

But, even with a culling strategy, if you are shooting high-res footage, over the years it does build up.

I do like your idea of an editing SSD drive. It would just need to contain the current project and event. The SSD could be synchronized with an external Raid5 drive daily for permanent storage. Have you done any numeric comparisons of speed improvements over disk-based editing?

Regards

Peter

"Bought a license for Retrospect 9 as it seems to be the most competent backup tool, especially at spanning multiple backup drives."

Peter,

Never heard of Retrospect. Will have to look it up. How does if compare with Super Duper, which is what I use for system backups. If you will, PM me your # to pass along to my friend in Singapore.

Steve

 

Peter,

Never heard of Retrospect. Will have to look it up. How does if compare with Super Duper, which is what I use for system backups. If you will, PM me your # to pass along to my friend in Singapore.

Steve

Hi Steve,

Retrospect was the best backup tool for Macs for decades, especially for larger Mac environments where one administrator backed up many client Macs. But, then, through some M&A, it fell into the hands of a company, EMC, that didn't see it as strategic so it languished. A year or so ago, the original development team bought it back and produced V9.

SuperDuper is good but basic. Retrospect is more sophisticated. For example, it has a rules system to include or exclude anything that can be codified in a rule. I have set up a rule to exclude "render files" in my video backup. It can also handle a many-to-many backup strategy. I have 3 drives backing up to one backup set - which currently contains 3 drives, but will grow incrementally as the source drives fill. Retrospect is quite smart on managing backup sets, keeping only one copy of identical files, cleaning out old data and advising when a new drive is required. Retrospect defaults to an incremental backup model but can do archival and exact duplication approaches. It can also do "proactive" backups, triggered when a device appears on the LAN, e.g. a laptop or an external drive. It also handles tape backup, if required. Another strength is to do a selective restore, selecting a file, files, folders, or anything through a set of sophisticated selection options.

More of a learning curve, than the simpler tools, but more powerful with lots of options and controls.

Regards

Peter

Often listed as a RAID 0+1, the RAID 10 needs a minimum of 4 disks. The upper layer mirrors the two striped sets below it, however, if one drive of each striped set fails, the data would be lost.

Steve, what you are talking about is RAID 0+1. RAID 10 is mirrored first then striped, hence 1+0, not 0+1. It can take the failure of 2 drives so long as it's not both in the mirrored volumes. hence double the safety plus if you have a spare drive, the rebuilds are much faster.

RAID 5 is cheaper and works for most single users who want easy backup. However the rebuilds are slower and is more processor intensive.

 

Steve, what you are talking about is RAID 0+1. RAID 10 is mirrored first then striped, hence 1+0, not 0+1. It can take the failure of 2 drives so long as it's not both in the mirrored volumes. hence double the safety plus if you have a spare drive, the rebuilds are much faster.

RAID 5 is cheaper and works for most single users who want easy backup. However the rebuilds are slower and is more processor intensive.

Half the problems people have with Raid is figuring out what the numbers mean...

Have you done any numeric comparisons of speed improvements over disk-based editing?

 

Regards

Peter

I've not done any read/write speed comparisons myself...but I based my SSD purchasing decision on tests done by good Blogs e.g. http://www.anandtech.com/show/5272/ocz-octane-128gb-ssd-review/4

Good SSD is still pricey...but I feel worth it for NLE. This is particulary true for the large, HBR files I work with on the Edius timeline (original, without conversion).

I enjoy your travel guide videos!

 

 

I've not done any read/write speed comparisons myself...but I based my SSD purchasing decision on tests done by good Blogs e.g. http://www.anandtech.com/show/5272/ocz-octane-128gb-ssd-review/4

 

Good SSD is still pricey...but I feel worth it for NLE. This is particulary true for the large, HBR files I work with on the Edius timeline (original, without conversion).

 

I enjoy your travel guide videos!

I already use a 512Gb SSD drive as my primary drive for the MacPro. But with OS, apps and other data, it is already too full to use for source video. In any case, I believe it is faster to have the source video on a different drive from the app and OS.

So I have followed your advice and ordered another SSD from OWC. This one is SSD on a PCIe board, so I don't lose a disk bay in my Mac Pro. http://eshop.macsales.com/shop/SSD/PCIe/OWC/Mercury_Accelsior/RAID

They claim is runs at 820 MB/s so should be zippy for FCP X source video (event folder). It arrives tomorrow so I'll let you know how it performs.

Regards

Peter

I already use a 512Gb SSD drive as my primary drive for the MacPro. But with OS, apps and other data, it is already too full to use for source video.

 

In any case, I believe it is faster to have the source video on a different drive from the app and OS.

 

 

Regards

Peter

Yes, that's a very important consideration. My i7 SandyBridge workstation has 4 internal drives: 1) Operating system; 2) Misc data; 3) 1.5Tb Video clip project data; and 4) SSD for timeline video data.

Internal transfer via SATA from drive 3 to the SSD is very quick...then the NLE only has to deal with the SSD.

For field editing I believe that some laptops have provision for two internal HDD. I had such an HP a couple years back but I'm not sure what's currently available. This would allow the second drive to be an SSD dedicated to the NLE. Much better than having everything on a single drive.