Need more disk space

By background, my Mac Pro has 7 TB of internal drive:

Main HD: 1 TB - 600 Gb free - OS, Apps, iTunes library, etc

Images: 2TB - 1 TB free - full of my photography - high res keepers only

Movies: 2 TB - 175 Gb Free - all the editing versions and other stuff that make up a movie, plus all the different saved formats (web, bluray, DVD, QT, etc)

Scratch: 2TB - 215 Gb Free - video capture and narration capture files from the last 2 years of my UW video.

Main, Images and Movies are backed up to external firewire LaCie drives of equivalent size. Scratch is not backed up although could be recreated from original HDV tapes or a pair of 320 Gb portable hard disks that I have been using as a backup from the camcorder's internal flash memory.

My current concern is about the shrinking space on Movies and Scratch. I used to zap the video capture files when I had done editing. But sometimes I would like to be able to pull a bit of older video into a FCP project. Getting it back from the original is so time consuming that it would be convenient to keep it all on disk.

My first question is: What do you do? Keep it all on disk? Use something like FCP's media manager and just keep the video content that you actually chose to use? Archive it in some easily retrievable system? Just zap and forget?

My second question is: If the answer to question one requires extensive disk storage, what storage system do you use? Several external FireWire disks? SAN system? Raid box? What type of RAID? And, which model works well for you?

Regards

Peter

Hey Peter,

You certainly have a lot of internal drive storage. Never fill a drive, internal or external, more than 85% of that drives capacity. If you do you run the risk of not only disc slow down but of crashing the drive and possibly losing data. There are several brands of drives that have gone in and out of favor as a company was sold to another or quality control slipped. I still have 3 fairly old Lacie's that are still running great. However, when they came out with their first 500gig drives a few years ago(when 500 gig drives were considered very large) there were many reports of problematic drives. The BBC just sent me a free 500 gig Lacie D2 quadra but I have not had it long enough to say anything significant about it. So then I went to G-Tech and have 2 of their drives which have held up nicely over the years and I would still recommend G-Tech. OWC, (Other World Computing) produces a very solid external drive and I use one for my complete system backups which I do once a month. It is partioned to accomadate back ups done with Super Duper for both my new 12 Core Mac as well as the Mac Book Pro. I do use the Disk Utility 'repair disc' function on a very regular basis, just as I repair permissions on the boot drive both before and after software downloads and updates. This is a part of regular maintanence.

Other than 1 Western Digital drive a company sent me, which I use only for archiving, these days I primarily use the Data Robotics Drobo Pro with a 16 terabyte capacity. Be sure the drives you get spin at 7200 however. They sent me 5 drives that spin at 5400 and in a bit more than a year, I can't say I've had a problem but it would be best for you to go 7200. You can set it for single or double redundancy raid configuration. While I do not own one, CaliDigi has an excellent rep as well and may be worth investigating. Just don't fill up those drives.

I would strongly recommend you back up all your capture scratch and media to another drive. I not only do that but do a 3rd back up to a different drive which is divided up by species folders.

FCP's Media Manager can be very buggy and I refuse to use it. Just don't trust it. However, I do use the Render Manager at the end of all my projects. Since there may be mulitiple renders of many of the same things I may have tried or redid, I just delete all the render files for the finished project and then re render. This saves significant HD space.

Hope this helps a bit,

Steve

I like the Wiebetech units with the ability to bring drives in and out like floppies. Easy to make clones to keep offsite. I also have a few other items. The OWC Quad Bay is a good one, have it set up as RAID 5 with 4 two terrabyte drives and hooked via eSATA, get good throughputs.

With the price of drives so low (just grabbed 5 2 Terrabyte drives for $80 to expand things). 3 Terrabyte drives are out now, but pricey. (Around $220 a pop.) Though Media Manager has a bad rap in many circles (and many people do not trust it like Steve mentioned), I have not had issues with it within the last few generations of the Studio/Suite. Make sure to use handles if you are going to use Media Manager and also test on another computer to make sure it all reconnects properly. I find it a good way to consolidate projects for storage or travel, more than "officially" using it for projects. In other words I would not delete any original footage. I keep all my original files and images and video files get backed up automatically on a daily basis to local drives , and clones of those drives for offsite. I have drives set up to keep renders for projects (and all Color renders also.) Really depends on what you are doing and whether the time to re-render things will be problematic. Depends on the project, but I have found it is easier to have them stored for longer projects rather than re-rendering. Partly because I had a couple of projects that required alot of rendering from Color and Final Cut and the time I needed them the renders took forever. Or so it seemed when you get the call you need to get something NOW!!

I use this set up from SeriTek with my MacPro and it works excellent: http://www.firmtek.com/seritek/seritek-2eEN4/

I have a friend using one of these from Sonnet that he swears by and seem to work well too:

http://www.sonnettech.com/product/fusion400qr5.html

Humm I've currently got:

Internal

1x500mb system

1x750mb

2x2tb

External

1x1tb firewire

2x1tb esata/usb

2x2tb esata/usb

The 2x2tb drives are in a 4 drive enclosure with 2 spare slots at the moment. Sadly esata hasn't worked since I updated to Snow Leopard, but as those drives are mainly archives I've been able to live with USB for now. I keep everything I can on disk as I'm often going back and working with material I shot 4 or 5 years ago.

I'd definitely recommend an external enclosure connected via either firewire or esata (when it works).

Cheers, Simon

I use this set up from SeriTek with my MacPro and it works excellent: http://www.firmtek.com/seritek/seritek-2eEN4/

I have a few of those also and they work well. There is some compatibility with certain non-Seritek eSATA enclosures, but the new Seritek Cards seem to do better.

It's always good to have a SAN at home if you have a few computers. I run a SAN rack system at home (I buy mine 2nd hand from a production house which upgraded to FC) with an old computer acting as a server.

Ethernet (GBe/Cat5 is cheapest) but once the speed demands go up, it is not feasible. Since you are in SG, an enclosure with circulation and dehumidifier is probably best, if you don't keep it in an ac room 24/7. I like rack enclosures because once you upgrade, the older drives can be stored in them and used as a spare server. 10Gbe is better and the nice thing is you can direct connect to the server.

I sometimes edit uncompressed so I use the older drives as a RAID 0 stripe to maintain 200MB/s. If you are truly safety oriented, an offsite server is also a good idea as a precaution. I have them in 3 locations across the planet

As a practice, 7200/10k rpm drives aren't necessary if you aren't running multiple work stations. Power efficient drives like the Samsung and Western Digital GP cut power demands significantly from drives. Note though that if you keep the server running 24/7, it may be better to buy server quality drives for better reliability. I run RAID50/60 partitions in my latest setup but I'm dependent on the RAID card not imploding.

One other thing to speed up things is to go to SSD on the main drive.

Thanks for the advice.

I'll head out to the Singapore computer shops and see what I can find from Wiebetech, Sonnettech, DroboPro or FirmTek.

Any other brands recommended?

Any to avoid?

One shop offered me a system from Onnto but a search on their name uncovered a bad report so I wont be going back to that shop.

Regards

Peter

Go to the Storage Studio on the 4th or 5th flr @ Sim Lim Sq. They have a lot SAN recommendations. I've bought drives and enclosures from there and they know what they are talking about, which is very rare around Sim Lim.

I use a 4 disk stripe (4x2TB) in my Mac Pro that is constantly backed up to a NAS box over gigE. I have a 5th and 6th drive in it for boot and scratch. I also have an eSata Wiebetech 4-bay unit that takes bare 3.5" drives like disks, but I mostly use it for offsite backup.

I also have a SAN. I've gone iSCSI because it seems to be the cheapest reliable SAN, and I don't want to have to use a dedicated box as a server on the storage side. I'm using a ReadyNAS Pro with 6 drives in it for an iSCSI box and get 100+ MB/s sustained throughput to it over Cat 5e. I use the iSCSI volume for still image work (fast enough -- can't even tell it isn't local), and local RAID for video.

If you do go ReadyNAS, I'd recommend not mixing NAS and iSCSI. Throughput becomes bursty. Run a cable directly from the Mac Pro's 2nd gigE port to the iSCSI box, bypassing switches.

Go to the Storage Studio on the 4th or 5th flr @ Sim Lim Sq. They have a lot SAN recommendations. I've bought drives and enclosures from there and they know what they are talking about, which is very rare around Sim Lim.

Thanks Drew.

5th floor.

I will go and check them out.

Regards

Peter

I have just purchased this unit which will hopefully deliver a decent price/storage/ speed solution

http://www.pccasegear.com/index.php?main_p...oducts_id=13799

Does anyone know whether you can populate lets say 4 of the 5 drives on install then add additional drive later. Also can you mix the size of drives?

That depends on which RAID solution you use. RAID requires the minimum of 2 drives. Then the various combinations that would require however many drives minimum. With you can mix and match partitions with software RAID, the smallest drive size would be the base of the partitions used. Hardware RAID requires equal sized drives for maximum efficiency. Or you could just do JBOD but then you'll have many drives on your systems, which may confuse you. I use a RAID 50 and 60 combo on hardware RAID, with RAID 0 for the faster stuff that's already backed up.

I usually just donate my older server drives as a RAID unit to schools or give it to thrift stores for resale to charity. Just make sure you secure wipe your data sector by sector first.

That depends on which RAID solution you use. RAID requires the minimum of 2 drives. Then the various combinations that would require however many drives minimum. With you can mix and match partitions with software RAID, the smallest drive size would be the base of the partitions used. Hardware RAID requires equal sized drives for maximum efficiency. Or you could just do JBOD but then you'll have many drives on your systems, which may confuse you. I use a RAID 50 and 60 combo on hardware RAID, with RAID 0 for the faster stuff that's already backed up.

I usually just donate my older server drives as a RAID unit to schools or give it to thrift stores for resale to charity. Just make sure you secure wipe your data sector by sector first.

you've lost me a little there Drew, I was going to set it up as RAID 5, all as a single partition. After thinking about it I will probably fully populate it from the start anyway but it would be good later on when bigger drives become available to increase each drive at a time.

BTW what's JBOD

JBOD=Just a Bunch Of Drives

I'm going to convert my RAID 50 to RAID 60, although I'd rather have RAID100 but my RAID card doesn't support it, which is a pain. I find RAID 5 a bit insecure now with the long rebuild times due to the large disks available now, which is why I went 60. If a 2nd drive fails during rebuild of a RAID 5, the whole array data is lost! With RAID 60, that isn't true.

Makes sense, so RAID 60 is faster and allows multiple drive failures?

Pretty cool, the Thecus one only allow RAID 0,1,5, 6 and 10.

Is 10 better than 5?

I've ordered the Sonnet Fusion D400QR5 with 4 x 2Tb WD 7200 drives. That'll give me 6Tb of online storage with hardware Raid 5. I'll see how it goes using Firewire 800 connection but, if too slow, might invest in the Sonnet eSATA card for my MacPro.

I selected Sonnet because they seem to be a strong, state-of-art player in the drive enclosure business, they are one of the most Mac-friendly and they have a regional support capability here in Singapore. Bit more expensive but probably worth it in the long run.

My plan is to use two of my internal drives for WIP, one for FCS files and one for capture. Once a project is complete, I plan to use FCP's media manager to copy the used footage (plus a large handle) to the Sonnet drive. I've already got a dozen projects that I've finished over the last couple of years, ready to be "archived" to the new drive. Then I plan to re-capture the used footage from some older projects and archive the used captured footage off to the Sonnet too.

By archiving only the used footage I'll have nearly all the good stuff and none of the bad stuff. If I use my first "long edit cut" for the archive copy, I'll get most of the good stuff. I might lose a few gems that I did not include in the long edit but that'll be insignificant compared to the wasted disk space. I can't imagine a situation when I would want to dig through the footage that I rejected during the first long cut.

Does this sound reasonable or can you see flaws in my logic?

Regards

Peter