Big storage

I'd be interested in a group buy too Eric. I'm planning to buy the same setup - the NV with either 400 or 500gb drives.

James

Great, guys! I'll work on it.

OK, but please don't post how little it ends up costing you! I will be green with envy. :-)

Alex

yeah me too...

/paul

I read somewhere that there are two types of hard drive - those that have failed and those that will fail - a bit cynical but...

Seriously though, I have a collection of 12 varying hard drives - with two configured as a RAID set-up. BUT I also back up ALL my Raw files onto DVD too and am about to start adding a second set burned onto Delkin Gold DVDs - not a perfect solution but the best I have been able to come up with so far.

If anyone has spotted a fireproof DVD safe, I'd be interested to hear about it.

After working with too many external drives (USB and/or Firewire) and experiencing various problems, we settled on a "pseudo-server". Simply an Antec case which can house many peripherals and has sufficient ventilation. This was an affordable option since we already had the necessary drives.

We bought a proper motherboard, and fitted the "monster" with a total of seven (7) drives and Windows XP. One HD is the C:\ drive and three others are permanently mounted. The last three are mounted in ventilated drawers. Those three drives are actually mirrors of the permanent ones. All but C:\ are 300Gb Seagate (5-year warranty). We did not go RAID for fear of our limited knowledge and total hate of computer problems.

Once a week, we swap the drawers with those we keep off-premises.

We use a freeware called SyncBack to keep all drives synchronised and it works fine.

Photo bank and presentations (Watchout and Powerpoint) are also copied on DVDs.

Sounds like paranoia but we can't affford to loose that stuff.

Not the most sophisticated and expensive proposal but it works... so far.

If my posting works, there should be a picture of the "monster" enclosed.

Enough computer talk... let's go back to u/w photography.

Michel

Forgot to tell you that there are so many fans in the "monster" that it sounds like a B-17 ready to take off!

Michel

Once I got past a compatability issue (it didn't like the maxtor drives I had) I fell in love with my ReadyNAS NV.

Count me basically in on a group buy as well, but it will need to be in August (I'm gone much of June/July) for me to participate, so don't wait for me if everyone else is ready to go. I'd be just as happy to buy a barebones system which I'll populate with SATA drives from the local CostCo.

The group buy is underway. If you're interested, please contact me ASAP. Continental USA only.

I know this may be a low-tech solution, or at least a s/w solution, but are people converting their RAW files to Adobe Digital Negative format (.dng)?

This is by far the best way to recover about half the space your RAW files take and have a format that is much more usable in the future when camera manufacturers "discontinue" their support for old formats.

it is very easy to do on a trip, simple open up all your keepers in CS RAW and save as .dng. Then you can delete the original RAW files (I know it took me courage the first time!). The dng format works exactly like RAW, but no sidecar files are then necessary and I save about half on my huge E900 RAW files which are 18 MB!

Just something that I think should be part of everyone's workflow and saves a lot of space.

Jack

You don't actually have to open the RAW files in Camera RAW. Instead, use the DNG converter: http://www.adobe.com/products/dng/

Also, some RAW formats are already compressed. For example, my Canon 1ds Mk II RAW files are smaller than your Oly RAW files.

but certainly, if your camera produces huge raw files, converting to DNG will save a lot of space!

You guys have just to much $$$$$.

I have a couple of the 500Gb LaCie drives on my G5.

I can attest to the issue of RAID failing and becoming unrecoverable. Happened to me once.

I now have a tape drive that uses LT02 Ultrium tapes (200 Gb native).

http://www.quantum.com/Products/TapeDrives...-2HH/Index.aspx

I now use one of the 500Gb drives as my main, with a daily backup to the other 500Gb drive and a weekly backup to the tape. Tapes go off-site to the safe deposit box at the bank.

May seem like overkill but the value really is in the data in the end

I just had my first hard drive failure and so thought I would add onto this post as I am looking at better back up solutions ..

I am thinking about just a couple of USB2 or firewire (I have firewire ports I never use)

I was thinking there must be some software that can smart detect changes to a hard drive and update another drive automatically. (like a mirror I guess)

I don't get what RAID is .. but a lot of systems say they come with it .. but then there's different levels I looked here and got more confused

My current set up is 2x 200 gb Hard Drives.

Drive 1: Programs and settings

Drive 2: My personal files and media .. (almost full .. now dead)

I have a 120gb external USB drive that has Microsoft Back up files on it ..

I would like to have 2 back ups of everything .. That will probably be about 200gb worth of data. I feel getting a seperate box is a little over kill for me, but I don't know.

I think something like The Lacie Brick with SyncBackAuto Sync the drives or perhaps incremental updates ?

However there is the other option of something bigger that seems more prof ?

oh and I saw this other software that looks quite good too

I was hoping to get osme opinions ?

I want it to be a live back up of my current system and files as I change it .. and I want it to be quick and simple and zero work for me .. I am lazy.

I am thinking about just a couple of USB2 or firewire (I have firewire ports I never use)

I was thinking there must be some software that can smart detect changes to a hard drive and update another drive automatically. (like a mirror I guess)

I use the program syncback. But there are countless other like it.

I don't get what RAID is ..

RAID stands for 'Redundant Array of Independant Disks' (or you can replace the I with Inexpensive, depending on who you ask). The commonly used RAID levels (or types) are 0,1,5.

Raid0: you dont want this if you want your data to be safe, unless you dont care about the data. This basically concatenates the disks into 1 large virtual disk. This means any disk failure will destroy the whole virtual disk. 2 500 GB disks give you 1 TB of data.

Raid1: this is sortof what you describe further down. It's also called mirroring. Each disk in a raid1 array has a counter disk, so it's always an even number. 2 500 GB disks gives you 500 GB of useable data, the other 500 is a live mirror. If any disk fails, the array can continue on the other disk while you rebuild the array with a new disk. This is normally all without interruption to the desktop machine.

Raid5: Here you always have whats called a checksum disk. Raid5 can only be done with 3 or more disks. With 2 disks it would just sortof be raid1 with 2 disks. In raid5 the raid software takes each block on all available data harddisks, and produces a checksum which is written on the block on the checksum disk. Simplified you could look at it like this. Say disk1 has the number 5 on block 1, and disk2 has the number 7 on block 1. Then the checksum disk has the number 12. Now if disk1 fails, the software sees 7 on disk2, and 12 on the checksum disk, and knows disk1 had a number 5 there. In case a disk fails in a raid5, the system continues in a degraded (slower) mode because it has to calculate all that data back. So, a raid5 with 3 x 500GB has 1TB of useable data with 1 disk 'lost' to the checksum disk.

In both raid1 and raid5 you can configure a 'spare' disk, which will automatically replace a failed drive. In both cases the raid software will have to rebuild the data, which can take anywhere from hours to days. During that time you are extremely vulnerable. Another disk failure will mean full loss of data. (there are other methods that can survive multiple disk failures but those are normally not for home use)

I think something like The Lacie Brick with SyncBackAuto Sync the drives or perhaps incremental updates ?

That would probably be fine for some time to come. Just make sure you stash one of your 2 backups someplace else

I dont think true image is what you want. I would avoid any kind of Windows software based disk mirroring. If such a thing even exists. Things like Syncback are not live mirrors. I run mine once a night automatically.

I want it to be a live back up of my current system and files as I change it .. and I want it to be quick and simple and zero work for me .. I am lazy.

If you absolutely want a live backup you probably need to get a raid1 or raid5 system. Imho you should get 2 though, so 1 raid system is a backup of the other, and then store the backup someplace else. I realise thats a hassle, but theft or fire are much bigger dangers than disk failure.

Ive actually been looking into this for home use recently also. I currently use firewire based external disks. I have 9 external disks now. 3 main disks, 3 backups, and 3 external backups I keep at my office. But this doesnt scale well. If only because windows can only handle so many firewire/USB2 harddisks.

Im now looking into eSATA arrays. Im probably going to buy a disk array that holds 5 eSATA disks. This is in theory much faster than firewire or USB2, as it basically acts as an internal harddisk. All you do is pull the SATA cable outside the desktop case. Unfortunately I'll have to buy 2 arrays, as I will not rely on RAID alone for my backups. Im in the lucky position that I can keep my 2nd array on another system and keep the 2 arrays mirrored over my DSL line. Too bad that ReadyNAS isnt eSATA.

Cor

This is a great thread and still going.

Giles: check out Eric's journal entry about storage, he has a good one.

A "quick and dirty" way to get started is to buy an external HD and have your desktop do a backup to it (incremental) every night.

Next up is to put a RAID card into your computer, but that means you should start from scratch and reinstall windows on that new partition.

I have a RAID 1 setup in my desktop with two partitions (4x 200 gig disks) with one 200g partition for system and one for files. I back that up periodically to a 500gig external Seagate drive.

And now, thanks to Eric and Wetpixel I have a 1.6TB ReadyNAS NV. It has 4 x 400gig drives in it with 1093GB of usable space.

Now I do my backups to that.

Cheers

James

Cor Thanks a lot .. thats some really usefull information for me .. and most of it phrased in a way a pretend geek can understand it !

Giles: check out Eric's journal entry about storage, he has a good one.

<{POST_SNAPBACK}>

A "quick and dirty" way to get started is to buy an external HD and have your desktop do a backup to it (incremental) every night.

<{POST_SNAPBACK}>

Next up is to put a RAID card into your computer, but that means you should start from scratch and reinstall windows on that new partition. 

<{POST_SNAPBACK}>

Does that sound feasable and sensible ? I am thinking a 500gb RAID 1 systems and the back up drives are probably 250gb each.

PS ... my old hard drive is fine .. it's the chipset on the drive that is screwed. My last resort is to send it back to the shop i bought the pc from in the UK .. however I was wondering if there was a cheaper solution in the states that anyone knew of to get the harddrive ghosted. Now I am off home to see what my back up contains.

...Simplified you could look at it like this. Say disk1 has the number 5 on block 1, and disk2 has the number 7 on block 1. Then the checksum disk has the number 12. Now if disk1 fails, the software sees 7 on disk2, and 12 on the checksum disk, and knows disk1 had a number 5 there...

I just found this thread, and most of it (including the above "simplified" explanation) are WAY over my head!

A few questions:

1. It sounds like the group purchase has already taken place(?) - is that the case (and I'm too late)?

2. I didn't see any discussion of "online" archival services, where you upload files to a website and pay them to store archives for you, which I assumed would be a good backup resource. Are none of you using anything like that? If you are, which one(s)?

Hi Bruce,

You can probably still get in on the Group Buy. I'd contact Adam Tow for details as he was the organizer. Do a search on his name and you'll find Adam's website in a jiffy.

Cheers

James