What is the optimal image storage strategy?
I have been using C1 since version 10 which doesn't mean that I am an expert. I need help, I'm an artist and I typically shoot 100-300 images a day, 42mp Sony raws, sometimes I shoot tethered but most often not, I import the images into C1 on my MBP and after they are imported I put the card into my 5K iMac and do it over again. I have a 10TB drive partitioned 4TB for time machine and 6TB free for whatever, C1 is backing up my images there. I really only edit on the iMac, bigger and better screen, and have been using the MBP as a fail-safe backup. What I think I would like to do is have a drive that acts like a server where the images go and I'm not making duplicates of duplicates and if I feel like tweaking any of them on my MBP all of the adjustments will be there when I next view them on the iMac. If I need to get another large drive to back up the back up I'll do so. I guess what I'm looking for is a road map of how to structure my storage. I have 10's of thousands of raw files dating back over 20 years all sitting parked on drives not connected to anything. I know how to connect them up and import them, once I have the strategy for importing them all it will take is time. I was a motion picture cameraman for 30 years and I didn't have to worry about any of this, now that I'm back to making art I need some help in organizing my legacy so that it doesn't disappear because of an electrical surge due to lightning. I'm open to any ideas that will make this process simpler and more robust. C1 will soon have iPad capability so having a central place that can be accessed from any device would be very nice. Any ideas? I know that there are dozens of you out there that have solved this problem quite elegantly. The genius of C1 is it's ability to data manage, I need advice on how to hook it all up beyond the desktop iMac. Thanks in advance for any advice you might give me.
-
Since nobody else has offered a solution, and since there's no "one right one", I'll suggest that you look at the post-processing forums at fredmiranda.com and DPReview.com. Variants of your question have been asked on both many times and lots of suggestions have been made. One or more may work well for you.
0 -
In the past - many years ago - I experienced silent corruption of files and it made me really feel bad, even though in the end it affected only a handful of files. With SSD it should be much less frequent, but it can happen. Backup do not necessarily work in this case - in mine didn't: in spite of having multiple ones, in the end the corrupted file ended up in the backup... Only a read-only backup would work (they are pretty good with RAW files given their nature), such as those on optical disk; I do have them too, but I make them when I accumulate enough culled images to fill one, and in that case it hadn't happened yet.
So I took the habit to compute a MD5 fingerprint of each file and store it as an extended attribute of the file. In this way, the info is stored with the file, but without affecting its contents. See here for more info:
https://en.wikipedia.org/wiki/Extended_file_attributes
I wrote a small tool for this task: it scans my folder, assigns the fingerprint attribute when it sees a new file, and compares the current fingerprint with the previous one, emitting a warning in chase they changed. I run it periodically, before performing a backup.
Thanks to extended attributes the tool doesn't require a database catalog to maintain, and attributes are copied along with files everywhere.
The tool is available as open source - please note that it's NOT a product, that is I can't maintain it, it's just for internal use. But if you have questions about it, I'd be glad to aswer.
https://bitbucket.org/tidalwave/solidblue3-p-src/src/master/
For people that don't want to deal with programming and scripts, I'm pretty sure there are similar applications out there ready to use.
0 -
If you want to avoid Fabrizio's scenario, one possible way is too include your picture folder / drive in TimeMachine, but if bits are rotting, that wont help. Currently I'm using two (yes, I know, not enough...) drives to backup the RAID. Optical media? DVDs are 4.7 GB, right? None of my main catalogs is that small...
Problem is, witn each backup we can't check, if all files are still physically ok, we'd even backup a corrupt catalog without knowing. And I don't dare to say "from a couple of 100.000 images I never had a problem", because I don't think I'd remember each problem and even if it would be true - who tells me that right now all of them are still ok?
EtMRS, I've read a couple of articles of a couple of NAS causing problems, some of them due to electrical issues. And which NAS is as fast as TB3?
I use an external SSD for the catalogs, a SoftRaid with 4 HDs for RAW storage and for backups just ordinary USB disks, cheap, slow, simple - but not only one. After importing, I start SuperDuper for an incremental backup. After finishing a project, again SD, but a different drive as target. Also, the catalogs have two targets for backup which are not on the other drives.
0 -
one possible way is too include your picture folder / drive in TimeMachine, but if bits are rotting, that wont help.
Why not? An advantage of Time Machine is that it keeps ALL versions of a file, not just the latest. If your file was corrupted 9 months ago then go back to the version that is 10 months old.
My current Time Machine backups started in Jan of 2019. If something was corrupted before that I'd plug in the previous Time Machine disk. Yeah, when my TM disks start to get full I start over with a larger disk instead of letting TM erase old files.
0 -
Lots of interesting stuff...
1. NAS with BTRFS. Yes. I do have it. Probably it makes fingerprints much less useful, but I've kept the habit of doing them. Consider that not all filesystems offer data integrity: e.g. APFS doesn't (it does only for metadata). Which is a big shame, considering that the first major filesystem with this feature was ZFS, released 15 years ago (and Apple refused to integrate it, taking instead ten years to develop APFS). The cost in user time to compute and check them depends on the hardware speed. The SDD of my MacBook Pro 16 is rated at 3Gb/s; in practice I see that computing MD5 of large files works at 450 MB/sec. 1 TB is done in a hour considering that peak speed is not kept all the time. You can schedule it once in a while overnight, or during lunch, and you don't waste your personal time on it.
2. Keeping all the old copies. Yes, it's a good strategy (it works with many software tools in addition to TimeMachine). You just need to go large when you buy disks.
3. Optical disks. I also do backups on optical disks, even though it's still partial (I'm not consistent with it). I started with 4,7GB DVDs and then went with 8GB DL. There is also BluRay, with a standard size of 80 GB. Actually one year worth of photos of mine fits it. It's quite slow to do, but you can do once in a year. The big advantage of optical disk of course is that they are not sensitive to electromagnetic fields. I mean: my NAS is behind by a surge protector + emergency power supply, but what could happen if a thunderbolt falls ten meters off my house? The problem with optical disks is that they are far from being eternal, indeed the last time I checked the literature (honestly many years ago) they were considered reliable for 5-10 years. I know that there are relatively recent technologies such as M-Disc specifically designed for long term storage, but so far I didn't have the time to study the argument. I can say that my oldest optical backups are still good after 15 years... (how do I know? Fingerprints, of course!).
In theory you could schedule periodic checks of fingerprints of your optical disks and replace copies when you detect a fault - I wrote a small tool that does this taking note of the time of the last check for each unit, so it can warn me when a new check is due, but the thing is so boring and time expensive that I don't do it in a consistent way.
4. There's the cloud, of course, which works if you aren't concerned by privacy. I have a lot of objections to cloud, so I don't use it. In theory if you strong encrypt your stuff locally before pushing it to the cloud you should be safe. The problem is what happens in case somebody steals your key...
0 -
Speaking of optical disks: once I read about the vice versa project of scanning films: plotting digital prints on film as film has a rather long lifetime and doesn't need electrical power to remain stored. Also, as "reading device" a projector is a rather simple thing compared to the many different electrical storage systems. My negatives survived 40 years and the way they still look let me expect 20...40 years more. Although no archival room conditions are taken to store them.
And frankly, I don't need to know my pictures will survive me.
0 -
You can plan a lot beforehand, but "reliability" unfortunately not often sticks to the (too optimistic) plans of hardware manufacturers in terms of expected service times. And since in cloud arrays somewhere in the internet also bad things can happen (like fires or floodings) it might be good to expect occasional failures and not rely on a single backup system. Especially the idea of cloud storage is tempting to transfer your responsibility to somebody else. I prefer to keep the risk on my side.
0
Post is closed for comments.
Comments
7 comments