Friday
Dec072007
If your proprietary RAW workflow dies
Friday, December 7, 2007 at 9:13PM
Nobody likes to think about faithful friends dying, but Micah Walter has brought up again a subject that I try not to think about very much: data lock-in. As data-locked-in computer users go, I'm not very locked in. I mostly live in text files, source code, email and the web. I have some confidence that my source control repositories will be readable for some time to come and that any migrations from them will be automatically handled by the next source control system to come along (you can already see that happening with Git-svn).
My photography, however, is another story and it rather scares the pants off me when I think about it too much. I have hundreds of gigabytes and tens of thousands of images on my hard drive, all of which represent precious memories and some of which are pretty good photographs in their own right. Pictures of my children's births, their first birthdays, family holidays, great trips I've taken, places I've been and so on. Years of my life in visual form.
But all of that is locked into proprietary formats. All of it. Firstly, the RAW files are Canon CR2 files from an EOS 350D and a 30D. That's the first thing that scares me. What happens when the 30D is a twenty-five-year-old camera? I'll only be in my early 50s when that situation arises and hopefully still doing a lot of photography. What will the computers and operating systems look like in the year 2032? Is Apple really committing to build a RAW decoder for the 30D into every future version of Mac OS X, Mac OS XI and the new-for-2030 A.D. Mac OS XII? What if Apple 'starts over' again with another OS once X has seen better days? Will they build RAW converters for prehistoric digital cameras?
If you think this is a silly question, think about the word processors that were common in 1982 and ask yourself how many of their files can be read in Pages or Microsoft Word today with zero loss of fidelity.
So that's just the RAW files themselves. I'm also locked into Aperture. I'm not locked in through metadata, but I am locked in through Aperture's proprietary database of all the image adjustments I've made to each of my photos. This concern isn't even one for the future. It's one for right now. If I wanted to, how would I take all my RAW masters and move to Lightroom, preserving the edits I've made in Aperture and keeping all of those edits non-destructive? I simply don't think this can be done today.
Now I have no evidence that Aperture is dying - I do have some circumstantial evidence that suggests the exact opposite - but one never knows what the future holds. We can't even predict five years hence in technology, never mind what could happen within the useful lifetime of important photographs shot in RAW today.
What to do? I honestly have no idea, and it's worrying to think about. Textual documents you can always read and sometimes get some content out of them, maybe even recreate important documents in modern formats. You can't recreate images. The best you can hope for is that you'll be able to transcode the RAW format into some modern file type. You can take metadata with you through XMP, but what about those image adjustments?
My photography, however, is another story and it rather scares the pants off me when I think about it too much. I have hundreds of gigabytes and tens of thousands of images on my hard drive, all of which represent precious memories and some of which are pretty good photographs in their own right. Pictures of my children's births, their first birthdays, family holidays, great trips I've taken, places I've been and so on. Years of my life in visual form.
But all of that is locked into proprietary formats. All of it. Firstly, the RAW files are Canon CR2 files from an EOS 350D and a 30D. That's the first thing that scares me. What happens when the 30D is a twenty-five-year-old camera? I'll only be in my early 50s when that situation arises and hopefully still doing a lot of photography. What will the computers and operating systems look like in the year 2032? Is Apple really committing to build a RAW decoder for the 30D into every future version of Mac OS X, Mac OS XI and the new-for-2030 A.D. Mac OS XII? What if Apple 'starts over' again with another OS once X has seen better days? Will they build RAW converters for prehistoric digital cameras?
If you think this is a silly question, think about the word processors that were common in 1982 and ask yourself how many of their files can be read in Pages or Microsoft Word today with zero loss of fidelity.
So that's just the RAW files themselves. I'm also locked into Aperture. I'm not locked in through metadata, but I am locked in through Aperture's proprietary database of all the image adjustments I've made to each of my photos. This concern isn't even one for the future. It's one for right now. If I wanted to, how would I take all my RAW masters and move to Lightroom, preserving the edits I've made in Aperture and keeping all of those edits non-destructive? I simply don't think this can be done today.
Now I have no evidence that Aperture is dying - I do have some circumstantial evidence that suggests the exact opposite - but one never knows what the future holds. We can't even predict five years hence in technology, never mind what could happen within the useful lifetime of important photographs shot in RAW today.
What to do? I honestly have no idea, and it's worrying to think about. Textual documents you can always read and sometimes get some content out of them, maybe even recreate important documents in modern formats. You can't recreate images. The best you can hope for is that you'll be able to transcode the RAW format into some modern file type. You can take metadata with you through XMP, but what about those image adjustments?
Reader Comments (10)
Already twittered this a bit, but for other readers I wanted to note that http://cybercom.net/~dcoffin/dcraw/ seems pretty committed to keeping the images off every camera usable. And of course excepting DNG it's focused just on camera output, not proprietary image/database formats like Aperture or Lightroom's.
I'm kinda dealing with the same thing in iPhoto, trying to free some hdd space by batch replacing some unnecessary RAW photos with JPEG versions, but not wanting to completely lose my occasional adjustments. At least with iPhoto, most of the other metadata (albums, ratings, comments) is publicly accessible.
Glad to see this discussion started, but unfortunately, like with office documents, this happens in most every problem domain for reasons of both practicality and economics.
As I start to amass a pile of NEF raw images next to my pile of JPEG images, I've asked myself the same questions. I can't say that I'm entirely comfortable with RAW as an archival format (and that is ignoring the meta data and adjustments issue that you bring up.)
Anecdotal data point: I nearly lost a small number of images I took with a Kodak DC 120 many years ago when I made a platform transition (i.e. no more Graphic Converter on Classic) that orphaned them. As it turns out I was able to convert them, and the images are technically lousy and not good for much besides small prints. But they were the first (and so far only) casualty in the proprietary camera files saga.
JPEG support is likely to outlive any proprietary camera format on whatever platforms we'll be using in the future. But then you give up the benefits of RAW today (or complicate your archival strategy.)
I haven't decided which set of compromises is best for me, so for now, I keep shooting a mix of RAW and JPEG (depending mostly, but not entirely, upon which camera I'm using.)
Congratulations, Fraser, you just sent a huge shiver down my spine. If anything, it's vaguely comforting that there aren't *any* nondestructive editing programs available that don't suffer from this problem. I'm not really convinced that (present-day) DNG is a solution to the data lock-in, either. I wish there were a way we could effectively pressure/lobby camera manufacturers to develop and support an open format for their RAW files. I don't see what the advantage is to keeping those proprietary, anyway...
@blalor: RAW formats will be forever and of necessity somewhat tied to the implementation of the sensor hardware with which they were captured, I suppose.
There's no real good reason why they can't be documented, though, and I've yet to hear even a plausible justification for Nikon encrypting the white balance information in NEFs (are they still doing that?).
The encrypted NEF data is the biggest reason that I bought Canon gear when I got back into photography a couple of years ago. That'll show 'em! :-)
I'm sure Canon will still be around, so we can at least hope that they will provide some way of bringing forward "old" CR2 files into whatever their new editing software is... But you might have to buy the latest camera to get it.
What about TIFF as a way of preserving as much data as possible in a non-proprietary format..? Seems to me that's been around a long time. I know it won't get us from the RAW back to the adjusted image, but it's a start.
This is an issue for all of data archiving - what about museums that have digital archives..? And it's not just the format that the data is saved in, it's the media too. You see people worrying about whether their CDs and DVDs are of 'archival quality' and will be readable in 30 years time, but who still has an 8in floppy drive, a 5.25in one, even a 3.5in or an Iomega Zip drive..?? I've heard reports that even USB floppy drives don't work reliably on all current systems let alone future ones.
What are we going to do....
I don't see this as being the problem it is made out to be. It *is* a problem, but just not the one it is made out to be.
There are 8" floppy drives around and people who will charge you plenty of $$ for using them. There is a business model for data recovery and archive transfer, but for it to be applicable to your needs, your data must be worth at least as much as these services are charging.
So the key to surviving the changes that lie ahead is in knowing which of your images are actually valuable and beeing able to access them all in one place. This will minimize your costs and maximize the payback.
Aperture stores its metadata in a sqlite3 database. The schema it uses is kind of obtuse from what I understand of it, but at least the database format itself is completely open.
I'd imagine that if Aperture were to die out, there would be a considerable market for software that could read those databases and convert them into a format that could be read by whatever replaces Aperture.
Hell, there might even be enough of a demand *today* for someone to develop a tool that can convert Aperture databases into Lightroom's format....
I think I might start printing my 'important' photos. It worked well as an archive mechanism in the days of film, and let's face it, in 20 years will I just want to look at the pictures or will I still be adjusting the white balance et al on ancient shots? I sincerely hope it's not the latter as I've barely time to process the images I'm capturing right now. You might not be able to read that wordperfect document from a few years ago but I think paper has a useful life of a lot more that that. Yes it's susceptible to fire/theft etc., but stuff happens, right?
Without trying to pooh-pooh digital photography: Rollei's current polyester film bases are archival rated for an estimated 500 years or so with correct processing and storage.
I recently dug out some 75 year old negatives that my grandfather took and scanned them with reasonable success (more to do with my dubious scanning technique than anything else).
In a money-no-object way, what might be advantageous is a 6x9 film printer: transfer your valuable raw images onto medium format film. To me the advantage is clear: a film transparency contains the entire image in a platform-independent, workflow-independent physical medium not subject to cruft. What was done to the image in a workflow could still be encoded -- or just written down -- for future application.
While digital is doubtlessly the future in terms of what we can do with our photographs, I personally think it would be foolish to discard the valuable work in terms of long-term storage that was done for film.