An idea for the industry re: RAW vs JPEG (test balloon)

[Rob]

Well, I am definitely working without a net here, so this may sound like a dumb idea or it may mean something. It may be something that is discussed a lot, and I just haven't seen it, or just way off base considering the new technology that is always coming out, such as with the HDR CCDs available, etc.

But, here goes -- if this is anything interesting to people, then I'd like to work it up into something more formal.

The basic idea is this:

RAW and Jpeg

Right now, there are two basic choices with images coming from digital camera: RAW and JPEG.

RAW is really for people who are more serious and typically takes a lot more time and knowledge to deal with. Whereas the JPEG conversion by typical cameras is just too simplistic and hard to deal with -- the highs and lows are usually cut off, and host of other problems can present itself.

I've been spending a lot of time researching the differences lately. With Sagelight, I try to serve both ends -- the higher end, professional level, including RAW support; and the easy more intuitive methodology of getting something that looks as great as possible with little effort.

What I've found is that JPEG really isn't that bad. It's the highlight and shadows getting cutoff where most of the problems come from. But, you don't necessarily want to restrict the rest of the image range, either (especially in 8-bits per-channel) just to protect something you may or may not need.

In some testing, I've taken highlight-protected RAW images, converted them to 8-bits per-channel and then saved them as a JPEG image. I then reloaded the and edited them. The results were much better than a typical JPEG from a camera.

Jpeg images, even at 8-bits per-channel, really is plenty to work with if you're not worried about getting very aggressive with your image, which most images don't require.

The problem with JPEGs from the Camera

The problem, of course, is that these cameras insist, rightfully, on attempting to generate an image that does not need editing. Something that is bright, noise-free, saturation, and balanced. Of course, 30% of the time this fails in some significant way, usually by cutting off the highlights. I use the term 'failure' to mean that it could be much better -- noticeably. Much more of the time, the image could be much better but looks ok if you're not looking at it artistically.

I firmly believe that not only is there a set of users that want to bring out the superior image without having to get into the complexities of RAW, but that this group represents 70% of the people who would use an editor in the first place.

Just Preserve the High and Low Data

With 16- or 8-bit JPEG, if only the highlight and shadow data were preserved. I don't have Highlight Recovery as the default when you load a RAW image for a reason. This is in order to keep the midtones as broad as possible so they don't become too compressed themselves in cases where there is no relevant highlight data.

So, how about this with Jpeg images from camera for some much higher quality: something like "JPEGr", if you will. A passive jpeg (8- or 16-bit) that carries the original image from the camera, but also has a "recovery data channel" which contains the upper- and lower-recovery information from the original image. It would be stretched -- anything from 0-.4999 represents the original 0-.249999, and anything from .5-.9999 represents the original .75-.999999 of the highlight information per-channel.

This data would be typically highly compressible without too much loss, so would probably be about half of the basic JPEG image size. This could be a specific type, i.e. jpgr, or could just be attached as some sort of EXIF information to the file. Either way, the additional information would be completely passive, so the camera can do what it wants and it would not interfere with any image viewers or editors.


Passive Metadata, making it easier to edit images in high-quality

Editors that are aware of this meta information can use it to either automatically restore highlights and shadows, or use it as an RD channel, akin to a layer, to allow the user to use curves (or whatever) on this information to restore it into the image.

The upshot is that this allows very quick editing at the JPEG level, but with a high-end spectrum that would deliver essentially the same professional results you see with many RAW programs -- but with far less hassle or need to invest the knowledge base one needs for RAW.

With RAW, for example, you have to think more in terms of noise reduction, sharpening, smoothing, color balancing, lighting, etc. etc.

The reason this comes about is because I've seen that the internal processing of cameras these days is not only better than it was before, but is also so tuned to the particular quirks and vagaries of that particular camera, it's more-or-less doing what you'd do anyway when you pull it out as a RAW. So, why not just let the camera do what it wants to do, as long we we can get the full histogram (without annoying users who don't care)?

This would be particularly useful in cameras (like my Powershot 780) where RAW makes less sense. But, also this would work well in larger cameras where one wants to edit their pictures in a significant way but doesn't really care about going to the RAW level.

What do you think? I also am thinking of the idea that these camera manufacturers could provide a proprietary .DLL (like a plug-in) to do the post-processing of the image AFTER it comes out of the camera as a RAW image. Sagelight and other editors could then call this DLL with various parameters. This would lead to greater control as well as higher quality because algorithms could be much more accurate since they don't need to save time as the camera does (for example, the L channel could be used for color noise reduction instead of the Y channel, which is mostly what happens now, leading to color loss

Anyway, those are my thoughts. If anyone wants examples of the RD channel attachment I'm proposing, let me know.

Again, this could be been-there-done-that in the industry, or a non-sequitur in the scheme of things, but, either way, I'm certainly seeing a way to get much better results with far less effort than using straight RAW.

Rob

[adb]

I usually work on a set of 20 or so images. Most of the time I now use RAW's because of the increased flexibility given by Sagelight. I feel that what would help, would be a couple of batch processes to pre (noise reduction etc) and post (sharpening, conversion to jpgs of a given MB limit, etc) process the full set of images.

[wwilson]

First off, I am not sure the industry is really looking to make third party image processing easier. Just look at how hard it's been for Adobe to get industry players to provide raw profiles and native DNG capability (whatever you may think of DNG, it's really the effort that Adobe has had to put into it that I am noting). Just recently I read a review of the new Olympus E-PL1 camera where the reviewer made the observation that this may be the best jpeg output of any camera yet. What this suggests is that the camera makers prefer to keep both raw formats and jpeg processing proprietary to themselves because this is where they think they can differentiate their cameras and/or drive innovation.

I know that there are many people who use a particular brand of camera because of it's 'look', the way it represents color, the way it balances detail and noise. Some brands, like Nikon, record the camera's settings for things like 'bold' or 'vivid' such that their raw processor can reproduce the look, and various levels of professional and amateur photographers are setting up their camera's to achieve a 'look', by setting saturation, color, noise and sharpening, contrast, etc.....

However, I do think that you are on to something with the notion of expanding jpeg, but it's the innovation that is going to drive such changes, not us technologists.
One idea that I have is that if you worked on the FlashPix extension (JPEG extension area APP2, of which there can be more than one) data then perhaps the mechanism for delivering more jpeg information is already at hand in an industry standard format.

[Rob]

Hi, adb.

I usually work on a set of 20 or so images. Most of the time I now use RAW's because of the increased flexibility given by Sagelight. I feel that what would help, would be a couple of batch processes to pre (noise reduction etc) and post (sharpening, conversion to jpgs of a given MB limit, etc) process the full set of images.

Thanks for the comment -- that's nice to know. I am working on the batching for the next version. I also want to look at the idea of having an batched/automatic adjustable RAW post-process, so that for whatever camera you have, it can just apply these settings, and then you can adjust them to suit the picture. It's a shame that all editors to things differently, sometimes dramatically. It would be nice to have established camera profiles that have the information to do basically what the camera does to get the same basic low-level pre-processing that you can control.

It would also be nice to do this as a process that pre-processes a number of RAW files but leaves them in an "editable" state, but not going all the way with them -- just the standard and user-defined pre-processing -- and saving them as 48-bit tiffs or whatever.

Rob

[Rob]

Hi, wwilson.

First off, I am not sure the industry is really looking to make third party image processing easier. Just look at how hard it's been for Adobe to get industry players to provide raw profiles and native DNG capability (whatever you may think of DNG, it's really the effort that Adobe has had to put into it that I am noting).

I know that the camera manufacturers want to keep a lot proprietary. So would I. The post-processing I'm seeing in these cameras, I think, warrants it. I definitely prefer some cameras over others specifically because of how the image is post-processes. There was a time where I wouldn't touch a certain camera manufacturer's cameras because of the jpeg compression algorithms. They've come past that, thankfully, but it shows how much of a difference there is between cameras. I think it's more now because of these 12MP CCDs in cameras that, realistically, don't need them.

With Adobe and DNG. I think there is a reason for that. I don't know all there is to know about the DNG format, but I do understand there are limitations, and I think its fair to be wary of a format coming from a product manufacturer than, say, a non-profit group or association. As a camera manufacturer, I think I'd feel that things are still changing so much that I don't want to be roped into supporting something that may limit me in the future. If it became the standard, then a company could suffer the consequences for supporting it and then wanting to move away for a valid reason. The Fuji RAF format comes to mind -- it might be compatible with DNG, but I can certainly see where it is stretching into areas that could outgrow the current DNG standard. I also understand that there is a loss of quality between some RAW formats and DNG.

Some brands, like Nikon, record the camera's settings for things like 'bold' or 'vivid' such that their raw processor can reproduce the look, and various levels of professional and amateur photographers are setting up their camera's to achieve a 'look', by setting saturation, color, noise and sharpening, contrast, etc

For me, that's exactly the issue. I don't want to take away the things the camera wants to do with the image, but actually add to it by preserving the post-processed information that is cut-off in favor of making a picture -- that cannot be understood contextually by the camera -- presentable to the user on a generic basis.

My thought is this: let the camera do what it wants to do. But, for people who want to enhance or otherwise edit the picture, the only way right now it to lose ALL of that information and to start from scratch, not to mention deal with an unwieldy RAW-sized image. I'm really suggesting a JPEG version of the FUJI RAF format, but whout the separate sensors. I'm proposing, essentially, an S channel but based on the RAW information that is necessarily cutoff or too compressed in the JPEG image. This also helps preserve the midtone data (otherwise you'd have to compress that in favor of the entire histogram).

I've been doing a lot of research over the past couple months, and I've found just this one thing: if we could preserve just this data -- without respect of the camera doing whatever it wants to do -- 20 times as many people could enjoy the benefits of making pictures that much better, not just the people who currently use RAW. As I noted, JPEG really does have the range if we could just extend the low and high ends of the histogram. I would also imagine that 16-bit JPEG will come into play soon enough with these cameras. So, with 16-bit data as well as the shadow/highlight preservation, you could do 95% of what you would ever do with a RAW image with no perceptible difference.

I'll look into FlashPix -- it's always so much easier to start in a place that's already going!

Rob

[lutz]

Hi,

I am not sure how welcoming the "industry" or even the public is to new file formats. Thinking for example of JPG2000 or JPEG XR (formerly Windows Media Photo and HD Photo) - both formats likely substantially better than JPG and both more or less dead.

Your idea (if technically feasible) sounds of course very appealing. It might have a few similarities with the XDepth system which is a bit more than one year old but also does not seem to have caught on. This seemed to be very promising:
http://raw.xdepth.com/xdraw_news.php

[Rob]

Hi, Lutz.

I know it's hard to get anything going. As I mentioned in the previous post, I find myself getting quite evangelistic about this particular issue, because it's clearly ruining pictures that are in actuality just fine. I think this is because it really would be that easy to make enhancing images for the regular person that much more interesting, not to mention getting much higher-quality images.

In this case, I have developed examples and prototypes. Also, this was the reason I put it out as a passive format. I think part of the problem becomes requiring software, changing standards, etc. In this case, it would be passive so you'd only use the data if you want to use it, and the current implementation doesn't change at all. It's metadata, basically, so it's not changing anything.

I'm hoping that with the examples I've been working out, this could be more viable. I mean, pictures are much more interesting than text.

You would think more advanced JPEG standards would have taken off. I think the real problem is backwards-compatibility. If a camera, for example, could write out a 16-bit JPEG that an 8-bit reader can just passively read without knowing the difference, I think you'd see that come along much faster. I don't think anyone wants to be the first to support something that suddenly requires extra software or more money -- or is just plain different (and, therefore, less desirable) than the competition.

I'm looking at something very discreet, and it would be great to see it be refined (or joined with some existing project) into something that becomes more compelling. Especially with before/after pictures.

With XDepth Raw... I just looked at the website. It just looks like a commercial product. It looks nice, but I'm not catching on to how what they're doing could or should change or add onto industry standards -- though, it looks like they're using lossless JPEG compression, and you'd think that would catch on (but I don't know how good that is, either).

Rob