Dark NEF image compared to other RAW viewer

[runfalk]

I tried out the library with NEF files from my Nikon D7000. I have used Geeqie before and I find that the images are much darker in rawloader:

On the left the raw file is displayed by Geeqie and on the right the same image decoded by rawloader. I decoded it into a PPM image and then used imagemagick to convert that to a TIFF.

The code I used for converting to PPM is based on the one from the README:

fn srgb_to_ppm(filename: &str, image: &rawloader::SRGBImage) {
    // Write out the image as a grayscale PPM
    let mut f = BufWriter::new(File::create(format!("{}", filename)).unwrap());
    let preamble = format!("P6 {} {} {}\n", image.width, image.height, 255).into_bytes();
    f.write_all(&preamble).unwrap();

    for pixel in &image.data {
        f.write_all(&[*pixel]).unwrap();
    }
}

let nef_raw = rawloader::decode("taipei101.nef").unwrap();
let nef_srgb = nef_raw.to_srgb(0, 0).unwrap();
srgb_to_ppm("taipei101.ppm", &nef_srgb);

To convert the PPM to TIFF I used:

$ convert taipei101.ppm taipei101.tiff

I can provide the NEF if necessary.

[pedrocr]

Thanks for the report. I don't know your level of experience with RAW files so forgive me in advance if this is too basic of an explanation. Two things are happening:

First what geeqie is showing you has nothing to do with the raw data. Geeqie is simply reading the embedded jpeg thumbnail in the NEF that's been processed by the camera. It's as if you had loaded a camera JPG instead of a NEF.

Second rawloader's main purpose is to decode raw data, as in the mosaic bayer data (one color per pixel) that the sensor actually captures so that other software can do all the interesting things it may want to do with that data. For convenience and as an example rawloader also ships a very simple raw pipeline that is able to create linear or sRGB images from that data. Raw images are linear and require a base curve to look good. Right now the rawloader pipeline includes a very simple curve:

https://github.com/pedrocr/rawloader/blob/a771d48f1fffe3b6b1d1b36800720ef053481cf1/src/imageops/ops/curves.rs#L14-L18

There's probably a better standard curve that works for more cases, I haven't invested a lot of time making it better but it should be at least reasonable for most cases.

Eyeballing the NEF I'd say the first effect is the biggest and that fancy auto-adjust features like Nikon's Active D-Lighting are enabled in the camera and it has adjusted the picture from a darker RAW into that JPG thumbnail. I can have a look at the NEF to see if that's the case.

[runfalk]

Thank you for the thourough reply and explanation. I am indeed quite the novice when it comes to raw formats. Funnily though I actually learned about debaying earlier yesterday when I tried to wrap my head around the code in rawloader.

I was under the impression that the raw image's embedded JPEG has way too low resolution to use on a computer. When I've tried to open NEFs they sometimes show a tiny thumbnail.

I made a side by side comparison with Photoshop's CameraRaw as well (without touching the sliders):

Here is the NEF i use: taipei101.nef.zip

It does indeed seem like Active D-Lighting is set to Normal, which I assume is on. Does this mean Photoshop recognizes that the image has been shot using D-Lighting and apply another curve to it? To me it would be preferable if the output was similar to what I see in CameraRaw.

I am willing to experiment with this. Are you interested in pull requests for features like this? Optional automatic rotation is the thing currently on my TODO list.

Here is the EXIF data from exiftool:

ExifTool Version Number         : 10.40
File Name                       : taipei101.nef
Directory                       : .
File Size                       : 18 MB
File Modification Date/Time     : 2017:03:24 11:31:08+01:00
File Access Date/Time           : 2017:03:25 10:29:58+01:00
File Inode Change Date/Time     : 2017:03:24 11:31:08+01:00
File Permissions                : rwxr-xr-x
File Type                       : NEF
File Type Extension             : nef
MIME Type                       : image/x-nikon-nef
Exif Byte Order                 : Big-endian (Motorola, MM)
Make                            : NIKON CORPORATION
Camera Model Name               : NIKON D7000
Orientation                     : Horizontal (normal)
Software                        : Ver.1.04
Modify Date                     : 2017:01:18 16:11:27
Artist                          : 
Jpg From Raw Start              : 180480
Jpg From Raw Length             : 1224663
Y Cb Cr Positioning             : Co-sited
Subfile Type                    : Full-resolution Image
Image Width                     : 4992
Image Height                    : 3280
Bits Per Sample                 : 14
Compression                     : Nikon NEF Compressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 1405152
Samples Per Pixel               : 1
Rows Per Strip                  : 3280
Strip Byte Counts               : 17718873
X Resolution                    : 300
Y Resolution                    : 300
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Reference Black White           : 0 255 0 255 0 255
Copyright                       : 
Exposure Time                   : 1/125
F Number                        : 8.0
Exposure Program                : Manual
ISO                             : 100
Sensitivity Type                : Recommended Exposure Index
Create Date                     : 2017:01:18 16:11:27
Exposure Compensation           : 0
Max Aperture Value              : 3.5
Metering Mode                   : Multi-segment
Light Source                    : Unknown
Flash                           : Off, Did not fire
Focal Length                    : 18.0 mm
Maker Note Version              : 2.10
Quality                         : RAW
White Balance                   : Auto1
Focus Mode                      : AF-C
Flash Setting                   : Normal
Flash Type                      : 
White Balance Fine Tune         : -2 0
WB RB Levels                    : 2.296875 1.24609375 1 1
Program Shift                   : 0
Exposure Difference             : 0
Preview Image Start             : 16760
Preview Image Length            : 105735
Flash Exposure Compensation     : -3
ISO Setting                     : 100
External Flash Exposure Comp    : 0
Flash Exposure Bracket Value    : 0.0
Exposure Bracket Value          : 0
Crop Hi Speed                   : Off (4992x3280 cropped to 4992x3280 at pixel 0,0)
Exposure Tuning                 : 0
Serial Number                   : 6549208
Color Space                     : sRGB
VR Info Version                 : 0100
Vibration Reduction             : On
VR Mode                         : Normal
Active D-Lighting               : Normal
Picture Control Version         : 0100
Picture Control Name            : Standard
Picture Control Base            : Standard
Picture Control Adjust          : Default Settings
Picture Control Quick Adjust    : Normal
Brightness                      : Normal
Hue Adjustment                  : None
Filter Effect                   : n/a
Toning Effect                   : n/a
Toning Saturation               : n/a
Time Zone                       : +08:00
Daylight Savings                : No
Date Display Format             : D/M/Y
ISO Expansion                   : Off
ISO2                            : 100
ISO Expansion 2                 : Off
Auto Distortion Control         : Off
Lens Type                       : G VR
Lens                            : 18-55mm f/3.5-5.6
Flash Mode                      : Did Not Fire
Shooting Mode                   : Single-Frame
Contrast Curve                  : (Binary data 578 bytes, use -b option to extract)
Shot Info Version               : 0220
Firmware Version                : 1.04a
AF-C Priority Selection         : Release
AF-S Priority Selection         : Focus
Number Of Focus Points          : 39 Points
Focus Tracking Lock On          : 3 Normal
Focus Point Wrap                : No Wrap
AF Point Illumination           : Auto
AF Assist                       : On
Battery Order                   : MB-D11 First
AF-On For MB-D11                : AE/AF Lock
MB-D11 Battery Type             : LR6 (AA alkaline)
Beep Pitch                      : Off
No Memory Card                  : Enable Release
ISO Display                     : Show ISO Sensitivity
Grid Display                    : Off
Viewfinder Warning              : On
Shooting Info Display           : Auto
LCD Illumination                : Off
File Number Sequence            : On
Screen Tips                     : On
Beep Volume                     : Off
Reverse Indicators              : + 0 -
Easy Exposure Compensation      : Off
Exposure Control Step           : 1/3 EV
ISO Sensitivity Step            : 1/3 EV
Center Weighted Area Size       : 8 mm
Exposure Delay Mode             : Off
CL Mode Shooting Speed          : 3 fps
Max Continuous Release          : 100
Auto Bracket Set                : AE & Flash
Auto Bracket Order              : 0,-,+
Func Button                     : Virtual Horizon
Preview Button                  : Preview
OK Button                       : Select Center Focus Point
AE Lock Button                  : AE/AF Lock
Command Dials Reverse Rotation  : No
Command Dials Change Main Sub   : Off
Command Dials Aperture Setting  : Sub-command Dial
Command Dials Menu And Playback : Off
Shutter Release Button AE-L     : Off
Release Button To Use Dial      : No
Metering Time                   : 6 s
Remote On Duration              : 1 min
Self Timer Time                 : 20 s
Self Timer Interval             : 0.5 s
Self Timer Shot Count           : 3
Image Review Time               : 4 s
Live View Monitor Off Time      : 10 s
Menu Monitor Off Time           : 20 s
Shooting Info Monitor Off Time  : 10 s
Flash Sync Speed                : 1/250 s
Flash Shutter Speed             : 1/60 s
Flash Control Built-in          : TTL
Flash Warning                   : On
Modeling Flash                  : On
Live View AF Area Mode          : Face-Priority
Live View AF Mode               : AF-C
Playback Monitor Off Time       : 10 s
NEF Compression                 : Lossless
Noise Reduction                 : Off
NEF Linearization Table         : (Binary data 46 bytes, use -b option to extract)
WB GRBG Levels                  : 256 588 319 256
Lens Data Version               : 0204
Exit Pupil Position             : 97.5 mm
AF Aperture                     : 3.6
Focus Position                  : 0x06
Focus Distance                  : 3.35 m
Lens ID Number                  : 154
Lens F Stops                    : 5.33
Min Focal Length                : 18.3 mm
Max Focal Length                : 55.0 mm
Max Aperture At Min Focal       : 3.6
Max Aperture At Max Focal       : 5.7
MCU Version                     : 156
Effective Max Aperture          : 3.6
Raw Image Center                : 2496 1640
Retouch History                 : None
Shutter Count                   : 7925
Flash Info Version              : 0104
Flash Source                    : None
External Flash Firmware         : n/a
External Flash Flags            : (none)
Flash Commander Mode            : Off
Flash Control Mode              : Off
Flash Compensation              : 0
Flash GN Distance               : 0
Flash Color Filter              : None
Flash Group A Control Mode      : Off
Flash Group B Control Mode      : Off
Flash Group C Control Mode      : Off
Flash Group A Compensation      : 0
Flash Group B Compensation      : 0
Flash Group C Compensation      : 0
External Flash Compensation     : 0
Flash Exposure Comp 3           : 0
Flash Exposure Comp 4           : 0
Vari Program                    : 
Multi Exposure Version          : 0100
Multi Exposure Mode             : Off
Multi Exposure Shots            : 0
Multi Exposure Auto Gain        : Off
High ISO Noise Reduction        : Normal
Power Up Time                   : 2017:01:18 16:11:01
AF Info 2 Version               : 0100
Contrast Detect AF              : Off
AF Area Mode                    : Auto-area
Phase Detect AF                 : On (39-point)
Primary AF Point                : C6 (Center)
AF Points Used                  : (none)
Contrast Detect AF In Focus     : No
File Info Version               : 0100
Memory Card Number              : 1
Directory Number                : 104
File Number                     : 5802
AF Fine Tune                    : Off
AF Fine Tune Index              : n/a
AF Fine Tune Adj                : 0
Retouch Info Version            : 0200
Retouch NEF Processing          : Off
User Comment                    : 
Sub Sec Time                    : 50
Sub Sec Time Original           : 50
Sub Sec Time Digitized          : 50
Sensing Method                  : One-chip color area
File Source                     : Digital Camera
Scene Type                      : Directly photographed
Custom Rendered                 : Normal
Exposure Mode                   : Manual
Digital Zoom Ratio              : 1
Focal Length In 35mm Format     : 27 mm
Scene Capture Type              : Standard
Gain Control                    : None
Contrast                        : Normal
Saturation                      : Normal
Sharpness                       : Normal
Subject Distance Range          : Unknown
GPS Version ID                  : 2.3.0.0
Date/Time Original              : 2017:01:18 16:11:27
TIFF-EP Standard ID             : 1 0 0 0
Aperture                        : 8.0
Auto Focus                      : On
Blue Balance                    : 1.246094
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 4992x3280
Jpg From Raw                    : (Binary data 1224663 bytes, use -b option to extract)
Lens ID                         : AF-S DX VR Zoom-Nikkor 18-55mm f/3.5-5.6G
Lens Spec                       : 18-55mm f/3.5-5.6 G VR
Megapixels                      : 16.4
Preview Image                   : (Binary data 105735 bytes, use -b option to extract)
Red Balance                     : 2.296875
Scale Factor To 35 mm Equivalent: 1.5
Shutter Speed                   : 1/125
Create Date                     : 2017:01:18 16:11:27.50
Date/Time Original              : 2017:01:18 16:11:27.50
Modify Date                     : 2017:01:18 16:11:27.50
Thumbnail TIFF                  : (Binary data 57816 bytes, use -b option to extract)
Circle Of Confusion             : 0.020 mm
Depth Of Field                  : inf (1.27 m - inf)
Field Of View                   : 67.1 deg (4.44 m)
Focal Length                    : 18.0 mm (35 mm equivalent: 27.0 mm)
Hyperfocal Distance             : 2.02 m
Light Value                     : 13.0

[pedrocr]

Almost all cameras have a large JPG in the raw thumbnail. That's often what's used to display on the camera screen when you are in RAW only mode and on external screens through HDMI for example (in which case 1080p thumbnails are common). Your camera goes one step further and includes a 4928x3264 JPG thumbnail which seems to be a full sized image. They probably did that to allow for 1:1 in-camera viewing. The reason you probably saw very small thumbnails is that some cameras include several thumbnails, some really small, and in some cases the format is readable as a normal TIFF but only showing that one small thumbnail. So if you use software that isn't raw aware you get that small poor image.

Anyway, raw processing is fast enough these days that there's no real point in using the thumbnails, it will only lead to poor color matching and confusing interfaces. rawloader can generate a 500px wide thumbnail from a 24MP image in 200-300ms which together with caching is more than enough for all uses.

As for this specific NEF it does seem the default curve makes it a bit too dark. A better default curve would be nice and PRs are welcome for that. Automatic rotation would also be welcome and it was on my TODO list. The biggest issue for that is getting the metadata from all the different formats. Once you know the orientation it's easy enough to apply it. The most performant way is to do it in src/imageops/demosaic.rs by doing coordinate swaps/flips. A cleaner but slightly slower way is to just do a new image operation that just does the flipping of the pre-demosaic data (but then also needs to adjust the bayer pattern).

[runfalk]

I've tried the Canon EOS600D this weekend. Using Rawloader with it seems to produce darker images too. Though it is not as apparent as with the NEF files. I don't know if Canon has an equivalent to Active D-Lighting.

I made a side by side comparison of what Rawloader and CameraRaw (without adjustments) produces:

Like my NEF above it is darker. It seems that Rawloader produces slightly red-tinted images in both cases. Which part of the decoding process could be responsible for that?

[pedrocr]

Raws are a matter of interpretation. For me the image on the left is better but raws are not about the defaults, but letting the user change things at will. A red tint could be because of a different whitebalance setting or again a different curve.

[runfalk]

In an artistic sense for this particular image I agree that the left one is more appealing. However, the one on the right is what the camera preview shows as well as being much closer to reality. I'd find it less confusing if the default settings produced something similar to what I've already seen on the camera, or another previewer. I have no objections about allowing the current result as an optional profile/preset.

[pedrocr]

I'm all for good defaults and the curve I set is pretty crude. Feel free to change the curve to something you think is more reasonable and do a PR with that. It's all nicely done in parameters anyway:

https://github.com/pedrocr/rawloader/blob/a771d48f1fffe3b6b1d1b36800720ef053481cf1/src/imageops/ops/curves.rs#L14-L18

[pedrocr]

I don't know what I was smoking at the time but the original curve made no sense. I've set what looks like a more reasonable default curve in imagepipe. You may want to try the converter program there (or chimper) to see if it looks a bit better.

[pedrocr]

Transferred the issue to imagepipe as that's where the curve code now is.

[runfalk]

Hey there. I tried the converter program in imagepipe and put the result compared to my original post. I think the new curve looks a lot better, but still very red.

[pedrocr]

The red is a different issue. The camera probably used a different whitebalance and/or color matrix. The comparison there would have to be to the Adobe tools to be meaningful and even then we wouldn't know what they're doing exactly.

[Lucostus]

Hey, foremost, love your work. I am too very new to raw in general and learned a lot from reading through the repos you have.
I still face the issue that the output of imagepipe doesn't look at all like the original shot and how I captured it irl. I tried to play with the curves and couldn't bring it anywhere close to what I see in the preview of apple (looks to me like the original shot; also i guess it's not jpeg). I use RW2, Panasonic.