Eliminating white balancing when using DSS

[Guy]

Hi Ivo

In a number of conversations e.g. here here you talk about using DCRAW to preprocess the subs from Canon DSLRs and convert them to TIFFs before using DSS.

Obviously the main reason to do this being to stop DSS white balancing the image so that we get the best stacked image for StarTools.

As I understand it from here the command is:
dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 *.CR2

When I use this command the images are rotated and changed so much that I have to reduce the star detection threshold in DSS to around 2% where for the CR2 version it was at 90%. In StarTools the image is very faint.

The rotation can be removed using -t 0 (autoflip image=none)

My questions are:
-k 0 sets the darkness level to 0 - isn't that like setting the black point to 0 in DSS - which I thought was not best practice?
I'd also be interested to understand why the saturation level is set to 32767.
The Canon 1000D camera I am using is 12-bit - does the saturation level need to be set differently for this - or what is the effect if I remove the -S command completely?

Any advice you can give would be appreciated.

Guy

[admin]

Hi Ivo

In a number of conversations e.g. here here you talk about using DCRAW to preprocess the subs from Canon DSLRs and convert them to TIFFs before using DSS.

Obviously the main reason to do this being to stop DSS white balancing the image so that we get the best stacked image for StarTools.

As I understand it from here the command is:
dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 *.CR2

When I use this command the images are rotated and changed so much that I have to reduce the star detection threshold in DSS to around 2% where for the CR2 version it was at 90%. In StarTools the image is very faint.

The rotation can be removed using -t 0 (autoflip image=none)

My questions are:
-k 0 sets the darkness level to 0 - isn't that like setting the black point to 0 in DSS - which I thought was not best practice?
I'd also be interested to understand why the saturation level is set to 32767.
The Canon 1000D camera I am using is 12-bit - does the saturation level need to be set differently for this - or what is the effect if I remove the -S command completely?

Any advice you can give would be appreciated.

Guy

If my understanding of these parameters and dcraw's inner workings are correct, -k 0 and -S 32767 assure that the full dynamic range as recorded by the sensor is used;

The "k" parameter equates full darkness to a recorded 0 value, while -S (channel saturation) is set to something much higher than what your 12-bit A/D converter can produce (which would be 2^12 = 4096). This ensures that all values that your A/D converter could possibly produce are kept intact and that there is no truncation going on in any of the channels. Of course we'd only be using 4096 (12 bits) out of the 65536 possible values (16 bits) possible in a 16-bit TIFF, but that's ok (depending on how the software that you use to import the data deals with the superfluous dynamic range of course). The reason why we'd want the full dynamic range as recorded (instead of a "white balancing prepped-version" is that we still have airglow and/or light pollution to subtract before white balancing. Prepping (clipping) channels before subtracting airglow/LP (reducing dynamic range even more!) is just a waste of dynamic range. Better to subtract LP/airglow first and then see if (and how much) we still need to clip.

At the end of the day, the point of this parameter-specifying-marathon is to end up with readings that are closest as possible to raw photon counts for all 3 channels. If you find any better options/parameters that get you better results to that end, please do share them though!

Hope this helps...

[Guy]

Thanks for the explanation.

I think these settings need to be changed for my setup. DSS seems to struggle to find stars and the result is a TIF which, when loaded into StarTools looks much fainter than the equivalent if using the CR2 files as input to DSS.

I'll do some tests with changing the dcraw parameters and hopefully find some parameters which work better for me.

I'll report back if I find out anything useful.

Guy

[Guy]

Hi

I've done sone tests and I think I have a better understanding of what dcraw is doing now.
Please correct me if I'm wrong.

DCRaw Action
setting -S tells dcraw the value of the input image to consider saturated
setting -k tells dcraw the value of the input image to consider black

If you don't set these parameters explicitly dcraw derives the values (possibly taken from the image itself or a database it has which relates to the camera)
for my Canon 1000D the values appear to be - S 3651- k 254
So by default:
254 on the input image maps to black (0) on the output tif
3651 on the input image maps to saturated (65535) on the output tif

If using -S 32767:
32767 on the input image maps to saturated (65535) on the output tif
however the input image here only has values up to 3651 so its maximum output is (3651/32767)*65535=7302
That is why the images were so faint - we are only using less than an eighth of the range.

For sub-frames converted to TIFF using the following:
Command: dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 -t 0*.CR2
'Scaling with darkness 0, saturation 32767, and
multipliers 1.000000 1.000000 1.000000 1.000000'
Output: Max Value 7302 Min 406 mean 625

Similarly with -k 0
0 on the input image maps to black (0) on the output tif
however for the input image here values never go below 254 so the output image is never black

So the values to use are camera dependent.
For a 12-bit camera like the 1000D we could use -S 4095 and -k 0 - but even that won't use the full output range due to the actual range of possible input image values being less than this.

Command: dcraw -v -r 1 1 1 1 -4 -T -S 4095 -k 0 -o 0 -q 0 -t 0*.CR2
'Scaling with darkness 0, saturation 4095, and
multipliers 1.000000 1.000000 1.000000 1.000000'
Output: Max Value 57959 Min 3248 mean 5008

Effect of DSS

DSS uses these values in defining the threshold - which is why the threshold needs to be set as low as 2% when using TIFs generated using -S 32767.
Comparing the result of using DSS with CR2 subs against using TIF subs I go the following results:

CR2 Files as Input:
DSS Output: Max Value 65280 Min 373 Mean 2735

Command: dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 -t 0*.CR2
DSS Output: Max Value 7306 Min 78 mean 627

So If you use TIFF sub-frames DSS outputs files with similar range to the input files.
Using CR2 subframes a the output TIF it uses a much larger span of the available dynamic range.

Effect of StarTools

The DSS output processed straight from Raw CR2 files initially has a red hue
The DSS output processed from TIF files generated using -S 32767 -k 0 initially has a green hue (regardless of which 'linear' load option you choose)
It feels like it is harder to get a decent result with the DSS output generated from TIF files

Ivo - in what way will StarTools process these two files differently? Is it having to work harder with the TIF-derived input file?

Is there a reason why we shouldn't omit -S and -k and let dcraw choose the values from its database - and so use the full range of output values where 0=black and 65535 is saturated?

Guy

[admin]

Hi

I've done sone tests and I think I have a better understanding of what dcraw is doing now.
Please correct me if I'm wrong.

DCRaw Action
setting -S tells dcraw the value of the input image to consider saturated
setting -k tells dcraw the value of the input image to consider black

Correct! (e.g. in that it is line with my understanding of the inner working as well)

If you don't set these parameters explicitly dcraw derives the values (possibly taken from the image itself or a database it has which relates to the camera)
for my Canon 1000D the values appear to be - S 3651- k 254
So by default:
254 on the input image maps to black (0) on the output tif
3651 on the input image maps to saturated (65535) on the output tif

If using -S 32767:
32767 on the input image maps to saturated (65535) on the output tif
however the input image here only has values up to 3651 so its maximum output is (3651/32767)*65535=7302
That is why the images were so faint - we are only using less than an eighth of the range.

Spot-on!

For sub-frames converted to TIFF using the following:
Command: dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 -t 0*.CR2
'Scaling with darkness 0, saturation 32767, and
multipliers 1.000000 1.000000 1.000000 1.000000'
Output: Max Value 7302 Min 406 mean 625

Similarly with -k 0
0 on the input image maps to black (0) on the output tif
however for the input image here values never go below 254 so the output image is never black

So the values to use are camera dependent.
For a 12-bit camera like the 1000D we could use -S 4095 and -k 0 - but even that won't use the full output range due to the actual range of possible input image values being less than this.

Command: dcraw -v -r 1 1 1 1 -4 -T -S 4095 -k 0 -o 0 -q 0 -t 0*.CR2
'Scaling with darkness 0, saturation 4095, and
multipliers 1.000000 1.000000 1.000000 1.000000'
Output: Max Value 57959 Min 3248 mean 5008

Effect of DSS

DSS uses these values in defining the threshold - which is why the threshold needs to be set as low as 2% when using TIFs generated using -S 32767.

Yes, exactly! "Saturated" stars will have a max value much smaller than what the maximum 16-bit dynamic range allows. That's indeed why the threshold needs to be much lower for DSS to detect the stars.

Comparing the result of using DSS with CR2 subs against using TIF subs I go the following results:

CR2 Files as Input:
DSS Output: Max Value 65280 Min 373 Mean 2735

Command: dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 -t 0*.CR2
DSS Output: Max Value 7306 Min 78 mean 627

So If you use TIFF sub-frames DSS outputs files with similar range to the input files.
Using CR2 subframes a the output TIF it uses a much larger span of the available dynamic range.

Correct. We can see here that DSS performs some "processing" (it's really dcraw used internally by DSS that is doing this).

Effect of StarTools

The DSS output processed straight from Raw CR2 files initially has a red hue

And this is where things become important. The red hue is due to color balancing.

The DSS output processed from TIF files generated using -S 32767 -k 0 initially has a green hue (regardless of which 'linear' load option you choose)

This is indeed the expected result (and in line with what, for example, PixInsight produces) on most cameras; white balancing typically cranks up (e.g. multiplies) the red channel a whole lot, multiplying any noise in there in the process as well. The latter is why we don't want color balanced data (yet).

It feels like it is harder to get a decent result with the DSS output generated from TIF files

Would you be able to tell me in what respect you feel it is harder? Colors? Noise?

Ivo - in what way will StarTools process these two files differently? Is it having to work harder with the TIF-derived input file?

Is there a reason why we shouldn't omit -S and -k and let dcraw choose the values from its database - and so use the full range of output values where 0=black and 65535 is saturated?

Guy

There are two main reasons;

1. Giving StarTools non-white balanced data makes sure that the noise level are the same across all three channels. This is important for two reasons; 1. It allows us to create a luminance frame that has each channel contribute signal (and noise!) evenly, resulting in the best possible luminance frame.2. It allows StarTools to commence the tracking of noise evolution much better (since it can assume noise grain is visible in equal measures from the start, no matter the channel).

2. White balancing (as I alluded to in my previous post) necessarily destroys some dynamic range in the highlights (blowing out stars). The reason why we'd want the full dynamic range as recorded (instead of a "white balancing prepped-version" is that we still have airglow and/or light pollution to subtract before white balancing. Prepping (clipping) channels before subtracting airglow/LP (reducing dynamic range even more!) is just a waste of dynamic range. Better to subtract LP/airglow first and then see if (and how much) we still need to clip. I.e. we can (likely) retain more detail in the highlights by white balancing after airglow/LP subtraction.

Does this make sense?

[Guy]

Thanks Ivo.

OK, I think we are homing in on the area I'm confused about.
I understand the whole reason for doing this is to avoid DSS doing any white balancing - so StarTools gets the best possible data to work with.
I was thinking that if we used dcraw and set -r 1 1 1 1 then dcraw would, for each colour, linearly translate the input values - without altering the balance between each of the R, G & B channels - so no white balancing.

If I understand what you are saying correctly if we omit -S and -k this would cause dcraw to white balance even though we were
still using the -r 1 1 1 1 parameter. Is that right?
Also, I get the impression from your reply that you would expect StarTools to process the stacked images that were preprocessed with -S 32767 -k 0 the same or better than the CR2 stacked image. Is this correct?

When processing the image from stacked subs preprocessed with -S 32767 -k 0 in StarTools it felt like it was more difficult because:
- when first loaded the image is very much fainter - you can't see any stars - just faint grey smudges
- the stretched image looks noisier (although I don't have a quantitative measure of this - and there is quite a bit of nebulosity in the test image)
- getting the colour right seemed more difficult (again a very subjective thing)

Also, when processing the image preprocessed with -S 4095 -k 0 in StarTools on first loading the image was much brighter - you could see the stars clearly and also their colours.

I did some further tests - looking at the RGB results of an image processed by dcraw with different -S and -k values

Code: Select all

 S	-k	  Max	Ravg	Gavg	Bavg		R%max	G%max	B%max
Lights
32767	0	7302	626	  694	 625		8.57%	9.50%	8.56%
8191	0	29211	2504	2778	2502		8.57%	9.51%	8.57%
4095	0	58429	5008	5558	5005		8.57%	9.51%	8.57%
3700	0	64667	5542	6151	5538		8.57%	9.51%	8.56%
3651	0	65535	5617	6234	5612		8.57%	9.51%	8.56%

32767	254	6847	 118	 187	 118		1.72%	2.73%	1.72%
8191	254	28048	 486	 770	 484		1.73%	2.75%	1.73%
4095	254	57959	1005	1591	1001		1.73%	2.75%	1.73%
3700	254	64603	1120	1774	1116		1.73%	2.75%	1.73%
3651	254	65535	1136	1799	1132		1.73%	2.75%	1.73%	- dcraw defaults for my camera.

Darks
32767	0	7304	512	512	512		7.01%	7.01%	7.01%
32767	254	6849	6	6	6		0.09%	0.09%	0.09%

The table shows the maximum values, RGB channel average and the percentage of the maximum that the channel average represents.
(all processed with dcraw using -v -r 1 1 1 1 -4 -T -o 0 -q 0)

So the maximum values change with different -S values but the mean value remains the same proportion of the total range.
To me this indicates that the 3 colour channels are being processed separately and linearly by dcraw when using these commands - and so preserving the raw colour channel balance.

Looking at the darks, there seems to be an offset in CR2 files which -k 0 doesn't take account of. As a result the tiff files will never go below a threshold for files preprocessed using -k 0. For my camera dcraw defaults to -k 254 to eliminate this.

Based on these results would it be OK to preprocess using settings other than -S 32767 -k 0 (e.g.-S 3651 -k 0) for this camera?
Would there be any difference in the way StarTools processed the file?
Are we introducing white balancing by using settings other than -S 32767 -k 0?
Does the permanent offset introduced by using -k 0 cause StarTools problems?


As always - thanks for the great support and a great program.

Guy

[Guy]

Hi Ivo,

I've done some further analysis and prepared three different files to show the results I got with different approaches to using DSS.
I did my tests on an old image taken with an unmodified DSLR.

1) using CR2 sub-frames - no preprocessing with dcraw
The resulting file can be viewed or downloaded here

The RGB histogram looks like this:

DSS output using CR2 Subs - no preprocessing

NGC2237-CR2-All-LDF-Hist.jpg (17.34 KiB) Viewed 16293 times

The red bias can be seen easily.

2) using TIF sub-frames preprocessed using: dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 -t 0 *.CR2
The resulting file can be viewed or downloaded here

The RGB histogram looks like this:

DSS output using TIF subs - preprocessed with dcraw -S 32767 -k 0

NGC2237-tiff-S32767k0t0-All-LDF-Hist.jpg (16.27 KiB) Viewed 16293 times

As you can see, the image has a green tint indicating it has not been white balanced.
However, this is the image that, when first loaded into StarTools, looks like a few very faint grey smudges.
Is this going to cause StarTools any problems?

3) using TIF sub-frames preprocessed using: dcraw -v -r 1 1 1 1 -4 -T -o 0 -q 0 -t 0 *.CR2
The resulting file can be viewed or downloaded here

The RGB histogram looks like this:

DSS output using TIF subs - preprocessed with dcraw -S and -k not set

NGC2237-tiff-NoSk-t0-All-LDF-Hist.jpg (17.39 KiB) Viewed 16293 times

This image has a green tint too.
It uses the full dynamic range of the tif for my 12-bit camera and when loaded into StarTools it shows the stars clearly.

Ivo, this last image looks to me like it is not white balanced and is still linear. Is that right?
Is there a reason I shouldn't use this last image as an input to StarTools?

Thanks and regards,
Guy

[admin]

Hi Guy,

Apologies for the delay in reply. Too much going on...

If I understand what you are saying correctly if we omit -S and -k this would cause dcraw to white balance even though we were
still using the -r 1 1 1 1 parameter. Is that right?

No, not quite. The -S and -k options merely have to do with the dynamic range allocation and clipping/setting of blackpoint + highlights (see below).

So the maximum values change with different -S values but the mean value remains the same proportion of the total range.
To me this indicates that the 3 colour channels are being processed separately and linearly by dcraw when using these commands - and so preserving the raw colour channel balance.

You are of course absolutely right in your conclusion.

To be on the safe side, the -k 0 and -S 32767 settings make sure that the dcraw command as published works for every camera (e.g. there is always enough dynamic range to record photon counts 1:1) and nothing is accidentally thrown away, distorted, clipped, or otherwise modified.

You are of course right that it stands to reason that if your camera only produces values in the rrange of 0...4095 max (e.g. 12-bit), then 4 bits of dynamic range in a 16-bit tiff (ranging 0..65535) go unused. The full signal will indeed sit in the lower 1/16th of the dynamic range, hence why you can barely see anything and why DSS needs to be pushed more to start picking up stars.

Based on these results would it be OK to preprocess using settings other than -S 32767 -k 0 (e.g.-S 3651 -k 0) for this camera?

Definitely.

Would there be any difference in the way StarTools processed the file?

There should not be.

Are we introducing white balancing by using settings other than -S 32767 -k 0?

No, but inaccurate clipping of blacks and highlights may result in some loss of dynamic range, distortion of the linearity in the highlights, or both. See also -H parameter.

Does the permanent offset introduced by using -k 0 cause StarTools problems?

It shouldn't, but it shouldn't hurt either.

You should notice that in the 1.4.x. versions file #2 and #3 are almost(!) look the same when loaded (#2 is automatically normalized).

You will notice, however, a subtle difference in the highlights where #3 is clipped a bit more than #2. You can see this more clearly when using the Layer module and subtracting a normalized version of #2 (in foreground) from #3 (in background), or simply blinking the two images.

When processing #2 or #3, however it makes (virtually) no difference;

comparison.jpg (293.81 KiB) Viewed 16273 times

Process flow for both (in 1.4.322);
--- Crop
Parameter [X1] set to [1568 pixels]
Parameter [Y1] set to [1497 pixels]
Parameter [X2] set to [1741 pixels (-2165)]
Parameter [Y2] set to [1666 pixels (-936)]
--- Wipe
Default.
--- Develop
Parameter [Digital Development] set to [75.00 %]
Parameter [Dark Anomaly Filter] set to [8.0 pixels]
--- Color
Parameter [Dark Saturation] set to [Full]

In short, in your case, I believe you can safely use either #2 or #3; both should work.

Hope this helps!

[Guy]

Hi Ivo,

Thanks - thats cleared things up completely.

To summarise (to make sure I've got this right - and for anyone following this):
- To stop DSS white balancing:

• preprocess all sub-frames - Lights, Darks, Flats etc. - with dcraw using one of the command lines below.
• stack the resulting .tif (or .fits) files
• In StarTools open the resulting image file using the 2nd Option 'Linear, was Bayered, is not whitebalanced'

- Preprocessing with dcraw using the command line:

Code: Select all

dcraw -v -r 1 1 1 1 -4 -T -S 32767 -k 0 -o 0 -q 0 -t 0 *.CR2

will work for all cameras with StarTools 3.5 and 4.x. However:

• with Deep Sky Stacker you may need to use a lower threshold to get a reasonable number of stars detected.
• when loaded in Startools v3.5 it may initially look faint - although it will be processed normally. In StarTools 4.x it looks fine and is processed normally.
• when used with other software it may look faint.

- Preprocessing with dcraw using the command line:

Code: Select all

dcraw -v -r 1 1 1 1 -4 -T -o 0 -q 0 -t 0 *.CR2 

will avoid these small limitations but only if your camera is supported by dcraw. This is not usually a problem but it is important to check. If in doubt don't use these settings as they may cause problems with clipping.

Also worth mentioning:

• There is a useful article on dcraw by Guillermo Luijk - with explanations of how to help make this preprocessing in Windows easy.
• The perceived problems I had with processing the fainter image in StarTools 3.5 were just my perceptions - there is no difference to the resulting output

Ivo, I appreciate the time you took to explain this - its clearly a busy time for you. The comparison of processing both images was unequivocal proof there was no difference - and is yet another handy feature of StarTools.

I will definitely be using this preprocessing from now on as it is pretty quick and easy and gives StarTools slightly better data to work with.


Guy

[ecuador]

Interesting thread. I was under the impression that when using RAW files DSS does calibration before debayering, which, as I understand the process, is the right way to go and should make a difference. So, using DCRAW first would debayer the images, so they would not be whitebalanced, but their calibration would not be as good. Am I correct in that?

Also, the problem with DSS white-balancing has only to do with the colors of our image when processing Star Tools? I.e. if I can get the colors I want out of an image even though DSS has whitebalanced it, would there be a benefit from trying to get non-whitebalaned data?