M31 background too busy

[steve72614]

Hi,
I've been experimenting with Star Tools on some old images of M31 from a dark site with a less than ideal 200mm lens. I used DSS on 20 .CR2 files with no background calibration or white balancing specified. I am aware that converting the .cr2 to .tiff might prevent some inadvertent white balancing performed by DSS, so adding dcraw preconversion to .tifs is on my to do list.

In processing the .fts output from DSS, I followed the suggested procedure on the last page of the Star Tools unofficial manual. Is there a way to prevent the background stars from being so bright, like jelly beans, and still see the detail in the galaxy? The background stars don't look realistic to me. Did I push development so hard that the stars are really exaggerated noise? It just doesn't look like the stock photos I've seen of Andromeda.

Thanks for looking.

Steve
-----------------------------------------------------------
StarTools 1.3.5.289
Mon Apr 17 15:02:42 2017
-----------------------------------------------------------
File loaded [D:\users\steve\my_docs\astronomy\our astrophotographs\2017-01-23 pictures\output FITS\Autosave.fts].
---
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [6.6 pixels]
Parameter [Outside ROI Influence] set to [10 %]
--- Crop
Parameter [X1] set to [10 pixels]
Parameter [Y1] set to [10 pixels]
Parameter [X2] set to [5192 pixels (-10)]
Parameter [Y2] set to [3455 pixels (-10)]
--- Bin
Parameter [Scale] set to [(scale/noise reduction 20.00%)/(2500.00%)/(+4.64 bits)]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [3 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [87 %]
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [Off]
Parameter [Outside ROI Influence] set to [15 %]
--- Deconvolution
Parameter [Image Type] set to [Deep Space]
Parameter [Mask Behavior] set to [De-ring Mask Gaps, Hide Result]
Parameter [Radius] set to [1.2 pixels]
Parameter [Iterations] set to [6]
Parameter [Regularization] set to [1.00 (optimal noise and detail)]
Parameter [Mask Fuzz] set to [8.0 pixels]
--- Contrast
Parameter [Expose Dark Areas] set to [No]
Parameter [Compensate Gamma] set to [No]
Parameter [Precision] set to [128 x 128 pixels]
Parameter [Dark Anomaly Filter] set to [3 pixels]
Parameter [Aggressiveness] set to [75 %]
Parameter [Dark Anomaly Headroom] set to [15 %]
Undo.
--- Contrast
Parameter [Expose Dark Areas] set to [No]
Parameter [Compensate Gamma] set to [No]
Parameter [Precision] set to [128 x 128 pixels]
Parameter [Dark Anomaly Filter] set to [3 pixels]
Parameter [Aggressiveness] set to [75 %]
Parameter [Dark Anomaly Headroom] set to [15 %]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [1.74]
Parameter [Detail Size Range] set to [40 pixels]
Parameter [Noise Suppression] set to [Off]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [Straight CIELab Luminance Retention]
Parameter [Dark Saturation] set to [2.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.18]
Parameter [Green Bias Reduce] set to [1.70]
Parameter [Red Bias Reduce] set to [1.00]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet De-Noise
Parameter [Scale 1] set to [90 %]
Parameter [Scale 2] set to [90 %]
Parameter [Scale 3] set to [90 %]
Parameter [Scale 4] set to [90 %]
Parameter [Scale 5] set to [0 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Scale Correlation] set to [3]
Parameter [Color Detail Loss] set to [20 %]
Parameter [Brightness Detail Loss] set to [5 %]
Parameter [Grain Size] set to [2.5 pixels]
Parameter [Read Noise Compensation] set to [Off]
Parameter [Smoothness] set to [75 %]
File saved [D:\users\steve\my_docs\astronomy\our astrophotographs\2017-01-23 pictures\output FITS\M31 StarTools out from FTS.tiff].

-----------------------------------------------------------

[happy-kat]

Why would you want to convert your cr2 files to tiff before using in DSS?

[admin]

Hi Steve,

Looking at your image, it appears that either something went wrong during stacking, or your focus was perhaps off. A typical stellar profile would be a bright (sometimes overexposed) point, fading out into neighbouring pixels according to the diffraction pattern of your optical train (a roughly a circle for non-stopped down lenses, spikes for mirrors, possible spikes for stopped down lenses).

There is one thing that may also cause what we're seeing - if there is something wrong with the FTS image (values encoded that outside the bounds of what the FTS specifies as max/min values), then StarTools will discard those pixels as "corrupted" and use a "neutral"/interpolated value instead that will not interfere with the many algorithms that would otherwise choke on anomalous data. If you have been saving your FTS files as flotaing point, may I suggest saving them as 32-bit Integer FTS files instead.

If you'd like us to have a look at the data, do please feel free to upload it somewhere.

Hope this helps in the meantime,

[admin]

Why would you want to convert your cr2 files to tiff before using in DSS?

The answer is that DSS insists on color balancing the data. There is a small boost in signal fidelity and noise reduction by handing StarTools data that has not been color balanced yet.

Color balancing causes noise from different channels to be blended in with other channels with different weighting factors, making noise levels in the individual channels a bit of a hodgepodge as opposed to "virgin" unbalanced data where noise vs signal is a known quantity that StarTools can track from the very beginning.

Does that make sense?

[steve72614]

Thanks for your responses. Retried everything with dcraw input and have some more questions. Hope it's not to tedious a list.

Per Ivo's advice, today I ran the CR2 files thru dcraw to produce a tiff with all the multiplier values of 1, and all the dcraw options he suggested in another post about DSS whitebalancing. The Canon Tool reports the DSS input CR2 files had a slightly red bias, and don't look to have an unnatural color cast. The images were taken at a pretty good dark site on a good night. I then ran the fts thru ST.

DSS still produced a fts with a heavy green emphasis.

I noticed that the HDR Reveal DSO Core module is what caused some of the background noise to look a little like the stars. When sampling a star field for white, and then setting the mask to the whole image, the result was still too much green. I had to raise the green bias reduction slider to about 1.7 to get this result. Why might the sampling of a star field for white not remove the green cast?

I used a 1% read noise compensation and 7 pixel grain size in the final noise reduction step. DSS had reported a sky background of roughly 0.45% for the input images. I'm guessing that the final noise reduction noise floor could always safely be set at least as high as the DSS sky background value reported in the DSS file list. Is this the correct idea?

During the ST noise reduction step I saw what looked like a bunch of hot mostly red pixels. I'm not really clear on how to make the proper adjustment on DSS hot pixel removal. Does someone know how to adjust this DSS parameter?

I used the Autosave.fts file from DSS, so I don't think its a floating point .fts file. I'm hoping the DSS default is integer data for the Autosave.fts file.

The lens I used for these pictures would not quite reach infinity focus, as it was for a different brand of camera, and I used a lens adapter. I have since got another lens that achieves good infinity focus. Wondering how much of my problem with "busy background" is related to the poor focus.

Do anyone know why Deep Sky Stacker does not white balance .tiff input files?

Since binning my DSS autosave.fts image in Star Tools is necessary anyway, how does this compare to having DSS process the .CR2 to superpixels, achieving the equivalent of a 2x2 binning before feeding the result to Star Tools? I'd imagine the white balance would still be a little off?

Anyway, I am happier with this result than the first. Dcraw, and some more aggressive final noise reduction were the main differences from the first attempt.

Thanks,
Steve

[admin]

Thanks for your responses. Retried everything with dcraw input and have some more questions. Hope it's not to tedious a list.

Please feel free to ask away - it's the only way we can learn!

Per Ivo's advice, today I ran the CR2 files thru dcraw to produce a tiff with all the multiplier values of 1, and all the dcraw options he suggested in another post about DSS whitebalancing. The Canon Tool reports the DSS input CR2 files had a slightly red bias, and don't look to have an unnatural color cast. The images were taken at a pretty good dark site on a good night. I then ran the fts thru ST.

DSS still produced a fts with a heavy green emphasis.

Great! The green (or sometimes teal) is an expected "color" for the data of most DSLRs/OSCs non-color-balanced data (specifically it is the non-color balanced skyglow and light pollution signature/bias).

I noticed that the HDR Reveal DSO Core module is what caused some of the background noise to look a little like the stars.

Hmmmm... Would you be able to post a before and after screenshot? Which version of StarTools are you running?

When sampling a star field for white, and then setting the mask to the whole image, the result was still too much green. I had to raise the green bias reduction slider to about 1.7 to get this result. Why might the sampling of a star field for white not remove the green cast?

At this point' I'd love to have a look at the data to see if there are any quick pointers I could give you.
Strange... Only a light pollution filter or (heavily) modded DSLR would skew colours this severely as you describe. Normally the Color module's initial settings/analysis (with full mask set) should do well on a widefield of M31.

I used a 1% read noise compensation and 7 pixel grain size in the final noise reduction step. DSS had reported a sky background of roughly 0.45% for the input images. I'm guessing that the final noise reduction noise floor could always safely be set at least as high as the DSS sky background value reported in the DSS file list. Is this the correct idea?

The read noise compensation is really only meant to be used in cases where noise is severe and noise influences other than the usual shot/photon noise start taking over (for example read noise), or (technical jargon alert!) where there are so few samples per pixel that the Gaussian noise model is no longer a good approximate fit for the Poisson distribution of shot/photon noise (brilliant article explaining the difference for those interested in the nitty-gritty)

During the ST noise reduction step I saw what looked like a bunch of hot mostly red pixels. I'm not really clear on how to make the proper adjustment on DSS hot pixel removal. Does someone know how to adjust this DSS parameter?

Dithering between frames should take care of such noise, while bias frames may also help (also, don’t' forget to shoot flat frames if you aren't already!)

The lens I used for these pictures would not quite reach infinity focus, as it was for a different brand of camera, and I used a lens adapter. I have since got another lens that achieves good infinity focus. Wondering how much of my problem with "busy background" is related to the poor focus.

Do anyone know why Deep Sky Stacker does not white balance .tiff input files?

That's because there is no "universal" way to white balance TIFF files; the CR2 files specify what camera was used to acquire the data. DCRAW then looks up the model and the right multiplication factors/matrix. When TIFF files are used, it presumes the image (note we're not talking about "data" any more) was already white balanced.

It's the murky difference between data vs image; the less you do the the raw photon counts, the closer you are to the definition of data. The more you process the data, the closer you are to the definition of an image/picture.

Since binning my DSS autosave.fts image in Star Tools is necessary anyway, how does this compare to having DSS process the .CR2 to superpixels, achieving the equivalent of a 2x2 binning before feeding the result to Star Tools? I'd imagine the white balance would still be a little off?

It would end up being roughly the same result, though DSS may be able to better align and stack the subs with less noisy sub-frames.

[steve72614]

Ivo,

Thanks for your responses! I've posted the autosave files for M31 up on box.com if you want to look at them. The one derived from dcraw produced tifs is what I'm most interested in.

The link to the files is here --
https://app.box.com/s/zi1lbat16d6moe1crl6ri7k6agssbjq8

My Star Tools version is 1.3.5.289. I'm running the 64 bit windows version on windows 7. I've got 4GB of RAM.

Steve

[admin]

Hi Steve,

Thanks for uploading!

It really looks like your woes stem from focus and/or collimation issues.

Just to illustrate, this is one of the stars from the data (not stretched, not processed, just a simple crop);

Autosave from DSS using TIF input files from dcraw.jpg (1.14 KiB) Viewed 10722 times

...which should really look like this (I used the Repair module's redistribute algorithm);

Autosave from DSS using TIF input files from dcraw_redistribute.jpg (1.02 KiB) Viewed 10722 times

As you can see, there is no real core or stellar profile to your stars, they more resemble "patches" of light rather than points of light. DSS may also have trouble stacking on top of that, given it probably looks for a signature matching the above "repaired" image, rather than the above "unrepaired"/"patch" image.

You will almost certainly see your problems go away with better focusing and/or collimation.

I can see some hot pixels as well. These will go away if dither between frames (or use bias frames).

Other than that, the colours (roughly) come out as expected by sampling the whole image (which the Color module does by default), though there are some odd colors going on in the cores. I suspect these might go away once DSS has less trouble stacking (though let me know if they don't!). You are right that for this image, pulling back the green was appropriate (the MaxRGB mode bears that out - pull it back just enough for the green dominance in the core to go away).

There is very little detail to recover for HDR, given that the detail is "smeared out" just like the stars. I would probably leave the HDR module alone until you are resolving data properly.

Have you acquired any new data yet in the meantime?

[steve72614]

Ivo,
I switched to better lens for this image of the Orion Nebula captured in February. I reprocessed the picture with Star Tools.
https://www.flickr.com/photos/stevestar ... 459202505/

I used dcraw conversion, and darks, flats, and bias frames for DSS. DSS produced a much flatter image as expected.
My one remaining concern is Star Tools "Wipe" function, which I wasn't able to adjust to produce an evenly lit image.

I was tempted to just crop it, to make the gradient less obvious. By using flats, I have reduced somewhat the need for the Wipe function.

Thanks!

Steve

[admin]

Hi Steve,

That data looks much improved - congrats!
If you'd like me to have a look at the stack again, do let me know. I would have though Wipe would have taken care of the gradients...