Fancy processing my M78 stack, anyone?

[Melty]

Hi,

I have been really struggling to get a good image out of an HDR DSS stack of M78 data i compiled a few weeks ago. All in all, I have over 7 hours of data.

All my images from Startools exhibit an unattractive brown mottle, where there should be smooth dust surrounding this reflection nebula.

A fellow imager, has kindly had a go of processing my stack and has produced an image which was so good, it literally made my jaw drop. His weapon of choice is Pixinsight.

I've been using Startools for a good year now, with some pretty good images under my belt, but this image has defeated me, and I am lost in trying to smooth out the noise.

Here is a sample workflow of my numerous attempts:

StarTools 1.4.340
Sun Jan 20 12:45:56 2019
-----------------------------------------------------------
File loaded [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19\Lights\M78 (No Drizzle).fts].
---
Type of Data: Linear, was not Bayered, or was Bayered + white balanced
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [2.20]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [24.07 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- Crop
Parameter [X1] set to [4121 pixels]
Parameter [Y1] set to [3121 pixels]
Parameter [X2] set to [135 pixels (-4505)]
Parameter [Y2] set to [353 pixels (-3119)]
--- Bin
Parameter [Scale] set to [(scale/noise reduction 89.20%)/(125.68%)/(+0.33 bits)]
Image Size 3557 x 2470 pixels
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [Temporary AutoDev] set to [Yes]
Parameter [Precision] set to [512 x 512 pixels]
Parameter [Dark Anomaly Filter] set to [20 pixels]
Parameter [Drop Off Point] set to [100 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [90 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [92.06 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
Mask used (BASE64 PNG encoded)

--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [Straight CIELab Luminance Retention]
Parameter [Dark Saturation] set to [2.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.07]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.75]
Parameter [Mask Fuzz] set to [1.0 pixels]
File saved [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19\Lights\M78 (No Drizzle).tiff].
--- Wavelet De-Noise
Parameter [Filter Type] set to [Gaussian Noise Diffusion]
Parameter [Grain Size] set to [6.5 pixels]
--- Wavelet De-Noise
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Scale Correlation] set to [6]
Parameter [Color Detail Loss] set to [200 %]
Parameter [Brightness Detail Loss] set to [200 %]
Parameter [Grain Size] set to [30.9 pixels]
Parameter [Read Noise Compensation] set to [100.00 %]
Parameter [Smoothness] set to [100 %]
--- Layer
Parameter [Layer Mode] set to [Darken]
Parameter [Cap Mode] set to [Clip]
Parameter [Brightness Mask Mode] set to [Off]
Parameter [Filter Type] set to [Gaussian (Fg)]
Parameter [Blend Amount] set to [100 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Filter Kernel Radius] set to [1.0 pixels]
Parameter [Offset X] set to [0.0 pixels]
Parameter [Offset Y] set to [1.5 pixels]
Parameter [Brightness Mask Power] set to [1.00]
File saved [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19\Lights\M78 (No Drizzle).tiff].

Then over to GIMP to stretch out detail Startools just doesn't bring out (in my hands).

My earlier efforts went along the lines of: AUTODEV->CROP->BIN->WIPE->DEV->DECON->CONTRAST->HDR->SHARP->COLOUR->TRACKING OFF/DENOISE->LAYER (fix trailed stars)->GIMP

Here is my FITS file.

https://drive.google.com/open?id=16Z9f7 ... wcMuVM_s3a

And one of my noisey efforts:

https://drive.google.com/open?id=1b0mxe ... RMXplxoWxr

I would love to know what Startools can muster with more experience hands.

Cheers

[almcl]

Had a very quick go, not that I am any sort of expert, using the work flow Richard Konrad outlined a few weeks ago for dealing with noisier images (mine are mostly pretty noisy, so it was of great interest).

Not sure if it's any better and there are several things I am not sure about - there seems to be some walking noise and light pollution as well as some banding which I've attempted to crop out, and I haven't tried dealing with the star halos, but see what you think?

[Melty]

Thanks for making the effort and trying. That looks similar to my earlier efforts.

I still can't get a result comparable to Pixinsight.

Tips anyone?

[happy-kat]

It would help if you want comments please share how many light files and what calibrations files etc.
On my monitor this looks OK but it may look dark on other monitors or mobiles.
StarTools.
Resized 75% and saved jpeg slightly lower quality which has stripped it a bit, the colour was better before the resize in the emissions.

M78 (No Drizzle).ftsv2rslq.jpg (460.64 KiB) Viewed 8634 times

[admin]

Hi,

Appreciate everyone's advice here already.

I had a look at the dataset as well and there are a number of issues/concerns I have regarding your dataset, even before going into processing;

• High Dynamic Range imaging (e.g. blending exposure times) is rarely needed. The only exception I can think of would be M42. In the case of most/all other objects, stars would blow out much earlier than the actual DSO.
• Blending different exposure times will mean blending different noise amounts and stats. This makes noise propagation tracking impossible; noise reduction or measurements that work in one area, would be completely inappropriate in another. My strong advice; just don't do it! If you absolutely need a HDR image (I would not think M78 is not an object that requires this though), process the images separately and then composite them in StarTools in the Layer module (see here).
• The dataset appears to be pre-color calibrated. If using DSS, please observe the setting seen here.
• The dataset does not appear to be calibrated with flat frames. Flat frame calibration, especially when doing DSOs, is not optional. Currently, your dataset shows "dark nebulae" that don't really exist. They are dust donuts caused by minute particles of dust on your sensor. If you're after the smooth faint brown dust, you cannot go without flats.
• As others already mentioned, walking noise (streaks) is visible. Please dither between frames. Walking noise is correlated noise that noise reduction routines find hard to distinguish from real detail (as it doesn't look like random Poisson noise). Bias frames will also help reduce walking noise, but dithering is less time consuming and tends to to the trick as well.
• The unfortunate fact is that, if, after 7 hours of imaging, you still cannot be sure whether something is real detail or a dust bunny in your data, you are not spending your precious time on the right things (namely sorely needed calibration frames!).

We could show you some trickery and "hacks", sure.
However, would you learn anything useful from it that you can use later on? Probably not!

M78 (No Drizzle).jpg (108.7 KiB) Viewed 8620 times

EDIT: The above is probably something i would be the most comfortable with, given that some of the darker detail just doesn't exist in reality...

It would all mostly entail working around the above issues which really just require changing/improving your acquisition methods.
It is unfortunately rarely a good idea to try to address acquisition issues using post-processing.
The only exception is where post-processing workarounds are advisable is when these acquisition issues cannot be helped in any other way, for example due to fixed circumstances or budget gear.

I hope the above "tough love" is useful!

[Melty]

Happy-Kat and Ivo - thanks so much for chiming in and taking time to process my stack.

Sorry for not posting my dataset content previously, but I must say from the outset your surmisation that I am not utilising calibration frames is incorrect - I nearly always religiously get into the habit of shooting calibration frames with all my images (Darks, Flats and Bias frames). If anything, I have been known to go too overboard, if that's possible.

Quite why the flats have not removed the dust donuts (which I was aware of during processing) is beyond me. I have made a couple of custom LED flat-field lightboxes from day one for this very purpose, that has served me well up to this point. The only group i failed to do flats for in this stack was the 60s group. Could that be the source of the noise issues?

The reason for opting for a HDR stack was because my long exposure subs are with CLS filter, and the short ones are without (as so to regain acceptable star colour). I normally stack Kappa-Sigma. I will look into the Layer module in greater detail (i normally only use it to correct elongated stars) in the future.

My mount is a simple Star Adventurer tracker, so any dithering is only available in RA. I have also been led to believe dithering in RA only may actually introduce walking noise, so hence never bothered. I am looking to upgrade to a better full-blown EQ mount, though.

My Data consists of:

141x 30s, 12x 300s, 62x 180s, 20x 60s, 19x 360s lights (in separate DSS groups)
325 Flats
166 Darks
302 Bias

I have since had another attempt at processing the data, this time increasing the dark anomaly filter and denoise grain size up to the max within the process. The results are certainly much better, but still nothing like the image from the other software program. I can't seem to get a filter/grain size big enough to smear out the noise. The dust bunnies/motes were Healed out. Sadly, looks like I lost or overwrote the image, too.

But here's the Pixinsight image I've just figured out how to upload, from said fellow imager, for comparison.

Cheers.

[admin]

Happy-Kat and Ivo - thanks so much for chiming in and taking time to process my stack.

Sorry for not posting my dataset content previously, but I must say from the outset your surmisation that I am not utilising calibration frames is incorrect - I nearly always religiously get into the habit of shooting calibration frames with all my images (Darks, Flats and Bias frames). If anything, I have been known to go too overboard, if that's possible.

That's good to know!
Something must have gone wrong applying them then. They should have reduced unevenness, as well as walking noise, but they have not.
An easy way to find out is to just stack the light frames only without calibration and see what that looks like.

The reason for opting for a HDR stack was because my long exposure subs are with CLS filter, and the short ones are without (as so to regain acceptable star colour). I normally stack Kappa-Sigma. I will look into the Layer module in greater detail (i normally only use it to correct elongated stars) in the future.

Then coloring would still be off in places unfortunately. In addition you're creating severe star halos by doing HDR stacking.
For your intended purpose, indeed try shooting with and without CLS in place. Use the CLS filtered stack for your detail (luminance) and the non-CLS filtered stack for your color information. Then process both stacks separately and combine in the results in the Layer module. This will give you the best of both worlds.

My mount is a simple Star Adventurer tracker, so any dithering is only available in RA. I have also been led to believe dithering in RA only may actually introduce walking noise, so hence never bothered. I am looking to upgrade to a better full-blown EQ mount, though.

Even manually dithering/pushing the mount between frames may help.

I have since had another attempt at processing the data, this time increasing the dark anomaly filter and denoise grain size up to the max within the process. The results are certainly much better, but still nothing like the image from the other software program. I can't seem to get a filter/grain size big enough to smear out the noise.

The simple answer is that we're almost certainly not looking at Poisson (shot) noise. It's uneveness/"signal" that originates from another source, most likely introduced during the stacking or calibration process.
Unlike PI's traditional noise reduction routines that will happily smooth/obliterate everything in the image you pass it, StarTools noise reduction routines use "3D" data, the third dimension being time (signal evolution). If the "signal"/feature was there in your linear data, StarTools will do its best to keep it. In StarTools you will have a harder time destroying features in your image that are not the result of Poisson noise; StarTools assumes clean data (e.g. photon counts), where the only undue influence is Poisson (shot) noise.

IMHO, the PI result is heavily artifacted, showing boundaries that do not exist in reality (for example the star halos). It's one of the objections I have against using TGV for astrophotography purposes; it's a prime example of terrestrial-oriented noise reduction routine that looks for edges and contiguous areas. Such hard boundaries and smooth contiguous areas are virtually never the case in outer space.

[Melty]

Hi, Ivo

I think I'll try abandoning HDR, re-stacking into separate files as you suggested, and take it from there, when I get the opportunity.

So to conclude, ive never heard of the term Poisson noise, but are your saying it's not a case of my micro 4/3 camera's sensor introducing the mottling as a result of the long exposures, rather the way I stack and calibrate?

Sorry for all the questions, but I'm always looking for ways to improve my AP with your fab software.

Thanks

[admin]

Hi, Ivo

I think I'll try abandoning HDR, re-stacking into separate files as you suggested, and take it from there, when I get the opportunity.

So to conclude, ive never heard of the term Poisson noise, but are your saying it's not a case of my micro 4/3 camera's sensor introducing the mottling as a result of the long exposures, rather the way I stack and calibrate?

It really looks like that is the case, yes!

Poisson noise is by far the biggest (virtually "only") source of noise in our images - provided all other sources of unwanted signal are taken care of properly.
It is essentially caused by your sensor's inability to count every incoming photon. Sensors will "miss" some photons randomly at a rate called the sensor's Quantum Efficiency.

So, even if two separate photosites/pixel on your sensor are hit with, say, exactly 100 photons. One photosite/pixel may count 93, while the other might count 87. This discrepancy is visible to us as fine noise grain. It is known as Poisson noise or "shot" noise. If the noise grain is not fine (e.g. you're experiencing mottle/clumps or streaks), something else is at play; such issues are usually caused (and fixed) during acquisition, stacking or incorrect RAW conversion/debayering.

StarTools focuses primarily on Poisson noise, as it is a constant "unknown quantity" throughout your processing that everyone has to deal with. It is the only form of noise that is "unfixable", no matter who you are, what exposure lengths you use and what your equipment is. It is therefore managed and tracked with the utmost care at every stage of processing.

Sorry for all the questions, but I'm always looking for ways to improve my AP with your fab software.

Please keep them coming - asking questions is the only way you and I learn stuff!

[Melty]

Just been playing with the v1.5 pre-release, for which Ivo was kind enough to link me to.

Back to processing old data, and this dataset came to mind. Ideal for back to back version comparisons.

I'd more or less given up on it, because it was:

1) A mish-mash of different exposure lengths which didn't quite stack (DSS Entropy) or process the was I'd liked.
2) Many of the subs were way over-exposed (histogram peak way over halfway on lots of subs - I know better now).
3) Taken before I got to grips with dithering.
4) No matter what I'd tried, I couldn't get Startools to control the noise.

Enter v1.5, with exactly the same data

OK, I'm under no illusions of it making NASA APOD (I may have overcooked the second stretch a tiny bit), but the absolutely incredible improvement I see over my previous effort has left me stunned. I really couldn't be any happier.

Dust lanes, which were previously invisible, are clear to see and the improvement in noise reduction is off the charts. Once again, well done Ivo - it's a remarkable improvement over previous versions!

Here's the log file:

File loaded [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19 (overexposed)\Lights\Autosave.fts].
Image size is 4640 x 3472
---
Type of Data: Linear, was not Bayered, or was Bayered + white balanced
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [Off]
Parameter [Outside RoI Influence] set to [50 %]
Parameter [RoI X1] set to [2761 pixels]
Parameter [RoI Y1] set to [1342 pixels]
Parameter [RoI X2] set to [2039 pixels (-2601)]
Parameter [RoI Y2] set to [1801 pixels (-1671)]
--- Crop
Parameter [X1] set to [4345 pixels]
Parameter [Y1] set to [2959 pixels]
Parameter [X2] set to [469 pixels (-4171)]
Parameter [Y2] set to [269 pixels (-3203)]
Image size is 3878 x 2692
--- Bin
Parameter [Scale] set to [(scale/noise reduction 33.33%)/(900.18%)/(+3.17 bits)]
Image size is 1292 x 897
--- Crop
Parameter [X1] set to [0 pixels]
Parameter [Y1] set to [896 pixels]
Parameter [X2] set to [1197 pixels (-95)]
Parameter [Y2] set to [49 pixels (-848)]
Image size is 1197 x 849
--- Crop
Parameter [X1] set to [0 pixels]
Parameter [Y1] set to [848 pixels]
Parameter [X2] set to [1197 pixels (-0)]
Parameter [Y2] set to [21 pixels (-828)]
Image size is 1197 x 829
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [Precision] set to [512 x 512 pixels]
Parameter [Dark Anomaly Filter] set to [6 pixels]
Parameter [Drop Off Point] set to [100 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [90 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [92.78 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [73 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
Mask used (BASE64 PNG encoded)

--- Contrast
Parameter [Expose Dark Areas] set to [Yes]
Parameter [Compensate Gamma] set to [No]
Parameter [Precision] set to [512 x 512 pixels]
Parameter [Dark Anomaly Filter] set to [3 pixels]
Parameter [Aggressiveness] set to [75 %]
Parameter [Dark Anomaly Headroom] set to [15 %]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal All]
Parameter [Dark/Bright Response] set to [1.00]
Parameter [Detail Size Range] set to [1000 pixels]
Parameter [Strength] set to [1.2]
--- Wavelet Sharpen
Parameter [Structure Size] set to [Large]
--- Color
Parameter [Cap Green] set to [No]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Artistic, Detail Aware]
Parameter [LRGB Method Emulation] set to [Straight CIELab Luminance Retention]
Parameter [Dark Saturation] set to [2.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.00]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.36]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [96.18 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- Color
Parameter [Cap Green] set to [No]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [Straight CIELab Luminance Retention]
Parameter [Dark Saturation] set to [2.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.00]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.36]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Entropy-driven Detail Enhancement
Parameter [Resolution] set to [Medium]
Parameter [Channel Selection] set to [All]
Parameter [Strength] set to [100 %]
Parameter [Midtone Pull Filter] set to [20.0 pixels]
Parameter [Midtone Pull Strength] set to [50 %]
Parameter [Dark/Light Enhance] set to [50% / 50%]
--- Wavelet De-Noise
Parameter [Filter Type] set to [Distance Weighted Outlier Rejection]
Parameter [Grain Size] set to [4.5 pixels]
--- Wavelet De-Noise
Parameter [Scale 1] set to [95 %]
Parameter [Scale 2] set to [95 %]
Parameter [Scale 3] set to [95 %]
Parameter [Scale 4] set to [95 %]
Parameter [Scale 5] set to [0 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Scale Correlation] set to [6]
Parameter [Color Detail Loss] set to [12 %]
Parameter [Brightness Detail Loss] set to [12 %]
Parameter [Grain Dispersion] set to [20.0 pixels]
Parameter [Non-linear Response <] set to [Off]
Parameter [Smoothness] set to [65 %]
Mask used (BASE64 PNG encoded)

--- Shrink
Parameter [Mode] set to [Shrink]
Parameter [Mask Grow] set to [1 pixels]
Parameter [Iterations] set to [1 pixels]
Mask used (BASE64 PNG encoded)


--- Heal
Parameter [Quality] set to [Medium]
Parameter [New Darker Than Old] set to [No]
Parameter [Grow Mask] set to [2 pixels]
Parameter [Neighbourhood Samples] set to [0]
Parameter [New Must Be Darker Than] set to [Off]
Parameter [Neighbourhood Area] set to [200 pixels]
Mask used (BASE64 PNG encoded)

--- Heal
Parameter [Quality] set to [Ultra]
Parameter [New Darker Than Old] set to [No]
Parameter [Grow Mask] set to [4 pixels]
Parameter [Neighbourhood Samples] set to [0]
Parameter [New Must Be Darker Than] set to [Off]
Parameter [Neighbourhood Area] set to [200 pixels]
File saved [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\M78 (Startools v1.5).tiff].
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Tame Highlights]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [1000 pixels]
Parameter [Strength] set to [3.9]
File saved [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\M78 (Startools v1.5).tiff].