grain disp> 30, or fill in remaining dark patches? part1

[skygaze]

I'm over my head in this deep end of noise suppression; all I can offer are some curious observations.I know I can somewhat mitigate high ISO noise grain by dropping the gamma down to say 0.7 and lowering the dark saturation.... the images shown here are at full saturation or even stretched to exaggerate the problem so as to make it obvious for discussion purposes. 400x400 section whose LL corner is at the center of the frame.

blotches_sm.jpg (28.82 KiB) Viewed 6563 times

These dark patches are, interestingly, semi-regular across the entire frame. They are there regardless of Nikon D5300, Canon 60Da, 5Diii, or even whether ISO 1600, 2500 or 6400 (just worse at higher ISO). But only semi-regular. Identical settings, exposures, dark/bias frames, identical section of the sensor, but different part of the sky (before denoising, but you can guess where the patches will be). Most interesting, the patches are in different places, so they are not creeping in from the bias, flat, or dark frames (shared by both exposures) :

Noisy_background_comp_notsame_sm.jpg (93.15 KiB) Viewed 6563 times

.
Using a large dark anomaly makes it worse (1 on left, 30 on right):

DarkAnomaly_sizecomp.jpg (170.07 KiB) Viewed 6563 times

con't next post

[skygaze]

Interestingly, binning 50% also makes the dark blotches stand out more:
no bin on left, 50% bin on right

nobin_bin_comp.jpg (164.96 KiB) Viewed 6561 times

That's not really a surprise since binning should increase the bright over the dark.

Now here is why I ask for a grain dispersal than 30 pixels (perhaps a naive ask due to limited understanding):

GrainSize30pix_stretched_sm.jpg (11.02 KiB) Viewed 6561 times

One can still see blobs/clumps, and those are the ones that end up in the final.

Out of curiosity as to the source of the dark patches, I compared 3 light frames (left) to the full 47 light frames at right (deliberately stretched to exaggerate):

lights3vs47_sm.jpg (58.92 KiB) Viewed 6561 times

The more frames I add, the bigger the holes get. Perhaps a result of "wandering noise"? I know nothing about that. Or is it that the inaccuracies in the tracker allowed a 2,3,4 pixel shift which permitted the darker spot to meander E-W (N is to the right)?

Here are some semi-ignorant wild-ass suggestions... perhaps they apply only to these higher ISO or equivalently low signal (short exposure due to tracking mount limitations at a particular focal length):
1) increase the noise dispersal to values greater than 30 pixels)
2) can your noise tracking algorithm be adapted to "see" that those individual green and red and blue pixels are spatially uncorrelated, and therefore noise?
2b) I am drawn to the idea of dealing with that uncorrelated colour noise before the colour module, in part to avoid the lavender blotches, and to avoid skewing the sample color balance.
3) detect these large dark areas and "fill" them to the background noise level so the stretch doesn't go to black there. Denoise 2 is not good here on the times I tried (seems to be a matter of taste)

And as I said at the start, I understand that obviously more signal is better, but I am limited by the equipment I have.
Lots to digest. Regards,
Alister.

[admin]

Hi,

I hope the posts here and here give you sufficient suggestions and background information.

In short, if you artificially introduce detail in your datasets StarTools will preserve that detail. This is a feature not a bug. Blotches and patterns are man-made artefacts. Addressing them during post-processing is not optimal nor - most of the time - needed!

[skygaze]

Better data is the route to go. No argument there!
In terms of previously acquired data (e.g. comets in the field), I can't go back and retake it.
Wouldn't a grain dispersal of 60 help?
Thanks Ivo,
Alister.

[admin]

The more frames I add, the bigger the holes get. Perhaps a result of "wandering noise"?

That's precisely what is going on. A single frame will not show the blobs at all.

Better data is the route to go. No argument there!
In terms of previously acquired data (e.g. comets in the field), I can't go back and retake it.
Wouldn't a grain dispersal of 60 help?
Thanks Ivo,
Alister.

You could also try the 'Non-linear Response <' parameter.

What you are trying to fix is not residual shot noise carried over from a smaller scale, nor does it have the distinct shape (a roughly Gaussian profile "blob") that the Denoise module is designed for to detect and attack.
Your blobs/mottling only exist at one scale only, plus have unique shapes that do not conform to one well-understood signature (e.g. Poissonian or Gaussian grain). The blobs fall more in the category of artefacts than noise. As such they cannot reliably be predicted, measured or statistically modelled in isolation of the rest of the image. The only way to get rid of them is through targeted manual intervention, which falls outside of the scope of a denoising algorithm.

[skygaze]

I appreciate you taking the time to give me a detailed explanation!
It makes more and more sense. Indeed the blobs are non-Gaussian. Even though they can't be "treated properly", I can't stop wondering if a grain size of 60 might mitigate some of it (hoping that from a programming point of view it's a change of a value).
Cheers,
Alister.
Looks like Tuesday night I will get a crack at the manual dither, and nearly the same field to boot (T2 near M81&82).

[hixx]

Hi Skygaze,
I just stumbled over this extremely interesting topic.
one question, please... did dithering work out the blobs for you?

clear skies,
jochen