Blotches patches mitigating high ISO noise

[skygaze]

Please, if you know a better approach/workflow (than what is below) for minimizing blotches or patches, let us know. I've seen them on other forums and with Nikon, Canon 60Da, 5Diii, so it is a not-uncommon experience. Ivo, I will do a separate post in feature request about an idea with extra images. The following images are a 400x400 region where the lower left corner is the image center. Just using suggested values:

blotches_sm.jpg (28.82 KiB) Viewed 5854 times

Using gamma at 0.7:

20200513_Siril_nobin_DA30_gam70_grain30.jpg (21.16 KiB) Viewed 5854 times

If you look closely, you can still see the blotches, but you can adjust to suit your taste. Like most things in processing, there is a trade-off: eliminating blotches means less nebula.

Almost needless to say the best solution is to improve your signal to noise, and dither (in my case I am positing 15 pixels). Yeah thanks, I can't with my current equipment, and it's not much help for last month's comet... so in the meantime, here is my workflow to mitigate the blotches:

1. autodev - crop - wipe as usual.
2. I do NOT bin, because it makes my dark patches more obvious (binning increases the contrast between dark and not-dark)
3. In wipe, try leaving dark anomaly at 1 or 2 pixels - setting it at 30 just seems to emphasize dark blotches.
4. 2nd autodev : ignore detail at 10 (doesn't seem to really matter); shadow linearity at 15%; [you might want to experiment here]
KEY: gamma at 0.7; different cameras may be better at 0.6 [my 60Da], or 0.8. This pushes the grain fainter! The lower the value, the less you will see the grain. But it also means the fainter the nebular parts get. You'll typically want to keep some.
5. Color: KEY: dark saturation to 1.2. With our problem of high ISO noise, our darker pixels carry a lot of color; setting this value quite low desaturates the noise (as well as the faint parts of the nebula), but as a benefit I don't have oodles of green pixels influencing the overall color balance. Make sure you click on mask, lasso some stars, do, and click sample to get a new color balance. Go back into mask, clear and invert to full, and return to the main color interface. Click keep.
6. Noise reduction: I've been setting grain size at 30, but it's not big enough as far as I can tell.
7. Anything else you need to do, you can at this stage.
8. I move over to GIMP (you can use Photoshop etc) to bring up curves, and play with the high and low end and saturation until I like the result, pushing the blotches to the edge of visibility, keeping as much nebulosity as possible.

Hope this helps! Again, if you have something better, please contribute. Regards,
Alister.

[admin]

Hi Alister,

Blotches and other correlated patterns should not occur in your datasets.
Are you using flats and do you dither sufficiently between frames?
Are you using the optimum ISO for your camera?

[Bobby_1970]

Please, if you know a better approach/workflow (than what is below) for minimizing blotches or patches, let us know. I've seen them on other forums and with Nikon, Canon 60Da, 5Diii, so it is a not-uncommon experience. Ivo, I will do a separate post in feature request about an idea with extra images. The following images are a 400x400 region where the lower left corner is the image center. Just using suggested values:

blotches_sm.jpg

Using gamma at 0.7:

20200513_Siril_nobin_DA30_gam70_grain30.jpg

If you look closely, you can still see the blotches, but you can adjust to suit your taste. Like most things in processing, there is a trade-off: eliminating blotches means less nebula.

Almost needless to say the best solution is to improve your signal to noise, and dither (in my case I am positing 15 pixels). Yeah thanks, I can't with my current equipment, and it's not much help for last month's comet... so in the meantime, here is my workflow to mitigate the blotches:

1. autodev - crop - wipe as usual.
2. I do NOT bin, because it makes my dark patches more obvious (binning increases the contrast between dark and not-dark)
3. In wipe, try leaving dark anomaly at 1 or 2 pixels - setting it at 30 just seems to emphasize dark blotches.
4. 2nd autodev : ignore detail at 10 (doesn't seem to really matter); shadow linearity at 15%; [you might want to experiment here]
KEY: gamma at 0.7; different cameras may be better at 0.6 [my 60Da], or 0.8. This pushes the grain fainter! The lower the value, the less you will see the grain. But it also means the fainter the nebular parts get. You'll typically want to keep some.
5. Color: KEY: dark saturation to 1.2. With our problem of high ISO noise, our darker pixels carry a lot of color; setting this value quite low desaturates the noise (as well as the faint parts of the nebula), but as a benefit I don't have oodles of green pixels influencing the overall color balance. Make sure you click on mask, lasso some stars, do, and click sample to get a new color balance. Go back into mask, clear and invert to full, and return to the main color interface. Click keep.
6. Noise reduction: I've been setting grain size at 30, but it's not big enough as far as I can tell.
7. Anything else you need to do, you can at this stage.
8. I move over to GIMP (you can use Photoshop etc) to bring up curves, and play with the high and low end and saturation until I like the result, pushing the blotches to the edge of visibility, keeping as much nebulosity as possible.

Hope this helps! Again, if you have something better, please contribute. Regards,
Alister.

I am finding very similar issues when I use Startools on my images.
I have used EOS450d, Sony A57, and also a ZWO 178MC.

My workflow looks pretty similar to yours yet whatever I do I always end up with a blotchy or noisy background. I have used "heal" to remove this somewhat but im not sure that the right way to go tbh.

Sorry to jump in but im keen to know a solution for this myself.

[skygaze]

Hi Ivo,

Blotches and other correlated patterns should not occur in your datasets.
Are you using flats and do you dither sufficiently between frames?
Are you using the optimum ISO for your camera?

These blotches are not correlated, only somewhat as they are similar in character and appear semi-periodic but are different in specifics from target to target with identical bias/dark (see my post in feature request with detailed comparison images).

Using ISO 1600 for 5Diii as recommended (thanks for that link on the tutorials page).

NOT dithering as my current equipment will not let me. In June I am sending my StarAdventurer back to the manufacturer because I can not get phD to calibrate (which would allow me to dither). I am positing a dither of half the size of the blotches so, 25/2 = 12 pixels (or should it be 1x? Unsure if dither already includes a +/-) which for a 100mm lens on a 5Diii amounts to 18 deg * 12/5760 = 2.25 arcmin.
I see dithering as a key mitigation.

Bobby I would be interested to see how you use heal - is it one by one or does it cover the whole image in one shot?

[Bobby_1970]

Hi Ivo,

Blotches and other correlated patterns should not occur in your datasets.
Are you using flats and do you dither sufficiently between frames?
Are you using the optimum ISO for your camera?

These blotches are not correlated, only somewhat as they are similar in character and appear semi-periodic but are different in specifics from target to target with identical bias/dark (see my post in feature request with detailed comparison images).

Using ISO 1600 for 5Diii as recommended (thanks for that link on the tutorials page).

NOT dithering as my current equipment will not let me. In June I am sending my StarAdventurer back to the manufacturer because I can not get phD to calibrate (which would allow me to dither). I am positing a dither of half the size of the blotches so, 25/2 = 12 pixels (or should it be 1x? Unsure if dither already includes a +/-) which for a 100mm lens on a 5Diii amounts to 18 deg * 12/5760 = 2.25 arcmin.
I see dithering as a key mitigation.

Bobby I would be interested to see how you use heal - is it one by one or does it cover the whole image in one shot?

I am no expert at all, far from it. I barely understand some of the terms talked about.lol

But

What I have done previously, is use the mask command to select the areas of "blotchiness". I used "similar brightness" to select them.

Then in the Heal command select the "new must be darker" option, and change the "medium" to "high"

I am probably not explaining it very well. Also for heal to work you still need to have at least 10% of the pixels not masked.

This still allows for a large amount of the background to be selected, have a play around and see what you think.

[admin]

Hi Ivo,

Blotches and other correlated patterns should not occur in your datasets.
Are you using flats and do you dither sufficiently between frames?
Are you using the optimum ISO for your camera?

These blotches are not correlated, only somewhat as they are similar in character and appear semi-periodic but are different in specifics from target to target with identical bias/dark (see my post in feature request with detailed comparison images).

Using ISO 1600 for 5Diii as recommended (thanks for that link on the tutorials page).

NOT dithering as my current equipment will not let me. In June I am sending my StarAdventurer back to the manufacturer because I can not get phD to calibrate (which would allow me to dither). I am positing a dither of half the size of the blotches so, 25/2 = 12 pixels (or should it be 1x? Unsure if dither already includes a +/-) which for a 100mm lens on a 5Diii amounts to 18 deg * 12/5760 = 2.25 arcmin.
I see dithering as a key mitigation.

Bobby I would be interested to see how you use heal - is it one by one or does it cover the whole image in one shot?

Hi,

The pixels that make up these blotches are - by definition - correlated. E.g. they form detail (a blotch) that is not real signal, and is the a result of a local signal error. The only local signal error in a properly calibrated dataset should be shot noise. E.g random per-pixel fluctuations in signal and nothing else. If multiple pixels in a local neighbourhood fluctuate in the same way, that is correlation and looks like blotches, streaks, etc. Not withstanding cosmic rays, the origin of such correlated fluctuations is never the real celestial signal (or its noise component) itself - it almost always an issue that is artificially introduced; e.g. not dithering, bad flats and/or not calibrating with bias or dark frames if you instruments needs those.

In short, blotches are virtually never an artefact of a particular camera or sensor, but almost always find their origin in acquisition techniques and pre-processing.
If you cannot dither with your current setup, you may even benefit from nudging the scope manually every x frames. Anything you can do to not have the same co-located photosites on your sensor collect the same signal will greatly help. Be sure to use a stacking algorithm that rejects samples (e.g. median, kappa sigma, windsorised clipping, etc.), rather than simply averages them.

Finally, if blotches are absolutely unavoidable, as a last resort, you can configure the noise reduction modules in StarTools to attack large scale noise (e.g. mottling/blotches), by increasing the Grain Size/Grain Dispersion and Scale 5 parameters in the De-Noise module, or Grain Size/Grain Limit parameters in the Denoise 2 modules.

Let me know how you go!

[skygaze]

Ah I see what you mean by correlated. I was thinking of it in the sense of one blob to another (in a pattern) or blobs on image 1 correlated to blobs on image 2. You are correct of course.

In the upcoming night I will go for manual dithering. Just increase the gap between images to give me the time to adjust and let vibrations die. Good idea.

Scale 5 on denoise1 at 100 is definitely an improvement, the blobs reduce by almost 1/2 their size. Any chance you can put in a scale 6 and larger grain dispersion (say to 60)? I do get your point about preventing the problem in the first place, but this could help previous shots.
Cheers and thanks.
Alister.

[skygaze]

Woohoo! Bobby, thank you so much for suggesting Heal.

On my latest attempt, in autodev I kept gamma closer to 1.0 (0.8 instead of 0.7) than previous.
- in colour, I continued to push dark saturation back to 1.2
- In denoise1, I used grain size at 30.9, and this time scale 5 at 99%.
- Heal: use the mask command to select the areas of "blotchiness". "similar brightness" to select them. Use grow/shrink to suit.
because I am filling in dark patches, not suppressing bright stuff I switched your values around:
New Must be darker at 99%; Quality LOW; Neighborhood Area 290; Neigh Samples 86; New Darker than Old NO.

Here is my recent try compared to my previous "best" processing (top right of original post):

Scale5Heal_comp_sm.jpg (40.84 KiB) Viewed 5796 times

I *like* it!
Alister.