Another histogram clipping question

[almcl]

Seeing the thread about histogram clipping in the Feature Requests prompts me to ask a question of my own.

For some while I have been concerned that the histogram (as shown in film dev) always appears cut off at the left at the end of processing:

after superstructure still looks clipped.jpg (33.72 KiB) Viewed 6706 times

So I've just checked my last set of images to try discover where I am getting this effect and it seems that after Autodev -Bin -Crop -Wipe-2nd Autodev the histogram is well over to the left.

I saved at this stage and loaded it into ImageJ and got this result:

histogram after 2nd autodev imagesJ.jpg (95.97 KiB) Viewed 6706 times

That seems to me to indicate a degree of clipping: 10 mega pixels in the first three bins of the 256 bin histogram.

I may be misunderstanding clipping, and of course it's easily fixed by upping the Skyglow in Film Dev by 1%, but I wonder if it's something in the settings in the Autodevs that I am doing wrong? Using the 'Ignore Fine Detail' slider in conjuction with the 'Outside ROI' one does a magnificent job on background noise, but may be introducing this problem?

[admin]

That histogram indeed doesn't look amazingly healthy...
Be careful with the outside ROI parameter, from the docs;

It is important to note that AutoDev will never clip your blackpoints outside the RoI, unless the 'Outside RoI Influence' parameter is explicitly set to 0% (though it is still not guaranteed to clip even at that setting). Detail outside the RoI may appear very dark (and approach 0/black), but will never be clipped.

E.g. don't set it to 0.

The Wipe module will clip dead pixels and dark anomalies. Other than that, I don't know what else could be causing clipping. Perhaps sharing the dataset and log would shed some light on this?

[almcl]

Thanks, Ivo.

As far as I recall the Ignore ROI was 5% and the Ignore fine detail at about 5.5 .

I've put the stack and the extracted log file in a dropbox folder:
https://www.dropbox.com/sh/d4nzfrxvahfj ... _GJXa?dl=0

Please ignore the satellite trails, dust bunnies and horrific vignetting - I am having calibration issues at the moment despite having what appear to me to be good flats, dark flats, bias and dark frames avaialable in a multitude of combinations, so end up cropping most of the frame out.

[admin]

dust bunnies and horrific vignetting - I am having calibration issues at the moment

That looks to be part of the issue. You have some pretty extreme noise correlation going on in your data;

Selection_558.jpg (209.18 KiB) Viewed 6671 times

That's very bad.

Those dark clumps will be filtered by the "Dark Anomaly" filter and the "Ignore Fine Detail <" filters during Wipe and AutoDev, and many of thse outliers may be clipped as they do not constitute reliable background readings. As it stands AutoDev and Wupe are working as expected.

This correlation should be urgently addressed. Are you dithering? The datasets are quite big - you're not drizzling, are you?

The satellite trails should not be in your image - they should have been stacked out if, you used any sort of outlier rejection method (median, kappa/sigma) which is highly recommended.

[almcl]

Thanks for taking a look, Ivo, very much appreciated.

I am dithering (every frame) and no, I don't drizzle. The 30 or so light frames are stacked with a Sigma clip, usually in DSS (but see below).

The output from the ASI 2600 mc pro is a fairly massive .fits file ~ 50Mb. It's presently running at Gain 100, offset 30 and -10°C. Most commonly I am using 180 sec exposures. The statistics look like this:

light stats.jpg (34.03 KiB) Viewed 6664 times

But I *think* the problem is coming from the flats which either over correct, under correct , introduce artefacts that weren't there before and/or fail to remove the dust bunnies even though they can be seen on both lights and flats, with a bit of a stretch. I've tried flats with ADUs of 10k, 15k, 20k and 25k. None of them seemed to work or made much difference. I've tried stacking in DSS, ASTAP (40 minute wait to see the results) and Astroart which just choked and refused to process (at least I think that is what happened - its user interface is a nightmare). I've also tried with and without darks, with and without bias, and with or without dark flats.

One thing that worries me about the flats is the histogram :

flat histogram.jpg (35.68 KiB) Viewed 6664 times

which looks markedly different to that of the lights:

light histogram.jpg (33.79 KiB) Viewed 6664 times

In case it helps, I'll put an individual light, flat and bias frame in the dropbox folder referenced above. Light file prefixed L, flat file prefixed F.

Grateful, as ever, for any insights you may care to offer.

[Burly]

What source are you using for you flats panel ?

[almcl]

Tried several, the method used for the last 7 years was a sheet of white acrylic illuminated by a suitably distanced mains LED bulb, but during this saga I have tried using the stretched T-shirt dawn sky method and more recently an EL tracing panel held in place over the end by a wooden frame.

All similarly unsuccessful. The acrylic panel worked fine with my Canon DSLR.

[woldsstargazer]

I save at each major stage of processing in ST and check the histogram. Your problems could start at the Wipe stage. Definitely check here and go back if you are clipping and change settings eg gradient aggressiveness. Superstructure & denoise are other areas where blackpoint can be clipped so again save before these stages & redo if the histogram is clipped. You also want a nice inverted U at the autodev or filmdev stage and again in the colour module.

[almcl]

Your problems could start at the Wipe stage. Definitely check here and go back if you are clipping and change settings eg gradient aggressiveness.

Yes, indeed the problems do start with wipe. Nothing I've tried in the parameters in the wipe module can prevent it clipping the background:

clipping.jpg (31.89 KiB) Viewed 4854 times

So it is now dealt with towards the end of processing, in the film dev module.

Probably if I knew what 'correlated noise' was and how to prevent or deal with it, (as per Ivo's earlier advice) the wipe module might perform better.

[admin]

Your problems could start at the Wipe stage. Definitely check here and go back if you are clipping and change settings eg gradient aggressiveness.

Yes, indeed the problems do start with wipe. Nothing I've tried in the parameters in the wipe module can prevent it clipping the background:
clipping.jpg


So it is now dealt with towards the end of processing, in the film dev module.

Probably if I knew what 'correlated noise' was and how to prevent or deal with it, (as per Ivo's earlier advice) the wipe module might perform better.

Almost nothing in StarTools clips your data (unless specifically stated in the UI). Wipe - at most - will clip some dark anomalies (because - well - they are anomalies), but that's the extent of it.

The histogram you are displaying is an 8-bit histogram; it will be rather useless when trying to evaluate high bit-depth data that has most of its detail compressed in the extreme lower band of the dynamic range. Incidentally, the latter is precisely Wipe's job.

Why are you concerned about clipping if I may ask?

Correlated noise is noise grain that is bigger than one pixel; if two pixels influence each other's noise levels, it is said to be "correlated noise". Correlated noise may look like bigger-than-one-pixel "blobs" of noise, or may look like tiny vertical or horizontal single-pixel width (or height) "rectangles", extending out two or three pixels.