StarTools 1.9 Beta

General discussion about StarTools.
decay
Posts: 497
Joined: Sat Apr 10, 2021 12:28 pm
Location: Germany, NRW

Re: StarTools 1.9 Beta

Post by decay »

Nice GIF, Mike :) And I thought I only had a licence to blink. ;)
Mike in Rancho wrote: Wed Jun 28, 2023 5:42 pm almcl's process does turn out much cleaner through SVD, though the question becomes what was done in order to achieve that.
Maybe almcl could be that kind and give us his ST log? So we could try to figure out what was done ... ?

Best regards, Dietmar.
Stefan B
Posts: 475
Joined: Sat Oct 31, 2020 8:59 pm

Re: StarTools 1.9 Beta

Post by Stefan B »

I didn't get it to work reproducibly either. almcl's log would be helpful maybe.

My SVD results regarding deringing were quite a bit different depending on my processing up to SVD. Bin, crop, wipe, Optidev should have been the same, while contrast, HDR and sharp were a bit different. Different stars showed the ringing which I wasn't able to get rid off. (Different sample stars were selected in SVD for sure, too.)

It will get pretty cumbersome to elucidate which differences in 'pre-processing' are crucial for the different ringing effects in SVD...

Regards
Stefan
almcl
Posts: 265
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Re: StarTools 1.9 Beta

Post by almcl »

Here's the log file (masks removed to keep size down).

I think the RoI in the final stretch is probably the most important bit, as most of the rest are just the defaults.



File loaded [E:\Astronomy\someone elses files\M 8, 2023-06-13, 80x6min, (l-eNhance), ATR3CMOS26000KPA_stacked.fits].
Image size is 6224 x 4168
---
Type of Data: Linear and was Bayered, but not whitebalanced
--- RoI Optimized Develop
Parameter [Ignore Fine Detail] set to [Off]
Parameter [Outside RoI Influence] set to [15 %]
Parameter [RoI X1] set to [0 px]
Parameter [RoI Y1] set to [0 px]
Parameter [RoI X2] set to [6224 px (-0)]
Parameter [RoI Y2] set to [4168 px (-0)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %]
--- Bin
Parameter [Mode] set to [Fractional]
Parameter [Scale] set to [Scale 50.00% / +2.00 bits / +1.00x SNR Improvement]
Image size is 3112 x 2084
--- Crop
Parameter [X1] set to [38 px]
Parameter [Y1] set to [30 px]
Parameter [X2] set to [3077 px (-35)]
Parameter [Y2] set to [2043 px (-41)]
Image size is 3039 x 2013
--- Wipe
Parameter [Synthetic Dark/Bias] set to [Off]
Parameter [Gradient Edge Behavior] set to [Absorb 50%]
Parameter [Synthetic Flats] set to [Off]
Parameter [Sampling Precision] set to [256 x 256 px]
Parameter [Dark Anomaly Filter] set to [4 px]
Parameter [Gradient Falloff] set to [0 %]
Parameter [Synth. Bias Edge Area] set to [100 %]
Parameter [Gradient Aggressiveness] set to [75 %]
Parameter [Correlation Filtering] set to [Off]
Redoing stretch of linear data
--- RoI Optimized Develop
Parameter [Ignore Fine Detail] set to [2.5 px]
Parameter [Outside RoI Influence] set to [15 %]
Parameter [RoI X1] set to [247 px]
Parameter [RoI Y1] set to [352 px]
Parameter [RoI X2] set to [2763 px (-276)]
Parameter [RoI Y2] set to [1833 px (-180)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %]
--- Contrast
Parameter [Expose Dark Areas] set to [Yes]
Parameter [Brightness Retention] set to [Off]
Parameter [Precision] set to [256 x 256 px]
Parameter [Shadow Detail Size] set to [10 px]
Parameter [Locality] set to [50 %]
Parameter [Shadow Dyn Range Alloc] set to [50 %]
--- HDR
Parameter [Signal Flow] set to [Tracked]
Parameter [Quality] set to [Medium]
Parameter [Gamma Shadows (Lift)] set to [1.00]
Parameter [Gamma Highlights (Tame)] set to [1.25]
Parameter [Gamma Smoothen] set to [20.0 px]
Parameter [Context Size] set to [50x50 pixels (1.65% image W, 2.48% image H)]
Parameter [Shadows Detail Boost] set to [30 %]
Parameter [Highlights Detail Boost] set to [30 %]
--- SNR-aware Wavelet Sharpening
Parameter [Structure Size] set to [Large]
Mask used (BASE64 PNG encoded)

--- SNR-aware Wavelet Sharpening
Parameter [Protection] set to [Shadows/Highlights Softclip]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [4 px]
Parameter [Amount] set to [300 %]
Parameter [highsnrsizebias] set to [85 %]
Parameter [lowsnrsizebias] set to [0 %]
Parameter [Dark/Light Enhance] set to [50% / 50%]
Mask used (BASE64 PNG encoded)


Mask used (BASE64 PNG encoded)

PSF samples used (5 PSF sample locations, BASE64 encoded)

VFMAAAUA3wvdBzcKywAsCeIE9QmyBAgJKgboAMkC

--- Spatially Variant PSF Deconvolution
Parameter [Synthetic PSF Model] set to [Circle of Confusion (Optics Only)]
Parameter [Synthetic PSF Radius] set to [1.5 px]
Parameter [Synthetic Iterations] set to [Off]
Parameter [Spatial Error] set to [1.00]
Parameter [Deringing Focus] set to [84 %]
Parameter [Dyn. Range Extension] set to [1.00]
Parameter [UNKNOWN] set to [13 %]
Parameter [Sampled Iterations] set to [10x]
Parameter [UNKNOWN] set to [100 %]
Mask used (BASE64 PNG encoded)

--- Crop
Parameter [X1] set to [936 px]
Parameter [Y1] set to [504 px]
Parameter [X2] set to [2137 px (-902)]
Parameter [Y2] set to [1685 px (-328)]
Image size is 1201 x 1181
File saved [E:\Astronomy\someone elses files\M 8, deringing.jpg].
Mask used (BASE64 PNG encoded)
Skywatcher 190MN, ASI 2600 or astro modded Canon 700d, guided by OAG, ASI120, PHD2
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: StarTools 1.9 Beta

Post by admin »

It's indeed an interesting failure case for the de-ringing.
I will have to do some troubleshooting to see what is happening here, but it appears the issue is some sort of extreme non-linear response in the troublesome stars. Similarly odd, is a complete lack of overexposure anywhere - it may be that you got your exposure times perfect, but I think something else might be going on here... :think: Indeed, you will notice that if you change the linearity setting down in SVDecon, the ringing becomes less.

Is there anything special about your camera in the way it deals with overexposure, highlights, or the top of the dynamic range. Can you confirm its response is linear?
Ivo Jager
StarTools creator and astronomy enthusiast
decay
Posts: 497
Joined: Sat Apr 10, 2021 12:28 pm
Location: Germany, NRW

Re: StarTools 1.9 Beta

Post by decay »

Let us know, if we can be of any help, Ivo. Fiddling with different settings now that we have almcl's log (thanks, @almcl)? But I'm sure, you have a much more sophisticated way to figure out things ... ;)

Dietmar.
Mike in Rancho
Posts: 1166
Joined: Sun Jun 20, 2021 10:05 pm
Location: Alta Loma, CA

Re: StarTools 1.9 Beta

Post by Mike in Rancho »

Thanks for the log, almcl! :D

I ran through it quickly as a test, seems to match up well with what you posted and was used in the blinky gif. But, with the ability to zoom to 400% or better, I'd say there's still what I would deem unacceptable ringing. That may wash out and not be very perceptable once we get to Color, but I didn't go that far.

Some differences. I loaded as bicolor rather than straight RGB, considering that more appropriate to the acquisition. But really that should just change the synth L by something like a 1.44% G bump. Noisier? Maybe. I also had stretched no-ROI, but the ROI here is fairly mild. I did use Contrast (can't recall if defaults or if I softened it by lowering the locality slider) but HDR only mildly, not defaults which I felt a bit strong here. Also I skipped Sharp.

In SVD I copied and then loaded your 5 PSF's. At the outset, the appearance is what we have been seeing of late. At least with my data, Stefan's, maybe Ron's. And again interestingly, some "other people" data I try works out much differently. Including a buddy with a RASA11 and dark skies, and he always takes like 2 minute full spectrum OSC subs and blows out countless star cores lol. Go figure.

I didn't fiddle too much (I should be on the freeway heading to the office lol) but just tried your 13% fuzz, which helped. The focus made little to no visible change on pre-post tweak toggle after that.

Now, some of the variance in results could be the stretch and the use of Contrast, HDR, and Sharp such that certain affected stars weren't surrounded by as bright a background, but I'm not entirely sure. More experimenting needed later.

By toggling back to the sampling screen, I maybe (?) noticed what appeared to be a distinct difference in the deringing effect between stars that are masked (though not necessarily selected) and those that aren't. This may explain some issues I've raised before as it seemed to be hinting at some sort of dichotomy - i.e. vast differences between the deringing controls as to different stars. I had wondered if there was a background support mask, but perhaps it's still just the sampling mask (don't know if we still call it an apod anymore)?

Anyway, masked stars don't seem to need as much deringing settings as non-masked. Hence, you can get side-by-side stars that behave completely differently, even if they seem fairly similar otherwise. And one can also get tons of little tiny stars with weak or no deringing, requiring some custom pixel clicking in Shrink later.

EDIT: On further thought, I suppose there could also be something about the star characteristics themselves that led to some being in the sampling mask while others were rejects?

Or I'm just completely off my rocker. :lol: I'll go back at it when I have more time later.

:D
Stefan B
Posts: 475
Joined: Sat Oct 31, 2020 8:59 pm

Re: StarTools 1.9 Beta

Post by Stefan B »

Hi Ivo,

thanks for looking into that.
admin wrote: Thu Jun 29, 2023 8:20 am Similarly odd, is a complete lack of overexposure anywhere - it may be that you got your exposure times perfect, but I think something else might be going on here... :think:
Mmhh...pretty unlikely that the exposure time is perfect. According to stats in NINA I always have several hundred pixels at maximum ADU if I remember correctly. Would it be helpful if I share some sample lights?
admin wrote: Thu Jun 29, 2023 8:20 am Indeed, you will notice that if you change the linearity setting down in SVDecon, the ringing becomes less.
Yes, I noticed that, but it doesn't vanish completely.
admin wrote: Thu Jun 29, 2023 8:20 am Is there anything special about your camera in the way it deals with overexposure, highlights, or the top of the dynamic range. Can you confirm its response is linear?
The camera is one of the Touptek clones with the color IMX571 sensor which now a lot of people use. It's used with NINA with NINA's native driver. The camera has an extended full well mode which I have never used until now. I don't think that I have used it or activated it inadvertently. The high conversion gain was used which reduces read noise (but also the full well capacity as far as I now). For gain and offset I used 100 as I always do. AFAIK the camera's response is linear.

I then stacked in ASTAP with the settings I also always use. No fiddling there. Should be the settings recommended on the ST website.

Regards
Stefan
Stefan B
Posts: 475
Joined: Sat Oct 31, 2020 8:59 pm

Re: StarTools 1.9 Beta

Post by Stefan B »

Independently from the non linearity issue: I am wondering why deringing works with the one slider control but not with the two slider control.

My assumption was that the two slider controls give you a two dimensional search space for a combination of settings. Since the sliders have 0-100% you have 101 x 101 = 10.201 possible combinations in the search space where your optimal combination is hiding. Likely there's are region of similar combinations giving you similar results being okay.

My guess was that the one slider control was just a one dimensional line through the two dimensional space reducing the 10.201 combinations to 101 possibilities (0-100%). But that's probably not true, right? The two slider controls works differently than the one slider control?

Regards
Stefan
Mike in Rancho
Posts: 1166
Joined: Sun Jun 20, 2021 10:05 pm
Location: Alta Loma, CA

Re: StarTools 1.9 Beta

Post by Mike in Rancho »

Good questions I think, Stefan.

I suppose you could think of the two right sided sliders as setting up those various permutations. :think:

To me I was thinking of fuzz as...blending or mask fuzz. Which explains why low means stronger deringing. So whatever underlying deringing structure is being brought back in, it's fuzzed and I presume from the outer edge?

The focus I presumed to be the strength of the "gradient" that fuzzing has. Could be wrong on that.

I think things are close, but the deringing structure that gets fuzzed in strikes me as slightly undersized, and the focus slider seems underpowered, if it is suppose to affect the gradient of the fuzzing.

Away from that specifically, I played with some additional data but still don't know why some stars get those severe black eyes, other than they seem to not be outlined in the apod mask. But I have other data that doesn't get black eyes, regardless of the sample mask. :confusion-shrug:

Finding out what is underlying that might ease up our reliance on deringing too.

I also noticed that some pre-processing steps, probably mostly OptiDev, do affect how SVD draws the white (and sometimes blue) outline mask including which stars are offered up. And in turn, if that now causes a problem star to be outlined in the mask when it wasn't with a different stretch, its black eye goes away.

Weird, huh?
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: StarTools 1.9 Beta

Post by admin »

Hi all,

I just released a new beta version. I finally found the issue causing the outsized ringing due to apparent non-linearity.

I was being a... not so smart person. :roll:

The issue was, that I was not taking into account the effect that synthetic luminance generation has on overexposing star cores.
Take blending, say, a duo/tri/quad band image (like @Stefan B's dataset - many thanks to you and @Mike in Rancho :bow-yellow: for helping to crack the case);

Anything not Ha will overexpose quicker than the Ha. The synthetic luminance will then in places become a blend of over-exposing cores/singularities and actual data from the Ha. To the eye, such areas actually look like plausible stellar profiles (they transition nicely), however such areas are highly non-linear and will quickly trip up deconvolution, no matter how hard the deringing algorithm tries to keep things in check.

I am now keeping track of such areas from the time they come in existence and treat them as any other over exposing singularities in your dataset. I'm now getting much better results with these types of datasets and hope you do too.

Please let me know how you go!
Ivo Jager
StarTools creator and astronomy enthusiast
Post Reply