StarTools 1.8.518 Beta 1 now available

[admin]

Any luck yet replicating wild pixels in SVD? If you've tried that is (I imagine you are very busy!). Or, anyone else notice it yet?

I'm having a bear of a time replicating this issue.
I have seen artifacts on occasion, but that typically involved poor sampling or pushing decon too far.

I can post some samples/screenshots if that would be helpful? But so far I am finding it to occur in nearly any dataset, whether mine or actual good data lol, and whether binned or full scale, or with other modules used first or straight to SVD after Wipe and AutoDev.

Anything that will help me reproduce this reliably (decon settings, dataset) would indeed be super helpful - thank you!

I'm not quite sure what the process is that's going on here, but for a couple stars (and a few others strewn about that are outside of this crop) it almost seemed like some ringings were created that are offset from the star itself. I tried altering settings of course, but couldn't manage to shake it. So kind of the same as wild pixels in that way. I also changed around my sample stars in case I had picked a few clunkers, but that didn't fix things.

If you see "ghosting" (a kind of "offset ringing"), then that is typically a sign of a bad sample somewhere. Usually it is a sample that is accidentally incorporating some other star (a faint companion for example). If it occurs in an area far away from any samples, try setting a sample nearby.

This happened to me a few times and lo-and-behold, I had clicked on a dud sample somewhere.

Also (perhaps obvious), growing the blue boxes ("Sampled PSF Area") will may cause neraby stars for all samples to become included...

Does any of this help?

[Mike in Rancho]

Always helpful, Ivo. And sorry for making all this extra work for you. I'm good at that sort of thing though.

Yes the ghosting was a one-off, and I was thinking the same thing - I must have grabbed the edge of another star. I changed some blue boxes around and when that didn't work, looked for the "erase all blue boxes and start over" button. I couldn't find it. I ran auto-apod mask again, but the blue boxes persisted! They can be hard to find in a big image. I could have just canceled out and started SVD again.

Good to know that ghosting is a clue that there is a wonky blue box somewhere.

Along that line, I wouldn't mind clarification on star sampling. Is it just other apod star outlines that you want to keep out of the box (obviously that), or also non-outlined red "stuff" that may be within the blue box? Pretty hard in a messy, nebulous, or starry full image.

Anyway, the main issue of course is the wild pixels. Odd that I seem to be the only one. I have a hard time finding any dataset which doesn't create them at SVD defaults. Often tons of them. To me they stick out like a sore thumb. My data, Elf data, CN user data, doesn't much matter, though some are worse than others and some can seem to recover a bit easier (but no guarantee on that).

So I figured the easiest way to show this would be some of ST's own tutorial data, and I downloaded the M42 "real world" dataset from the YT page. I replicated the workflow as best as possible. ST1.4 is a bit before my time, and perhaps a bit quaint and archaic.

As always happens, wild pixels appear in SVD with the selection of the very first sample star, no matter how nice and green-cored. This data doesn't seem to have the greatest selection, and many are red-centered, but I boxed maybe 5 or 6 spread around the image.

Settings otherwise left default, although I had already changed to centroid PSF which eliminated some of the wild pixels, and shuffled around a few others. Here's a screenshot with circles. Some may be hard to see because I think I scaled this down too much?

STOrionTutSVDWildPixelsmarked.jpg (106.23 KiB) Viewed 5433 times

Of course this is just a zoomed in view. The wild pixels are literally all over the image, in orbit around various stars. Now, maybe this data doesn't support the default 10 iterations? I backed off one by one. When I got down to 1x iterations, the very last wild pixel in the M42 image finally disappeared. But at that point, is deconv even doing any good?

I then replicated the workflow in 505, although if I got the same blue boxes or not is a toss up. I tried. 10x iterations was not a problem. Well, it may in fact be a wee bit strong, so I bumped the deringing up. But the detail of M42 resolved quite nicely.

And that is exactly what I have found with almost all data.

Here's the workflow, but it's really just copying the log you had posted on YT and mostly mimicking your lasso of the data flaws. I may not have perfectly matched what happened there, as it appears Wipe had some different settings back then, which perhaps evolved into the current vignetting controls.

Code: Select all

-----------------------------------------------------------
StarTools 1.8.506alpha
Sun Aug 01 23:15:37 2021
-----------------------------------------------------------
File loaded [F:\ASTROPHOTOGRAPHY\Tutorial\Startools\M42_Output1.tiff].
Image size is 4304 x 2762
--- 
Type of Data: Linear and was Bayered, but not whitebalanced
--- Bin
Parameter [Scale] set to [scale 50.00% / +2.00 bits / +1.00x SNR improvement]
Image size is 2152 x 1381
--- Crop
Parameter [X1] set to [14 pixels]
Parameter [Y1] set to [11 pixels]
Parameter [X2] set to [2126 pixels (-26)]
Parameter [Y2] set to [1374 pixels (-7)]
Image size is 2112 x 1363
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [Off]
Parameter [Outside RoI Influence] set to [15 %]
Parameter [RoI X1] set to [0 pixels]
Parameter [RoI Y1] set to [0 pixels]
Parameter [RoI X2] set to [2112 pixels (-0)]
Parameter [RoI Y2] set to [1363 pixels (-0)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %]
Mask used (BASE64 PNG encoded)
--- Wipe
Parameter [Synthetic Dark/Bias] set to [Off]
Parameter [Gradient Edge Behavior] set to [Absorb 50%]
Parameter [Synthetic Flats] set to [Off]
Parameter [Sampling Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [3 pixels]
Parameter [Gradient Falloff] set to [0 %]
Parameter [Synth. Bias Edge Area] set to [100 %]
Parameter [Gradient Aggressiveness] set to [75 %]
Parameter [Correlation Filtering] set to [Off]
Mask used (BASE64 PNG encoded)
Redoing stretch of linear data
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [3.0 pixels]
Parameter [Outside RoI Influence] set to [15 %]
Parameter [RoI X1] set to [814 pixels]
Parameter [RoI Y1] set to [195 pixels]
Parameter [RoI X2] set to [1717 pixels (-395)]
Parameter [RoI Y2] set to [920 pixels (-443)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %]
Mask used (BASE64 PNG encoded)
--- Spatially Variant PSF Deconvolution
Parameter [PSF Resampling] set to [Intra-Iteration + Centroid Tracking Linear]
Parameter [Synthetic PSF Model] set to [Circle of Confusion (Optics Only)]
Parameter [Sampled PSF Area] set to [15x15]
Parameter [Synthetic PSF Radius] set to [1.5 pixels]
Parameter [Synthetic Iterations] set to [Off]
Parameter [Spatial Error] set to [1.00]
Parameter [Deringing Fuzz] set to [20.0 pixels]
Parameter [Deringing Detect] set to [50 %]
Parameter [Dyn. Range Extension] set to [1.00]
Parameter [Linearity Cutoff] set to [85 %]
Parameter [Sampled Iterations] set to [10x]
Parameter [Deringing Amount] set to [0.80]

Anyway, unsure if this is a self-inflicted wound from me doing something really dumb. But I'm at a loss.

[almcl]

Mike

Having followed with interest, I can say you are not alone in this. I had a go at some previous data this morning and got a similar although not as pronounced effect:

Selected star

Selected star1.jpg (31.3 KiB) Viewed 5430 times

Before SV PSF Decon:

Selected star2.jpg (22.61 KiB) Viewed 5430 times

After SVPSFD

Selected star3.jpg (30.53 KiB) Viewed 5430 times

A lighter pixel has been generated at the bottom of the central star for no very good reason that I can see?

[hixx]

Mike, almcl,
Very interesting discussion
1) How does SVDECON behave when reducing the spatial error setting somewhat?
In some screenshot it reads 1.0 - a lower setting might make the algorithm less prone to random pixels

2) In almcl's screenshot, interestingly SVD seems to "resolve" a double star (bottom left)
Does anyone know if that is real or wild?
It might be the "lighter pixel" below the center of the sample star is also a double - real or wild ???

3) In general, it might also be just decon'd noise pixels - we should not take this SVDECON output as real output though - if it was a noise pixel, tracking would identify it and DENOISE will get rid of it almost completely. But the SVDECON preview does not take this into account. What happens if you run your SVDECON output through COLOR and DENOISE ? Will those wild pixels be removed/mitigated?

clear skies,
jochen

[almcl]

That's a very interesting point, Jochen!

Unfortunately none of my star atlas/ planetarium programs seem to go into that much detail, although Martin Meredith's Pretty Deep Maps shows the 'resolved' star as a double, it isn't shown in that orientation (sorry, this one is upside down compared to first post).

Pretty deep maps.jpg (35.04 KiB) Viewed 5413 times

Looking at individual subs before stacking or processing, the small star is definitely a double, but the bigger one that eventually produces the wild pixel doesn't show any signs?

Individual sub.jpg (98.01 KiB) Viewed 5413 times

[Mike in Rancho]

Thanks folks, yes good to know I am not alone!

almcl, yes, I too have had the very faint ones that are just a slight brightening of a pixel towards the edge of a star halo, though for the example I chose to show the more separated and brighter ones. But assuming this is universal (though again all data behaves differently), most of us should be able to find them. But it takes a bit of hunting and flipping the before/after and pre/post tweak toggles. That makes them easy to find.

hixx, I forget which post above I mentioned, but yes I did test all the way to the end to see if tracking noted them as noise and erased them. But they didn't go away through color, SS, or denoise, and ended up in the final image.

And yeah I also tested for resolved double stars - by checking Stellarium, the Digital Sky Survey, as well as processing in 1.7 to see what deconv resolved in that version, and even 505alpha since that is the closest analogue to what SVD is doing. In all those cases they are not double stars.

Plus, as I noted, the pixels can shift position as you change settings. If I knew how to make an animated gif I guess I could show it. But, for example, at defaults a star may have some bright orbiting wild pixels created. Changing to centroid resampling may make some go away, but others appear, or may even just shift the position of the bright pixel to a different orientation about the very same star. This is true of some other settings as well. That also convinced me that these are not resolved actual detail.

I have tried dynamic range, spatial error, and all the deringing functions. Again things just shift around some. Maybe I didn't play with spatial error enough? However I thought that setting was to help with certain fringe areas such as field curvature at the far edges? And this can happen everywhere in the image, in all data good or bad, at original or binned/cropped resolutions, and whether other modules were used first or if one goes to SVD straight after wipe and autodev.

Also, again, the difference is from 505a to 506a, with settings identical except we have some new deringing controls in 506, and the data in 505a is okay with the chosen iterations, sampling, spatial error, etc etc.

About the only things I haven't tried is intentionally changing blue box sizing (though I've made sure my stars are all bounded within the squares - I'm not sure how close the star outline can be to the edge, but you can see in almcl's sample he is fully within the box, not up to the edge), or custom changing the apod mask. However, early on I do believe I confirmed that the wild pixels can end up around stars that are apod masked, and stars that aren't apod masked.

Anyway, I don't want to send everyone on wild pixels hunts, but give a few shots to the toggle buttons and see what pops up. You might need to be zoomed in and pixel peeping a little bit. Which is something I do routinely with modules, particularly deconv, to check for ringing, see how well stuff resolves, and look for artifacts.

EDIT: I did just check the instructions for SVD, and in the sample image (he he) for star sampling there are apod outlines fully within a blue box, and some where the star outline goes right to the edge of the blue box. So either should be okay?

[Mike in Rancho]

In case it helps, I also just tried the same Orion data and workflow in (windows) non-GPU, with similar results.

Adding spatial error (it starts at 1.0 all the way to the left now) did not fix them all. As with many of the settings, some may vanish or get better but other wild pixels just shift about.

I started removing blue boxes from the 6 I started with to only 3 in a triangle about M42, all in clear space, and none towards the edges where there might be field curvature. Same deal. Also, adding more spatial error (I went from 1.0 to 1.05 to 1.10 I think) actually made quite a few more wild pixels.

Different issue: because I'm that kind of guy , I managed to crash non-GPU. Following the posted workflow, I went into Wipe and hit the mask button before it was finished with its initial processing. Poof! ST vanished. I started up again and went more slowly, and it stayed running.

Not that I use non-GPU at all. And I haven't seen that behavior out of GPU, which is pretty robust about button-pushing, even with the blue boxes now in 506.

[admin]

I really appreciate all you guys' help trying to get to the bottom of this!

The plot thickens. I followed @Mike in Rancho's log and screenshots to the letter with the ST 1.4 tutorial (2019 seems like a lifetime away indeed! ) and...

...got no aberrant pixels at all, no matter what samples I set.

Then I thought, maybe it's a GPU issue (for optimization purposes, I'm leaving math optimizations to the GPU drivers instead of forcing the exact same implementation for all GPUs), but Mike blew that theory out of the water.

@Mike in Rancho; would you be able to indicate which one sample you used to make the pixels appear with the tutorial data?

[Mike in Rancho]

Happy to help, Ivo. Looking at that 1.4, ST has come a long way, and 1.8 will do it yet again.

So, just to get things started to have you or anyone else even see a hot pixel (maybe), I ran through the workflow again for the M42 tutorial. Even better, I did it simultaneously on my laptop and desktop. Although the hand-drawn Wipe masks probably didn't exactly match. My fault for picking this data. Even so I used the new circle mask creator for the dust donuts, and the line draw plus 20 grows for the bad column areas. So pretty close to matching.

Just about any star creates wild pixels for me, but I decided to choose one sort of towards the center with a pretty green center, already fits in the 15x15, and about as circular as I could find in the apod outlines. It's also easy to find in the star patterns to the left of M42.

STOrion506SampleStar.jpg (112.7 KiB) Viewed 5386 times

After processing, a wild pixel should actually appear on that very star (not sure that's always the case), and many, many others throughout the image. My desktop machine did the same exact thing. The wild pixels were also mostly matching from computer to computer, but not perfectly. Unsure if that's due to slightly different Wipes? Also, the blue boxing of the same star was shifted from other computer to the other, by like 1 pixel over.

Here over to the right is a good concentration of wild pixels.

STOrion506WildPixelsmore.jpg (132.5 KiB) Viewed 5386 times

Again I did nothing here to start suppressing them, this is pure defaults and 1 sample selected.

Both machines are Win10 x64 running GPU ST.
Laptop i7-8550U, 32GB, 940MX.
And my infamously color module crashing desktop, Q9550, 4GB, msi Cyclone N460GTX 1GD5/OC.

[alacant]

Does the debug version I created for @jackbak work though?

Hi Ivo
Yes, i couldn't get it to crash with this version. Must try harder!
Here is the openCL log anyway:
HTH,
Steve

[Num platforms
] :: [1]
[Using platform index
] :: [0]
[Num GPUS
] :: [1]
[Using device index
] :: [0]
[Found GPU
] :: [0]
[Intel(R) HD Graphics Haswell Ultrabook GT2 reserved] :: [0]
[
] :: [0]
[Device Found
] :: [1348381280]
[Creating context
] :: [0]
[Device Index
] :: [0]
[Device Used
] :: [1348381280]
[Creating queue
] :: [0]
[Init complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]
[Registering function
] :: [-1]
[Building program
] :: [0]
[Creating kernel
] :: [0]
[Registering function complete
] :: [0]