StarTools 1.8.518 Beta 1 now available

[Mike in Rancho]

Thank you for reporting (and sharing the dataset @Mike in Rancho!)

I have not seen any at all on Linux, so I'm wondering if it might be something specific to the Windows version...

As "usual" I cannot replicate this on Linux (the reference platform/OS for ST). So I'm going to have to do some wider testing. Do they appear on the non-GPU version as well?

Non-GPU!! (I looked for the barfing emoji, could not find)

Ok, so I sacrificed and took one for the team. It was painful.

Despite identical workflow, I could not make an apples to apples comparison. Non-GPU seems to be a wee bit off, unfortunately. Here's what I was seeing--

ST nonGPU screen SVD what the.jpg (209.86 KiB) Viewed 8959 times

I even did some slightly different workflows and even a different stack (same target though), each taking incredible amounts of time lol, and it was really just...bizarre. Digging a little deeper, it appeared to me that non-GPU was creating a different apod mask than GPU.

So I figured I'd save the mask from GPU and load it into non-GPU. But, ST would not let me run a second instance (even tried 511, thinking different folder and all). So, I fired up the desktop. I think I've mentioned it before, old quad core but a decent gpu, runs ST great except crashes color. But color isn't needed here. Crazy, that old beater was like a race car compared to the new laptop using non-gpu. I saved the apod mask to a thumbdrive (same data and workflow) and loaded it into non-GPU. Didn't help, stuff was still crazy all over.

One thing I don't know is if this issue affects only the apod masks, or also the sharp mask (which was run in this workflow), thereby leading to different starting inputs into SVD, or if the differences between GPU and non-GPU results go beyond masks?

Later I took saved apod masks for this data and workflow from each of the laptop GPU, laptop non-GPU, and the desktop GPU already saved, and compared them in Gimp layers. For sure different. The two GPU made masks, despite very different machines, looked identical, and a subtraction proved that to be so. Oddly, when I subtracted the GPU and non-GPU masks, the result was sorta kinda like wild pixels and the other bizzaro ringing artifacts (like a curved chain of single pixels) that were resulting in non-GPU SVD.

So...hmmmm. Peculiar.

[admin]

Thank you for reporting (and sharing the dataset @Mike in Rancho!)

I have not seen any at all on Linux, so I'm wondering if it might be something specific to the Windows version...

As "usual" I cannot replicate this on Linux (the reference platform/OS for ST). So I'm going to have to do some wider testing. Do they appear on the non-GPU version as well?

Non-GPU!! (I looked for the barfing emoji, could not find)

Ok, so I sacrificed and took one for the team. It was painful.

Despite identical workflow, I could not make an apples to apples comparison. Non-GPU seems to be a wee bit off, unfortunately. Here's what I was seeing--


ST nonGPU screen SVD what the.jpg


I even did some slightly different workflows and even a different stack (same target though), each taking incredible amounts of time lol, and it was really just...bizarre. Digging a little deeper, it appeared to me that non-GPU was creating a different apod mask than GPU.

So I figured I'd save the mask from GPU and load it into non-GPU. But, ST would not let me run a second instance (even tried 511, thinking different folder and all). So, I fired up the desktop. I think I've mentioned it before, old quad core but a decent gpu, runs ST great except crashes color. But color isn't needed here. Crazy, that old beater was like a race car compared to the new laptop using non-gpu. I saved the apod mask to a thumbdrive (same data and workflow) and loaded it into non-GPU. Didn't help, stuff was still crazy all over.

One thing I don't know is if this issue affects only the apod masks, or also the sharp mask (which was run in this workflow), thereby leading to different starting inputs into SVD, or if the differences between GPU and non-GPU results go beyond masks?

Later I took saved apod masks for this data and workflow from each of the laptop GPU, laptop non-GPU, and the desktop GPU already saved, and compared them in Gimp layers. For sure different. The two GPU made masks, despite very different machines, looked identical, and a subtraction proved that to be so. Oddly, when I subtracted the GPU and non-GPU masks, the result was sorta kinda like wild pixels and the other bizzaro ringing artifacts (like a curved chain of single pixels) that were resulting in non-GPU SVD.

So...hmmmm. Peculiar.

Thnak you.

I'm trying my hardest to replicate this, but having zero luck.
From the screenshot, it doesn't appear as if the apodization mask is working at all.
You mention something about the Sharp mask though. Please note that you cannot use this mask for Decon (unless you invert it). An apodization mask != star mask. Might that be the source of the issue?

Can you confirm the big blown out stars were adequately covered by mask pixels in the mask editor?

If none of this helps us, would you be able to test without Wipe's correlation filter applied?

[Mike in Rancho]

Thnak you.

I'm trying my hardest to replicate this, but having zero luck.
From the screenshot, it doesn't appear as if the apodization mask is working at all.
You mention something about the Sharp mask though. Please note that you cannot use this mask for Decon (unless you invert it). An apodization mask != star mask. Might that be the source of the issue?

Can you confirm the big blown out stars were adequately covered by mask pixels in the mask editor?

If none of this helps us, would you be able to test without Wipe's correlation filter applied?

Oh no, such lack of faith. No I was not using a mask from Sharp in SVD. Both were auto-generated at the outset of their respective modules. And I imagine I would have noticed the complete failure of SVD. I was, inartfully it seems, mentioning the Sharp mask only in the context of GPU vs non-GPU, since that was last night's specific test. GPU and non-GPU created quite different apod masks, so I wondered if it was a mask generation issue in general, and noted that Sharp had been used in the workflow prior to SVD. The concern being that the data did not enter into SVD in the same state, again GPU vs non-GPU.

In any event, there's certainly something separate going on with apod mask generation in non-GPU. Knock on wood I don't use non-GPU.

I did more testing (GPU only) tonight whilst sitting here (laptop is busy imaging). As far as I can tell, Wipe correlation filter to off did not make a difference. Or if so, very little. Hard to tell since it all seems fairly dynamic.

Skipping Sharp completely did seem to help with a couple of the stars that are prone to pixels. As perhaps did a ROI in AutoDev to reduce the stretch, and giving a grow to the apod mask. And maybe even extra super careful star sample selection. But even those didn't necessarily mean no wild pixels at all.

I did take a number of screenshots of stars in various configurations - apod mask, mask editor, results - so maybe can mosaic some together.

What does count as adequate coverage for a big star in the SVD mask? i.e. how far out into the halo? I did try one and even two grows (expanding blue boxes as needed), but that didn't seem to be enough in and of itself.

I will have to be more scientific in my method, and perhaps choose a better dataset as an example.

[Mike in Rancho]

Ok, for more WP info, I decided to make this a little easier, though it will perhaps have to span several posts depending on attachment limitations.

Here is an easier to deal with dataset. https://drive.google.com/file/d/1odjUJf ... sp=sharing

Same underlying DSS stack, but pre-cropped, rotated and saved by ST, still linear.

I then proceeded with a default workflow: Compose bicolor from DSLR. Bin 71%. Wipe - defaults, no corr filter. A/D defaults with IFD 4.0. Contrast defaults. HDR defaults. Sharp defaults. SVD automask.

This is the generated mask.

[Mike in Rancho]

#2

I selected 7 green (or mostly) stars that seemed reasonable, spread throughout the image. Here is their code: VFMAAAcAsAjpBW8DPgQbBccAfgDJA+8FeAQzAuUAKAihBOIGFQE=

Zooming in on a couple of the affected stars in the nebula area (there may be one more off-screen), here is what the apod sampling screen looks like to show the star perimeters chosen (note none of the sample stars are in this FOV).

screenshot SVD apod mask.jpg (81.93 KiB) Viewed 8890 times

And here is the same in the mask editor view.

screenshot SVD mask ed.jpg (93.37 KiB) Viewed 8890 times

And finally the results view.

screenshot SVD result.jpg (107.16 KiB) Viewed 8890 times

You can see the obvious aberrant pixels mostly lower left of the one star, but there is also a faint WP just to the right of the upper bright star.

[Mike in Rancho]

#3

Here is the results view after I did some select grow blob on the affected stars. It does help the deringing to work better. But, does not eliminate WP's. Even if I grow that particular star mask really big, the WP's still remain inside the mask perimeter.

screenshot SVD result aft grow blobs wp stars.jpg (124.58 KiB) Viewed 8889 times

And here, I regenerated the apod mask, and then did a global mask grow. I believe I had to remove one offending side star that impinged on a blue box after this grow.

screenshot SVD result aft 1x global mask grow.jpg (123.73 KiB) Viewed 8889 times

This may or may not be the best dataset for testing. Firebrand18 stated he may have some glob data that is more prone to WP's?

The only real changes to WP's seem to be with selection of the number and location of sample stars. But is is hard to eliminate fully. Sometimes with just one well-selected star, maybe. But really once you start trying to sample across the image, which is SVD's whole point I think, then WP's more readily appear.

Which makes me think that the spatial transforms being done must have something to do with it. Particularly since the WP's, while somewhat variable, do most often appear in quasi-fixed spots as to particular stars, and multiple stars can often have WP's hanging off of them in exactly the same orientation.

It also seems to me, from toggling and pixel peeping, that the WP's may be generating just off the edge of some mask structure. Perhaps the initially-created apod? Odd though that changing the apod/mask editor does not seem to "move" them. More testing on that needed, maybe. Is there an alternate mask going on that is unseen?

Oh and all this was in Windows GPU, of course. Let me know if there's anything else to try, test, or data to attach.

[jackbak]

Now that I am NOT using the intel GPU my masks are working but I am also getting wild pixels in my M8 - Lagoon data. I can really ramp them up
by increasing the iterations in SVDecon (10x - default is way too high) 2x I can sneek by with.

[admin]

Thank so much for you thorough analysis and help. It has helped me find some issues in the non-GPU version. Due to a bug in the Tracking code, it is indeed much easier to produce copious amounts of artefacts on this version (GPU code backport went wrong and was incomplete ).

Mask generation is indeed a little different on the non-GPU version too. I have also found minute differences when allowing the GPU driver to select its own preferred/native precision and mathematical operation implementations, vs forcing high precision, though these are mostly imperceptible.

I have also been able to produce precisely one tiny artefact with the Tulip dataset around a strongly over-exposing star (I believe it's the same star as in your screenshot, just 180 degrees rotated). However, that is really the extent of what I have been able to produce so far...

Screenshot-1.jpg (46.71 KiB) Viewed 8818 times

It is perhaps worth mentioning though, that normally Deconvolution is extremely sensitive to noise or artefacts and singularities (such as over-exposing star cores, or hot/dead pixels, etc.). We are quite spoiled in "StarTools land" with the signal evolution tracking engine, but is perhaps worth remembering that in any other traditional application, deconvolution is generally advised against (for example by the author of PI here and here), unless you have exceptionally clean data at hand. Such is the challenge of implementing this type of signal recovery algorithm.

The next alpha will contain the mentioned bug fixes, as well as a tweak to the correlation filter in the Wipe module (in addition to some other stuff). I'm targeting the end of this week for ETA (no promises though!).

Thanks again! This has been of tremendous help.

[firebrand18]

@Ivo, you hit the nail on the head with "It is perhaps worth mentioning though, that normally Deconvolution is extremely sensitive to noise or artefacts and singularities (such as over-exposing star cores, or hot/dead pixels, etc.)." That is probably the crux of the problem in my case as shooting from a Bortle 8 where noise and other spurious junk makes its way into subs, even with narrow-band filters (I use L-Extreme).

In a heavy nebula image (currently processing the Tulip) good luck trying to select clean sample stars; never mind WP, I get deformed stars no matter how I play with the settings. Have to resort to synthetic decon in these cases, which still does a pretty nice job with deconvolution. Had much better success with my Glob images using SVDecon with only 1-3 WP's, easily healed out.

On the subject of 64-bit GPU (any version); I cannot launch it; assume because my Nvidia GTX-1060 6MB on a pretty powerful Intel i7 PC is not supported? If so, any options; I feel left out!

[jackbak]

I am about to pull the trigger on a moderately priced (for today's bloated prices) used AMD RX 570 as I have been told that AMD drivers are built-ins for
linux. I sure hope this is not a waste of $400 US. I just can't redo my complete computing world for Kubuntu - I do everything from banking to astronomy in linux, there is no Windows in my world.

@firebrand18 let us know what GPU solution you latch onto.