Wipe woes

[Mike in Rancho]

This is actually an issue I've had for a while. Not sure why, maybe the Newt, maybe trouble taking flats, maybe just horrific LP and stray light...

But in using Wipe, and often with very necessary high aggressiveness, it seems to work pretty well except for the edges. Often just two (one short, one long) or mostly just the two, end up with a coagulated mass in a line just inside the frame, outside of which is darkness.

With settings and even masking, I haven't found a good way around this. Cropping seems to make no difference (unless way way cropped like a small portion of the image) as the same result just sets up again. Mostly I've handled this by cropping after the fact and/or just using Auto-OptiDev to suppress it or just not stretch very much. Occasionally I can get gradient edge behavior to help a little, but often that just swaps things around to different edges and isn't helping me get all four right. Dropping to 128 sample helps a little too.

I've also tried Wipe at full scale, right after crop, before binning, without a great difference I don't think, once one finally OptiDevs it. And since that stuff is sitting there as "real" no amount of IFD or anything will really make it go away.

The worst offender is the L filter, followed by the RGB filters which aren't nearly so bad. The NB is fine.

Because of the differences in filters I'm usually already running separate Wipes and saving, in order to then recompose. But that only helps so much too.

Though I don't know the programs very well, I have tried gradient extraction with Siril and PI (ABE anyway, don't know DBE yet), and they seem to not have this issue as to edges, though they may be suboptimal to Wipe in other ways and I'm not sure how much data they could be erasing with their subtractions or division.

I'm attaching just an L master here, 2 hours on M78, in case anyone can get a flat field out of this. This was the PI stack but the one from ASTAP acts mostly the same I think. It's been an ongoing thing.

https://drive.google.com/file/d/1TirtBZ ... sp=sharing

TIA

[dx_ron]

Impressive gradient you got there...

Uncalibrated1 flattens it right out. At too much cost? Perhaps

mike_uncal1.jpg (498.65 KiB) Viewed 13372 times

mike_uncal1_autodev.jpg (445.17 KiB) Viewed 13372 times

[Mike in Rancho]

Thanks Ron!

I think I did try that, and indeed it takes a hatchet to the fainter stuff. As you can see there's little to no structure now from Barnard's loop which has sort of been erased from existence. (I felt a disturbance in the Force....)

Now, that can be set aside somewhat as the eventual plan here is to NB Accent the Ha filter in.... But as we know NB Accent only gets refinement from it's version of AutoDev, no parallel enhancement modules.

And while the L is the worst, this also happens to a lesser degree with the RGB filters, I think R and G mostly. So the hammer of uncalibrated could wipe some color data too on those filters.

Yes my LP and gradients when I shoot towards the south get difficult, and the two nights were pretty close on timing but not identical, maybe shifted by half an hour or an hour?

The uncal setting probably preserves the high SNR M78 area though.

This particular stack used WBPP with all the defaults on, so includes the features that are supposed to help with this sort of thing. I haven't checked to see if its any better than the same stack in ASTAP, but I know both are kind of the same general issue in Wipe of that coagulation line and darkening beyond it. Then again, I'm also not sure I used LN properly (separate thread too).

I suppose I could just sacrifice the edges by cropping in after Wipe, though it's a decent sized region that would be chopped off all around. And I don't see this edge behavior with other gradient removal algorithms, even if they aren't as smart as Wipe.

[admin]

Hi Mike,

We are indeed looking at some pretty extreme uneven lighting. The lighting signature indeed shows a suspicious rounding and darkening towards the corners (as well as secondary darkening in the center). I indeed say we're looking at some sort of complex flats issue rather than just bad LP gradient.

With the given data, Wipe behaved quite well (in my opinion) with the "Uncalibrated 1" preset on the 50% binned dataset.

Note that, for the behavior near the edges (where data becomes sparse), you can configure how the module should "make up " for the missing data using the Gradient Edge Behavior (I just noticed "Bounce Back" is mentioned twice in the 1.9 alpha - the third option should read "Absorb 50%").

Note also that the dataset as-is (e.g. without crop) appears to have some progressively worsening signal quality at the bottom edge (and to a slightly lesser extent, right edge), so this you may wish to crop a few pixels or increase the Dark Anomaly Filter significantly to help Wipe overcome the worsening background fidelity in those areas. They're really the worst sort of stacking artefacts, as they are harder to detect (in AutoDev/OptiDev, at >100% zoom, look for areas of noise where the noise increases in contrast versus other areas). Try configuring your stacker to auto-crop the dataset to the area where all pixels have data from all frames to avoid this.

All that said, Wipe will always struggle when uneven lighting undulation frequency encroaches on that of actual structural detail, as objectively figuring out what is detail and what is gradient becomes impossible when the two intertwine.

When the gradients you are chasing increase in undulation frequency (typically only the case in very wide fields if flats are good), then increasing Sampling Precision should help as well.

Does this help?

[Mike in Rancho]

Thanks again, Ivo.

Sorry I didn't mention it - I figured we might be losing Absorb 50 for a different type of bounce back as a work-in-progress, so didn't say anything. But as long as we are at it...there's two Uncal1 buttons.

It's always possible flats went awry here too. On these nights I wasn't as OCD about making sure the scope was as close to directionally-oriented for flats as for the lights as I have at other times. That said it wasn't that far off. Still, some remnant vignetting issue seems to exist, and I saw that after trying ABE (degree 1 for the linear LP gradient) also.

WBPP does now have an autocrop option, I guess similar to DSS intersection mode, and hopefully across multiple filters and with retention of registration. In fact, I don't believe you can't use it in DSS across multiple filters, even all to one reference, as you could lose registration and/or end up with different file x-y sizes. Alas, WBPP said it failed the autocrop on this data.

I believe the reference here was one of my R subs, so the L may be slightly shifted - though not by much as I nailed my rotation match and NINA plate solved the sequence starts. Then there's also dithering, though my pixel amount is fairly tight. Nonetheless, I had cropped before Wipe (once mouse wheel "click" turn off each edge). As a test I did a triple click crop, so way way past any potential registration or dithering shifts, with no changes in the matters discussed. I also did my typical 35% bin.

Uncal1 was okay, but not great I would say. Too bright in the center, and I know it's "somewhat" empty there between the Loop and M78. Relatively emptier, let's say. Some alteration of settings helped some, and I did need the grow opposite axis, but I still ended up with a (false) bright bottom edge. Ultimately could be an extra crop job, I suppose.

Which direction is an increase in sampling precision? 128 seems more precise than 256, but I'm guessing it might be the other way. Still, 128 seemed more helpful (to the edge problem) than higher sampling numbers.

This is what I ended up with, and I think is kind of okay, except for that bottom edge. Will put the O/D shots in following post.

M78 L Wipe-Uncal1 adjusted.jpg (167.3 KiB) Viewed 13347 times

[Mike in Rancho]

So after an OptiDev on the Wipe as run above, it comes out as this (no ROI setting yet). I think I still need to tune out some vignetting feel to things here.

M78 L after Wipe Uncal 1 adj OptiDev.jpg (152.56 KiB) Viewed 13346 times

Siril BE (after altering a few points...kind of annoying and I have no experience using it) loaded and then OptiDev. Also may be showing a slight vignetting appearance, at least on the right side.

M78 L after Siril BE then OptiDev.jpg (154.04 KiB) Viewed 13346 times

And ABE run twice, once for linear gradient as a subtraction and then a division for what seemed remnant vignetting. This didn't seem too bad, but overcorrected the top right corner, left me with a questionable left edge, but seemed the best at not erasing the segment of Barnard's Loop?

M78 L after 2 passes ABE then OptiDev.jpg (159.12 KiB) Viewed 13346 times

All were cropped and binned/resampled to be matching, or close. Some flaws can be found in all, I'd say. Possibly user flaws! But still, not sure of the best way to attack this.

At least I think I'm getting closer to acceptable with the Wipe version - though that does have to be saved out as just a 16-bit tiff and thus also loses all the header data goodies like exposure time for subsequent composing.

[admin]

Hi Mike,

I would still maintain the Wipe version is the most trustworthy (though ideally you would not have to use the more aggressive settings of course).

You can actually see Wipe's "carefulness" in action with regards to the edges. The way it leaves the "remnant" is warranted, as it is a real "feature" (between quotes, as it appears artificially induced to me) of your dataset;

Untitled.jpg (188.6 KiB) Viewed 13287 times

By isolating the gradient's progression though a narrow dynamic range stretch, we can see that sudden bump in brightness that does not appear to be part of the gradient's progression. "Sudden" is what Wipe is "allergic" too; as "sudden" means "high frequency" versus the slow and steady progression of the low frequency gradient. Hence Wipe backing off there in order not to destroy this "detail".

My money would be on some sort of amp glow or other heat source near that edge, which wasn't entirely eliminated (change in temperature? ) during calibration (or else perhaps a very coincidental distant light pollution dome). Before going down that rabbit hole though, I would love to understand how it is possible that the gradient is so severe.

It *does* appear to be a very wide field, so extremely strong light pollution is certainly possible. Can you confirm that there was indeed very strong light pollution emanating from the region in the lower right corner?

[Mike in Rancho]

Appreciate the analysis, Ivo!

Well after "all" this time, seems I still have much to learn about Wipe.

I imagine too that Wipe should be the best, though sometimes I am a bit hesitant about complexities tripping it up. And it seems I'm throwing a lot of complexities at it.

The LP is probably pretty strong here, and likely to be a mix of several LP domes, and then of course I am tracking through them all. Max alt at transit was perhaps 55°, that corner toward the ground, south - southwest, and things do get even brighter to the southwest. The L file here was taken first so maybe 2 through 1 hours pre-meridian.

Still, I'm not sure if that is the bottom edge issue I was having, which was both lower, thinner, and spanned a good 2/3 of the image width.

The place you have the pointer is, however, in the region where Barnard's Loop's strongest portions are weaving through the frame. But again I don't recall having much trouble at that spot, after trying to dial in Wipe, anyway.

That is kind of a thing in Wipe though, especially if separately working a mono file. It can be difficult to tell whether and where Wipe is backing off, and if it has and you hit Keep, then that noise or brightness maybe gets converted into false structure. Absent reality checking against something, you might never know. Probably too much trouble for a low-incidence issue, but it'd be great if Wipe would alert you somehow as to what it is doing. Like, here's what I subtracted and divided, or a perhaps a reject map you could toggle to (a la maxRGB), maybe in various colors depending on how much backing off is being done. Neon orange for the worst? Some stuff is obvious, especially in RGB, like stacking artifacts, where Wipe creates super bright clearly geometric shapes. But not so much with smaller lumps or edge issues maybe.

Other than the tangled mess of directionalized LP, there shouldn't be too much of an issue here. Even any miscal with the flats seems fairly minimal (at least after looking at things using a single pass of a dumber, linear gradient only, extraction.)

All that aside, I did manage to come up with some improved Wipe results IMO, using Uncal1 for LRGB but backing off the sliders until it seemed I was matching reality the best I could when looking at reference images, but otherwise having a flat field where appropriate. The Ha I just used NB preset (unsure if it even did much of anything).

Compositing those last night gave me a decent result but I thought it could be better. Really I have been unsure what to do when compositing pre-Wiped (and cropped/binned) tiffs. Generally I've thought to run Wipe again, but at zero aggressiveness, in case the RGB channels still need that initial floor or balance setting. But I could just be confusing myself. I have in fact run into occasional trouble with pre-Wiped recompositing, such that strange lines and shapes are created in the image that can show up (either from edge behavior seeming to go bonkers) in either Wipe or later in NB Accent (where it seems perhaps tied to the ROI). Anyway...

Tonight I tried again but skipped Wipe entirely on the recomposite, figuring I'd let Color admonish me and just try to balance the best I could. That seemed to work fine, and frankly with my LRGB filter set and matching color exposures, the bias after star sampling ends up pretty small and not too different than if I had run Wipe.

So, I think I came up with my best version of this data yet, and better with getting Wipe closer to "right" than the ones I did using ABE.

M78 5h WBPP preCBW ST9 4B.jpg (462.87 KiB) Viewed 13268 times

Of course quite a bit of imager's choice in just how much to feed in the narrowband, but I think this balance came out pretty well. A bit more might have worked. Maybe not bad for 5 total hours across two nights in Bortle 8 to 9 (could be a true 9 in this direction).

Some of the Ha structure, even if faint in a few areas, does seem to match up moderately correctly with what seems to be an okay reference: https://apod.nasa.gov/apod/ap171229.html

Don't know the particulars of that APOD or how the Ha was added in, of course.

If useful at all (files to challenge Wipe ) or if anyone else wants to take a crack at some Wipe, compositing, and NB Accent practice, I've put all 5 masterlights right out of PI, as well as the 5 pre-cropped/binned/Wiped and then saved ST tiffs that I used for the image inserted above, all here: https://drive.google.com/drive/folders/ ... sp=sharing

[hixx]

Note that, for the behavior near the edges (where data becomes sparse), you can configure how the module should "make up " for the missing data using the Gradient Edge Behavior (I just noticed "Bounce Back" is mentioned twice in the 1.9 alpha - the third option should read "Absorb 50%").

This was a duplicate entry in the english Wipe XML which I fixed. Should be good in Alpha 3 (536)

Jochen

[admin]

Appreciate the analysis, Ivo!

Well after "all" this time, seems I still have much to learn about Wipe.

I imagine too that Wipe should be the best, though sometimes I am a bit hesitant about complexities tripping it up. And it seems I'm throwing a lot of complexities at it.

The thing is that you can't address what you don't know exists.
One man's "tripping up" is another's faint detail that is retained. StarTools will always favor the latter use case, but I accept that some just want it over and done with a bit quicker in the face of multiple adverse conditions, in lieu of eking out the most possible detail under pristine skies.

The LP is probably pretty strong here, and likely to be a mix of several LP domes, and then of course I am tracking through them all. Max alt at transit was perhaps 55°, that corner toward the ground, south - southwest, and things do get even brighter to the southwest. The L file here was taken first so maybe 2 through 1 hours pre-meridian.

Thank you for confirming that; the dataset has already been archived in my "wipe test" folder.

Still, I'm not sure if that is the bottom edge issue I was having, which was both lower, thinner, and spanned a good 2/3 of the image width.
The place you have the pointer is, however, in the region where Barnard's Loop's strongest portions are weaving through the frame. But again I don't recall having much trouble at that spot, after trying to dial in Wipe, anyway.

The issue indeed extends to about 2/3rds of the width of the image (with 1/6th on each side). The image with arrow is to show the start of an anomaly in the gradient at the bottom - it's not just confined to that area. The smallish height of the anomaly means its undulation frequency is high, and is thus (by default) retained as detail, while the slower undulating gradient is removed. The issue quickly recedes as soon as the bottom pixels with poorer SNR are cropped away (the low SNR also existing in that area may be a coincidence, or may be related to whatever happened to the dataset).

That is kind of a thing in Wipe though, especially if separately working a mono file. It can be difficult to tell whether and where Wipe is backing off, and if it has and you hit Keep, then that noise or brightness maybe gets converted into false structure.

Under normal conditions, Wipe backing off happens over large areas and any "false structures" (gradient/LP remnant) sizes are extremely large and easy to identify (they are normally defined by the threshold - Aggressiveness - between gradient undulation frequency and detail undulation frequency). Think the "halos" around dark anomalies or stacking artifacts. Or in other words, any remnants would only happen at the scale of non-detail, and would therefore always be easily identifiable.

Under difficult conditions (high aggressiveness), Wipe backing off obviously happens at smaller scales. Remnants at these sizes will only be hard to identify if their sizes were explicitly allowed to encroach on the size of the detail in your image (because now, gradient can look like detail and vice versa). When you are at that stage, you implicitly accepted that the signal in your image cannot be 100% objectively identified by you, nor algorithm (any algorithm!) as real celestial signal or gradient. The undulation frequency of unwanted celestial signal and gradient are too similar. You are now well and truly in the subjective realm where sample setting and/or further aggressive settings will be needed to try to "rescue" the dataset.

Absent reality checking against something, you might never know.

Quite the opposite; Wipe's algorithm is the only algorithm (that I know of) that can guarantee it does not remove signal below a certain size limit ("undulation frequency"). And normally that size limit does not encroach on the size of detail in your image; the two undulation frequencies of gradient and detail are typically well separated. However, the steeper the gradient (aka the higher the undualation frequency), the smaller this separation becomes, to the point of overlapping.

it'd be great if Wipe would alert you somehow as to what it is doing. Like, here's what I subtracted and divided, or a perhaps a reject map you could toggle to (a la maxRGB), maybe in various colors depending on how much backing off is being done. Neon orange for the worst? Some stuff is obvious, especially in RGB, like stacking artifacts, where Wipe creates super bright clearly geometric shapes. But not so much with smaller lumps or edge issues maybe.

We used to have something like that in 1.3 ("output model only"), but there was of limited use for the actual output (it wasn't a true model as such, given there is subtraction and division going on at the same time, so no real way of using that output). I'd have to think about how to visualize this in a useful manner. It should be possible to show a difference map at the very least.

Compositing those last night gave me a decent result but I thought it could be better. Really I have been unsure what to do when compositing pre-Wiped (and cropped/binned) tiffs. Generally I've thought to run Wipe again, but at zero aggressiveness, in case the RGB channels still need that initial floor or balance setting. But I could just be confusing myself. I have in fact run into occasional trouble with pre-Wiped recompositing, such that strange lines and shapes are created in the image that can show up (either from edge behavior seeming to go bonkers) in either Wipe or later in NB Accent (where it seems perhaps tied to the ROI). Anyway...

If shot under substantially similar circumstances, you should be able to use Wipe in one go, without having to apply it separately.

Tonight I tried again but skipped Wipe entirely on the recomposite, figuring I'd let Color admonish me and just try to balance the best I could. That seemed to work fine, and frankly with my LRGB filter set and matching color exposures, the bias after star sampling ends up pretty small and not too different than if I had run Wipe.

If some sort of normalization was performed already (either a previous Wipe or something else), Wipe would indeed not be strictly necessary to achieve good coloring (the important thing for coloring is that something is done to remove any channel bias).

So, I think I came up with my best version of this data yet, and better with getting Wipe closer to "right" than the ones I did using ABE.

That certainly looks very nice; lovely contrast between the blue reflection and red Ha! It's very cool that these sorts of shots are possible from such a light polluted area - no one would be the wiser how much you "suffered" to get to this image!

If useful at all (files to challenge Wipe ) or if anyone else wants to take a crack at some Wipe, compositing, and NB Accent practice, I've put all 5 masterlights right out of PI, as well as the 5 pre-cropped/binned/Wiped and then saved ST tiffs that I used for the image inserted above, all here: https://drive.google.com/drive/folders/ ... sp=sharing

Much appreciated!