Wipe needs Dark Anomaly bump; why?

[TelescopeGreg]

I've been troubled with a strange color cast when using the default settings in Wipe. In a post on Cloudy Nights, Ivo pointed me to tweak the Dark Anomaly up a bit, and that's gotten around the problem (Thanks!). 3 pixels has been sufficient, but this month I've run into an image (a stack, specifically) where I had to bump it up to 7, and that's feeling wrong. Re-stacking in a different app brought the need back to 3 pixels, but now I'm wondering what is going on. Is it a problem with my camera, the telescope, or my stacking workflow?

Here is an example of what it looks like with defaults. Prior to this point in the processing I had used AutoDev with an RoI around the neblua in the middle, followed by a slight crop around the edges to remove any unseen stacking artifacts. This image was stacked with DSS using 13 Lights, 20 Darks, 10 Flats, and 10 Dark-flats. Not the best data, I know, but the color gradient issue is present in any image. In this case, setting the DA to 3 pixels would give a relatively even gradient across the image.

ST Wipe color gradient (resized).jpeg (75.79 KiB) Viewed 4826 times

Using ASTAP for stacking the same data required a Dark Anomaly of 7 pixels this time, in order to even out the gradient. Some images stacked with ASTAP are fine with 3, some with 2. 7 is strange. Adding to the puzzle was that changing the Synthetic Dark/Bias through various settings created very distinct colored "hot pixel" bands. I'm guessing this is related to how the Lights were blended together during the stacking process, and not representative of the camera's sensor. At least I hope that's the case. Here's an example:

ST Wipe color gradient ASTAP with synth (resized).jpeg (111.88 KiB) Viewed 4826 times

So it seems clear that the stacking process is involved, but I'm still left with the puzzle of what is going on under the covers to cause the effect in the first place. Any thoughts? Anything to worry about?

Thanks!

[Mike in Rancho]

Wow those are some amazing color, er, gradients? Very bright and happy.

I've used DAF of 10 or so in the past (probably incorrectly lol). Any more than that and things start taking a long time to process.

Not sure on the DSS and ASTAP differences. Are the settings the same, as far as kappa sigma and whatnot? I know ASTAP has no bias tab, so you either take dark flats or throw your bias in that folder. Uncertain if the alignment algorithms match. Also ASTAP offers a choice of demosaicking, which may be different from DSS.

For the second image, did you think you needed artificial darks or bias? If I remember right, when you use the presets for uncalibrated, with artificial bias going on the aggressiveness has to be lowered, starting at about 50? 75 might be too much.

Speaking of, how does the Wipe look if you just hit the uncalibrated preset? If not needed, that's sometimes really taking a hammer to things, but it seems to be able to tame most anything. And it can probably be adjusted from there, such as turning off vignetting, or taking the bias up to aggressive.

Will be interested to see what Ivo says though. Oddly enough, for my just-taken 2.5 hrs of RGB (UV-IR cut filter only on the D5300f/s), I have quite a few color splotches around the image in Wipe, and of course they can propagate later into Color. Nothing like yours, but still rather colorful.

What happens when you take your first Wipe example and switch to Luminance? It almost has that look that Wipe gets when it is angry at you, like for not getting all the stacking artifacts completely cropped off. Or not telling it to ignore them using the mask.

[TelescopeGreg]

Hi Mike,

Yeah, stacking artifacts around the edges was my first thought, but even cropping way in didn't help. The colors were a bit different, but fundamentally the issue remained. Playing with the DAF setting was simply a "Hmm, wonder what this button does?" sort of thing. I did try the other presets, and Uncal1 was a bit better than Base, but the colors came out very off when I get to the Color module. Same for Amp Glow.

The images were all stacked with lights and darks, flats and dark-flats. I don't do Bias, as it just confuses me, and the slight time advantage of using bias instead of dark-flats has zero benefit. (I frankly don't know why we even talk about Bias anymore, but that's a different thread.) I did try changing the stacking settings in DSS, specifically locking in bi-linear and bi-cubic alignment instead of Auto. Also turned off the color channel alignment. No substantive changes. Also tried Median and Average for the Lights; no change. I don't really have enough Lights for K-S to work, but I don't think that would help either.

ASTAP's stacking mode is whatever it was last time for the first pass at Tulip, which didn't have this trouble.

{shrug} I guess if DAF of 7 isn't "bad", perhaps I should just leave it that way and be happy. DSS often does a better job at multi-night sessions than ASTAP, so I expect I'll be back on DSS for the final image here, as long as the clouds give way before the Moon returns.

I'm still just curious what is going on. Why the dramatic gradient in the first place?

[Mike in Rancho]

A couple things you might try, that I have noticed will affect Wipe - especially with color gradients and splotches.

One is to crop in tighter, if your data allows it and you are ok with that FOV. Since I think this is the Pacman, and there is pretty much nothing interesting around it, this might be possible.

The other is to go into Wipe maintaining full resolution, or perhaps less of a bin, like 71% instead of 50 or below. If your CPU+GPU can handle processing the bigger image, that is.

[TelescopeGreg]

Well, the images posted are pretty much the full FoV - I'm imaging at 910mm focal length with an APS-C camera, so there's not much space around objects to crop. Processing speed isn't great - I haven't figured out how to get the GPU involved in processing. The image was already binned 2x2 in the camera, so about 2k x 3k pixels in dimension.

I guess the base question is whether the gradient seen is "normal", and if not, what's causing it? It doesn't seem to be dependent on where in the sky I'm imaging. They all seem to have the same sort of issue, though I haven't compared the particular metrics (color direction, size, etc.).

[admin]

If you see "pretty" colors like these then you have different dark anomalies in different channels.

Most of what dark anomalies are is explained in the Wipe docs.

Ideally, dark anomalies should not occur on your data at all, and you should try to figure out what is causing them (you can typically find them at the center of the halo/patch that surrounds them). If they are small, then they are typically dead or "cold" pixels, and if they are small than the Dark Anomaly Filter can mostly take care of them by filtering them out. Properly dithering during acquisition should take care of them.

The larger Dark Anomalies are in size, the higher you will require to set the Dark Anomaly Filter parameter value to try to filter them out. It should ideally not be relied on as a permanent solution though!

EDIT: posting the dataset would be most helpful if the above isn't immediately useful.

[TelescopeGreg]

Hi Ivo,

Thanks for filling in some of the underlying operation of Wipe. I'm thinking that the issue is "simply" that I don't have much time on the target as yet (about an hour, with very poor sky transparency), and don't dither, so the noise "holes" are probably due to some combination of the two. I presume that the different levels of the issue demonstrated by the two stacking programs is a matter of their internal processing algorithms.

To verify, here are the two stacked images discussed above. The first is the better of the two, from Deep Sky Stacker. I need a DAF of about 3 pixels to clear things. The second is the same data set stacked with ASTAP, and needs something around 7 pixels to do the same. (To be fair to ASTAP, I've also observed the reverse in the past, with ASTAP creating an image that did better in this regard than DSS...)

DSS
https://www.dropbox.com/s/34th8xdvstuv3 ... S.fts?dl=1

ASTAP
https://www.dropbox.com/s/c5qcqgedbl1x7 ... .fits?dl=1

Greg

[Mike in Rancho]

Hi Greg,

I looked at the DSS stack (unusually small sized for a FITS?), so I could create the screen of incredibly pretty colors right in front of me! Maybe even better when dropping the default DAF from 1 to 0.

Anywho, I noticed a black dot (dark anomaly?), so zoomed in and found many more strewn throughout. As you note, DAF 3 seems to be able to take care of it. I went through the image while zoomed in Mask, and unmasked a whole bunch of black DA's using the new circle creator.

This made a pretty big difference, and allowed for use of DAF 2 in order to clear things up.

I may have missed some, or perhaps there are some others that are anomalies but not dark? I noticed some bright spots that had the same vector and didn't match the field elongation of the stars in the corners.

Seems like you may have already predicted the issue - perhaps hot and cold pixels and no dithering?

[TelescopeGreg]

The file size is a small mystery. The capture was done with the camera binned 2x2, so roughly 3k x 2k pixels. Individual subs are about 13 mb. For some reason, ASTAP's stack is 2x the size of the DSS stack. No idea why, since it's from the same original data set. I presume there's some check-box parameter that's different between the two programs.

I looked at the fits file statistics in CCDciel, and the minimum pixel in an unprocessed Sub is about 2x the mean Dark pixel (~1,100 vs ~500), so I think the exposure is ok. The stacks, after full calibration with Darks and such, run the pixels a lot lower - down to 5 in DSS, and all the way to 0 in the ASTAP stack. That's probably why the ASTAP stack is harder on the DA than DSS.

Using the Auto Mask in ST is a cool idea! I didn't realize it could look for dead pixels, but there it is. I set it to max (20 pixels, 100 threshold), and wow there are lot of them. Lowering the threshold to 10 still has a bunch, though far fewer, What does that mean in terms of my camera and/or the stacking parameters? I did a quick test, going back to last month's Tulip Nebula image. Comparing it to this month, both using the same camera, exposure, binning, etc., the two images' DA patterns are completely different. So, it's (thankfully) not the camera, right?

BTW, my logic on not dithering during capture is from two perspectives. Primarily, my AVX mount is being pushed about to its limit with what I have on top of it, and throwing a half-dozen arc-second "Crazy Ivan" skew into the mix every few subs is not going to help in that regard. I also image for only a few hours per session, so I get a natural dither by combining multiple sessions since they are never exactly lined up. This was only the first night's data. There's also a little "light dithering" caused by the 1.5" RMS random walk that the mount is doing while guiding. To that point, I did notice that the guiding was unusually good that night for some reason, so perhaps the better the guiding the more DA issues I have? Might be significant.

Next opportunity for imaging might be tonight, but Seeing is predicted to be terrible. We've had a lot of wind lately, and it's stirring things up. Fortunately, no power outages or new fires here.

[Mike in Rancho]

Actually I just searched for and unmasked the anomalies manually. I always forget that there's a whole slew of extra things to choose in the automask, including hot and dark pixels.

And while it makes for some good posts on CN about accelerating one's learning curve, isn't it really about time to retire the AVX?

I liked the reference though, but must say that the Crazy Ivan storyline was far better in the book than the movie.

Based on what I picked up over the weekend, I hope you are using that UHC for your clear night tonight? You've already picked up enough RGB in case you want to do some normal spectra stars. And coupled with the built-in IR cut of your camera, you'd have a reasonable facsimile of an L-eNhance. Just wider in the red.

Huge difference when the filter subs of the Pacman started coming in, compared to my RGB session. And of course ST processes it so much nicer too.

I picked up 8 hours of bicolor, uncertain if I want to try for more even though the nights are perfect right now. Dark (but the moon), clear, and cold. I hate the cold but the camera likes it, no darks needed. Two nights up until 4am though and I'm still recovering.

20 pixels seems a bit high for a hot/cold pixel search, even if your AVX's natural dithering may have smeared it around? I presume you are using kappa sigma rejection in stacking. Question it, did it work?

Of course, the DAF works also, that's what it's for.