Processing results not as I expected

[admin]

Hi David,

Most of these issues stem from anomalies in the dataset;

When doing the wipe, I saw this

It appears you have left stacking artefacts around the edges.
You will notice every workflow (including the one above) starts with inspecting the dataset and cropping away stacking artifacts.
Without doing so, you are effectively instructing Wipe to remove gradients from much-darker-than-real-background pixels. As a result Wipe will construct a gradient model that backs off in those areas, as Wipe (and StarTools as a whole) never clips your data unless explicitly allowed to. Wipe in general is extremely sensitive to dark anomalies like stacking arifacts, dust donuts, trees and anything else that might be darker than the true recorded background level. If you have such dark anomalies and you cannot (or don't want to) remove them, you can mask these out however.

Star mask generation

Star mask generation uses two algorithms in tandem; one for "small" star detection of stars that do not over expose, and one for "big" stars that do overexpose.
If you are using data where overexposing stars have had their values modified to no longer be unity (e.g. full brightness), the second algorithm will - rightfully - no longer regard these pixels as overexposing stars, as they are not overexposing any more. E.g. StarTools treats pixels at unity as singularities with an unknowable value.

Out of all stackers I am aware of, DSS in is a bit "unique" in its treatment of singularities, in that it treats pixels at full unity as values at full unity, rather than unknowable quantities. This, for example, results in overexposing stars dimming during some of its calibration steps, compared to other over exposing stars in the same image, as well as colour discrepancies in highlights due to treating colour channels at unity as having their value at unity, rather than as an unknowable quantity.
Regardless, you can correct difficulties with mask generation by lowering the "Threshold" parameter from 100.00 to whatever percentage of unity the dimmest overexposing star in your image sits at.

Hope this helps!

[perdrix]

Hmmm... I'm struggling to reproduce your results - could you send me the log from your session to see exactly what you did? From the sound of it you are suggesting that I need to crop the other edges, but by how much! I didn't see much in the way of anomolies at the left/right/top when inspecting the AutoDev'd image. Of course I'd already cropped off the horizontal lines at the bottom.

Regarding DSS's treatment of 100% saturated pixels: Could you suggest how we should improve that ? I can readily add code to special case this - my guess would be to leave it saturated regardless when doing any image calibration step (subtraction of dark, multiply by flat etc.). But that is just a SWAG (Simple Wild Assed Guess).

Currently when scaling DSLR RAW data the DSS code scales from the ADU value that the RAW file says represents saturation up to 65535. For example:

00000095 2019/09/22 08:34:15.951 031812 00009a5c >No White Balance processing.
00000096 2019/09/22 08:34:15.958 031812 00009a5c >White balance co-efficients being used are 1.000000, 1.000000, 1.000000, 1.000000
00000097 2019/09/22 08:34:15.965 031812 00009a5c >Maximum value pixel has value 13257
00000098 2019/09/22 08:34:15.972 031812 00009a5c >Saturation level is 10230
00000099 2019/09/22 08:34:15.978 031812 00009a5c >Applying linear stretch to raw data. Scale values 6.406158, 6.406158, 6.406158, 6.406158

In this case saturation level is 10230 everything is scaled by 6.406158 to give 65535 for any input ADU of 10230 or above.

Thanks
David

[admin]

Hmmm... I'm struggling to reproduce your results - could you send me the log from your session to see exactly what you did?

This previous post contains the log with added comments. Where parameters deviated from the defaults, they were mentioned. I did not include the BASE64 mask.

From the sound of it you are suggesting that I need to crop the other edges, but by how much! I didn't see much in the way of anomolies at the left/right/top when inspecting the AutoDev'd image. Of course I'd already cropped off the horizontal lines at the bottom.

Depending on the image, they can be single lines of pixels. If you see this behaviour, the dark anomaly should be right next to it (if it is in the form of a halo remnant, it is at the center of the halo). Another DSS quirk I noticed that, even if intersect is selected, often a single row and column of stacking artefact pixels can be seen top, left, right and bottom, causing this precise "look" if not remedied by cropping. More info about prepping your data for Wipe and these exact symptoms can be found here.

Regarding DSS's treatment of 100% saturated pixels: Could you suggest how we should improve that ? I can readily add code to special case this - my guess would be to leave it saturated regardless when doing any image calibration step (subtraction of dark, multiply by flat etc.). But that is just a SWAG (Simple Wild Assed Guess).

Currently when scaling DSLR RAW data the DSS code scales from the ADU value that the RAW file says represents saturation up to 65535. For example:

00000095 2019/09/22 08:34:15.951 031812 00009a5c >No White Balance processing.
00000096 2019/09/22 08:34:15.958 031812 00009a5c >White balance co-efficients being used are 1.000000, 1.000000, 1.000000, 1.000000
00000097 2019/09/22 08:34:15.965 031812 00009a5c >Maximum value pixel has value 13257
00000098 2019/09/22 08:34:15.972 031812 00009a5c >Saturation level is 10230
00000099 2019/09/22 08:34:15.978 031812 00009a5c >Applying linear stretch to raw data. Scale values 6.406158, 6.406158, 6.406158, 6.406158

In this case saturation level is 10230 everything is scaled by 6.406158 to give 65535 for any input ADU of 10230 or above.

That all seems perfectly correct of course! It's been a while since I used DSS (I'm a Linux user and the latest versions don't play nice with my version of Wine any more), but I seem to remember flat fielding in particular was causing this. I will have to investigate and let you know if/how I can consistently replicate this.

[perdrix]

When I ran wipe on a completely new run of the stack (without WB) it did much as I expected it to, but when I pressed keep I was back to an un-stretched image for the follwoing steps.

Is that normal?

To answer my own question: I don't think so. Even though I couldn't see much I worked through the rest of the steps, turned off tracking and de-noised, and then Auto-Streched.

Result a B/W image:

Startools mono.jpg (346.91 KiB) Viewed 6320 times

-----------------------------------------------------------
StarTools 1.5.368
Mon Sep 23 09:30:40 2019
-----------------------------------------------------------
File loaded [D:\Users\amonra\Documents\Astrophotography\Flame (NGC2024) and Horsehead\NoWB 16bit.TIF].
Image size is 5202 x 3465
---
Type of Data: Linear and was Bayered, but not whitebalanced
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [Off]
Parameter [Outside RoI Influence] set to [50 %]
Parameter [RoI X1] set to [0 pixels]
Parameter [RoI Y1] set to [0 pixels]
Parameter [RoI X2] set to [5202 pixels (-0)]
Parameter [RoI Y2] set to [3465 pixels (-0)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %] <============ Image shows as Monochrome ater Auto-Dev
--- Crop
Parameter [X1] set to [41 pixels]
Parameter [Y1] set to [170 pixels]
Parameter [X2] set to [5176 pixels (-26)]
Parameter [Y2] set to [3438 pixels (-27)]
Image size is 5135 x 3268
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [1 pixels]
Parameter [Drop Off Point] set to [100 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %] <==== Image after Wipe is good and colour
--- Contrast
Parameter [Expose Dark Areas] set to [No] <===== Image now mono and not stretched!
Parameter [Compensate Gamma] set to [No]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [1 pixels]
Parameter [Aggressiveness] set to [75 %]
Parameter [Dark Anomaly Headroom] set to [15 %]
Mask used (BASE64 PNG encoded)


Mask used (BASE64 PNG encoded)


Mask saved [D:\Users\amonra\Documents\Astrophotography\Flame (NGC2024) and Horsehead\NoWB 16bit_mask.tiff].
Mask used (BASE64 PNG encoded)

--- Deconvolution
Parameter [Image Type] set to [Deep Space]
Parameter [Mask Behavior] set to [De-ring Mask Gaps, Hide Result]
Parameter [Radius] set to [1.5 pixels]
Parameter [Iterations] set to [6]
Parameter [Regularization] set to [1.00 (optimal noise and detail)]
Parameter [Mask Fuzz] set to [4.0 pixels]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness & Color]
Parameter [Algorithm] set to [Reveal All]
Parameter [Dark/Bright Response] set to [1.70]
Parameter [Detail Size Range] set to [1000 pixels]
Parameter [Strength] set to [2.6]
--- Life
Parameter [Detail Preservation] set to [Linear Brightness Mask]
Parameter [Compositing Algorithm] set to [Multiply, Gamma Correct]
Parameter [Inherit Brightness, Color] set to [Brightness]
Parameter [Output Glow Only] set to [No]
Parameter [Airy Disk Sampling] set to [128 x 128 pixels]
Parameter [Airy Disk Radius] set to [15 pixels]
Parameter [Glow Threshold] set to [0 %]
Parameter [Detail Preservation Radius] set to [20.0 pixels]
Parameter [Saturation] set to [100 %]
Parameter [Strength] set to [100 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Entropy-driven Detail Enhancement
Parameter [Resolution] set to [Medium]
Parameter [Channel Selection] set to [All]
Parameter [Strength] set to [100 %]
Parameter [Midtone Pull Filter] set to [20.0 pixels]
Parameter [Midtone Pull Strength] set to [50 %]
Parameter [Dark/Light Enhance] set to [50% / 50%]
--- Wavelet De-Noise
Parameter [Filter Type] set to [Distance Weighted Outlier Rejection]
Parameter [Grain Size] set to [6.4 pixels]
--- Wavelet De-Noise
Parameter [Scale 1] set to [95 %]
Parameter [Scale 2] set to [95 %]
Parameter [Scale 3] set to [95 %]
Parameter [Scale 4] set to [95 %]
Parameter [Scale 5] set to [0 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Scale Correlation] set to [6]
Parameter [Color Detail Loss] set to [12 %]
Parameter [Brightness Detail Loss] set to [12 %]
Parameter [Grain Dispersion] set to [6.4 pixels]
Parameter [Non-linear Response <] set to [Off]
Parameter [Smoothness] set to [65 %]

I'll upload the files (16bit TIF, 32bit TIF Intgger, 32bit TIF Rational, 32bit FITS) to Dropbox and share them to you

[perdrix]

Argh! Just realised I hadn't tirned off all background calilbration in DSS.

Let me try again. No it behaved the same - reverting to unstretched image when I did a Keep after running Wipe

I'll replace the files with the No calibration versions.

David

[szymon]

That's normal after a wipe. Recall that at the start of your processing, you do an autodev to stretch to the maximum to see what data you actually have, highlight any issues with it etc. You then crop and maybe bin that, before doing a wipe. Wipe does a temporary autodev to show you the result, and when you keep the wipe you get back to an unstretched state. You then either autodev to get a "good" stretch, or if that doesn't work then you use the manual develop module. This is the first "real" stretch of the data -- and it's done on cropped binned wiped data.

-simon

[perdrix]

OK that helps a lot - or I thought it did, but:

I loaded the image, Auto-Dev gave stretched monochrome image, I then cropped this, and ran Wipe, clicking on "Color" (hey guys only the Yanks spell it that way!) showed me colour in the image, so I did a Keep.

I then returned to Auto-Dev expecting the resulting image to be in colour - but it was still monchrome!??? Surely that can't be correct?

To Ivo, the files you asked for so you could check your processing of the various bit-depths and Integer/rational images are in my Dropbox and have been shared to you.

[admin]

Hi David,

Thanks so much for uploading the datasets - it's really appreciated! I'll go through them and see if there is anything that needs fixing.
Thank you Simon for a great explanation.

OK that helps a lot - or I thought it did, but:
I loaded the image, Auto-Dev gave stretched monochrome image, I then cropped this, and ran Wipe, clicking on "Color" (hey guys only the Yanks spell it that way!) showed me colour in the image, so I did a Keep.
I then returned to Auto-Dev expecting the resulting image to be in colour - but it was still monchrome!??? Surely that can't be correct?

If you chose to import your dataset as "Linear and was Bayered, but not whitebalanced" (e.g. the subject of the DSS feature request ), then Compose mode is used to process luminance and color separately.

Processing luminance and color separately in other software, means that you would have to process two datasets separately in sequence and the combine the two into a composite. In StarTools, however, the two datasets are processed in parallel (many benefits to this).

Throughout your processing you will mostly work on the luminance (detail) portion of the signal (which is obviously mono). In the case of the Wipe module, however, there are two results to inspect (as indicated by the message on the screen under the image). The "Color"/"Luminance" module will allow you to inspect Wipe's result of working on the color and luminance portion of your signal. This button is not not functional if Compose mode is not engaged.

Compose mode is typically used for processing LRGB or narrowband datasets, but is also used for the special case of DSLR datasets where no color balancing has been performed and the data used to be Bayered. This is because, now, it can be asserted that green channel data is 2x as reliable/clean (has twice the amount of samples) as the red and blue channels. Weighing the luminance signal accordingly will yield a cleaner luminance signal, while the coloring remains unaffected by the reweighing.

I fully appreciate that if you are used to DSS only for post-processing, then there seem to be a whole lot of nuances and "hoops to jump through". I hope, however, the rationale and - ultimately - results make it clear why these nuances and hoops exist!

Clear skies!

[admin]

David,

Thanks to the datasets, I found an issue with the 32-bit rational TIFF import. This will be fixed in the next release. The others imported ok and as expected. Thanks again!
Your latest DSS stacks (32-bit rational used below) shows improved signal again (e.g. less noise and improved faint background detail). I am really stoked we can now soon get this sort of signal from DSS. Kudos!

NoWB 32bit Rational.jpg (168.13 KiB) Viewed 6309 times

[happy-kat]

As an observer to this interesting thread, when comparing this to the process on the previous page there is a big step change. Looks wow. Thank you both, as a user perhaps the trickle effect will help me too.