Processing results not as I expected

[perdrix]

I've just spent a good few hours using Startolls to re-process one of my "Flame and Horsehead" images from the raw output after stacking with DSS.

When I intially used Auto-Dev, I ended up with a totally blown out monochrome image, so I cancelled out of that and used develop with a DDP of 99% and a gamma of around .23

This gave a quite good result so I moved through the rest of the stages, and turned tracking off. The de-noise stage ran OK and produced this:

Test before final Auto-Stretch.jpg (119.2 KiB) Viewed 7209 times

which isn't at all bad - it has pulled out lots of detail in the nebulosity but has unfortunately totally blown out Alnitak.

Following the instructions I then re-ran Auto-Dev - YUK! The best result I got out that was by using a small ROI on the Flame nebula with this result:

Autodev.jpg (258.5 KiB) Viewed 7210 times

which was really disappointing.

I failed to add another attachment so will reply to this with another image

[perdrix]

By contrast here's what I produced just using DSS:

Flame and Horsehead2.jpg (181.98 KiB) Viewed 7209 times

And while it is noiser (no wavelet de-noising) I think that the stretch is much better as Alnitak isn't blown out, and the nebulosity shows up pretty well (though I freely admit that Startools pulled out a lot more detail in the Flame nebula).

So, I'm assuming that I did something wrong in the Startools processing, but quite what I can't work out.

Guidance please.

Thanks
David

[admin]

Hi David,

That is indeed certainly not as expected.

In general, workflows that yield decent results for most data, are fairly simple and replicable. They tend to roughly follow the workflow as shown here;
https://www.startools.org/modules/introduction/quick-start
Note also the specific notes about Deep Sky Stacker.

In your particular case, it may be that DSS was used to calibrate the channels (often yielding some sort of mono image), or that Wipe was not used at all to remove calibrate the signal (I can see a significant background bias/gradient in the file called Autodev.jpg that Wipe normally should have easily taken care of). The very extreme values you needed to use in the Develop module seem to point to an issue with the initial data (whether perhaps already stretched, or not calibrated yet with Wipe).

Would you be able to post the relevant section of StarTools.log, so we can get a better idea of the steps you are taking?
To best help you, however, would you be able to share your dataset (Dropbox, Google Drive, etc.), so we might diagnose the problem?

Thank you!

[perdrix]

Found the log file - it's quite big - I've emailed it to startoolsastro@gmail.com

I've uploaded the TIF file (copy of original stack from DSS but not compressed) to DropBox and sent a share invitation to startoolsastro@gmail.com

Admit I used Wipe in a previous attempt but cancelled out as it didn't appear to do much useful to the image. Didn't use it this time round

Hope that works

D.

[admin]

Hi David,

Much appreciated!
Something really odd is going on with the dataset. I cannot get it to open to show anything useful on any of the applications (OS image viewers, The GIMP) I have here. It's a good demonstration of why TIFF (aka "Thousands of Incompatible File Formats") support is so hard to nail down. Only ImageMagick's display utility showed something recognisable. I can see it's a 32-bit TIFF though with an alpha channel (for some reason). Is this output, straight from DSS?

When I load it into StarTools, the image is entirely black and I cannot see any highlights (e.g. over exposing stars). This is likely the cause of our problems. It could be I have a bug in StarTools. I will investigate.

For the time being, I converted the dataset into a 16-bit TIFF;

Code: Select all

convert Test.TIF -depth 16 +compress Test16.TIF

Opening this gave me something more recognisable that is congruent with a linear dataset. However, unless you imaged under exceptionally dark skies, the dataset's background levels seem to have already been calibrated by something. I cannot see any background bias levels, as usually caused by the various sources of skyglow.

Nevertheless, processing from there was reasonably straightforward. Below is an extremely simple flow that you will recognise as similar to most flows in tutorials, videos, etc.;

--- Auto Develop
Default settings. To see what we're working with.
We can see some stacking artifacts, a defective sensor row, noise, a gradient, oversampling and tracking error.
--- Bin
Trading "useless" resolution for a better signal.
Parameter [Scale] set to [(scale/noise reduction 25.00%)/(1600.00%)/(+4.00 bits)]
Image size is 1300 x 866
--- Crop
Getting rid of stacking artefacts.
Parameter [X1] set to [3 pixels]
Parameter [Y1] set to [17 pixels]
Parameter [X2] set to [1295 pixels (-5)]
Parameter [Y2] set to [862 pixels (-4)]
Image size is 1292 x 845
--- Wipe
Get rid of Gradient. Default settings, except parameter [Dark Anomaly Filter] set to [5 pixels] just in case to make sure dark anomalies (e.g. the defective sensor row and any dark noise) is disregarded as representative of the real background level.
--- Auto Develop
Now that the gradient is gone and no longer requires dynamic range to describe it, we can do the final stretch.
Default parameters works ok, with two tweaks.
Parameter [Ignore Fine Detail <] set to [6.0 pixels] to make the detail detector blind to fainter fine noise.
Parameter [Shadow Linearity] set to [8 %] to allocate less dynamic range to the background.
You should - hopefully - notice Alnitak is kept under control very well, while still allocating plenty of dynamic range for the other detail.
--- Deconvolution
Usually worth a try. StarTools - thanks to signal evolution Tracking - tends to have a good handle on how far it can go without the signal falling apart. The earlier binning will have increased the signal-to-noise ratio sufficiently to recover some detail.
The auto mask generator is not working perfectly here, possibly because of the pre-calibration.
I semi-auto generated one like so; Mask > Auto > FatStars preset, set Fiter Sensitivity to ~ 5. > Do. Now click "Grow" three times. Finally click Invert to create "gaps" where stars are that need ignoring.
Back in the Decon module, I just used the default parameters.

... more to taste ...
You can use the HDR module, the Contrast module, the Sharp module, etc. to taste as well

--- Color
Final color calibration. As you're used to DSS, I chose the Legacy preset. Which emulates the way "classic" application desaturate the highlights.
The default color balance StarTools came up with was sufficient. But you can make sure in the MaxRGB mode that any dominant green can be ascribed to noise and is not readily visible. There are exceptions of course (e.g. O-III dominant emissions such as M42's core). Once you are satisified that any spurious green is not due to an incorrect color balance, you can set [Cap Green] set to [100 %] to enforce the removal of any spurious green.
--- Wavelet De-Noise
Switching Tracking off. StarTools has now had the longest time to monitor signal evolution, including its noise component.
Parameter [Grain Dispersion] set to [15.0 pixels]
Parameter [Smoothness] set to [70 %]

You should end up with something like this;

Test16.jpg (164.25 KiB) Viewed 7186 times

Hope this helps!

[perdrix]

Hey Ivo,

Thanks for looking at it. I'll give it another try using a 16-bit format.

Yes. it was almost straight out of DSS. Because the autosave.tif was compressed I reloaded it into DSS and saved it as a 32-bit rational TIF without compression (which is why GIMP and other apps had problems with it as most apps do NOT support 32 bit rational, this isn't really a problem with TIF, its a problem with lack of support for 32-bit per plane images) - Photoshop would have managed fine.

As an aside when I tried feeding Startools with a 32-bit integer TIF image it really had problems with displaying it right.

If you have problems handling 32-bit rational/integer images you should clearly shout when loading them

David

[perdrix]

Re: Tracking error - 4 minute unguided exposures can do that to you

And yes the nights I took those had exceptional clarity for urban UK! They turn the street lights off at midnight so that's when I started the imaging runs. No I didn't clamp the dark image areas in any way.
David

[admin]

Yes. it was almost straight out of DSS. Because the autosave.tif was compressed I reloaded it into DSS and saved it as a 32-bit rational TIF without compression (which is why GIMP and other apps had problems with it as most apps do NOT support 32 bit rational, this isn't really a problem with TIF, its a problem with lack of support for 32-bit per plane images) - Photoshop would have managed fine.

What I was trying to say is that, when I tried, The GIMP and other applications loaded and displayed the files in a similar way to StarTools. E.g. they did not complain, nor failed loading and showed either a black image (where alpha was ignored) or a checkered background due to the alpha channel being inappropriately set to full transparency. E.g. it's as if DSS did not make use of the dynamic range properly?

As an aside when I tried feeding Startools with a 32-bit integer TIF image it really had problems with displaying it right.

I'd be happy to have a look at the problematic dataset, to see if there is anything I need to fix!

The council turning of the street lights at midnight is amazing - I wish they did that here!

[perdrix]

When doing the wipe, I saw this:

Wipe.jpg (418.9 KiB) Viewed 7183 times

If the white area round the edges is where it thought there was a gradient - I think it is incorrect ...

D

[perdrix]

Star mask generation: I think you need to allow for bigger stars here as even with the slider for star size all the way to the right, the brighter stars weren't detected. I recently changed DSS to allow star diameter of up to 100 pixels as we had the same problem with creating star masks.

Mask.jpg (55.51 KiB) Viewed 7182 times