Problem processing Pillars of Creation dataset using HLA data

[kevinm]

Hi,

I've been using Startools 1.8.527MR2 to follow the excellent tutorial on processing the Pillars of Creation dataset from the Hubble archive.

First off I downloaded the pre-binned/cropped RGB tiff file from startools.org and loaded that into all three channels in the Compose module with the relevant exposure settings per-channel for synthetic luminance. Perfect result, lovely final image.

Then I wanted to go back to the raw data so I downloaded the 770MB combined file from HLA (color_hlsp_heritage_hst_wfc3-uvis_m16_f673n_f657n_f502n_v1_drz_sci.fits). Same workflow, loading this file into all three channels, same lovely outcome.

Finally I thought I'd go the whole hog and use the individual filter files from HLA, namely:

hlsp_heritage_hst_wfc3-uvis_m16_f673n_v1_drz_sci.fits
hlsp_heritage_hst_wfc3-uvis_m16_f657n_v1_drz_sci.fits
hlsp_heritage_hst_wfc3-uvis_m16_f502n_v1_drz_sci.fits

Loaded f673n into red/SII, f657n into green/Ha and f502n into blue/OIII per the filter wavelengths, exactly the same bin/crop workflow but the image is now strongly red rather than the expected green dominance from the Ha channel that you get in every other case.

FWIW I've downloaded similar filter-level data from the old WFPC2 camera and that works fine in the separate channels (but you have that wacky chevron image shape to deal with!).

The individual images look pretty much as you'd expect. I'm obviously doing something stupid here, but I've stared at this for long enough now so thought I'd see if anybody could point it out.

I normally use Startools for my own imagery, but the fact that you can process raw Hubble data with such ease and with such great outcomes blows me away. I plan to use this example to evangelise ST at my local astro club!

TIA

Kevin

[Mike in Rancho]

Hi Kevin,

Odd!

I downloaded the individual files to try it as well. Forgot to download the RGB combo to compare though.

Overwhelming red! Beyond the capability of Color to handle the bias.

I wonder if Wipe is not taking off the red floor, or if it is but it remains overwhelming?

I suppose maybe I might experiment with DAF or just re-weighting the channels in compose, and perhaps look at them individually too.

Will be interesting to see what the verdict is. Cool stuff though - I hadn't been to the HLA website before.

[admin]

The problem with some of the HLA datasets, is that many of them have FITS headers that don't have enough instructions (DATAMIN/DATAMAX fields) on how to deal with outliers. The HLA datasets contain copious amounts of anomalies; stacking artifacts, and lots of "undefined" pixel values.

Applications then have to estimate and figure out where true dynamic range starts (usually interstellar background) and ends (usually at over-exposing stars). Anomalous pixels that are brighter than the actual stars, will therefore be taken as the end of the dynamic range (ST doesn't want to clip your data).

If the DATAMIN/DATAMAX fields are not available or not reliable, I try to have ST detect such pixels to the best of its abilities, but it isn't always successful, particularly if they occur in clumps (they do in this instance).

Indeed, if you open up just the Ha or the O-III (the culprits here), you will notice the image is almost entirely black, with not a star visible.You may even be able to find some anomalous pixels by panning around.

For each (Ha, and O-III), you can use the FilmDev module to much more aggressively filter out these clumps of pixels, by using the Dark Anomaly Filter and letting it apply to bright anomalies as well;

StarTools_2825.jpg (72.68 KiB) Viewed 3867 times

You can see over-exposing stars (the blooming streaks) slowly appearing, with the anomalies still remaining much brighter. You can max out the dark anomaly filter (which really is just an 'anomaly' filter now), then keep and save the result. The Layer module will further allow you to linearly multiply the signal (Multiply Foreground Only) until highlights properly over-expose.

Incidentally, the Ha data does not seem to suffer from this at all.

Please note that properly calibrating the dynamic range is essential for the synthetic luminance weighting results to be correct as well!

TL;DR; Properly calibrating the dynamic range in all channels will fix the problem, which seems to have happened in the combined dataset, but not in the individual datasets.

[Mike in Rancho]

There's a whole lot of weird here, Ivo.

First, your technique sounds cool. I went looking in the documentation, and even just googled it (Startools and calibrate dynamic range), but came up empty. I think we may need more detail, knowledge of what ST is doing, and the creation of a new "Special Technique" for Jochen!

I tried, of course, but really couldn't figure out what I was looking for in either module -- FilmDev, following your instructions above and looking at the notes which say it can be used for hot pixels; or Layer either. Didn't know where I was supposed to stop with the linear multiplier.

In any event, I downloaded the full color file from HLA, and followed along with the YT tutorial the best I could. ST 1.6 was looking a little dated! And I think it may work a bit differently. I matched the compositing, bin, crop, and mirror (which doesn't end up in the log, FYI), but the FilmDev required a higher percentage of development to get the same "look" as your video.

Contrast of course has become new and improved, so a bit different. I could not get a decent deconvolution result out of this data, so just canceled it. Color was a good match, and everything came out pretty nice. I even used the provided defect file for heal. I was not going to go around lassoing dozens of black spots.

I will for sure try my own take on this data another time.

Next, I took that same HLA file and split it into R, G, B FITS using Siril, something that worked well for me on the Rosette last month. I tried to follow the same log steps, but alas, things were very different. In fact, it was much like the discrete SII, Ha, OIII files directly from HLA, except this time it was an overwhelming green cast that Color could not handle, instead of red. And FilmDev did not work at all, it took an AutoDev instead (but was still an abort in the end).

I haven't yet tried using ST for the channel splits to see if it will recomposite, though I presume that may work fine.

Kind of baffling, really.

[kevinm]

Thanks Ivo, that explains a lot. In fact a quick peek at the individual files in FITS Liberator shows just how dissimilar they are.

My fault for naively assuming that the individual files had just been combined in the colour file. The file names (and the FITS headers) suggest that they had been through the standard WFC3 processing pipeline, including Astrodrizzle, so I thought all the standard noise correction magic would already have been done. Looking more closely they are flagged as level 5 (HLSP) products which suggests they have been processed in some way outside of the normal pipeline. The combined file is level 4 so appears to be a standard product of the pipeline - probably why it works.

On the plus side, I've learnt a couple of new tricks in ST and also to be more wary of files in HLA. I'm new to this hobby and it's hard enough without being bowled the odd googly by the professionals

Kevin

[Mike in Rancho]

Well, I'm still grinding away on this one. But it is indeed fun data to work with. I will for sure go back to HLA for more now that I know it exists.

I also downloaded the FITS Liberator, which seems a handy tool. Oddly, when one loads the full color "all filter" file into it and reviews the panels (i.e. "RGB" I guess), the panels still do look very much like the individually-downloaded filter files, if not identical. Yet ST can manage things when the single file is composited. Go figure.

The Liberator is less useful though for comparing the "calibration" though, as the ST save then becomes a TIFF, unless there is another way to go about that. But after some playing around I did get the fixing to work, I think, even though I am somewhat unsure what is being done.

The FilmDev trick with anomalies sets the black side, and the Layer trick sets the white point? I really wasn't sure where to put the anomaly headroom, so just matched Ivo's initial example. And for the multiplier in Layer I just took things to the point that those star slashes became fully saturated. For the SII the Layer part wasn't needed, it seemed.

Oh I was also thrown off for a while by what I think is a typo above, that the Ha file is the good one. I think it's the SII. But I still ran a mild fix on that one anyway.

Here were the techniques I applied - which allowed for processing and seemed fine, though I'm not sure I got it right enough so that the exposure weightings were then correct?

Code: Select all

-----------------------------------------------------------
StarTools 1.8.527MR2
Sat Mar 05 00:07:49 2022
-----------------------------------------------------------
File loaded [D:\ASTRO\TUTORIALS\HST-ST-M16\hlsp_heritage_hst_wfc3-uvis_m16_f657n_v1_drz.fits].
Image size is 8000 x 8400
--- 
Type of Data: Linear
--- Photographic Film Development Emulation
Parameter [White Calibration] set to [Use Dark Anomaly Filter (May White Clip)]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [Off]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [33 %]
Parameter [Dark Anomaly Filter] set to [20.0 pixels]
--- Layer
Parameter [Layer Mode] set to [Multiply Foreground Only]
Parameter [Cap Mode] set to [Clip]
Parameter [Brightness Mask Mode] set to [Off]
Parameter [Filter Type] set to [Gaussian Low-pass(Fg)]
Parameter [Blend Amount] set to [150 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Filter Kernel Radius] set to [1.0 pixels]
Parameter [Offset X] set to [0.0 pixels]
Parameter [Offset Y] set to [0.0 pixels]
Parameter [Brightness Mask Power] set to [1.0]
File saved [D:\ASTRO\TUTORIALS\HST-ST-M16\RE-CAL Ha and OIII stacks\RE-CAL HA hlsp_heritage_hst_wfc3-uvis_m16_f657n_v1_drz.tiff].

File loaded [D:\ASTRO\TUTORIALS\HST-ST-M16\hlsp_heritage_hst_wfc3-uvis_m16_f502n_v1_drz.fits].
Image size is 8000 x 8400
--- 
Type of Data: Linear
--- Photographic Film Development Emulation
Parameter [White Calibration] set to [Use Dark Anomaly Filter (May White Clip)]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [Off]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [33 %]
Parameter [Dark Anomaly Filter] set to [20.0 pixels]
--- Layer
Parameter [Layer Mode] set to [Multiply Foreground Only]
Parameter [Cap Mode] set to [Clip]
Parameter [Brightness Mask Mode] set to [Off]
Parameter [Filter Type] set to [Gaussian Low-pass(Fg)]
Parameter [Blend Amount] set to [315 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Filter Kernel Radius] set to [1.0 pixels]
Parameter [Offset X] set to [0.0 pixels]
Parameter [Offset Y] set to [0.0 pixels]
Parameter [Brightness Mask Power] set to [1.0]
File saved [D:\ASTRO\TUTORIALS\HST-ST-M16\RE-CAL Ha and OIII stacks\RE-CAL OIII hlsp_heritage_hst_wfc3-uvis_m16_f502n_v1_drz.tiff].

File loaded [D:\ASTRO\TUTORIALS\HST-ST-M16\hlsp_heritage_hst_wfc3-uvis_m16_f673n_v1_drz.fits].
Image size is 8000 x 8400
--- 
Type of Data: Linear
--- Photographic Film Development Emulation
Parameter [White Calibration] set to [Use Dark Anomaly Filter (May White Clip)]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [Off]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [33 %]
Parameter [Dark Anomaly Filter] set to [11.0 pixels]
File saved [D:\ASTRO\TUTORIALS\HST-ST-M16\RE-CAL Ha and OIII stacks\RE-CAL SII hlsp_heritage_hst_wfc3-uvis_m16_f673n_v1_drz.tiff].

SVD still had trouble with the file and created a lot of artifacts where various bad or saturated pixels exist, particularly those vertical slashes through the bright stars. I thought I might try a heal on the hot/cold pixels prior to SVD, but that didn't work. In the end I just used my defect mask and the undo buffer to reverse deconvolution in those spots, with fair but not great results.

That said, SVD still resolved detail and double stars better than NASA even did, though they also applied quite a bit of cosmetic repair to their final image - including the bright star slashes and that odd swirly artifact in the upper left.

I will continue working on that dataset though.

[admin]

Apologies about the Ha <-> S-II typo. Hope it didn't cause too much of a wild goose chase!

FYI the way the anomaly filter works is as follows;

Say you want to normalise an image (e.g. linearly stretch the image so that the highest value in the image comes out as pure white, and the lowest value in the image comes out pure black).

Normally you would find the minimum and maximum, then subtract the minimum from all pixels and then multiply all pixels by (1.0 / (maximum-minimum)). Now you got a an image that perfectly spans the 0.0 - 1.0 dynamic range.

The problem arises when the minimum and maximum values are incorrect due to errors in the dataset (hot pixels, dead pixels, etc.), so you'd want to those incorrect values out before measuring the minimum and maximum.

To do that you measure the minimum and maximum of a filtered image, and apply the found minimum and maximum to the original (unfiltered image).

So, for example, take the original image and apply an outlier rejection algorithm (can be a Gaussian blur function, can be a median filter, etc.). You will want to use a filter that best matches the characteristic of the problematic data you're trying to filter. For example, a Gaussian blur with a large kernel leaves big structures intact and removes all small "crud". If the brightness of those big structures are still representative of the minimum and maximum values in your dataset, then great! You just found a way to measure the real minimum and maximum, without the small errors ruining your day.

That is, what the anomaly filter does in a nutshell. A variant of this, is to use just the minimum or just the maximum of the filtered image, leaving you with a "dark anomaly" or "bright anomaly" filtered minimum and maximum.

That said, SVD still resolved detail and double stars better than NASA even did, though they also applied quite a bit of cosmetic repair to their final image - including the bright star slashes and that odd swirly artifact in the upper left.

Developing SVDecon I had such a ball processing Hubble data. Pretty much anything that was suitable yielded such great results. It was especially an interesting exercise/test, because all PSFs are purely distorted by imperfections in the optics, given we're dealing with a space telescope.

But yeah, strange artefacts in the data ruined things more than once.

[Mike in Rancho]

Thanks again, Ivo. Though I think my head is absorbing too much learning.

Luckily this probably wont' happen much, absent weirdo Hubble data.

I may be getting a grasp here. So, keeping development itself OFF in FilmDev I presume avoids non-linearity. Changing the White Calibration to DAF filters white anomalies as well - as you said, now just anomalies in general. Basically filters on both sides now for the "junk" to fall through. ST senses the background level to be used on the dark side. And the white side is pretty much eyeballing it, but can run into the limits of what this FilmDev filtering can do. Hence the need of the Ha and OIII to also go through linear multiplication in Layer, so that the obvious oversaturated slashes through the bright stars in fact reach the level of being blown out.

Close?

In a little experimenting I also noticed that simply opening one of these anomalous files in ST itself, and then saving without doing anything, also seems to impart some minimal "fixes," though not enough to make it amenable to processing. But the histogram of such an ST-saved file was different than other 16-bit TIFF saves (say from DSS), and also could be stretched to reveal some shapes - even if highly posterized. But the opened and 16-bit saved version from DSS seemed to retain all the flaws from the original.

In any event, the Hubble repository seems like a lot of fun, and now I should be able to handle strangely scaled FITS files.

[admin]

Thanks again, Ivo. Though I think my head is absorbing too much learning.

Luckily this probably wont' happen much, absent weirdo Hubble data.

I may be getting a grasp here. So, keeping development itself OFF in FilmDev I presume avoids non-linearity.

Exactly.

Changing the White Calibration to DAF filters white anomalies as well - as you said, now just anomalies in general.

Correct!

Basically filters on both sides now for the "junk" to fall through. ST senses the background level to be used on the dark side. And the white side is pretty much eyeballing it, but can run into the limits of what this FilmDev filtering can do. Hence the need of the Ha and OIII to also go through linear multiplication in Layer, so that the obvious oversaturated slashes through the bright stars in fact reach the level of being blown out.

Close?

Right on the money!

In a little experimenting I also noticed that simply opening one of these anomalous files in ST itself, and then saving without doing anything, also seems to impart some minimal "fixes," though not enough to make it amenable to processing. But the histogram of such an ST-saved file was different than other 16-bit TIFF saves (say from DSS), and also could be stretched to reveal some shapes - even if highly posterized. But the opened and 16-bit saved version from DSS seemed to retain all the flaws from the original.

Indeed, the measures I was referring to in my earlier post do their best to find and fix any egregious or overly-suspect anomalies.
Such anomalies may be pixel values that lie outside any specified min/max values. "Special" values such as NaN or plus or minus infinity. Under certain conditions, another measure kicks in as well when it detects large swathes of pixels that all have the same value.

In any event, the Hubble repository seems like a lot of fun, and now I should be able to handle strangely scaled FITS files.

Though some of it is quite rough (enough for science, but perhaps not so much for pretty images), there are definitely some gems there. See if you can find the horsehead nebula close-up - another iconic Hubble image.