Flats & noise propagation; who's up for an experiment?

[admin]

Hi all,

As 1.5 development draws to a close, I've been looking at an idea I have had for a while, which is to feed flat frame stacks to the signal evolution Tracking engine.
As you know, gradients, vignetting, dust, etc. locally darken the signal. Flats counter that obviously by multiplying the signal by the correct amount. However, this also multiplies the noise component.

By recovering and feeding these per-pixel multiplication factors to the Tracking engine, all algorithms (including Denoise) should be able to achieve even cleaner results than they already are.

The prerequisites for this are 2 stacks; one completely calibrated as normal, and one stacked without flats applied but otherwise identical. If your flats show a fair bit of correction (due to dust, smudges, vignetting, etc.) and you're up for an experiment, please let me know!

Ivo

[szymon]

I am so in! Am just working on data from the eagle nebula where I have a number of different sets of flats (and I think I used the wrong ones when stacking originally). I'm going to get it "properly" stacked in DSS, but I would love to also see if ST can do a better job

[admin]

I am so in! Am just working on data from the eagle nebula where I have a number of different sets of flats (and I think I used the wrong ones when stacking originally). I'm going to get it "properly" stacked in DSS, but I would love to also see if ST can do a better job

I'd love to get my hands on the two stacks, where one has flats applied and the other hasn't! The idea is to compare the two and deduce the multiplication factors used on the signal to bring it up to "even it out".
To be clear, the idea is not to replace what DSS is doing, rather it is to take into account what DSS (or any other stacker) is doing. I think you were only recently introduced to StarTools (correct me if I'm wrong! ), but Tracking has been one of the mainstays of StarTools and is what sets it apart as a processing engine. It's a complex beast that I've been working on for almost a decade now. The core tenet is to keep track of the complete evolution of your signal from start to finish, in service of signal fidelity and noise/artifact mitigation. To most "mere mortal" astrophotographers, time spent under the night skies is precious and so is - by extension - every morsel of signal.

Right now, Tracking starts when you import the stack. The Tracking feature is getting quite mature now to the point where I'd like to go back even further; I want to measure what the stacker has been doing as well to estimate noise propagation at this much earlier stage. Flat frame calibration is a reasonably easy target for StarTools, as it does not (yet) require I write and maintain a custom stacker. It should - in theory - also be fairly impactful as uneven lighting and light-blocking foreign materials can really make the signal dip.

That's in theory, but the proof is in the pudding, so I'm looking for some dirty sensors...

[szymon]

Aha! Yes, that makes sense, and yes I'm new to StarTools and tracking . The idea sounds good!

When I get some time this evening, I will do my best to process the stacks correctly, and then I will upload what's needed. I'll also put the raw files out there in case you want to play with stacking them yourself. And hey, if I do have an actual issue with the flats, perhaps you can work out what I did wrong when taking them!

-simon

[szymon]

My apologies for the radio silence. We left for our summer holiday on Friday evening, and it turns out I've had no time to do anything at all over the past four days, as we've been visiting Iceland. There was really fast internet in the hotels we stayed in (my son ran a speedtest and got synchronous 100mbps download and upload, in a hotel in the middle of nowhere!), but I had no energy in the evening to even start the computer up. We're now in the lounge waiting for our onward flights to Boston -- we have a 2.5 week road trip next, through Maine, the Canadian Maritime Provinces, Quebec, New Hampshire, Vermont and all the way down to New York; I am hoping for some dark skies for my portable kit, there was no astrophotography here in Iceland as there was no darkness . In any case, I have some time and a fast internet connection, so I've gone through and tried to reprocess the data.

So, I can't work out whether the problems I have is in the lights, in the flats, in the way I am stacking it all, or what. The Oiii image seems good, the HA one doesn't. I figured the best thing to do would be to give you access to the raw data and let you stack it yourself as best you can, with and without flats, and then you'll have two "best practise" stacks and you can try your new algorithms out. All I ask is that you tell me what's actually wrong with the data (did I do something wrong in capturing, is it fixable, etc) and share the output.

Each Zip file contains four folders: LIGHTS, DARKS, FLATS, DARKFLATS. There are separate flats and darkflats taken with each filter, because I'm not sure whether that makes a difference so I took them anyway. I've also included an extra folder with some more flats that I believe were taken with the HA filter but earlier in the session.

HA: https://www.dropbox.com/s/pxe9vs4kbk5g4 ... A.zip?dl=1
OIII: https://www.dropbox.com/s/hkwymzeoi35il ... I.zip?dl=1
ExtraFlats: https://www.dropbox.com/s/ojc4veh3scqnq ... S.zip?dl=1

I hope it's useful! I'll be online on and off while we're on holiday so I will look in to see if these have done any good. I'll also hopefully be showing off new data captured with my mobile setup -- Canon 6D, Astronomik XL CLS Clip Filter, Baader Nano-Tracker, range of lenses -- processed in StarTools .

-simon

[admin]

Will let you know findings - if any - when I got some time. Much appreciated!

[admin]

I had a quick look at the Ha set and it appears the flats are mismatched with the actual defects in the light frames, while the O-III set's flats seem to over-compensate.

[dariv]

Ivo,
I can provide data for this experiment as well. I use darks, flats, and flat darks (or is it dark flats?)
Do you want one stack fully calibrated and one with darks only?

[admin]

Ivo,
I can provide data for this experiment as well. I use darks, flats, and flat darks (or is it dark flats?)
Do you want one stack fully calibrated and one with darks only?

That would be fantastic DaRiv!
Yes, exactly as you suggest; one stack fully calibrated and one with darks (and of course lights!) only.
Datasets from people at theskysearchers.com have already helped me understand that still including darks in the otherwise uncalibrated stack may (may!) be important to get this to work properly. I'm not giving up on the idea of accounting for darks too however.

So, if you don't mind, a stack with just lights (without darks!) would be very useful as well.

Thank you so much in advance!

[szymon]

Hi Ivo,

I'm not sure whether you're still interested in this data, but I have been concentrating recently on learning to collect clean data and good calibration frames and stacking properly, and I thought you might be interested this one set of stacks where I accidentally forgot to add flat frames

Here is the data:

https://www.dropbox.com/s/741c8unvum5n1 ... t.zip?dl=1

These are the four good stacks:

2019-08-27_03-51-44_Sii_19.60_300.00s_6@ 20xFD 20xF 4xD 6xSii_stacked.fits
2019-08-27_03-16-37_Ha_19.70_300.00s_12@ 20xFD 20xF 4xD 18xHa_stacked.fits
2019-08-27_01-58-58_Oiii_20.50_300.00s_6@ 20xFD 20xF 4xD 6xOiii_stacked.fits
2019-08-27_03-59-10_CLS_19.70_300.00s_1@ 20xFD 20xF 4xD 12xCLS_stacked.fits

These are the four bad stacks (missing flats):

2019-08-27_03-51-44_Sii_19.60_300.00s_6@ 0xFD 0xF 4xD 6xSii_stacked.fits
2019-08-27_03-16-37_Ha_19.70_300.00s_12@ 0xFD 0xF 4xD 18xHa_stacked.fits
2019-08-27_01-58-58_Oiii_20.50_300.00s_6@ 0xFD 0xF 4xD 6xOiii_stacked.fits
2019-08-27_03-59-10_CLS_19.70_300.00s_1@ 0xFD 0xF 4xD 12xCLS_stacked.fits

I hope it's useful. As a bonus, I'd love to see if you can come up with a nice rendering of the 'good' data in StarTools!

-simon