"Sharkmelley flats" for OSC/dslr

[dx_ron]

A few months ago a post by CN user Mark Shelley (aka sharkmelley) caught my eye. His observation is that some color gradients, especially circular or at the borders, can be due to two factors: 1) imperfect sensor linearity at the very low ADU values that typify most astro images; 2) crosstalk between OSC pixels from imperfections in the Bayer filter matrix. His idea is to shoot flats that match the sky brightness and color characteristics of your lights.

See here: https://www.markshelley.co.uk/Astronomy ... flats.html

I thought it was interesting, as I frequently struggle with generally poor color background in Wipe. I try what feels like my best and move on to the rest of processing. The other day it was cloudy and I decided to pursue his approach. I wrote down the color-channel-specific ADU average for a light frame taken without moon fairly near zenith and no major nebulosity (one of my Deer Lick Group subs). It came to R/G/B 1350/1875/879 (I expect the quite low blue is related to the L3 lum filter I use to avoid most of the AT130's lateral color issue at the blue end).

I went to PowerPoint and fiddled with colors until I was able to take a flat that matched fairly close to those ADU values (it looks sort of red-brown). I used it to calibrate ~5 hours of M81/M82.

Here is Wipe at 90 / DAF 3 with the normal flats:

normal-flat_Wipe90-3.jpg (734.11 KiB) Viewed 6916 times

and here with the "color-matched" flats, same Wipe settings:

rgb-flat_Wipe90-3.jpg (735.98 KiB) Viewed 6916 times

The differences are pretty subtle (though a bit harder to see in the jpgs, I think). Nothing like some of the dramatic improvements Mark shows with his dslrs. It's obviously a fair bit of fiddly work - how much and how fiddly will depend on how much the target background varies by target. And probably way too fiddly to try to implement for a dark-site trip. I also shot 100 of the low-ADU flats, because the SNR of the stacked flats must be much lower when the mean ADU is 1800 instead of 32000. (as an aside, I was impressed how well the new flats corrected the lights, because the lights were take 3 weeks ago and the rig had been disassembled into camera+filter-drawer+OAG / flattener / scope and stored for the intervening 3 weeks. I did intentionally avoid using the rocket blower on anything)

[Mike in Rancho]

I remember Mark raising this issue several times. In fact his original CN post goes back to something like 2019.

I have sometimes had, of course, quite a bit of seeming color and correction errors in raw stacks. Usually enough Wipe settings seem to take care of things, though perhaps with higher aggressiveness than I might otherwise like. And occasionally something like inverse vignetting correction will go out and take care of those edges and corners.

But as Mark says, the less gradient extraction needed, generally the better. And that's one reason I got WBPP to start utilizing subframe weighting and local normalization due to the many changing LP gradients I have when I do an imaging sweep.

Most such errors though I have attributed to stray light, or perhaps mirror flop, and so I have also tried to take direction-matched flats on most occasions. Which mostly just means I take separate flats for each side of a flip, and back up the mount to the midpoint of the imaging sweep. Not perfect, but perhaps close enough. I haven't considered color matched flats in a long time, but Mark does say mono can be affected too. Not sure if that's only for the linearity issue. Does cross-talk only affect OSC, or can a photon go into the "wrong" bucket in mono too?

But if it's a good idea, what is the proper implementation?

Did you use a statistics tool for the mean or median, or a histogram tool for the channel skyfog peaks (same thing either way I believe), jot down those ADU levels, and then subtract off your bias level from each, in order to get the correct relative channel balances? If it matters, are those bias corrected numbers then fed into a 0-100 or 0-255 color picker (such as in Gimp)? But with what scaling? I think in one example Mark set the G scaling to 100% and calculated the R and B relative to that.

Anyway that can probably get you the right hue, but then what intensity to pick? These flats are going to be incredibly dark at these ADU's, I think. Pretty amazing if they still properly work for vignetting and dust spots.

I'm not sure how I would correctly pull this off for mono. Not the least of which because my flat box is constructed around a tracing pad (and thus white or whitish), and again I have to plop it onto the OTA while mounted for direction-matching. Aiming at a computer screen in the horizontal would surely induce some mirror flop or collimation/focuser deflection.

That said I did go get my recent M33 data and ran some statistics on R, G, and B subs, each of which were 60s. Surprisingly I was brightest in the blue.

Well maybe my light pollution is blue-ish, but then I realized that these subs were not taken back-to-back-to-back, so the scope was likely pointed at a different LP situation when each sub I used was acquired. Oh well.

So, pulling this off, if the improvements would even be worth it, would take some real elbow grease for me, not to mention likely new hardware. Rats.

[dx_ron]

Yes - the idea is to have your flats match the average ADU of your lights. I simply loaded a bias-subtracted light into Siril and took the per-channel statistics Siril reported as correct. The implementation was completely trial and error. I think for a dslr Raw you might already have some weights for "white balance" available as a starting point, for the 571c I simply started with white and then kept tweaking the rgb values until I got pretty close to the same distribution as the light sub. I ended at 160-133-177. The screen brightness and exposure time were tweaked to get the same per-channel mean ADUs.

So yes, the flat is pretty dark. But that's not a problem for calibration. Remember that we don't actually use "light/flat". It is really "light / (normalized flat)". What goes into the division are numbers always between 0 and 1, with the brightest pixels (in the flat) have the corresponding pixel in the light divided by 1 (so they keep their starting value). Any pixels where the flat is dimmer will be divided by a number less than 1, so those pixels in the light will have their raw ADU boosted by the appropriate factor. The reason normally given for ~32k ADU flats is to "ensure that your flat is in the linear region of your sensor" - so quite the opposite of the ADU-matched flat approach, which wants to account for any non-linearity at the ADU values actually encountered across most of the light.

I really don't know how this would be applied to mono. I assume any low-ADU sensor non-linearity does not depend on the presence of a Bayer filter. As to how to implement - my first guess would be to take, for each filter, flats at an ADU that matches the lights for that filter. I'm not really convinced by Mark's "crosstalk" theory, and I'm not sure what kind of data he could come up with to test it. I do wonder if there's something inherently different about dslrs that make Mark's results so striking.

I also don't know the best approach to determine sky background color from lights that include a large region of nebulosity. Presumably M33's coloration is not the same as your LP background sky.

[dx_ron]

But as Mark says, the less gradient extraction needed, generally the better. And that's one reason I got WBPP to start utilizing subframe weighting and local normalization due to the many changing LP gradients I have when I do an imaging sweep.

Separate question - does applying local normalization in WBPP violate ST's expectation of true raw ADU values? It sounds like it might be a similar type of offense as Siril's per-sub 1st-order background subtraction.

I see that PI's price quietly jumped EU50 on Jan 1. If they'd made that generally known ahead of time they might have had a sale from me...

[Mike in Rancho]

Separate question - does applying local normalization in WBPP violate ST's expectation of true raw ADU values? It sounds like it might be a similar type of offense as Siril's per-sub 1st-order background subtraction.

I see that PI's price quietly jumped EU50 on Jan 1. If they'd made that generally known ahead of time they might have had a sale from me...

I had similar concerns. Mike doesn't want to break the rules. Unless of course I convince myself that it's totally warranted.

I think Siril's per-sub gradient extraction is more violative, since that's stepping on Wipe's toes. If more was known about it, and how closely it might be doing what Wipe can do? But it just might be too heavy handed a manipulation anyway.

I'd say PI's LN bells and whistles (note that AFAIK they do not offer a version of extraction-while-stacking) are more like what's available in DSS, except premium level. Now, Ivo has warned us against that too, hence the DSS settings recommendations. But there is a waiver clause (get out of jail free card?) for one particular type of background normalization in DSS, if required in order for sigma rejection to actually work as expected. So there's that. The other types of normalization in DSS and of course any white balancing or channel neutralizing remain verbotten.

So, I sought Ivo's blessing and suggestions, here: viewtopic.php?f=7&t=2755

In most cases I can tick off more boxes for using LN than not. Or at least I decide to weight them more heavily.

And my chaotic LP gradients during an imaging sweep, especially if a flip is involved, gets the strongest weighting.

Granted, I probably should look into how WBPP is choosing and creating the LN reference, and override the defaults if necessary. For my M33 last month I know the first night had some passing high clouds and I just left them in to be handled by subframe weighting and LN. I do think that PI chose from my second, cleaner night, for constructing the references, but I didn't give it more than a glance, really.

OT: There appeared to be an ST-curious returning old imager in EDSI, you think of chiming in? I considered, but ol' Bob is just so hyperbolic and overbearing I just left it.

[decay]

Interesting topic, Ron. Thanks for pointing out the article Mark Shelley wrote.

But I’m also not sure how relevant this is in the end. Agreed, his examples are striking, but I never heard of such problems with such strong colour casts at the borders nor I encountered them myself.

As Mike wrote, Wipe will usually and automatically take care of such (colour) gradients because gradients due to light pollution are usually much stronger for most of us. Maybe this is a combination of a nice dark sky (B3) and his specific equipment? But of course, you also mentioned having problems with background colour for yourself … maybe I’m just not critical enough with my own processing ...

As for non-linearity I’m able to follow up to some point. Yes, maybe some cameras do show non-linear response for low ADU values. And maybe some cameras more than others. And this results in what is being described. But I would assume, that this is relevant only for very low ADU values and therefore it should be possible to mitigate the problem by increasing exposure for both, lights and flats up to a reasonable value, leaving this non-linear range at the low-end?

(BTW, there is one more reason to leave some distance between the left margin of the histogram and the shoulder of sky background brightness respective flat exposure level. Using a higher ISO (or gain) setting means stronger noise signal and this may even clip at the left side for peak values. Not sure, if this is visible in histogram curve in all cases, but it will probably mess up calibration.)

I’m too not sure regarding bayer matrix cross-talk because of the angle of incoming light. Because some photons will pass and leave the dot/cell in which they entered and instead enter the next adjacent cell (having another colour)? Hm. Maybe. Maybe not. It somehow sounds too simple. But of course, my knowledge is not deep enough. And: If this is true for DSLR lenses having short focal lengths, this is probably not the case for our scopes having focal lengths of 750, 1000 mm or even more? Light is coming in nearly orthogonally to sensor surface, I would assume? No?

Mike, in case of a mono cam this would result in a coma-like effect? Stars being prolonged radially / towards the borders?

Regarding the colouring of sky and flats: For me, my lights show a (typical?) reddish / brown colour due to light pollution. (My camera is astro modded, this may of course cause an additional red cast.)

2024-01-07 17_39_20-Window.jpg (28.79 KiB) Viewed 6684 times

and ran some statistics on R, G, and B subs, each of which were 60s. Surprisingly I was brightest in the blue.

Well maybe my light pollution is blue-ish

Maybe there’s a lot of blueish illuminated advertising down in LA, Mike?

I tried a lot of different approaches to take flats: PC display, tablet, t-shirt, the white wall of my house and flats taken during dusk. And all possible and impossible other things. But I was never satisfied and so I decided to build a flat field box. This was a huge improvement and I had no more issues regarding flats since then. I used LED stripes having a warm-while colour instead of cool-white. To be honest and if I remember correctly, just because cool-white was not on stock. But because of this my flats unintentionally show a reddish/brown colour cast, just like the lights:

2024-01-07 17_42_11-Window.jpg (27.92 KiB) Viewed 6684 times

So haha, just good luck! And red is _much_ stronger than blue and green and therefore my flats have to be pretty bright. But they work quite fine.

Having read Mark Shelley’s article and our discussion here I would suggest using RGB LED stripes which allow user defined colour setting in order to build a flat field box.

Using dusk sky flats may indeed be a problem as they are pretty blue – instead of red. If you don’t like to build a box on your own, you might consider to buy one, Ron. In relation to the equipment you already bought, that should be quite affordable for the aperture of your refractor scope, I guess. And taking flats is _much_ easier and done in 2 minutes!

Best regards, Dietmar.

[Mike in Rancho]

Good additions to the discussion, Dietmar.

I'm unsure of the LP chrominance here. And I'm sure it's different in different directions. I flipped through my archives and found one sequence where I took R, G, and B in fairly quick succession - all within 7 minutes. It was west pier approaching meridian, so movement should have lessened the later taken shot, if I am thinking that through properly. My skyfog was still primarily blue-leaning. Or perhaps the target affected things (it had some reflection nebula up in Cygnus)? Anyway with blue normalized to 100, the R was 66 and G 88. Earlier in the sequence (down in more LP) it was 76-94-100. Not as strong but still blue. I don't know what's over there. Not LA! Victorville?

A flat panel with custom colors would be pretty cool, though. Or maybe use some kind of colored diffusion sheets? Currently I built my flat box around an A3 light pad, and have a few sheets of clear diffusion added. But now I see some A3 pads on Amazon that sort of have a color choice, or at least three light temperature choices. White, normal, and warm. I wonder if those would help or the match has to be more exact?

I set the rig up and took a couple test flats through the G filter, one normal and the other at about 920 ADU as fairly representative of G skyfog like the lights might have. The low ADU flat was quite noisy. I subtracted off bias and then divided one by the other. I think that's what Mark did?

Flat Linearity Division.jpg (306.94 KiB) Viewed 6582 times

I don't see much error in the vignetting profile here, autostretched. I even saved it out as a FITS and tried to hyperstretch it in ST (no clue how to stretch in PI other than mashing buttons) and still didn't see anything. Other than some banding. The exposure was so short that I may have picked up oscillation in the panel's LEDs.

So that's linearity I think. Not sure how to test for wavelength crosstalk. Maybe have to read Mark's page again, but at least I don't see obvious trouble regarding the intensity of my flats.

I then decided to take a flat at the same exposure (at near center ADU levels) through each filter, to see what my RGB set thinks of my flat box. Here, G was the brightest. Normalizing G to 100, I got 69-100-91. That's still decently strong in blue, and the overall result is sort of a teal.

Maybe I need more of an aqua blue flat than teal.

[dx_ron]

Very interesting.

I *think* the only reason to try to match flat color to LP color is *if* you are in the situation Mark described, where your sensor is not completely linear at the ADU values of background regions in your lights *and* you are taking flats at those same ADU levels. Otherwise I think the prevailing feeling is that color balance of the master flat is not very important. Except I guess for narrowband, and then it's more about getting stuck taking really long flats.

One way to check linearity is to take light subs at, say 30s and 120s. Then check to see if the R:G:B ratio is the same or different between the two exposures (after bias subtraction, of course). That keeps the sky background constant.

I actually take flats in a variety of ways. Bright daylight at zenith, twilight, and an LED panel. Which mostly depends on when I am set up and if it is clear during the day (can't take sky flats with any clouds).

[decay]

My skyfog was still primarily blue-leaning.

Yes, interesting insights about your LP chrominance, Mike. I thought I had read a couple of times, that urban / human caused LP usually tends to be yellow – orange – red – brown. I tried to find some information on this, but it was not as easy as I thought. I found this scientific paper:

https://academic.oup.com/mnras/article/ ... 01/1003276

Not sure, if it’s worth reading the whole text, but they found urban LP to be stronger at longer wavelengths (i.e. red) and that clouds will multiply the strength of these components.

So again learned something new. Blue-ish LP for Mike. Strange(r) things are going on there Maybe one day we will find an explanation for this.

I don't know what's over there. Not LA! Victorville?

Area 51?

I *think* the only reason to try to match flat color to LP color is *if* you are in the situation Mark described

Agreed. Good to know and to understand, but not relevant until it shows up.

Best regards, Dietmar.

[Mike in Rancho]

I *think* the only reason to try to match flat color to LP color is *if* you are in the situation Mark described

Agreed. Good to know and to understand, but not relevant until it shows up.

Best regards, Dietmar.

Maybe I'll have to re-read Mark's write-up and think it through again. I didn't think the situation described was Mark taking lights in a non-linear range of his DSLR, nor that he was taking low ADU flats and ran into this issue. I thought the low ADU flats came along later, as part of his experimentation/remedy for the intensity issue.

Next rainy day maybe I'll set up the rig inside again for some flats testing, but will try all the filters for any differences instead of just G.