Discussion idea - giving StarTools stdev data from stack?

[Cheman]

[*] pat yourself on the back and have a beer [/list]

[/color]

I'd be worried about excessive negative ringing around stars if I do that, though. I guess that actually might be a good thing, as it forces a sharp threshhold around the star if I do a feature-detect threshholding at a >0 value. Perhaps skip the fourth step, though, as that'd be of questionable legality (I'm not even in college yet!)[/quote]

Then this would be of questionable legality

[khyperia]

Although, I'm starting to think I should just go the "use DSS's intermediate files" route instead, since DSS seems to have figured everything out and has all the features in it, like supporting bayer filters, calibration, it has a nice GUI, etc., something that's going to take a long time for me to implement (and the only purpose of my program is to create standard deviation data). Heck, it'd probably work with other stackers, too, as long as they have the option to output intermediate calibrated/registered .fits files.

Ended up doing this just now. I've attached the Windows prebuilt, the source is available in the github (AstroStats folder) as a single cpp file (for linux). Should be simple enough as "g++ -std=c++11 -O2 -lcfitsio AstroStats.cpp". It doesn't need the Eigen library.

The Windows version has a file dialog built in, so it does *not* require knowledge of command line stuff. However, since mac/linux doesn't have a common file dialog API, it's only for windows (just wrapped in a #if _WIN32). I figured linux people should know how to use a terminal

To make it simpler, I've defaulted the output to be "standardDeviation.fits" in the directory of the first image file.

Usage:

Open up DSS, do your normal stacking routine, but on the "intermediate files" tab of "Stacking Parameters", make sure these two are checked.

DSS.png (13.35 KiB) Viewed 11852 times

which should generate a bunch of "blahblah.reg.fts" files alongside the light frames. Run AstroStats.exe, either passing in "path/*.reg.fts" if you're on linux, or by selecting those files in the file browser that pops up if you're on windows (windows command line with globbing works too). You should then get a standardDeviation.fits file in that directory.

Important:

It supports N-dimensional .fits HDUs. It does NOT support multiple fits HDUs, it will ONLY take the first one. It also copies over the fits headers from the first frame, hopefully carrying any bayer filter/etc information with it (Che, I need your data to test this, and also to see if I need to support multiple HDUs!).
Edit: whoops, you posted your data as I was writing this, thanks!
Edit2: Hmm... actually... that .fit file you posted earlier looks like it's monochrome (with a bayer filter, causing the gridding pattern). How did you convert the file to FITS? Did it come raw off the camera like that? If not, could you send me the original raw frame?

[Cheman]

thats the raw fits off the camera (QHY8 one shot color) So yes its monochrome at at that point.that is fed into DSS along with all the others and a master flat and a master bias. I can demosaic and send the color version if you want.

Che

[khyperia]

thats the raw fits off the camera (QHY8 one shot color) So yes its monochrome at at that point.that is fed into DSS along with all the others and a master flat and a master bias. I can demosaic and send the color version if you want.

Che

Okay, took me a bit to find the setting in DSS that allows you to demosaic an image, but once I did everything worked great. Thankfully color in FITS files is represented as a 3-dimensional image, so AstroStats.exe should still work on the calibrated *.reg.fit files.

I did, however, run into memory issues with a 2x drizzled version of the image you sent me. The amount of memory required is 5*imagewidth*imageheight*numcolors*8 bytes, which turns out to be almost 3 gigabytes (5*6080*4032*3*8bytes) - which is 37% more memory than what a 32 bit application can hold
I've recompiled cfitsio+astrostats to be 64 bit for my local machine, but unless you run into the same issue, I'll just leave the 32 bit one as is.

I *could* add massive complexity to my program to reduce the leading 5 constant to be a 2, but I don't think that's all that necessary, unless someone with huge images comes around who's still for some odd reason on 32 bit.

All in all, though, I think this is finally starting to come together! Finally we can start getting back to this thread's original topic!

[admin]

Just a quicky while I work my way through my other work load; For those wanting to experiment with the standard deviation data using the current version of StarTools, here is how you can (NOTE: much more can be done by creating dedicated code in StarTools to use the stddev data), but it's a start;

• Load the stddev data
• Indicate that the image is non-linear and don't engage Tracking (yet)
• Launch the Layer module
• 'Open' the image data in the foreground.
• Set Filter Type to 'Differential Adaptive Noise'
• Set Filter Kernel Radius to 4.5 pixels
• 'Keep' the result

This is your new noise reduced data set, but data will be corrupted in the highlights (where singularities exist).
Fixing those is fairly easy;

• Load t the image data (don't worry - the noise reduced data set will still be in the undo buffer!)
• Launch the Layer module
• Launch the Mask editor and run the Auto->Stars preset
• Click Invert and 'Keep' the mask
• Back in the Layer module, click Undo->Fg

Turn Tracking on and process as normal (you might want to clear,invert the mask).

Gotta run...

Awesome work!

[admin]

On more thing I should add; Seeing as you're not even of beer drinking age, I'm *extremely* impressed with what you're doing here. If you ever need a reference of a CTO-ish type person for a job, etc. I'm happy to provide one; your proficiency in this area is displayed on Github for all to see and I'm happy to vouch for the pretty extraordinary work you're doing here!

[Cheman]

Just a quicky while I work my way through my other work load; For those wanting to experiment with the standard deviation data using the current version of StarTools, here is how you can (NOTE: much more can be done by creating dedicated code in StarTools to use the stddev data), but it's a start;

• Load the stddev data
• Indicate that the image is non-linear and don't engage Tracking (yet)
• Launch the Layer module
• 'Open' the image data in the foreground.
• Set Filter Type to 'Differential Adaptive Noise'
• Set Filter Kernel Radius to 4.5 pixels
• 'Keep' the result

This is your new noise reduced data set, but data will be corrupted in the highlights (where singularities exist).
Fixing those is fairly easy;

• Load t the image data (don't worry - the noise reduced data set will still be in the undo buffer!)
• Launch the Layer module
• Launch the Mask editor and run the Auto->Stars preset
• Click Invert and 'Keep' the mask
• Back in the Layer module, click Undo->Fg

Turn Tracking on and process as normal (you might want to clear,invert the mask).

Gotta run...

Awesome work!

Just finished acquiring new data. NGC 6946. I've stacked in DSS and produced the standard deviation data from the intermediate reg files, but have not yet had the time to try it out. If anyone else wants to play with the data here it is. Image
https://www.dropbox.com/s/vt45v3j4qqvmh ... 206946.fit

standard deviation data
https://www.dropbox.com/s/34hvj1qh5b3pb ... ation.fits
Che

[admin]

Thanks Che,

Not using stddev data
http://www.startools.org/download/Tutor ... _final.png

Using stddev data
http://www.startools.org/download/Tutor ... _final.png

Exact same processing - (AutoDev dealt with the image a little bit different due to the higher noise level in the non-stddev-treated data).

I think it's pretty clear the added stddev data has (as expected) a pretty noticeable effect on the background where read noise is most prevalent.

A slight detail loss in the core is visible though, but once the procedure outlined before is just a real quick & dirty way of incorporating the stddev data into your wokflow with the current state of StarTools' engine. Proper integration should yield better results!

This is very promising!

EDIT: fixed links

[Cheman]

Ivo
for some reason I cannot access the "not using" data
Che

[admin]

Ivo
for some reason I cannot access the "not using" data
Che

I'm trying to do too much at once... Sorry, fixed the links now (and the one that *was* working was actually the noise reduced one)