Guess I'm getting rusty :)

Questions and answers about processing in StarTools and how to accomplish certain tasks.
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Guess I'm getting rusty :)

Post by admin »

gboulton wrote:
admin wrote: Ok... any sort of calibration in DSS will also significantly affect how much detail can be extracted, how well noise levels can be determined and typically causes clipping and bloating of stars (as is quite apparent here).
I don't really understand this statement...since this has never been an issue before, and it's not like DSS gives you a choice. You can either do Per Channel or RGB calibration...there's no "Don't do any" option. ONE of those two types of calibration has been done to every DSS image StarTools has ever processed. :confusion-shrug:
You can turn both calibration options off by making sure that under Stacking Parameters->Intermediate Files->'create a calibrated file for each light frame' and 'create a calibrated file for each light frame' is unchecked;

So you can switch from this;
Screenshot-Stacking Steps.png
Screenshot-Stacking Steps.png (38.05 KiB) Viewed 9932 times
To this;
Screenshot-Stacking Steps-1.png
Screenshot-Stacking Steps-1.png (36.8 KiB) Viewed 9932 times
The following is a checklist for DSS, which should (should! - I'd love to verify this with some data) yield the best results in your situation;
  • Make sure that under Stacking Parameters->Intermediate Files->'create a calibrated file for each light frame' and 'create a calibrated file for each light frame' is unchecked.
  • Make sure that under Settings...->Raw/FITS DDP Settings...->RAW Files tab->Color adjustments->Brightness, Red scale and Blue scale are set to 1.0
  • Make sure that under Settings...->Raw/FITS DDP Settings...->RAW Files tab->White Balance->Non of the options are ticked
  • For best results from a DSLR when not dithering and when few frames are available, under Settings...->Raw/FITS DDP Settings...->RAW Files tab->Bayer Matrix Tranfromation->Create super-pixels from the raw matrix
Note that the latter option reduces your resolution to 25%, but does not interpolate data and does not exacerbate noise grain due to interpolation. Always use this option if the CCD in your DSLR has a resolution that is waaaay to high to capture one pixel of real detail per pixel on your CCD. Also use this if seeing causes the same issue (1 unit of detail is smeared out over multiple pixels).
Bugger :( Would've liked to learn why the data looks like it does. Storage is so ridiculously cheap these days though!

It's not storage cost..it's OCD :)

As it happens, however, it occurs to me that i may not have purged them from the session laptop yet however...I frequently copy rather than move them over the network (on the off chance something bombs during the transfer) and sometimes don't go back and delete them from the lappy...I'll check next time I think about it and let you know.
It'd be so awesome if you could still retrieve that data - I'd like to write up some instructions for people using DSS, as getting the right, absolutely 'virgin' data has become more and more important and there are a number of pitfalls on the road to getting there!
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Guess I'm getting rusty :)

Post by admin »

gboulton wrote:Ivo,

Found the raws...will upload a zip tomorrow...upgrading our connection going to wait till I have some bandwidth
Yay! :bow-yellow: It'll help heaps getting some sort of DSS workflow together.
Ivo Jager
StarTools creator and astronomy enthusiast
gboulton
Posts: 122
Joined: Thu Dec 06, 2012 12:40 am

Re: Guess I'm getting rusty :)

Post by gboulton »

http://www.nightskyforums.com/astroimag ... 013-08-01/

Has a zip of a single light frame, and all the light frames.
-- Gordon

Image

My Astrobin
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Guess I'm getting rusty :)

Post by admin »

Thanks for that Gordon.

I spent all day battling DSS and dcraw, trying to keep them for modifying the whitebalance - something fishy is going on with that (red channel is being destroyed completely or even inverts data!) and I gave up trying to use unbalanced data.

I ended up with the following although using non-whitebalanced data got me much better detail (but destroyed the red channel completely);
Final3avg.jpg
Final3avg.jpg (104.48 KiB) Viewed 9927 times
May I ask what version of DSS you use?
I actually was unable to stack the CR2s as they were in DSS (3.3.2), as it recognised their resolution incorrect and I was also unable to convince DSS not to whitebalance the image - very annoying.

After converting the RAWs with the latest version of dcraw (which DSS uses as well, but an older version) to TIFF, I ended up with a stack.

You have a lot of streaks in the data, which you can minimise by using a bad pixel map and (especially recommended!) dithering between frames.

I did the usual Autodev, crop, wipe sequence and did a global stretch with Develop.
I then used the Life module's Isolate preset and a mask to isolate the nebula from the background.
Not much more was done, except color balancing. I used the latest alpha for this.

Thanks again for uploading the data. It will really help with development!

Cheers,
Ivo Jager
StarTools creator and astronomy enthusiast
gboulton
Posts: 122
Joined: Thu Dec 06, 2012 12:40 am

Re: Guess I'm getting rusty :)

Post by gboulton »

admin wrote:May I ask what version of DSS you use?
3.3.3 beta 47

using 3.3.2 gave me and several others some very odd results...including result files that were nothing but a long vertical bar of noise, and so on. At that time, someone (I can't for the life of me remember who or where) suggested the latest beta version available at the DSS Yahoo Group. That seemed to resolve that issue.
After converting the RAWs with the latest version of dcraw (which DSS uses as well, but an older version) to TIFF, I ended up with a stack.
I'm having more and more doubts about DSS as I learn more and more. I've been playing with PixInisght as well of late (I know..don't shoot me! I maintain there's plenty of room for MULTIPLE tools to get things done *heh*) and get stacks out of it that are FAR superior to DSS's results. I'm less and less sure about even keeping DSS around, to be honest.
You have a lot of streaks in the data, which you can minimise by using a bad pixel map and (especially recommended!) dithering between frames.
These were dithered...just like every set I've taken for the last year or so.
I did the usual Autodev, crop, wipe sequence and did a global stretch with Develop.
I then used the Life module's Isolate preset and a mask to isolate the nebula from the background.
Not much more was done, except color balancing. I used the latest alpha for this.
You got a lot more brightness out of it than I did...but some of that may well be the different stack, or possibly just stronger use of Life. That is, however, my basic workflow as well.

I'm definitely noticing that we have different tastes too. Your results frequently tend to "jump" out of the background. You like things bright and vivid, and will accept some halos or fuzziness to get there. I'm more a fan of perhaps less pronounced but sharper lines, and more subtle colors.

Not to say either of them is better or wrong...as I've said many times before...anyone who says anyone's processing is "wrong" is full of it...NONE of the objects we shoot would look anything like our images to the naked eye, EVEN if we could travel there to see them. Every one of these is simply our own interpretation of a much larger set of data than our eyes could ever gather.
Thanks again for uploading the data. It will really help with development!
You're welcome to make use of it however it may serve you in further development efforts. :)

Perhaps it's time to explore giving StarTools its own integration module? ;)
-- Gordon

Image

My Astrobin
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Guess I'm getting rusty :)

Post by admin »

gboulton wrote:
admin wrote:May I ask what version of DSS you use?
3.3.3 beta 47

using 3.3.2 gave me and several others some very odd results...including result files that were nothing but a long vertical bar of noise, and so on. At that time, someone (I can't for the life of me remember who or where) suggested the latest beta version available at the DSS Yahoo Group. That seemed to resolve that issue.
Aha! So there are in fact newer versions out there. That's good to know.
I'm having more and more doubts about DSS as I learn more and more. I've been playing with PixInisght as well of late (I know..don't shoot me! I maintain there's plenty of room for MULTIPLE tools to get things done *heh*) and get stacks out of it that are FAR superior to DSS's results. I'm less and less sure about even keeping DSS around, to be honest.
PixInsight's stacking and integration modules yield the absolute best (consistent) quality currently on the market. It's worth the license fee for that alone (and even then it is very reasonably priced for all the other tools you get).
You have a lot of streaks in the data, which you can minimise by using a bad pixel map and (especially recommended!) dithering between frames.
These were dithered...just like every set I've taken for the last year or so.
Well, looking at the data, the frames are just allowed to drift in a straight line between frames (which is indeed better than nothing!), whereas a preferred pattern would be spiralling out relative to the first frame, thereby spreading any pixel bias evenly across the frame, leaving a much less distinguishable pattern that is also much easier to noise reduce (tightly packed structures/pixels flowing in a single direction will stand out as 'detail', both to the human eye and to any isotropic noise reduction algorithm).
You got a lot more brightness out of it than I did...but some of that may well be the different stack, or possibly just stronger use of Life. That is, however, my basic workflow as well.
I'm definitely noticing that we have different tastes too. Your results frequently tend to "jump" out of the background. You like things bright and vivid, and will accept some halos or fuzziness to get there. I'm more a fan of perhaps less pronounced but sharper lines, and more subtle colors.
Not to say either of them is better or wrong...as I've said many times before...anyone who says anyone's processing is "wrong" is full of it...NONE of the objects we shoot would look anything like our images to the naked eye, EVEN if we could travel there to see them. Every one of these is simply our own interpretation of a much larger set of data than our eyes could ever gather.
You're totally right though - I tend to push the data in the face of noise - mostly using the Life module and a mask - where other people (probably rightfully) back off. Perhaps too much for some. I don't use the Life module at all when there is ample signal for the object to 'stand on its own two feet' so to speak, unless the image is too 'flat' or there is a buasy star field obscuring he object of my interest. As for colors, I always like to show the full range of star temperatures.

I actually found there's something 'special' about the way ST handles colors (which seemed like a no-brainer to me). Thanks to Tracking, ST totally separates the processing of luminance and color, which allows it to convey 'true color of an object' in the context of the total image, no matter whether it is very bright or very dark. E.g. an asnwer to the question of 'what would the colors look like if the whole scene were evenly illuminated?'
A perfect example is M42 or M31. Where most software portrays M42's core as white (because color data is stretched along with the luminance data in that software), ST brings out the 'true' color (teal/green) in relation to its surroundings, no matter how the image was stretched or exposed. Same thing for M31. Whereas most software protrays it as a brown/white, ST shows it as a blue, brown, purple, orange and yellow amalgate, much like the (in)famous Apple MacOSX wallpaper.

It's a more 'scientific' way to show colors of the objects, allowing you for example similar to easily see areas of similar gasseous composition independent of how bright they are, with the same being true for stars. For example, M42's colors are consistent in tone and consistently saturated throughout the image across different data sets, even with different exposure times (unless the core gets overexposed of course). I didn't fully realise the consequences of this 'feature' until a couple of weeks ago. What this also means though is that colors in ST are quite a bit more vivid. And this also means that color is not always what people expect when comparing to other software.

So a couple of days ago, I started adding a feature to allow people to use alternative ways of bringing color into their images, for example retaining even illumination but emulating human visual response to color and brightness (leading to less saturated highlights, leading to the image looking like 'you are there' with super human eyes), or using the luminance stretching history to manipulate highlight saturation (leading to the look you get from most other software). The differences can be quite dramatic. Stay tuned!
You're welcome to make use of it however it may serve you in further development efforts. :)
Perhaps it's time to explore giving StarTools its own integration module? ;)
Thanks again Gordon - it's much appreciated!
Perhaps for 1.5 - it needs to be done right though.
I have a lot of work to do for 1.4 first!
Ivo Jager
StarTools creator and astronomy enthusiast
gboulton
Posts: 122
Joined: Thu Dec 06, 2012 12:40 am

Re: Guess I'm getting rusty :)

Post by gboulton »

admin wrote: PixInsight's stacking and integration modules yield the absolute best (consistent) quality currently on the market. It's worth the license fee for that alone (and even then it is very reasonably priced for all the other tools you get).
Well said, sir.

I think there's definitely room for both it and StarTools in my toolbox. I'm absolutely blown away with PI's DBE tool. It's amazing. But the thing can't correct imperfect stars without 184 lines of machine code, 732 mouse clicks, and a dead chicken swung in a counter clockwise direction (presuming you're north of the equator of course...you aussies should swing clockwise, obviously).

I think, to be honest, the best description from a USER's point of view I can come up with is this : It certainly makes some complicated things very simple...but it maddeningly makes the simplest of things extremely complicated. Seriously, PixInsight? Crop needs to be a dynamically linked function that requires me to launch an "applet" and link it to an instance of an image?? What? Come on...it's CROP for chrissakes. :lol:
Well, looking at the data, the frames are just allowed to drift in a straight line between frames (which is indeed better than nothing!), whereas a preferred pattern would be spiralling out relative to the first frame, thereby spreading any pixel bias evenly across the frame, leaving a much less distinguishable pattern that is also much easier to noise reduce
:doh:

You know what? I was sitting here typing out a long thing about being sure PHD dithered exactly as you're talking about. I'm quite familiar with the concept, and quite understand how and why it should be done a certain way, and so on. And I'm thinking "What the hell is Ivo talking about?? I know this, and that's how PHD does it!"

Then it occurred to me...the last time I was imaging...back in April or May I guess?...I was having some difficulty with my ancient CG5. It was being cranky, and didn't want to settle down on the Dec axis after a dither. So GUESS WHAT I DID!
Image
Good lord, Ivo...I bet I'd have spent WEEKS trying to sort that out if you hadn't described it just as you did. Thanks!
You're totally right though - I tend to push the data in the face of noise - mostly using the Life module and a mask - where other people (probably rightfully) back off. Perhaps too much for some. I don't use the Life module at all when there is ample signal for the object to 'stand on its own two feet' so to speak, unless the image is too 'flat' or there is a buasy star field obscuring he object of my interest. As for colors, I always like to show the full range of star temperatures.
That is, I think, why this hobby can be so intriguing sometimes, besides just the fact we're imaging some pretty cool stuff.

There's such a hard core geeky technical side to a lot of it. The physics, geometry, electronics, and so on...it all appeals to my very black/white/on/off/1/0 nature. I do databases for a living, and fly airplanes for fun. In my world, I tend not to think in terms of "eh, about close enough kinda sorta." Either a value does or doesn't satisfy a where clause...either the plane is or isn't flying. There's not a whole lot of room to be guessing. :)

But then it comes time to process...and it's SOOOOOOOOOOOOO much about personal taste...what "looks right". I think that's why I so often refer over on /r/astrophotography to a goal of acquiring "enjoyable" images rather than "good or bad" ones. We can lose so much of what this OUGHT to be about (imo anyway) by chasing down who did a "better" job of getting this color or that just right.

Yesterday, my wife found a picture on facebook of a soldier holding a rabbit. The comments were a mine field of vitriol and argument and debate about the "meaning" of the image and so on and so forth...and finally, someone commented "Hey everyone...just shut up and enjoy the bunny."

Yeah...pretty much. :)
So a couple of days ago, I started adding a feature to allow people to use alternative ways of bringing color into their images, for example retaining even illumination but emulating human visual response to color and brightness (leading to less saturated highlights, leading to the image looking like 'you are there' with super human eyes), or using the luminance stretching history to manipulate highlight saturation (leading to the look you get from most other software). The differences can be quite dramatic. Stay tuned!
That should be an EXTREMELY powerful tool.
Perhaps it's time to explore giving StarTools its own integration module? ;)
Perhaps for 1.5 - it needs to be done right though.
Well, to be sure...we know you don't do anything until it's done right. :)

But I sure as heck think there's room for a piece of software that can manage to do 2 things:

1) Stack data and not completely ruin it
2) Do it without requiring a masters degree in IS and a 73 click 417 step process. :roll:
-- Gordon

Image

My Astrobin
User avatar
admin
Site Admin
Posts: 3382
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Guess I'm getting rusty :)

Post by admin »

gboulton wrote:I'm absolutely blown away with PI's DBE tool. It's amazing.
If it's not too much trouble, I'd love to see where DBE does a better job than Wipe (always looking to improve! :D )
But the thing can't correct imperfect stars without 184 lines of machine code, 732 mouse clicks, and a dead chicken swung in a counter clockwise direction (presuming you're north of the equator of course...you aussies should swing clockwise, obviously).
We swing live kangaroos here, much better results (if you can catch one), plus you can reuse them. Clockwise obviously.
Good lord, Ivo...I bet I'd have spent WEEKS trying to sort that out if you hadn't described it just as you did. Thanks!
Hehehe no problem :thumbsup:
You're totally right though - I tend to push the data in the face of noise - mostly using the Life module and a mask - where other people (probably rightfully) back off. Perhaps too much for some. I don't use the Life module at all when there is ample signal for the object to 'stand on its own two feet' so to speak, unless the image is too 'flat' or there is a buasy star field obscuring he object of my interest. As for colors, I always like to show the full range of star temperatures.
That is, I think, why this hobby can be so intriguing sometimes, besides just the fact we're imaging some pretty cool stuff.
There's such a hard core geeky technical side to a lot of it. The physics, geometry, electronics, and so on...it all appeals to my very black/white/on/off/1/0 nature. I do databases for a living, and fly airplanes for fun. In my world, I tend not to think in terms of "eh, about close enough kinda sorta." Either a value does or doesn't satisfy a where clause...either the plane is or isn't flying. There's not a whole lot of room to be guessing. :)
But then it comes time to process...and it's SOOOOOOOOOOOOO much about personal taste...what "looks right". I think that's why I so often refer over on /r/astrophotography to a goal of acquiring "enjoyable" images rather than "good or bad" ones. We can lose so much of what this OUGHT to be about (imo anyway) by chasing down who did a "better" job of getting this color or that just right.
Yesterday, my wife found a picture on facebook of a soldier holding a rabbit. The comments were a mine field of vitriol and argument and debate about the "meaning" of the image and so on and so forth...and finally, someone commented "Hey everyone...just shut up and enjoy the bunny."
Rogelio Andreo said it best; 'there are as many schools of astrophotography as there are astrophotographers'. Like you, I have little time for people that claim to know what the 'right' way to present an image is. The only distinction I make is this; there a the difference between someone who purposefully creates an image and is a master of the tools versus someone who mindlessly applies filters and cannot command the tools at his/her disposal effectively. I cannot and will not argue with any image that was created intentionally with a specific look and artistic vision in mind, as doing so would otherwise simply be insulting. I can and will argue with an image that was created by applying procedures in a destructive manner (ex. clipping, decon after stretching, etc.), or by *not* applying procedures, simply because you don't know about them (not applying procedures because you don't like them is of course a whole other matter!).
Ivo Jager
StarTools creator and astronomy enthusiast
Post Reply