Crash while loading in compose mode

[ldipenti]

Hello Ivo,

I've been trying to process a fairly large image that I assembled as a mosaic of 5x5 panels using the ASI533MC-Pro camera, so I think the result is around 200mpx, square format. 1.8.527MR2 MacOS version.
When I try to load the L layer in compose mode (~780MB image grayscale) it seems to load it correctly but then the spinner start moving and the app quits.

I then tried to load the OSC data (~2.4GB, always in compose mode) and the same happens, but then I tried to load the Lum layer in normal mode to see if it could handle such big files and it loaded it without issues.

I then copied the exact ST dir to my external disk and booted up Windows 10 in the same machine, using Bootcamp. I was able to load all the layers in compose mode and even do some processing before the app quitted, so I guess there's a MacOS specific issue somewhere.

My computer is an 2017 iMac Retina 5K with a Radeon Pro 580 8 GB GPU and 32 GB of RAM.

As an example I've uploaded the Lum layer (783MB) to the following link:

https://siasky.net/IAC6ZTY1cbk83loOyuI3 ... eb59jeMrZg

...but honestly I don't think it is an issue of my specific data files.

Bonus question: Is processing such large images a good idea in StarTools? I might need some other workflow.

Thanks!
Lucas.

[admin]

Hi Lucas,

Thank you for your message.

There may be a number of limiting factors when it comes to large datasets.
If swap space is sufficient, the next most likely culprit/limitation is the GPU driver.
This dataset is 211MP in size. A typical dataset @ 32-bit per pixel = ~844MB per copy, or 2.5GB per RGB copy. Many algorithms require multiple copies. Not only does this quickly fill up your GPU's memory, some GPU drivers and/or operating systems have hard (or sometimes soft) limits on how much memory can be allocated at once. I'm suspecting you may be running into such a limit.

You may be able to get things working on macOS by forcing ST to use the CPU instead of the GPU.

Bonus question: Is processing such large images a good idea in StarTools? I might need some other workflow.

This, really, is the million dollar questions. It is usually not a good idea in any software package. This is particularly so in ST, as you are leaving a lot of improved detail on the table.

Even binned to quarter resolution (e.g. bin 50%) your dataset remains oversampled;

Selection_742.jpg (29.18 KiB) Viewed 17855 times

E.g. above, you can see that, even at 50% bin, it still requires multiple pixels to describe - what should be - a single unit of detail; a non-overexposing star is a perfect sample of the smallest unit of detail in your dataset.
Meanwhile, I can see correlated noise in your dataset, even at 50% bin. E.g. noise is not single-pixel random, but tends to bleed into neighbouring pixel (zig-zag / zipper / vertical / horizontal / worm-like "detail" smaller than stars but bigger than one pixel).

To make a long story short, your dataset has an incredibly high resolution, but incredibly low detail and signal quality. If you cannot trade resolution for signal quality during acquisition or stacking, then binning your dataset in StarTools is your next best option.

Making use of the improved signal, will allow you to bring out more detail (for example using deconvolution), which, if you really wish, you can upscale to arrive at the original 211MP resolution (but with far more resolved detail!).

To make a long story short, unless undersampled, it's never about the X*Y resolution; it's all about the resolved detail.

I hope this helps and makes sense!

[ldipenti]

Hi Ivo, sorry for the late reply.

Thanks so much for the detailed response! I'll have all these wisdom tips in my mind the next time I process an image.

So trying to confirm if I understood correctly, my best bet with this dataset is to bin it to a % that gives me at least non-correlated random noise and starts sampled just right.
I could do this with the Windows ST, save it as a linear TIFF and then continue processing on MacOS, correct? Then I could try to upscale it to my needs. (I was trying to get a printable image of a good enough size for hanging on a wall)

Going back to the oversampled issue: the Astronomy Tools Website's CCD suitability page tells me that the C5 without reducer + an ASI533MC-Pro is slightly oversampled with OK seeing, so it's surprising to me that even binning at 50% still show oversampled stars, what do think could be my acquisition errors here? I don't have a really good tracking mount (just a AZ-GTi in EQ mode, but RMS have been acceptable, or so I thought: 0.8 to 1.5)

Thanks again!

[happy-kat]

Have you thought of processing each panel separately and then building the mosaic last?

[ldipenti]

I haven't thought about it and never seen mentioned. I use AstroPixelProcessor for stacking & stitching, have you had experience with that kind of workflow? I guess it would be very difficult to get the coloring seamless between panels if processing each one separately.

[happy-kat]

I used the startools log to process each panel the same way, the mosaic i built joined up ok and looked the same across all panels. I stacked each panel in DSS separately. The mosaic was a wide field star field with M42

[ldipenti]

What a great idea, thank you! I didn't give any thought to the log, I'll give it a go. In the meantime, do you have a detailed writeup on your experience? I'm curious about how consistent were the panel exposure times between every panel. It's my first serious attempt and even though I tried to shoot the same nr of subs per panel per night, sometimes it was just not possible.

[ldipenti]

Followup that Ivo might find useful:

On Windows I used the compose mode and combined the Mono narrowband stack with the color stack in "L, RGB" mode.
Then, I applied AutoDev so that I could see the preliminary results, rotate a couple of degrees to optimize my future crop and binned to 50%.
Then, I clicked on "Restore" and "Original" and saved the TIFF for further processing.
Then, I loaded the TIFF file with "Open..." as Linear and went my usual way... only to find out later that the color information wasn't there anymore. Is this expected? I thought that restoring to original only kept the binned & rotation.

Regarding my first report: On MacOS even though I couldn't load the Narroband mono stack (~800MB) in Compose mode as the L layer because it crashed as soon as I tried it, I was able to open it with the "Open" button. So maybe there's room for optimization there.

I'll try to bin both stacks (L & RGB) separately and then loading them up in Compose mode.

Regards,
Lucas.

[admin]

So trying to confirm if I understood correctly, my best bet with this dataset is to bin it to a % that gives me at least non-correlated random noise and starts sampled just right.

That's one possibility, sure. You might also be able to instruct your stacker to bin the data before stacking.This should make stacking a lot quicker, while possibly aiding the stacker with more precise alignment as well.

I could do this with the Windows ST, save it as a linear TIFF and then continue processing on MacOS, correct? Then I could try to upscale it to my needs. (I was trying to get a printable image of a good enough size for hanging on a wall)

Correct!

Going back to the oversampled issue: the Astronomy Tools Website's CCD suitability page tells me that the C5 without reducer + an ASI533MC-Pro is slightly oversampled with OK seeing, so it's surprising to me that even binning at 50% still show oversampled stars, what do think could be my acquisition errors here? I don't have a really good tracking mount (just a AZ-GTi in EQ mode, but RMS have been acceptable, or so I thought: 0.8 to 1.5)

It could be many things in your optical train that contribute to further diffraction or blurring; seeing could have been poor or variable, or focus could have slightly off. The stacker may have had trouble stacking stars that varied in shape due to poor tracking or poor filed flatyness, etc. Stars aren't quite round;

Carina_Mosaic_Narrowband_Mono-reg.jpg (97.88 KiB) Viewed 17554 times

Followup that Ivo might find useful:

On Windows I used the compose mode and combined the Mono narrowband stack with the color stack in "L, RGB" mode.
Then, I applied AutoDev so that I could see the preliminary results, rotate a couple of degrees to optimize my future crop and binned to 50%.
Then, I clicked on "Restore" and "Original" and saved the TIFF for further processing.
Then, I loaded the TIFF file with "Open..." as Linear and went my usual way... only to find out later that the color information wasn't there anymore. Is this expected? I thought that restoring to original only kept the binned & rotation.

This is expected behaviour indeed. Saving in StarTools only saves the image-as-you-see-it. It does not save the Tracking data, nor the separation (and processed parallel datastreams) of luminance, chrominance and narrowband accents.

Regarding my first report: On MacOS even though I couldn't load the Narroband mono stack (~800MB) in Compose mode as the L layer because it crashed as soon as I tried it, I was able to open it with the "Open" button. So maybe there's room for optimization there.

This lends weight to the theory of your GPU running out of memory. Did you try without GPU acceleration?

[ldipenti]

I could do this with the Windows ST, save it as a linear TIFF and then continue processing on MacOS, correct? Then I could try to upscale it to my needs. (I was trying to get a printable image of a good enough size for hanging on a wall)

Correct!

Followup that Ivo might find useful:

On Windows I used the compose mode and combined the Mono narrowband stack with the color stack in "L, RGB" mode.
Then, I applied AutoDev so that I could see the preliminary results, rotate a couple of degrees to optimize my future crop and binned to 50%.
Then, I clicked on "Restore" and "Original" and saved the TIFF for further processing.
Then, I loaded the TIFF file with "Open..." as Linear and went my usual way... only to find out later that the color information wasn't there anymore. Is this expected? I thought that restoring to original only kept the binned & rotation.

This is expected behaviour indeed. Saving in StarTools only saves the image-as-you-see-it. It does not save the Tracking data, nor the separation (and processed parallel datastreams) of luminance, chrominance and narrowband accents.

Ok, so I'm a little confused here. If I should be binning & saving the different stacks before attempting to compose them, how should I do it if saving as TIFF doesn't retain all the information? Is it doable with ST?

This lends weight to the theory of your GPU running out of memory. Did you try without GPU acceleration?

Haven't tried it yet on MacOS, will report back.

Thanks for your help so far!