1.5.369 crashes on wipe

[richbandit]

Hi Ivo,

I'll do my best to explain my process! FYI, my virtual memory is set to 'automatically manage paging file size' and currently allocated size is 5120MB.

I note that the log file from my last failed attempt shows details for files loaded, compose, crop, bin, develop - then nothing else shown in the log after that? I'm presuming that's because the subsequent wipe process didn't complete successfully?

Fits files are created with APP calibration and integration, same process I've always used, nothing different. The files used on my last attempt (as in log below) had been only processed for light pollution in APP because they were very 'dirty' (hence the 'lpc' suffix shown in the file name), but I've also tried with the original integration sets (several times!) with absolutely no processing whatsoever and I still get the same result in 1.5.369 - as mentioned, same files, same process, works fine in 1.4.352.

Here's the log for my last attempt:

-----------------------------------------------------------
StarTools 1.5.369
Tue Oct 08 15:34:56 2019
-----------------------------------------------------------
Loading luminance channel data
File loaded in LRGB module [D:\IMAGES\M31 Andromeda Galaxy\Further attempt with new cal files 08.10.19\St-avg-1800.0s-MWC_1_3.0_4-x_1.0_LZ3-NS-full-qua-add-sc_BWMV_nor-AAD-RL-noMBB-Lum-lpc.fits].
Loading red channel data
File loaded in LRGB module [D:\IMAGES\M31 Andromeda Galaxy\Further attempt with new cal files 08.10.19\St-avg-4500.0s-MWC_1_3.0_4-x_1.0_LZ3-NS-full-qua-add-sc_BWMV_nor-AAD-RL-noMBB-Red-lpc.fits].
Loading green channel data
File loaded in LRGB module [D:\IMAGES\M31 Andromeda Galaxy\Further attempt with new cal files 08.10.19\St-avg-4500.0s-MWC_1_3.0_4-x_1.0_LZ3-NS-full-qua-add-sc_BWMV_nor-AAD-RL-noMBB-Green-lpc.fits].
Loading blue channel data
File loaded in LRGB module [D:\IMAGES\M31 Andromeda Galaxy\Further attempt with new cal files 08.10.19\St-avg-4500.0s-MWC_1_3.0_4-x_1.0_LZ3-NS-full-qua-add-sc_BWMV_nor-AAD-RL-noMBB-Blue-lpc.fits].
--- Compose
Parameter [Luminance, Color] set to [L, RGB]
Parameter [Color Ch. Interpolation] set to [On]
Parameter [Lum Total Exposure] set to [0h20m (20m) (1200s)]
Parameter [Blue Total Exposure] set to [1h00m (60m) (3600s)]
Parameter [Green Total Exposure] set to [1h00m (60m) (3600s)]
Parameter [Red Total Exposure] set to [1h00m (60m) (3600s)]
Image size is 5524 x 3695
Type of Data: Linear, was not Bayered, or was Bayered + white balanced
--- Crop
Parameter [X1] set to [68 pixels]
Parameter [Y1] set to [75 pixels]
Parameter [X2] set to [5463 pixels (-61)]
Parameter [Y2] set to [3619 pixels (-76)]
Image size is 5395 x 3544
--- Bin
Parameter [Scale] set to [(scale/noise reduction 70.71%)/(200.00%)/(+1.00 bits)]
Image size is 3814 x 2505
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [91.53 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [10.7 pixels]

Following on from this, this is my exact same process - copied from v 1.4.352 log because that works! - of my WIPE process leading up to the crash using 1.5.369. Not sure if this may be slightly different from the log 1.5.369 would produce, but at least it shows my process:

--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [Temporary AutoDev] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [8 pixels]
Parameter [Drop Off Point] set to [100 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [85 %]

After adjusting the aggressiveness from the default 75% to 85% it seems OK. After that I've adjusted the dark anomaly filter up to 8 pixels, and on several occasions it's crashed at that point. But on the majority of occasions it happens once I've further adjusted the aggressiveness to another setting again (higher or lower than the previous 85%) …..

Hope this helps? Would it help if I was to send you my data files for you to try? (they may not be the best, but they're all my work so I'm proud of them!!).

Many thanks for your assistance, much appreciated!

Richard

[richbandit]

Hi Ivo,

As a further update, I've just tried setting my virtual memory manually instead of letting Windows manage it automatically - set initial amount to 5120, max amount to 10250 (so double the size available) …… still crashes, exactly the same!

Oh well, that was worth a try ….

Thanks,

Richard

[richbandit]

Hi Ivo,

I have a further update, not sure if this may help or not!

I thought I'd try a completely different dataset to see what happens. Same imaging train, nothing changed, just a different target taken a couple of weeks earlier.

This time, before the programme crashed at the same place in WIPE again, I had an error window pop up (not seen this before!) stating 'Host multiplatform message - out of memory'. Upon pressing the 'OK' button on this error box, the programme closed immediately …..

So, it would seem to be 1.5.369 is using/needing loads more memory at that point for some reason - I have 8GB RAM memory installed, plus my allocated virtual memory, surely that should be enough??

Anyway, a quick Google search revealed that this error message is indeed due to Windows not having enough memory available to complete a task, and the resolution is to amend a regedit setting responsible for allowed and available memory usage. So I followed the recommendations to increase that value in regedit accordingly (after saving a back up of regedit, just in case!), re-booted PC, gave it another go ….. and still the same problem!!

So that didn't seem to do the trick either. However, it seems to have now pointed towards an over-use/requirement of memory issue being at the heart of the problem? What do you think?

Thanks,

Richard

[richbandit]

Hi Ivo,

Apologies for making so many posts in quick succession, but I thought the more info you have the better you've have a chance to try and resolve this issue!

Looks like I'm able to confirm it's definitely a memory related thing.

I've now increased my virtual memory to the max allowed for 8GB RAM, which is 24.576GB (1024 x 8 x 4). Then with 1.5.369 I've followed the same procedures as previously advised I now find that I'm able to do several more iterations of adjustments in wipe before the inevitable crash, at which time I'm now getting the aforementioned memory error message every time.

During each iteration my hard drive is working flat out, so indeed the virtual memory is required. The dark anomaly filter routine runs and completes correctly, it's the final 'wipe' process (spinning 'working' wheel in bottom right hand corner) that then takes ages with the hard disk working overtime, until the error eventually pops up.

This suggests that the requirement is for more than 32GB of memory (my installed 8GB RAM + the more than 24GB of virtual memory), which seems somewhat excessive as I'm betting not many users would have the opportunity to have that amount of straight RAM available so as to be able to avoid the virtual memory disk access requirement …

Not sure what the solution is now, so I'll await your further most welcome thoughts for where to go next!

Many thanks again,

Richard

[admin]

Many thanks for the sleuthing Richard! It is very helpful.
This sounds like a memory leak and this has given me some valuable information.
Stay tuned!

[richbandit]

Thanks Ivo, glad you found the information useful and look forward to hearing further from you once you've managed to figure out what's going on.

Thanks,

Richard

[admin]

Hi Richard,

Could you try the latest alpha for me (1.6.378)? I fixed a memory leak in Wipe. I'm hoping that this may be your issue. If it is I might backport it to 1.5 or re-base the current alpha as 1.5.

Cheers!

[richbandit]

Hi Ivo,

Well done that man, looks like you've successfully managed to 'plug the leak'!!

Everything now works just fine, hardly any (if any at all?) hard disk access now, so looks like no paging taking place or required any more.

All processes, including WIPE that alerted me to the problem, now run through without any issues.

I was going to insert my image for you to see, but couldn't figure out how to upload/include that! If you could kindly tell me how best to do that in a post I'll be happy to share my valiant attempt!

Many thanks Ivo, and please keep up the excellent work on what is without doubt a brilliant piece of astro software!

Richard

[richbandit]

Hi Ivo,

OK, my bad - I should have looked up how to post images first, in your FAQ!

Uploaded it to my Dropbox, at: https://www.dropbox.com/sh/6pcajrla4z2l ... W6uka?dl=0

Hope this link works ….

Thanks,

Richard

[admin]

Nice Image Richard! (though I spot some green stars - color balance these out, or, as a last result, use the Cap Green function).

I just released a 1.5. Maintenance Release with this fix, as well as some other back-ported fixes from 1.6 alpha. Announcement here. Thanks again for reporting this, helping me diagnose the issue and bearing with me while I issued a fix!