SVDecon crashes on 1.8.512alpha

[PaulE54]

The non-GPU version does not seem to suffer the same crashes in SVDecon, but it is so painfully slow. When crashes occur, I do have to reboot the machine, not just close and re-open StarTools. I will upgrade to 516 alpha and see if this avoids the problem.

Paul

[admin]

The non-GPU version does not seem to suffer the same crashes in SVDecon, but it is so painfully slow. When crashes occur, I do have to reboot the machine, not just close and re-open StarTools. I will upgrade to 516 alpha and see if this avoids the problem.

Paul

Hmmm... requiring an entire reboot for things to "reset" themselves sounds like a wider system issue

Deconvolution in ST is probably the single most taxing thing I have seen a system do. It stresses all components (and drivers) to the max.

@PaulE54 what are the specs of your system (CPU, GPU, OS, etc.) and how old is it?

[PaulE54]

Hi Ivo
It's a desktop - around 5 years old. It has intel I5-6500 cpu, 16GB RAM, up to date Windows 10/64, OS lives on Samsung SSDs (although images typically live on a NAS accessed via Gig ethernet). GPU is a GeForce GTX960.
Any obvious weak spots - or settings I should tune?
I can attach recent startools.log file if any use for debugging.
Had a quick look and can't see anything obvious.

Thanks

Paul

[admin]

A GTX 960 should definitely be able to handle ST without issues (I ran a GTX 950 for the longest time), nor exhibit the GPU timeout troubles of weaker cards. An application should never be able to take your system down (the OS should always protect you), so this is starting to look more and more like a hardware issue.

It may be a power supply that's on its way out, or your GPU that's on its way out.

You could see if you can replicate any GPU instability issues with a stress testing tool like FurMark.

To test OpenCL specifically, you could run GeekBench.

You can also try to stabilise your GPU by underclocking it using a tool like AfterBurner.

A PSU problem is much harder to detect without specialised equipment...

[PaulE54]

Thanks for this Ivo. To be clear, the problem I'm seeing is this -
1. I turn the machine off nightly. After I switch it on, the first use of StarTools works fine, all modules work OK, and I can save my finished image.
2. If I then process a second or subsequent image the same day, once I have selected around 10 stars in SVDecon, StarTools crashes - it does not crash windows. No error messages are shown, the app just terminates. This behaviour is repeatable, and the only way I have found to fix it, is to reboot the machine. I can't see anything obvious in the StarTools log.
Whenever it crashes, there are very similar entries in the Windows Event log, e.g. -
Log Name: Application
Source: Application Error
Date: 08/10/2021 11:10:55
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: WINDOWS-QPFAD9K
Description:
Faulting application name: StarTools-Windows64-GPU.exe, version: 0.0.0.0, time stamp: 0x00000000
Faulting module name: StarTools-Windows64-GPU.exe, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000003e5f0
Faulting process ID: 0x59e0
Faulting application start time: 0x01d7bc22f2739d71
Faulting application path: C:\.......\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe
Faulting module path: C:\.....\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe
Report ID: 5cea3cba-c559-4acc-91ac-95642dcd93f5
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>100</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-10-08T10:10:55.1376650Z" />
<EventRecordID>50795</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>Application</Channel>
<Computer>WINDOWS-QPFAD9K</Computer>
<Security />
</System>
<EventData>
<Data>StarTools-Windows64-GPU.exe</Data>
<Data>0.0.0.0</Data>
<Data>00000000</Data>
<Data>StarTools-Windows64-GPU.exe</Data>
<Data>0.0.0.0</Data>
<Data>00000000</Data>
<Data>c0000005</Data>
<Data>000000000003e5f0</Data>
<Data>59e0</Data>
<Data>01d7bc22f2739d71</Data>
<Data>C:\....\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe</Data>
<Data>C:\....\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe</Data>
<Data>5cea3cba-c559-4acc-91ac-95642dcd93f5</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>
All the failures have the same fault code. From a quick google this looks like a memory access issue of some sort?
Is it worth running a memory test - perhaps I have a faulty RAM module?
All but one have the same faulting module. The sole exception failed in msvcrt.dll. There are a variety of fault offset values.

I will try the GPU tests you outline.
The system was built for me, so I guess it is possible the supplier used some rare settings which have caused some sort of "edge case".

I do have a digital multimeter, so I can monitor the DC voltages out of the PSU.
Hope this is helpful, and thanks again for your detailed input, and the fantastic software!

Paul

[admin]

Thanks for this Ivo. To be clear, the problem I'm seeing is this -
1. I turn the machine off nightly. After I switch it on, the first use of StarTools works fine, all modules work OK, and I can save my finished image.
2. If I then process a second or subsequent image the same day, once I have selected around 10 stars in SVDecon, StarTools crashes - it does not crash windows. No error messages are shown, the app just terminates. This behaviour is repeatable, and the only way I have found to fix it, is to reboot the machine. I can't see anything obvious in the StarTools log.
Whenever it crashes, there are very similar entries in the Windows Event log, e.g. -
Log Name: Application
Source: Application Error
Date: 08/10/2021 11:10:55
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: WINDOWS-QPFAD9K
Description:
Faulting application name: StarTools-Windows64-GPU.exe, version: 0.0.0.0, time stamp: 0x00000000
Faulting module name: StarTools-Windows64-GPU.exe, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000003e5f0
Faulting process ID: 0x59e0
Faulting application start time: 0x01d7bc22f2739d71
Faulting application path: C:\.......\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe
Faulting module path: C:\.....\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe
Report ID: 5cea3cba-c559-4acc-91ac-95642dcd93f5
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>100</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-10-08T10:10:55.1376650Z" />
<EventRecordID>50795</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>Application</Channel>
<Computer>WINDOWS-QPFAD9K</Computer>
<Security />
</System>
<EventData>
<Data>StarTools-Windows64-GPU.exe</Data>
<Data>0.0.0.0</Data>
<Data>00000000</Data>
<Data>StarTools-Windows64-GPU.exe</Data>
<Data>0.0.0.0</Data>
<Data>00000000</Data>
<Data>c0000005</Data>
<Data>000000000003e5f0</Data>
<Data>59e0</Data>
<Data>01d7bc22f2739d71</Data>
<Data>C:\....\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe</Data>
<Data>C:\....\StarTools_1_8_516_alpha\distro\StarTools-Windows64-GPU.exe</Data>
<Data>5cea3cba-c559-4acc-91ac-95642dcd93f5</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>
All the failures have the same fault code. From a quick google this looks like a memory access issue of some sort?
Is it worth running a memory test - perhaps I have a faulty RAM module?
All but one have the same faulting module. The sole exception failed in msvcrt.dll. There are a variety of fault offset values.

I will try the GPU tests you outline.
The system was built for me, so I guess it is possible the supplier used some rare settings which have caused some sort of "edge case".

I do have a digital multimeter, so I can monitor the DC voltages out of the PSU.
Hope this is helpful, and thanks again for your detailed input, and the fantastic software!

Paul

Thanks Paul. If it is just an application-level crash, then it is a little bit less likely the hardware is at fault. If you keep StarTools running and this happens, then it may be a memory allocation issue. If the app just "vanishes" without the OS complaining or errors, then this is usually StarTools catching a non-recoverable error and exiting.

Can you confirm you have sufficient virtual memory allocated? (see here)

It would really help being able to replicate this problem... Is there a dataset that consistently exhibits this issue?
Does the GPU or RAM usage slowly creep up? (see here on how to tell GPU memory usage)

Thank you,

[PaulE54]

Hi Ivo.
My virtual memory size is 2432MB. I don't know what might be a reasonable / required value.
The problem occurs with any of my datasets.

My normal working pattern is that I close StarTools after I've finished one image and then re-open it for the next image. Just a habit I've got into, don't know if this is right or wrong. The issue appears when running "re-opened" StarTools.
What I'm going to try now is keeping StarTools open after I finish with one image, save that image, don't close StarTools, and then open the next image and process that.
I'll also keep task manager open and see what the consumption of resources (CPU, GPU, RAM) looks like.

I'll report back when I know more.

Paul

[PaulE54]

Right, think I'm getting somewhere now. I processed a first image, then processed a second image without closing StarTools. During processing the second image I was interrupted twice, so had to step away from the machine for 10-15 mins and then resume working. Task Manager shows that SVDecon on my machine is using a LOT of RAM. - StarTools alone was using 11.2GB during the second SVDecon run, and with other processes also running, that meant 91% of my 16GB RAM was used.

In both cases when I took a break, RAM usage was high, and by the time I got back and resumed work, StarTools RAM usage had dropped back to around 1.3GB. And nothing crashed.

So this is starting to feel like a RAM utilisation issue of some sort. Paging file is on an SSD BTW.

Paul

[admin]

My virtual memory size is 2432MB. I don't know what might be a reasonable / required value.

That may very well be the culprit!
Only having a maximum of 2GB extra on top of 16GB is very, very little, giving your system a maximum of 18GB to work with for everything that is running.
If you absolutely must have a fixed size, then a good rule of thumb is to have that value at 2x your RAM (e.g. 32GB) or more.
Ideally, however, you would have it set to your OS' equivalent of "unlimited" (e.g. dynamically allocated by the OS as needed).

[PaulE54]

Thanks Ivo! I will try setting a higher value (plenty of room on the SSD) and report back.

Paul