Mac 1.9.542alpha: Crash when Sharp Module is loaded

[cytan299]

Hi Ivo,
I want to report a crash on StarTools 1.9.542alpha when I load the Sharp Module. I've attached the crash log from Mac and the StarTools.log file in the crash_log.zip file.
I can send more info if you need it, like the file that I've been processing.

Thanks!

cytan

[admin]

Hi Cytan,

Thank you for reporting this, as well as for attaching the very helpful logs!

It appears the crash happens in the graphics driver, which may be system-specific.
Are you able to reproduce this 100% of the time? If so, does the problem happen when first binning the dataset?

I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements...

[cytan299]

Hi Cytan,

Thank you for reporting this, as well as for attaching the very helpful logs!

It appears the crash happens in the graphics driver, which may be system-specific.
Are you able to reproduce this 100% of the time? If so, does the problem happen when first binning the dataset?

I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements...

Yes, ST 1.9.542alpha crashes immediately when I load Sharp Module. It did this twice and ST1.9 crashes at the same point when loading the Sharp Module, that's the reason why I posted this report. For comparison, ST 1.8.527MR2 does not crash when I do approximately the same sequence of processing actions.

cytan

[admin]

Hi Cytan,

Thank you for reporting this, as well as for attaching the very helpful logs!

It appears the crash happens in the graphics driver, which may be system-specific.
Are you able to reproduce this 100% of the time? If so, does the problem happen when first binning the dataset?

I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements...

Yes, ST 1.9.542alpha crashes immediately when I load Sharp Module. It did this twice and ST1.9 crashes at the same point when loading the Sharp Module, that's the reason why I posted this report. For comparison, ST 1.8.527MR2 does not crash when I do approximately the same sequence of processing actions.

cytan

Thank you. Does the problem happen when first binning the dataset?

[admin]

I just uploaded 1.9.544. It was missing a text string for an empty mask condition. Does this solve your issue?

[cytan299]

I just uploaded 1.9.544. It was missing a text string for an empty mask condition. Does this solve your issue?

1.9.544 also crashes when I start the Sharp Module. See attached zip file.

I don't bin, so I'm not sure what you mean in the previous message.

cytan

[admin]

I don't bin, so I'm not sure what you mean in the previous message.

In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.

[cytan299]

I don't bin, so I'm not sure what you mean in the previous message.

In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.

I did what you suggested and did a bin 50%. Now, no crash in the Sharp Module!

However. what I don't understand is that the same 25 MB file with approximately the same processing does not crash at all in 1.8.527MR2. So, the Sharp Module processing has been changed in 1.9.544alpha?

cytan

[admin]

I don't bin, so I'm not sure what you mean in the previous message.

In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.

I did what you suggested and did a bin 50%. Now, no crash in the Sharp Module!

However. what I don't understand is that the same 25 MB file with approximately the same processing does not crash at all in 1.8.527MR2. So, the Sharp Module processing has been changed in 1.9.544alpha?

cytan

Many thanks - this is an important datapoint!

Indeed, the engine code changes all the time. There are things like small performance tweaks/improvements by keeping data that is going to be re-used resident in GPU memory, etc.
25MP becomes quite big in memory; at worst, it can becomes 25MP * 64-bit values * 4 channels * 5 (L, R, G, B, NBAccent) datasets = ~4GB. You would typically keep multiple copies of that 4GB around for the algorithms (source, destination, modulator, etc.). Of course, I try to be frugal and not use multiple channels or datasets where I don't have to, so 4GB doesn't really happen. Math on the GPU happens in 32-bit rather than 64-bit so that saves some data too (it does however mean a "copy" of 64-bit translated to 32-bit is needed, so that's extra memory again, etc.).

There is a *lot* of data being shifted around and there is always a tradeoff between processing speed and memory requirements...

[cytan299]

In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.

I did what you suggested and did a bin 50%. Now, no crash in the Sharp Module!

However. what I don't understand is that the same 25 MB file with approximately the same processing does not crash at all in 1.8.527MR2. So, the Sharp Module processing has been changed in 1.9.544alpha?

cytan

Many thanks - this is an important datapoint!

Indeed, the engine code changes all the time. There are things like small performance tweaks/improvements by keeping data that is going to be re-used resident in GPU memory, etc.
25MP becomes quite big in memory; at worst, it can becomes 25MP * 64-bit values * 4 channels * 5 (L, R, G, B, NBAccent) datasets = ~4GB. You would typically keep multiple copies of that 4GB around for the algorithms (source, destination, modulator, etc.). Of course, I try to be frugal and not use multiple channels or datasets where I don't have to, so 4GB doesn't really happen. Math on the GPU happens in 32-bit rather than 64-bit so that saves some data too (it does however mean a "copy" of 64-bit translated to 32-bit is needed, so that's extra memory again, etc.).

There is a *lot* of data being shifted around and there is always a tradeoff between processing speed and memory requirements...

I'm glad that my datapoint helped. I hope this will be fixed in the later versions of ST1.9.

Just for completeness, the crash happens right at the start of Sharp Module when it is calculating the scales.

cytan