The mask issue should now hopefully resolved in 1.7.434 (please be careful with selective processing!)
1.7.434 implements some possible fixes. Is it behaving any better?
Not to be alarmist, but a complete shut down by the PC can be a symptom of hardware failure.
An application error should never be able to shut down or reboot the system, unless hardware failed to operate correctly.
Causes for this behavior may include overheating of the CPU or GPU (check your thermal interfaces and cooling solution), insufficient or out-of-spec power dellivery to the CPU or GPU (check your PSU), or an unstable overclock of the CPU or GPU. StarTools severely stresses your hardware and will quickly lay bare any such problems where other applications will not.