StarTools

1.7.401a mask function
I often create lasso masks and am finding that with 1.7 it is often squaring off the wibbly edge I drew, even if first it looks ok when switch to invert the squaring will occur.
draw an irregular shape on it's closure the mask is infilled and many of the edges are squared off.

in case my post got missed as it is a bug in 1.7.401 used on W10 Pro 64bit

Apologies for the delay in getting back to you on this.
Actually, this behaviour has always been the case from version 1.0!

The idea is that the tool only draws convex shapes. That is because most objects for which the lasso tool is used (stars, galaxies, globs, planetary nebula, dust donuts etc.) tend to be are spherical or elleptical in nature. This is almost always to due the nature of the physics governing the appearance of such objects (gravity, emissions, explosions, Airy disc/diffraction etc.).

Cases of contiguous masks with concave features are quite rare, unless you engage in truly arbitrary mask creation, which, though possible, is usually not appropriate if documentary photography is the goal.

I hope this helps & makse sense!