StarTools 1.8.518 Beta 1 now available

[Russ.Carpenter]

I'm respectful of all the hard work represented by this thread, but I thought it might be interesting to add some context. I use a six-year old iMac, which has 32 GB of RAM and a 4 GHz Quad-Core Intel Core i7 processor. I've used many iterations of version 1.8 over a very wide range of projects. The computer has never crashed or frozen, or otherwise shown anomalous behavior. I imagine I missed some small things, but the upgraded tools in version 1.8 just keep working.

I've also tried Version 1.8 on my little MacBook Air, which uses the remarkable M1 ARM processor. Same story--no issues.

Russ

[Mike in Rancho]

I tried today, solely with the intent of causing a crash.

516 behaves very well on my desktop. Win 10, i7-6700, 32GB, overclocked GTX 745. Binned to about 3K wide and got myself to SVD. Then I clicked and unclicked sample stars like a crazed madman (even no good crappy stars, I didn't care), but couldn't trip it up.

Then I tried the same thing on my laptop. Win 10, i7-8550U, 32GB, 940MX. The clicks and unclicks proved too much in very short order, and ST gagged and then crashed (vanished).

Oh and both are running the OS and ST from an Nvme SSD. I know ST has all those tracking files at work in the folder, but uncertain how much if any access is happening during SVD.

Will have to keep open some kind of monitoring app to see if it can be determined what might be under duress here, and why the two systems would have such different results.

EDIT: Granted, what I did is in no way normal usage. If I am reasonable with star selection on the laptop then it handles things fine, as long as I don't either click or unclick in succession too quickly. Just looking for good green cores keeps things slow enough. But it's the "oops I don't want that one" sudden unclick of a just-clicked star that can do it -- and you can tell from the little clock-thing that ST is struggling. Or maybe more accurately, the laptop is struggling.

[admin]

With regards to the SVDecon crashing thing, keep an eye on total memory usage and make sure your system can allocate virtual memory at will.

Beyond requiring memory to store data, StarTools will also stress your OS's memory manager's ability to keep heap fragmentation in check. If doing a lot of work, this can temporarily also increase memory usage and will likely require use of virtual memory.

Clicking on a sample, kicks of a huge amount of operations that all require memory (re)allocation...

[Mike in Rancho]

I pushed it again (on the laptop, I don't think I can crash the desktop) with some monitors running, and didn't really see anything being stressed out when I crashed it.

Hard memory is 32GB, and maybe usage got up to somewhere around 9 when I was in SVD? The virtual memory / page file thingy was also another available 32GB itself, so it was saying I had 64 to play with.

So I don't think I was out of resources that way. Maybe some bottleneck somewhere that doesn't show in the monitors?

I can almost tell though when I am going to be able to crash it. Things start lagging a bit, CPU gets high but not maxed out (GPU seems relaxed), and ST becomes slow to respond to (or even misses) a star click. That's when I know if I throw just a couple more clicks I can make it go boom. If I held off, it would recover.

Is there a "buffer" of some type that is holding the star clicks in a queue so ST can catch up and apply them?

[admin]

I pushed it again (on the laptop, I don't think I can crash the desktop) with some monitors running, and didn't really see anything being stressed out when I crashed it.

Hard memory is 32GB, and maybe usage got up to somewhere around 9 when I was in SVD? The virtual memory / page file thingy was also another available 32GB itself, so it was saying I had 64 to play with.

So I don't think I was out of resources that way. Maybe some bottleneck somewhere that doesn't show in the monitors?

I can almost tell though when I am going to be able to crash it. Things start lagging a bit, CPU gets high but not maxed out (GPU seems relaxed), and ST becomes slow to respond to (or even misses) a star click. That's when I know if I throw just a couple more clicks I can make it go boom. If I held off, it would recover.

Is there a "buffer" of some type that is holding the star clicks in a queue so ST can catch up and apply them?

Interesting... It definitely sounds like a similar issue to Paul's (which now appears to be resolved with the virtual memory tweaks). E.g. it sounds like a heap fragmentation issue, with the OS frantically trying to clean up / consolidate fragmented memory, so that it can be made available again to applications.

If a hard limit is needed, rule of thumb is to have ~3x the amount of physical RAM available as virtual memory (e.g. 96GB in your case, not 32GB).

Short of writing my own memory manager, it can be a tricky one to solve. I can see if I can reduce the amount of memory (re)allocations...

[admin]

The first beta of 1.8 has now landed.

* Changed parameter display in HDR module to include area size
* Reduced number of memory (re)allocations in SVDecon
* Added warning message to Color module if Wipe has not been used

Have a great weekend all!

[Mike in Rancho]

Interesting... It definitely sounds like a similar issue to Paul's (which now appears to be resolved with the virtual memory tweaks). E.g. it sounds like a heap fragmentation issue, with the OS frantically trying to clean up / consolidate fragmented memory, so that it can be made available again to applications.

If a hard limit is needed, rule of thumb is to have ~3x the amount of physical RAM available as virtual memory (e.g. 96GB in your case, not 32GB).

Short of writing my own memory manager, it can be a tricky one to solve. I can see if I can reduce the amount of memory (re)allocations...

Thanks Ivo. I don't think there's any need for rewriting at the moment, unless both of the folks who were having real troubles need it (and as you say, one appears to be resolved now).

For one, I was intentionally overdoing it just to stress the system+ST and see what happens.

Though I am curious why, other than it being a little slower system, I can do it on the laptop but not the desktop.

Both systems have Windows 10 managing the pagefile automatically. In advanced settings, the "recommended" size is about 5GB for each. And that is what it is currently set at by Windows on the desktop. (Both machines have 32GB RAM). So it seems odd that the laptop, also the exact same Windows-managed virtual memory, with the same 5GB recommended, has a pagefile size of 32GB.

Maybe too much to "manage" for the little mobile processor? Anyway both are on a 1TB SSD with plenty of room, so not sure what the issue would be.

I don't think I have anything running or background-running that should result in such a disparity. But maybe I need to look closer. Might be something going on behind the scenes on the laptop that is dragging it down.

Is there much paging done anyway? With ST going and me clicking away being silly in SVD, the RAM itself only got up to maybe 45% used, so I still had many GB of RAM still on tap.

[Mike in Rancho]

The first beta of 1.8 has now landed.

* Changed parameter display in HDR module to include area size
* Reduced number of memory (re)allocations in SVDecon
* Added warning message to Color module if Wipe has not been used

Have a great weekend all!

Thanks Ivo!

These Thursday night weekends always throw me off.

[admin]

These Thursday night weekends always throw me off.

Living in the future also means I get to go back to work sooner...

Though I am curious why, other than it being a little slower system, I can do it on the laptop but not the desktop.

Both systems have Windows 10 managing the pagefile automatically. In advanced settings, the "recommended" size is about 5GB for each. And that is what it is currently set at by Windows on the desktop. (Both machines have 32GB RAM). So it seems odd that the laptop, also the exact same Windows-managed virtual memory, with the same 5GB recommended, has a pagefile size of 32GB.

Maybe too much to "manage" for the little mobile processor? Anyway both are on a 1TB SSD with plenty of room, so not sure what the issue would be.

I don't think I have anything running or background-running that should result in such a disparity. But maybe I need to look closer. Might be something going on behind the scenes on the laptop that is dragging it down.

Is there much paging done anyway? With ST going and me clicking away being silly in SVD, the RAM itself only got up to maybe 45% used, so I still had many GB of RAM still on tap.

It actually nicely fits with the memory fragmentation theory. As memory gets allocated and freed, that nice contiguous chunk of free memory that your computer starts off with, gets full of alternating, disparate areas that are allocated and not allocated (do you remember defragging your harddrive? same sort of thing).

When an application requests a memory allocation, the OS has to find a contiguous area that is big enough and "re-use" it. As free memory gets more fragmented, it can become difficult to find such areas at all. The "easy" solution is to swap idle allocated memory to virtual memory and re-use its memory. Another strategy is to compact and re-arrange pages of memory. All this takes CPU power and resources. A fast CPU with more cores can stay on top of compaction etc. easier than a slower CPU with fewer cores.

The gaps/holes between regions of allocated memory can not really be counted as "used" memory, even though it may be technically unusable. So depending on how bad the fragmentation is, the OS may still struggle to find a chunk of contiguous free memory, even though there is technically plenty available - it's just fragmented.

Hope this helps!

[Mike in Rancho]

Living in the future also means I get to go back to work sooner...


It actually nicely fits with the memory fragmentation theory. As memory gets allocated and freed, that nice contiguous chunk of free memory that your computer starts off with, gets full of alternating, disparate areas that are allocated and not allocated (do you remember defragging your harddrive? same sort of thing).

When an application requests a memory allocation, the OS has to find a contiguous area that is big enough and "re-use" it. As free memory gets more fragmented, it can become difficult to find such areas at all. The "easy" solution is to swap idle allocated memory to virtual memory and re-use its memory. Another strategy is to compact and re-arrange pages of memory. All this takes CPU power and resources. A fast CPU with more cores can stay on top of compaction etc. easier than a slower CPU with fewer cores.

The gaps/holes between regions of allocated memory can not really be counted as "used" memory, even though it may be technically unusable. So depending on how bad the fragmentation is, the OS may still struggle to find a chunk of contiguous free memory, even though there is technically plenty available - it's just fragmented.

Hope this helps!

Thanks Ivo, it does.

Yeah always back to the future on the other side of the line down there, even though really your "night" only comes slightly after mine.

But...ST isn't your real job? Even more impressive.

I was never much of a defragger, but I sure had to fix computers after my dad decided he needed to defrag, or otherwise "clean" things up on his.

Makes sense then that the laptop can't keep up as well as the desktop. The 6700 CPU is just better and faster than the 8550U. Same cores/threads, even cache, but even beyond faster speed it likely transfers faster, and I haven't even looked yet to compare the memory sticks or the mobo/chipsets.

On a couple quick tests the new beta SVD does seem more stable on the laptop if I do early hyper star clicking. I can still crash ST on both machines in SVD, but again it's nowhere close to proper and real life usage. The "danger zone" seems to be after letting SVD get a little bit further in processing before doing a crazy click sequence, like 1/4 to 1/2 the way around the clock.

Hopefully it's all good and stable for those who were having troubles while properly using ST.