Memory management

[jimmy101]

Hi Ivo,

I have noticed when using Drizzeled 2x2 .fits files with a size around 125MB startools uses a hell of alot of memory. Now this is equally a great thing, but it will crash if memory+swap is exceeded.

I'm using 64bit Startools linux using a 12core CPU, 16GB RAM and still need around a 12GB swap partition to ensure it wont crash. No problems if I do this at all.

BTW this happens on all bleeding edge plus the latest one you sent link to. I'm unsure whether this is actually a bug at all (memory leak?), but its more the fact it crashes that could be a pain for people.

Maybe it uses all this memory because of it being able to "time travel".

Screenshot attached which is the memory used after crop and doing first wipe with autodev enabled - Hope it helps.

James

[admin]

Hi James,

StarTools is indeed an unapologetic memory hog due to the Tracking feature. A maximum of 28GB RAM can really be quite limiting when working with very large datasets. 12GB is also quite small for a 16GB system (for modern systems rule of thumb is having a swap space that is at least equal to your physical RAM amount).
The Tracking functionality uses spades of disk space too, but has its own swapping mechanism to prevent taking up system RAM or swap space.

What resolutions are you processing at? Do you need to process at such resolutions? Do you drizzle because otherwise your datasets are undersampled?

I'm not aware of any memory leaks (I do test regularly with processing multiple datasets in sequence without closing the app), but I'll keep an eye out for any unruly behaviour!

[jimmy101]

Thanks. Good to know it's using RAM for tracking. I gave away swap space some time ago with larger RAM these days. Will just up it then.

I drizzel as I'm using a RASA8 that undersamples with larger pixel cameras. Otherwise I don't have this issue.

[admin]

Understood. Let me know. I love the RASA design/line of scopes and what it allows "regular" people to do!