I am using the none GPU version and it is useable but I do need to wait quite a while though using a ROI helps for fiddling with settings.
I miss Reveal DSO Core, so I guess I will need to work out how to do it manually.
HDR module in 1.8.515 Public Alpha is extremely slow
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Last edited by happy-kat on Sun Oct 03, 2021 7:27 pm, edited 1 time in total.
-
Mike in Rancho
- Posts: 1166
- Joined: Sun Jun 20, 2021 10:05 pm
- Location: Alta Loma, CA
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Ah, the new HDR is certainly shaking things up! 
After a little testing, I have to say it is acting as advertised - the initial calculation takes time, but after that changes are nearly immediate. And remember on old HDR certain changes would also require a fairly hefty recalc.
We've become a little bit spoiled by ST-GPU methinks. Remember beforehand, when it was almost mandatory to use things like a preview box in some modules?
Brendan I believe you desktop might be okay, but the laptop could be lacking the oomph to keep up with ST as it advances. I understand though, just a couple weeks ago I was trying to extend the life of my Q9550 desktop. Well, this new HDR would have been the nail in the coffin I think, if I hadn't obtained a replacement computer.
From what I can tell so far, new HDR is very CPU dependent, and even though monitors do show GPU being used, it's not maxed out like the CPU's are. I ran some test data at it on two machines, image was 2380x1668, so a bit on the low side of the resolutions I've been processing at lately (I like to maintain a certain amount in order to avoid squaring off stars and to give SVD room to work):
On my laptop (i7-8550U, 32GB, 940MX), initial HDR took 4:46, but a context level of 25 instead of 50 only took 19.8 seconds. A change to a context of 75 took 2:21 just to get to the second little bar in the clock (so that means go mow the lawn and come back). All cores/threads were maxed out, and were being kept to a turbo speed of about 2.8Ghz. The 8550U is supposed to max turbo at about 4.0, and in certain modules it does (SVD), but not the way HDR is using it.
On my desktop (i7-6700, 32GB, GTX 745 OEM), initial HDR took 3:29, context level 25 took 17.7 seconds, and a change to context 75 took 1:39 to move one clock bar (mow half the lawn!). However, this CPU was able to run all the cores full out at 3.8GHz, which I think is the difference here.
Now, even though the GTX 745 is nothing to write home about, I used MSI Afterburner to OC it by 135Mhz on the core clock and 200Mhz on the memory clock. This does increase my benchmark by a noticeable amount on userbenchmark. However, the GPU overclock made no difference at all to my HDR times.
But, using different dataset at 2920x1945 resolution, this mild gpu overclock did shorten my SuperStructure wait time by nearly 10 seconds, from 1:43 to 1:34. Other modules, which generally don't take as long, like AutoDev, Wipe, and Denoise, were also sped up by a small amount. So it is something worth considering. MSI Afterburner is also free and is not limited to only MSI's own gpu's. Just do some Googling to find out what people have found for safe OC's of your particular gpu.
And although Ivo already noted the ancient nature of the Fermi gpu architecture, I am still of a mind to put my old N460GTX Cyclone into this tower, since it is faster from the get go than the GTX 745, and can also be seriously overclocked. I already tossed a new 700W PSU into this thing anyway, might as well use some of that capacity!
Nonetheless, from this initial testing, it seems to me that you have to look to your CPU regarding new HDR and how long it will take.
I still need to run old and new HDR side-by-side to see how the new settings and controls compare, and if there is anything we are now "missing out on" such that we would want to increase the context level and eat the longer wait times.
After a little testing, I have to say it is acting as advertised - the initial calculation takes time, but after that changes are nearly immediate. And remember on old HDR certain changes would also require a fairly hefty recalc.
We've become a little bit spoiled by ST-GPU methinks. Remember beforehand, when it was almost mandatory to use things like a preview box in some modules?
Brendan I believe you desktop might be okay, but the laptop could be lacking the oomph to keep up with ST as it advances. I understand though, just a couple weeks ago I was trying to extend the life of my Q9550 desktop. Well, this new HDR would have been the nail in the coffin I think, if I hadn't obtained a replacement computer.
From what I can tell so far, new HDR is very CPU dependent, and even though monitors do show GPU being used, it's not maxed out like the CPU's are. I ran some test data at it on two machines, image was 2380x1668, so a bit on the low side of the resolutions I've been processing at lately (I like to maintain a certain amount in order to avoid squaring off stars and to give SVD room to work):
On my laptop (i7-8550U, 32GB, 940MX), initial HDR took 4:46, but a context level of 25 instead of 50 only took 19.8 seconds. A change to a context of 75 took 2:21 just to get to the second little bar in the clock (so that means go mow the lawn and come back). All cores/threads were maxed out, and were being kept to a turbo speed of about 2.8Ghz. The 8550U is supposed to max turbo at about 4.0, and in certain modules it does (SVD), but not the way HDR is using it.
On my desktop (i7-6700, 32GB, GTX 745 OEM), initial HDR took 3:29, context level 25 took 17.7 seconds, and a change to context 75 took 1:39 to move one clock bar (mow half the lawn!). However, this CPU was able to run all the cores full out at 3.8GHz, which I think is the difference here.
Now, even though the GTX 745 is nothing to write home about, I used MSI Afterburner to OC it by 135Mhz on the core clock and 200Mhz on the memory clock. This does increase my benchmark by a noticeable amount on userbenchmark. However, the GPU overclock made no difference at all to my HDR times.
But, using different dataset at 2920x1945 resolution, this mild gpu overclock did shorten my SuperStructure wait time by nearly 10 seconds, from 1:43 to 1:34. Other modules, which generally don't take as long, like AutoDev, Wipe, and Denoise, were also sped up by a small amount. So it is something worth considering. MSI Afterburner is also free and is not limited to only MSI's own gpu's. Just do some Googling to find out what people have found for safe OC's of your particular gpu.
And although Ivo already noted the ancient nature of the Fermi gpu architecture, I am still of a mind to put my old N460GTX Cyclone into this tower, since it is faster from the get go than the GTX 745, and can also be seriously overclocked. I already tossed a new 700W PSU into this thing anyway, might as well use some of that capacity!
Nonetheless, from this initial testing, it seems to me that you have to look to your CPU regarding new HDR and how long it will take.
I still need to run old and new HDR side-by-side to see how the new settings and controls compare, and if there is anything we are now "missing out on" such that we would want to increase the context level and eat the longer wait times.
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Really appreciate the responses with tests on your own systems.
I'm fine with my laptop being out of spec, possibly my PC too, but it seems such a huge leap in processing requirement for HDR compared with any of the other modules. That's why I asked what the minimum spec would be actually to use the maximum Context setting - it seems orders of magnitude higher than anything else in StarTools. I'd also like to know what kind of machine I'd need to start thinking about, if this is the StarTools roadmap for the future.
It would be interesting to know what the 'average' spec is of StarTools users' machines, and whether they could also handle HDR to its maximum capability.
Anyway, I'm back on the previous version and I'll just keep tabs on responses here. As I say, I'm hoping some sort of compromise could be reached where a 'light' version of HDR could be accommodated. The previous one is/was ideal!
I'm fine with my laptop being out of spec, possibly my PC too, but it seems such a huge leap in processing requirement for HDR compared with any of the other modules. That's why I asked what the minimum spec would be actually to use the maximum Context setting - it seems orders of magnitude higher than anything else in StarTools. I'd also like to know what kind of machine I'd need to start thinking about, if this is the StarTools roadmap for the future.
It would be interesting to know what the 'average' spec is of StarTools users' machines, and whether they could also handle HDR to its maximum capability.
Anyway, I'm back on the previous version and I'll just keep tabs on responses here. As I say, I'm hoping some sort of compromise could be reached where a 'light' version of HDR could be accommodated. The previous one is/was ideal!
Not so much boldly going as randomly stumbling where plenty of people have been before
-
Mike in Rancho
- Posts: 1166
- Joined: Sun Jun 20, 2021 10:05 pm
- Location: Alta Loma, CA
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Well, old-HDR may have been ideal because it's the only one we knew. 
And after playing around with new-HDR, old-HDR becomes less and less ideal.
I got a chance to run side-by-side tonight with 512 and 515, and used the trusty ST Orion Tutorial data. It wasn't perfect, because like a dummy when I set up the new tower I forgot to run a monitor profile with the x-rite, but...oh well.
One thing I noticed in running each through the various presets was just how little the user settings changed in old-HDR from preset to preset, and yet, the results would change, sometimes drastically. This leads me to believe that much of old-HDR was buried in the algorithms, but with new-HDR, more power has been granted to us with these sliders. Ivo will have to let us know if I am off base here. Anyway, in new-HDR, the preset changes can actually be seen affecting all the new controls settings.
I think I am going to like new-HDR. And I don't think I'll need to run it twice, as I have often done with particular targets before, such as Reveal All followed by a Reveal Core.
And for nostalgia purposes, the new Equalize preset still makes a mess of things, just like it did before.
Anyway, for those who are balking at the initial calculation burden, I would suggest just immediately dropping the Context level to 25. Or something between 30 and 45, whatever you think you can handle for waiting it out a little bit longer. It is a little more small-detail oriented, but you can back off the appearance of that effect, if you don't like it, by adjusting the controls if you so choose. And if you do that, the first results really aren't going to appear so noticeably different than the default first results out of old-HDR. Even at level 25, and that should not take much calc time at all.
I say do a test, either in parallel or serially after saving a tiff right after old HDR, and see if you can fairly well match up the results even at a lower but faster context level. And then start trying out all the new stuff.
I have yet to test anything higher than the default 50 context to see what benefits that might provide. It would have to be pretty good though. One of these days I'll do a comparison when I can just walk away from the computer for a while, and see what exactly that does. That said, aren't we usually more interested in lowering the detail level in HDR? I don't think I ever felt the need to go up from the default 1000 in old-HDR, always down.
And after playing around with new-HDR, old-HDR becomes less and less ideal.I got a chance to run side-by-side tonight with 512 and 515, and used the trusty ST Orion Tutorial data. It wasn't perfect, because like a dummy when I set up the new tower I forgot to run a monitor profile with the x-rite, but...oh well.
One thing I noticed in running each through the various presets was just how little the user settings changed in old-HDR from preset to preset, and yet, the results would change, sometimes drastically. This leads me to believe that much of old-HDR was buried in the algorithms, but with new-HDR, more power has been granted to us with these sliders.
Ivo will have to let us know if I am off base here. Anyway, in new-HDR, the preset changes can actually be seen affecting all the new controls settings.I think I am going to like new-HDR. And I don't think I'll need to run it twice, as I have often done with particular targets before, such as Reveal All followed by a Reveal Core.
And for nostalgia purposes, the new Equalize preset still makes a mess of things, just like it did before.
Anyway, for those who are balking at the initial calculation burden, I would suggest just immediately dropping the Context level to 25. Or something between 30 and 45, whatever you think you can handle for waiting it out a little bit longer. It is a little more small-detail oriented, but you can back off the appearance of that effect, if you don't like it, by adjusting the controls if you so choose. And if you do that, the first results really aren't going to appear so noticeably different than the default first results out of old-HDR. Even at level 25, and that should not take much calc time at all.
I say do a test, either in parallel or serially after saving a tiff right after old HDR, and see if you can fairly well match up the results even at a lower but faster context level. And then start trying out all the new stuff.
I have yet to test anything higher than the default 50 context to see what benefits that might provide. It would have to be pretty good though. One of these days I'll do a comparison when I can just walk away from the computer for a while, and see what exactly that does. That said, aren't we usually more interested in lowering the detail level in HDR? I don't think I ever felt the need to go up from the default 1000 in old-HDR, always down.
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Hi all,
The new HDR module indeed is very CPU-heavy vs GPU-heavy, due to the type of calculations being preformed (lots of number sorting, which GPUs are very bad at). So GPU utilization will be somewhat minimal in places.
To mimic the old module (including its processing times), you can reduce the Context Size parameter to something much smaller. In a way it functions as a "quality" setting.
To mimic the Reveal core DSO setting, reduce Context Size, and only set Shadow Detail Boost, while leaving all other parameters (e.g. Gamma Highlight, Gamma Shadow and Highlights Detail Boost) alone. One difference though compared to 1.7, is that even at max settings, the detail recovered in 1.8 is much more carefully placed in its surroundings, and does not look artificial, while also perfectly conforming to the noise levels of its surroundings (the latter extends also covers the interaction with the Gamma Highlight/Shadow parameter - consistency is maintained).
It's easy to pick the 1.7 image;
Top left original, top right 1.7 HDR, bottom left 1.8 HDR (Shadow Detail Boost 100%, nothing else), bottom right 1.8 HDR Shadow Detail Boost at 100%, with Gamma Shadow at 100%.
(obviously using the extremes of the parameters is not exactly advisable, but this is just for demonstration purposes)
(re)design goal #1 spotted!
Design goal #3 was to offload as much as possible to a pre-calculation step, so that the rest of the module can be more responsive.
Finally design goal #4 was (as mentioned above) enhancing the quality of the result, avoiding artefacts (like the infamous "pancake look") from other methods as much as possible.
Quality always takes precedence over expediency and ST has a long history of using brute-force number crunching. That said, 20 minutes for a single image seems a bit excessive (though maybe not impossible at the highest settings). While I cannot see immediate avenues for significant optimisation, there are (almost) always ways to trade-off quality for faster processing times.
The new HDR module indeed is very CPU-heavy vs GPU-heavy, due to the type of calculations being preformed (lots of number sorting, which GPUs are very bad at). So GPU utilization will be somewhat minimal in places.
To mimic the old module (including its processing times), you can reduce the Context Size parameter to something much smaller. In a way it functions as a "quality" setting.
To mimic the Reveal core DSO setting, reduce Context Size, and only set Shadow Detail Boost, while leaving all other parameters (e.g. Gamma Highlight, Gamma Shadow and Highlights Detail Boost) alone. One difference though compared to 1.7, is that even at max settings, the detail recovered in 1.8 is much more carefully placed in its surroundings, and does not look artificial, while also perfectly conforming to the noise levels of its surroundings (the latter extends also covers the interaction with the Gamma Highlight/Shadow parameter - consistency is maintained).
It's easy to pick the 1.7 image;
-
- 4panelhdr.jpg (462.36 KiB) Viewed 6068 times
(obviously using the extremes of the parameters is not exactly advisable, but this is just for demonstration purposes)
Spot-on.Mike in Rancho wrote: ↑Mon Oct 04, 2021 9:00 am with new-HDR, more power has been granted to us with these sliders.
(re)design goal #1 spotted!
(re)design goal #2 spotted!And I don't think I'll need to run it twice, as I have often done with particular targets before, such as Reveal All followed by a Reveal Core.
Design goal #3 was to offload as much as possible to a pre-calculation step, so that the rest of the module can be more responsive.
Finally design goal #4 was (as mentioned above) enhancing the quality of the result, avoiding artefacts (like the infamous "pancake look") from other methods as much as possible.
Quality always takes precedence over expediency and ST has a long history of using brute-force number crunching. That said, 20 minutes for a single image seems a bit excessive (though maybe not impossible at the highest settings). While I cannot see immediate avenues for significant optimisation, there are (almost) always ways to trade-off quality for faster processing times.
Ivo Jager
StarTools creator and astronomy enthusiast
StarTools creator and astronomy enthusiast
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Interesting stuff, thanks Ivo.
I'll play around with this all some more and see what I can/cannot do with it.
I'll play around with this all some more and see what I can/cannot do with it.
Not so much boldly going as randomly stumbling where plenty of people have been before
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Do let me know your findings. I'm not a fan of things that regularly take 20m to run to be useful...
Ivo Jager
StarTools creator and astronomy enthusiast
StarTools creator and astronomy enthusiast
Re: HDR module in 1.8.515 Public Alpha is extremely slow
I've done some quick performance tests and the time for HDR to complete processing as related to Context Size is as follows:
So it is possible to get the processing down significantly by relatively small changes in Context Size.
Which brings me to a question about Context Size.
Since Context Size is in pixels the optimum size will depend on the original dimensions in pixels of the image and how much it has been binned as well as the subject and the detail you are trying to bring out.
Ivo, would it be helpful to have the Context Size related to something other than pixel size - to the size of the image for example?
It could perhaps be a percentage? So you could choose a Context Size of, say, up to 10% of the image?
It would mean that the same value would be used on an image regardless of binning.
It would also mean the default value would be appropriate in more cases.
Guy
Code: Select all
Context Size 50 40 30 25 20 10
%Processing time 100 50 15 6 4 2
Which brings me to a question about Context Size.
Since Context Size is in pixels the optimum size will depend on the original dimensions in pixels of the image and how much it has been binned as well as the subject and the detail you are trying to bring out.
Ivo, would it be helpful to have the Context Size related to something other than pixel size - to the size of the image for example?
It could perhaps be a percentage? So you could choose a Context Size of, say, up to 10% of the image?
It would mean that the same value would be used on an image regardless of binning.
It would also mean the default value would be appropriate in more cases.
Guy
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Just ran some quick tests on the PC (i5-4590M CPU @ 3.3GHz, Windows 10 with 8GB RAM, Intel 4600 GPU and 20 compute units).
The image size is 1779 x 1109, after binning, cropping, and then performing all the steps up to HDR just to try and approximate a 'real' situation.
Opening the HDR module with all other settings at their defaults, I get:
Context size 50px
Full size with no ROI - 3:00 minutes
With a ROI of about 5% of the full image - 20 secs
60px
Full size - 5:30
With ROI - 00:25 (ie 25 secs)
70px
Full size - 9:55
With ROI - 00:45
80px
Full size - 17:15
With ROI - 1:33
Anything below 50 px isn't too much of a problem, for example at 25px the full screen takes about 10 seconds. But I don't think there's much point me testing above 80px as it's clearly going to take quite some time.
So I could continue to do 'what if' with the ROI, but if I then decide that a higher context size suits the image, I'm going to have to, as Mike puts it, mow the lawn while I'm waiting for the full image to be prepared.
Hope this helps.
The image size is 1779 x 1109, after binning, cropping, and then performing all the steps up to HDR just to try and approximate a 'real' situation.
Opening the HDR module with all other settings at their defaults, I get:
Context size 50px
Full size with no ROI - 3:00 minutes
With a ROI of about 5% of the full image - 20 secs
60px
Full size - 5:30
With ROI - 00:25 (ie 25 secs)
70px
Full size - 9:55
With ROI - 00:45
80px
Full size - 17:15
With ROI - 1:33
Anything below 50 px isn't too much of a problem, for example at 25px the full screen takes about 10 seconds. But I don't think there's much point me testing above 80px as it's clearly going to take quite some time.
So I could continue to do 'what if' with the ROI, but if I then decide that a higher context size suits the image, I'm going to have to, as Mike puts it, mow the lawn while I'm waiting for the full image to be prepared.
Hope this helps.
Not so much boldly going as randomly stumbling where plenty of people have been before
-
Mike in Rancho
- Posts: 1166
- Joined: Sun Jun 20, 2021 10:05 pm
- Location: Alta Loma, CA
Re: HDR module in 1.8.515 Public Alpha is extremely slow
Nice testing everyone. And Guy, quite interesting with the way you converted into relative percentages of the time taken - that's a helpful way to consider it.
I have not yet played with the preview box, but have assumed that like with some modules in the past, you would hit an accept or do-all button, which would then lead to the waiting period.
I too am a little perplexed by the context size meaning. With 50 px at about the halfway point, I assume it goes to 100. One problem with a preview box - if we really are interested in the larger structures that such context will affect, the preview box would necessarily need to be larger in order to, well, preview it. No?
In the just prior old-HDR, I believe we were working with detail size, and the default began at 1000 pixels. Pretty big disconnect there, so I am wondering what (if any) the relationship is between new-context-pixels and old-detail-pixels.
Nonetheless, after just a few dataset tests, I seem to be getting pretty good results, and all sorts of readily modifiable detail enhancement, just using context size at 50 and below.
I have not yet played with the preview box, but have assumed that like with some modules in the past, you would hit an accept or do-all button, which would then lead to the waiting period.
I too am a little perplexed by the context size meaning. With 50 px at about the halfway point, I assume it goes to 100. One problem with a preview box - if we really are interested in the larger structures that such context will affect, the preview box would necessarily need to be larger in order to, well, preview it. No?
In the just prior old-HDR, I believe we were working with detail size, and the default began at 1000 pixels. Pretty big disconnect there, so I am wondering what (if any) the relationship is between new-context-pixels and old-detail-pixels.
Nonetheless, after just a few dataset tests, I seem to be getting pretty good results, and all sorts of readily modifiable detail enhancement, just using context size at 50 and below.