Hi Jeff,
Do you have a particular dataset in question that isn't ending up how you think it should? Or just general targets in mind?
I know the subject has been discussed here often, directly and indirectly, but you might have to go hunt through threads in the gallery and image troubleshooting sections. Plenty of varying theories of what can be done, as well as what
should be done (i.e. in terms of maintaining fidelity to the data or at least a good logic in presenting the data), too.
Some, maybe many, targets are just weak in OIII. In mono you will sometimes see people amass a much longer integration on OIII. That greater SNR permits a greater contribution to the luminance, though the relative chrominance remains the same. Not really possible with a duo though, which will have identical integration.
Using ST's typical Compose for duoband and bicolor matrices in Color, you will be held to relative Ha/OIII balancing as the logical way to display HOO narrowband. While still holding to that ideal, there's a couple things you can try. Throttling back the Ha, of course, to start. Toggle the MaxRGB screen to get an idea of overall Ha/OIII balance and if you can see stronger (with Ha relatively throttled) OIII regions start to reveal themselves. Of course this weakens the Ha, and in some targets can turn fringe areas into gray fog.
Try boosting the saturations, possibly by quite a lot, and balance the light and dark sats towards emphasizing the blue OIII, if possible. Reverting style back to Color Constancy may help pump the blue back up, also.
If a noisy OIII channel out of your duo is causing problems with this, it's possible some adjustments at the Compose stage can help. You can indeed extract the channels. ST can do it, but perhaps best done in your stacker to maintain 32-bit depth. Consider ditching the B channel entirely, depending on target, as it may be the weakest and noisiest. And if your duoband has an expanded OIII pass, such as an L-eNhance which also captures Hb, discarding B and only taking G as your OIII may remove pesky Hb (depending on the cutoffs of your Bayer filters). Then compose those two files into ST into R and G as normal R(GB)(GB) bicolor, or Ha into R and OIII into G and B but zero out the exposure time on either G or B.
The same thing may be possible using only Compose by not loading the B channel, and then picking normal R(GB)(GB) bicolor.
Depending on your estimate of the relative SNR in your channels, you can even make further adjustments using the exposure time sliders, which affect the creation of the Synth L. The idea behind the Synth L is the best overall SNR, and in narrowband that is not necessarily as well matched to exposure times as it would be in broadband RGB.
If you want to dismiss the need for relative emissions balancing, then yes you could extract and process separately as a grayscale, and see how a subsequent layer operation looks. However, it is a bit tricky to get the final results colorized for that later Layer usage, and not a good way to do it all on the fly, that I am aware of. But, I actually did something quite similar to this when I was processing some (six filter!) JWST data. After registration, each was processed separately in ST, and in Color I abused the sliders in order to get the hues I wanted. Step by step things were then blended in Layer, sometimes with different Layer operations (such as Lighten mode for one that I recall) depending on what was getting added in and where the data was spatially for layering in.
Of course that's taking a step (or three) towards "artist's impression," but I think NASA does the same thing, and hey, we don't have a six channel JWST compose module. Also, JWST is a mishmash of infrared wideband, narrowband, and medium band anyway. So the whole thing is far more about what can you show is where, spatially, with the colors, than it is relative emissions.
Whether you want to try a similar philosophy with bicolor Ha/OIII would be up to you.
More things you could try include a different channel mapping that may set off the blue better, perhaps H(H+O)O. Or even some of the tricolor SHO matrices. With data composed as R(GB)(GB), those mappings still end up as bicolors.
Finally as a bit of artistic boost you could see about an OIII boost in the Filter module. I think there's one in there, but I haven't used it in years.
Caveat though, things may still not appear similar to images of the same target you might find on the internet, with huge pools of incredibly deep blue OIII, yet at the same time incredibly strong red Ha everywhere else. Well there are a couple techniques for that, some of them quite popular, but probably best considered as airbrushing, IMO.
Star colors are a tough call. You can try to mostly bleach them to white, using increased highlight repair in Color, or color tame in Shrink if you use that (perhaps even as only deringing rather than iterations). A star mask and some Layer operations can also bleach the stars out, again if you'd rather they be white instead of the duoband, which can be odd-looking. Just the way most stars spit out Ha and OIII.
Another, and still logical, trick to try is capturing unfiltered OSC for the stars and blending those in, usually by a Color of Fg operation in Layer on a star mask, IIRC. Then you can say you've got duoband with RGB stars. A lot of posts here cover how to do that. I lack skill in pulling that off, at least where stars are in front of nebulosity.
Hopefully some of that helps?
Others may be able to chime in with better suggestions. I know we've got a number of duboand users here.
