My refraction algorithm had a glaring mistake in it. For sometime I've had refraction switched off while I worked on other components, each at a time. But eventually it had to be looked at. I started by adding a much higher resolution HDRI. It's quite green, as you'll see. That's because it's full of trees and green foliage.
The first test samples came out much like the image below. Render times ranged from mid 15 to mid 16 minutes, for 10 passes.
I dug into the code a little and realised I was running a section of the code twice over, when it wasn't necessary. The resulting issue meant render times were shot out, because rays were being sent out twice to calculate refraction. It was an error on my part, not quite understanding the code flow, that had big time consequences at render time.
Rewriting the code prevented the ray casting from being done twice, with no affect on image quality. Below is a render of the same shot, but done in less than a third of the time.