You'll likely get different answers from different people, but I for one do use single-AF. Reason one, my Oly is a bit older and only has 3 focus zones, which I deactivate to only the center anyway. Then if I want my focused subject to be off-center, I focus where I want with a half-press and reframe before completing the shutter operation. If I tried that in continuous mode, as soon as I moved for the reframe I'd lose my selected focus element, thus ruining shots like this:
Of course, I didn't get much DOF variation in that shot anyway, so its an even bet whether you think that shot is 'ruint' to begin with. 
The other reason is related to the Oly focus being a little slow sometimes, even with a focus lamp. If I was on continuous AF I'd run the risk of it running in and out the whole range, vs. just doing it once and then I have to re-press to have it try again. I probably lose shots but prefer the sensation of being a little more in control of the hunt. Not sure how either of these ideas would apply to either your shooting style or camera.
As to drifting as you push, that's probably a matter of technique improvement as well, kind of along there with breathe control to hold steady, or stretching/compressing your body to compensate for drift direction a little (as best you can when using a viewfinder) if you know you'd still gliding a little at the time you're shooting. Don't let the first try get you down too bad, especially if your prior experience was using the PnS in 'liveview' like mode at arms length with the screen, holding a rig up to your eye will take different 'muscles' so to speak.