Fair question, with no simple answer. Obviously any close interaction with an animal this big could be potentially dangerous. I think it's something that has to be considered case by case, or perhaps whale by whale.
There were actually two humpbacks that people from our boat swam with that day. The second humpback was lob-tailing - raising the lower part of its body and slamming its flukes onto the surface - when we first encountered it, and it continued to do so after three of our people entered the water, though it never did this in the direction of the swimmers. (There's a pic of this happening on my website, if you're interested).
I chose not to go in the water with this whale, because I didn't think it was safe. Even if the whale was trying to avoid endangering us - which some of us believed - I still felt there was a risk that someone might be hurt. The difficulty in keeping track of the whale and other swimmers in the waves (on the surface) and bad viz below were also factors.
The first whale, on the other hand, moved very slowly whenever snorkellers were close, repeatedly moving away then slowly reapproaching, positioning herself close to individual swimmers so that she could see them with one of her eyes, rolling on her back, thrusting a pectoral fin up into the air, etc. She (we think it was a she) was clearly aware of the swimmers and intentionally continuing and resuming the interaction.
I'd never been this close to an animal this big before, and I was very nervous at first. But as soon as I was in close to this whale, my worries quickly disappeared. At one point, I had to duck and weave a bit to avoid her pectoral fin, but the motion was very slow. (I'd decided to avoid touching the whale.) There was no doubt in my mind that this whale was aware of us and taking care not to unintentionally injure us. Of course that could simply by me/our fantasy and anthropomorphising, but I don't think so. You probably had to be there.
I should stress that this is very unusual behavior for humpbacks, at least in this area. People do attempt to get shots of whales underwater, but you don't expect any interaction. These whales are migrating, and tend to be moving fast. The procedure is that the boat drops the snorkellers a few hundred meters in front of a single whale. If the drop is done right and the whale doesn't change course, you might get a quick glimpse as it swims by, but that's the most you expect.
These two encounters were very different from the norm. I have no idea why. In both cases, when we first encountered the whales, they seemed to be taking a break from the migration, staying more or less in one place. The first whale was broaching when we first saw her, the second - as I mentioned above - was lob-tailing, all by itself. (This may be unusual as well.) Most of the whales we saw were moving, and in groups of two or three, and the lob-tailing behavior seemed to be part of real or mock fighting between two males over a female.
I hope this is helpful.