Gary - My wife/dive buddy and I have been to Cayman Brac eight times and to Little Cayman two times (that's about 40% of all the dive trips we have taken). If you don't want crowds, hustle-bustle, and crowded reefs, and if you don't mind a 45-minute additional flight on Island Air, I highly recommend either of these laid-back, barely discovered gems with near-pristine reefs. In November thru early December, there are NO crowds at all, and the weather should be nice (usually).
On the other hand, if you like crowds, shopping, night life, etc., Little Cayman or the Brac are not for you. The only people we have met who didn't like Little Cayman and the Brac were people who really weren't prepared for just how quiet and laid-back these islands are.
In any case, the Caymans have GREAT diving, arguably the best in the Caribbean.
Jim