The solution to the fogging problem is to use desiccant. Even if you aren't getting fog, you are keeping your camera in a very humid environment, and long term, that can lead to trouble. And sooner or later, you may hit conditions that will cause fog.
All that's really going on with fogging is, the air temp in your housing is dropping to the dew point. All the data I found on dew point uses big volumes (cubic meters), so let's say your housing is for an imax camera and happens to have exactly 1 cubic meter of volume inside, 1 cubic meter of air. Let's forget about the camera for now, and just talk about an empty housing with 1 cubic meter of air in it.
Let's say the air in the housing is absolutely dry. There is nothing in the housing but oxygen and nitrogen, no water, 0% humidity. Let's say the temp in the housing is is 25C or 77°F. At that temp, one cubic meter of air can hold 23 grams of water, which is 23cc's of water (that's cubic centimeters, or about 1.55 tablespoons or .77 fluid ounces, and if you're like me, that's enough math to make you want a pint).
OK, so you put your 23ccs of water in the housing, and wait a bit, and it all evaporates, and now the air in your housing is at 100% humidity, and it's still at a temp of 25C or 77°F (hey, geeks, please ignore that we've squeezed 1.000023 cubic meters into a 1 cubic meter housing and changed the pressure, and while were at it, let's ignore the change in temp due to the energy of vaporization, mmmkay?)
You go out of the room, to somewhere warmer. Now the air in the housing gets warmer, so can hold more moisture. As the temp of the air goes up, the relative humidity goes down. If there was a humidity gauge in there, you'd see the needle drop. Get the temp up to 30C or 86°F, and the air can hold 30.4 grams of water, and the humidity in there is only around 74%. Get it hotter, and it can hold even more moisture.
And now you go diving. Finally.
So what happens when anything on the inside of your housing hits a temperature lower than 25C or 77°F? You can't have more than 100% humidity. You hit the dew point. Some of the water condenses out. You get fog. The needle on the humidity gauge has hit 100 and tried to cross it.
If you have a metal housing, maybe that fog occurs on the metal and you never get condensation on your port. If you have a lexan housing, maybe it occurs on a metal push-button, since the button will cool off faster than the housing wall. Or maybe your camera runs hot, and as long as you keep taking pictures, you can keep your housing temp up over 77 degrees, and you won't get fog.
Or maybe, for whatever reason, your lens port cools off first, and you get fog where you least want it.
There are two ways to avoid it: keep your housing's temperature up, or reduce the amount of water vapor in your housing. My wife uses the first method, she only dives in warm water. I use the second method: I use dessicant.
Many people also use the second method by only opening their housing in an air conditioned room, with the idea that if you are starting with air at 80% humidity at 70°F, you won't get condensation until you get into water that's down in the 60's, so in the tropics you're just fine. Right? Yeah, well, usually, but why mess with it? why not use some desiccant? You never know when you're going to start out with a bit more humidity and hit a thermocline. But is this really a good idea? if your housing is just a few degrees away from hitting the dew point, that means the humidity in your housing is very high. How happy is your camera in an environment with a 90% humidity?
So if you get fog in your housing, you know that one of two things has happened: the temp has dropped below the dew point, or you got more water in your housing. Those are the only two reasons fog can happen. Those are the only two reasons. Really. Those are the only two things that can cause the needle on the humidity gauge to move.
Whatever temperature your housing is at on the surface after you close it does not matter. Under a towel, in the sun, it doesn't matter. How can it? There is still only 23cc's water in the housing, and it won't condense until the housing hits 76 degrees. It doesn't matter if the housing has been sitting at 80 in the shade or 800 in the sun. The air in the housing doesn't have a memory. It has 23cc's of water, and as long as you're above 77 degrees, no condensation will occur. It does not matter what the housing is made of. All that matters is the temperature and water content. That's it.
Now here comes the fun.
Let's take the housing and put a camera in it, and leave it in the sun before a dive. Regardless of everything I've said up above, we all know it's much more likely to fog. We've all seen it.
How can this be? How can it fog just because it got hot? How can it fog at a temperature higher than 77 degrees? It cannot. If there is only 23ccs of water in there, no fog can occur if the temp is above 77. It just plain can't. In fact, up above, we said that at higher temps, humidity % is lower. The needle on the humidity gauge drops as temp goes up. It should be less likely to fog. It should not fog.
But it does.
The needle on the humidity gauge has gone up, not down.
There must be more than 23cc's of water in the housing.
It must have come from the camera.
Here's what I think is going on, kids. Your camera has moisture in it from spending so much time in humid environments. And that time in the sun is getting your camera hot, getting that moisture moving, and it gets out in the housing where it can cause trouble.
I have two examples to support the theory.
The first is my dive computer. If I leave it in the sun, it fogs. It's sealed. No moisture is coming and going out of the housing. But when I leave it in the sun, the guts of the computer get up over a hundred degrees or so, hot enough to vaporize so much moisture from the guts of the computer that the humidity in there is so high that it condenses out on the cooler front window of the computer. There is so much moisture in my computer that the dew point in there is up in the 80's. When I take it out of the sun, the travels back to whatever other part of the computer is coolest and the window eventually clears up. It's not a happy computer.
The second example I have is my original digital camera. It got so much moisture in it that it started fogging on its own, out of the housing, just out on land. I tried everything to get it to clear, and it never did. I tried leaving it out with every door open, no luck. Over years and years of use in humid environments, the moisture had migrated into the lens barrel, which is pretty well sealed, and it wasn't coming out quickly.
here's a picture from that camera.

This camera was not in a housing, and the front lens was clean. The fog was in the guts of the lens. Kids, use desiccant. You are more likely to fog in cold water, you are more likely to fog when you assemble your housing in warm air with high humidity, you are more likely to fog in a housing made of a material that cools faster. But even if you are not fogging, why do you want your camera in a housing with high humidity? Use desiccant.