The sense of humor this season among most of the houseguests (well, the GT and Knockouts at least) is really dry, caustic and dark. They say rude things to each other all the time and know they are kidding. What Andy called his mom (which was odd, I grant), he calls other people in the house all the time, and they know he's kidding. A lot of comments by GTers get blown out of proportion on the internet because people don't hear the whole conversation going on.
With all the unapologetic creeps in the house, it's easy for me to think of Andy as a "good one". It's easy to find things about all the HGs to pick at and pick at, but with some perspective I've seen that he's being really smart and playing a shrewd game, and hurting no one in the process. He may not take loud stands like Amanda does, but he is more than willing to let evil people hang themselves without getting the blood on his hands. That's how you get far in BB.
The worst thing he could do right now is become HoH. Like Dan, the game he's playing means he has to be passive until there are fewer people in the housee. Then he can strike.