From seeing LaToya in interviews, she seems very much warmer. I think to her, AI was just a vehicle for exposure but she just wasnt going to get into the whole game of it all. Im sure she would have loved to win, but she wasnt going to act a different way to do so.
That being said I dont think Simon dislikes the AI contestants. I think he would say that Clay can sing but that his music might not be his style. From what I gather from interviews and such, Simon does not seem to like the same pop music he is guilty of putting out there.