Conrad & David resent him & don't want him in the family. Also, he's shown his unsavory side with the art gallery bit. Plus, from what his ex-wife said, I think there was a lot more than him being gay. She indicated he was big trouble & very bad news. Neither of these could apply to being gay. The only problem she could have with that is that she was hurt & decieved. What she indicated was much more than that.
I don't think there's anything more going on with Victoria & Patrick, but there is something unsettling about their relationship. The lack of resentment from him & her blind acceptance of him, & the hugs are a bit much.