Brian is low-key. Doesn't get worked up about anything. And now that he's got an alliance with Ted, I don't see him going. In addition, he's always seen "doing something". He went for water, he helped go fishing, he looked for food, etc.
Plus, he's got to be working behind-the-scenes on a coconut-hairgel manufacturing machine, ala Gilligan's Island.