In Survivor, anyone can be blamed for anything. Just takes a couple of people to target you, start the subtle wispering and appeal to enough people's righteous indignation (over anything) and a surprise vote will leave your head spinning. Sitting out too many challenges could give someone ammo at a later date. Todd just needs to make sure he does enough of everything - challenges, food gathering, boiling water, etc. and stayed tuned in to what the group is thinking.