Rory and John P. seem like nothing but trouble for the men. Getting rid of those two might help the rest unify rather than fight amongst themselves.
Eliza needs to go too, there's only so much of her you can take (unless she manages to keep her mouth shut). The older group knows she will go back to the younger group now that Dolly is gone and give the younger group an advantage. Unfortunately I think she'll stick around a while, pushing others to the fringe of the group and voting them out.