I think my big issue with the book was that I felt like i should have loved the characters more than I did. When bad things happened to them, i didn't care that much, because I thought some of them - primarly David and Norah (the birth parents) weren't the good kind people I needed them to be to really care. I never got past David being controlling, and I never got past Norah being so caught up in herself -- the drinking and then the affairs. Caroline (the nurse) on the other hand, I really liked. I saw all the good in her, so I found myself frustrated when I had to go too many chapters without her.
My other "issue" was that I questioned how much of David and Norah's marital issues were in fact due to Phoebe being given away / "dying" and how much was a reflection of the times. They were part of the generation that was effected the most by the women's movement, and the evolution from a traditional mom stays home and nurtures dad earns the living and provides family to the family structure we have today. A LOT of marriages in that period of time failed just like theirs, not due to a deep guarded secret and loss, but due to roles changing and each spouse not knowing how to handle it. So I was never totally convinced that their issues were in fact due to David's decision.
Lastly, I thought there were some siginificant events in the book that were glossed over, and tons of time given to insignificant events. The most glaring was David's death. It's just mentioned in passing, with one sentence -oh yeah, he died? I had to go back and re-read that because I thought that clearly I had missed something. I didn't. We get all of this depth and description over ancilary events like Norah's sister having cancer, but the key character of the book dies, and we get one sentence. It didn't make sense.