Series that aren't procedurals like CSI or medical dramas like Grey's Anatomy have a hard time getting big ratings. I think it's because people feel like, if they miss an episode or two, they won't know what's going on. Plus, those other shows are "easy" to get into because of that pre-set formula.
I still have hope that the critical acclaim will buy this show another season. I think it's only gotten better as this season has progressed.