I sometimes don't notice deviations, if they're subtle or it's been a while since I've seen the episode which contradicts. However, I agree, when I notice them they do really jar me out of the story.

I think the main thing that upsets me is the sloppiness it shows, because it makes me think that the author of the story didn't care enough to get their facts right. And then, as you say, it does make me start to doubt the rest of the story, especially characterization-wise.