This is where the mixture that y228 was talking about comes into play. You're right, if you only looked at statistics, you wouldn't get the whole picture in that scenario. Of course there's nobody that only studies statistics and doesn't watch any games--if you're a baseball fan, you watch baseball.
I've said this before, but there are different levels of fandom. You can be a perfectly "good" fan if you just watch the games, and you would have a solid understanding of baseball. Statistics, in my opinion, make that understanding deeper. There's room for both.