Here's a personal example: Jacoby Ellsbury.
When he first came up, I saw him make a few very nice plays. I knew he has speed, so I naturally thought he was a great defensive player. When he finally became the FT CF'er, a poster on that other site claimed he was not plus and had slow reaction times and took the wrong routes to balls hit to him. I started defending JE.
Then, I noticed his UZR/150 was negative, so I started watching every ball hit to him more closely. I did notice bad routes and slower breaks on the plays I was able to see on TV (not all that many). It's hard to know what my bias was before and after I started focusing on him more often.
To make the example even more complex, the next few years, I didn't notice the mistakes as much and went to check the numbers- sure enough, he had big plus numbers in 2010 and 2011 and was barely plus afterwards. The numbers seemed to support my observations that yes he was worse than I thought in 2009 but then did get better afterwards.
Was this just an example that supports my beliefs, so I use it, or not?