The problem with the "eye test" is not because it is not stat based. To me, the problem is that no one person observes every play of every player being compared. Sample sizes of non Sox player observations are too small for any definitive comparative evaluation.
Although I try like hell to not be biased when I observe our own players and compare them to our opponents' players in any given game or in the totality of all games I watch, I know that bias has to creep into my final judgement based on observations alone.
I know many people think I look at the numbers more than the game, or that I let the numbers form my opinions, that's not actually true most of the time. Usually, I observe something, formulate my opinion, and then check if the data supports my opinion. When it does not, I try to observe more closely and see if the data appears wrong to me or if perhaps my initial observations were faulty. For example, when Ellsbury first came up, I thought he was a great fielder. I was shocked to see his numbers did not support my opinion. I started watching more closely and realized he often got late breaks, took wrong angles and as I always knew, had a noddle for an arm. I then started arguing against posters who I used to be on their side with. Then, as time went by, Ellsbury actually started improving, and I had to change my opinion again based on my observations and the data. (In my opinion, he never became a significant plus on defense, although some data said he was.)
Now, back to observations. We both love defense, and I respect you for that and also your opinions that I do not always agree with. I have no issues with anyone saying, "JBJ is a great OF'er" based on what you see and no data what so ever. To me, the problem arises when someone wants to say JBJ is top 5 or 10 or 15 based on just personal observations. I just don't think that opinion is valid, even if that person might watch a lot of games without the Sox playing in them. I don't think that someone using data to say that JBJ is just the 10th or 14th best CF'er in MLB necessarily negates personal observations that indicate he is great. There very well might be 15 great defensive OF'ers in MLB today, and there is a chance JBJ is somewhere between the 10th and 15th best. In my heart, I don't think that's true, but I have seen a lot of great defensive OF'ers in games vs the Sox, and the assorted numbers seem to show JBJ is somewhere between 8th best and 14th best over a large enough sample size of 2-4 years. I'm okay with that. It doesn't bother me that the data looks a little off. It just makes me look more closely, or maybe think the data might be a little skewed, although not enough to render them useless. To me, UZR/150, DRS and the Fielding Bible are all vastly superior to Flg%, RF/9, CS% and all the other old data we used to use to supplement our opinions. They are all not perfect, but they are based on something one person cannot do: they are based on human observations of every play in every MLB game. Yes, bias can play a factor in this data. Yes, how the data is weighted and formulated might be somewhat flawed. Yes, it should never be used to definitively value a player with no arguments allowed. I apologize, if I have ever used data in a way that makes someone knowledgeable of the game feel like their opinion is not valued or respected. I know I throw numbers around in ways that may be taken as obliterating someone else's point of view, but I do not intend to come across like that.
We all know a good player when we see one. The eye test is not without value. The tough part comes when we try to compare a player to others we barely watch.