A lot of this is software as notin pointed out. Effectively cameras and computers are watching every ball hit - especially now with statcast at all the stadiums.
The defensive stats now are much better than the old fielding percentage days, simply because there is better data going in. I mean, consider fielding percentage. If we replaced a Red Sox outfielder with a paint bucket - and just left it there. If 1 fly ball landed in the bucket, the fielding pct of 1.000 would be amazing. But we can't say the paint bucket is a good substitute to like, a person. We just have vastly better information on players getting to the ball - which is really what defensive value is for the most part, no?
As far as being skeptical - any be-all stat without context deserves skepticism. Is UZR better than defensive runs saved or outs above average? I have opinions, but the real answer is that all of them together give good information. I mean, we accept that batting average, slugging, on-base all provide incomplete pictures of hitting ... but in combination it's not too shabby.