I hear what you're saying, and to some extent even one of the worst MLB starting SS is probably a better defender than any of us ever played with.
I can see someone calling all SSs between 11-20 "average", and I suppose one could push the envelope and claim Bogey is number 20, and therefor "average", but my observations tell me otherwise, and the combination of UZR/150, DRS and other data and rankings tell me he is not even the 20th best out of 30.
I keep seeing better plays by opposing teams' shortstops, and the numbers back me up.
What bothers me more than anything is that I don't see him getting any better. It looked like he started to improve a little in 2015, but he's done worse since then on defense.
I don't look at Bogey and see Jeter or some other really horrible defensive SS. Like Jeter, he makes most of the sure plays and doesn't look uncoordinated or unathletic. Maybe that's what keeps some people thinking he's halfway decent or even better.
Maybe he can show some growth in 2018. We may need it, if Devers doesn't improve on defense at 3B.