Median is absolutely not a reasonable definition of average. The two concepts are completely different.
The number sets of {1,2,3,10} and {1,2,3,1000000} have the same median. Does that mean they have the approximately the same average?
I do get what you're saying, and the use of median might be more appropriate in this case, as there is one certain player (Andrelton Simmons) who is that outlier that significantly influences the average. But if Boagerts is being measured against other shortstops defensively, I think "average" is probably the best way to describe his defense.
And all things considered, I think that is a very good thing...