Well, it wasn’t 11 different players. Three teams retired it for Frank Robinson. (Cincy, Baltimore, Cleveland).
And as it turns out, Arizona (Luis Gonzalez) and KC (Frank White) did retire it for non HOF talent.
Still an excellent question. I had to look it up…