Well, everybody's got their ideas of what a sport is, so there has to be some sort of objective reference point.
I would point out that a golfer has been SI's Sportsman of the Year 6 times since the award began in 1954.
And the only golfer to win the award twice was not white.
And continuing with that theme, check out the LPGA events sometime...the majority of the best players are not white.