AGon made a few comments that turned public opinion against him. Plus fans were unhappy in general with that 2011 team. It was a disaster. Epstein gone, Francona gone, chicken and beer...
Personally I don't think it was a racially dominated thing with AGon. I think most fans were happy to see Josh Beckett go too, and he was a white man who was instrumental in the 2007 title.