It's not a spacious argument at all. If he was still a capable CF, the Yankees could play him there and then play Matsui in LF, making their lineup that much better.
Compared to other corner outfielders, he's just not that productive of an offensive player. Is he good? Sure. Not great, like you claim.