So you'll go through 3 bench guys (Young to Panda to Castillo) for one AB in the middle of the game? This could have made sense in the 8th or 9th inning, but not that early in the game with the Sox already holding a lead.
People complain about managers not making enough moves, but fans tend to want to make moves just for the sake of making moves. Personally, I'd rather stick with Holt than needing to put Castillo in LF for the final 4 innings of a game.