I think he had that much of a negative impact on the whole team. I actually think the team was against him before they even met him, because it was suggested that he was hired to 'whip the team into shape'. He had a huge strike against him before he even started, which was probably very unfair to him, but I don't believe he helped his cause any when he actually started the job.
If memory serves me right, he left Lester in a game to fry. Pure speculation, but I had the impression that Valentine was punishing Lester for pitching so poorly, so to speak, and let it be known to Lester that that was what he was doing. Francona, OTOH, might have left Lester in the game, but he would have approached it from the angle that he needed Lester to take one for the team and save the pen arms. IMO, there was a big difference in the way the managers communicated things.