It's always magnified when there's a man on 2nd because they have to go through 3-5 signs each time they want to throw something. Instead of going through "fastball away, fastball in, cutter in, curveball away, curveball in, changeup" which would require like 30 signs, after the 2nd time they'll just come out and talk.
You don't notice it as much when there's nobody on, but he's probably shaking him off just as much.