I am thinking sort of like you. Unfortunately, if you voice this opinion about three or four people will hit you with all types of heretofore unseen data that would point a true analyst toward a mostly unfounded and assumptive conclusion.
I think that a runner on base is a big distraction to the pitcher but I have not stats to prove it. Just 48 years of watching moves to 1st and 2nd. Maybe I imagined the stretch, too.