Usually what happens is that pitchers have several "outlier" starts. Those are starts you know a pitcher can't consistently get, like 2 IP, 6 ER blowups or CG shutouts. But Peavy's only "good" outlier is a 9 IP 1 ER outing, and his two best after that are nearly identical 7 IP,1 ER outings. Those aren't outliers.
I think the 4.09 ERA resulting from removing his best and worst start is the best you can when looking for a "true" average.