Well, outliers is not necessarily a pure tit for tat sort of thing. You are looking at the appearances that DO NOT trend with the rest of his season. If you plot his ERA on an outing by outing list for instance, one set of histogram buckets might be:
ERA 0.00 to 1.50: 5 starts (5/13 = 38.5%)
ERA 1.50 to 3.00: 2 starts (2/13 = 15.4%)
ERA 3.00 to 4.50: 1 start (1/13 = 7.7%)
ERA 4.50 to 6.00: 2 starts (2/13 = 15.4%)
ERA 6.00 to 7.50: 0 starts
ERA 9.00 to 10.5: 1 start (1/13 = 7.7%)
ERA 10.5 to 12: 0 starts
ERA 12.00 to 13.50: 1 start
ERA 22.5 to 24.00: 1 start
Plotting a histogram like this shows the bad starts are clearly the outliers, much more than the good ones - where the clustering is taking place. Taking one or more of them out of the analysis is totally reasonable, but generally you don't want to unless there really are some sort of temporary circumstance. (an injury, an inside the park homerun, Coors Field, whatever)