I'm not certain that he'll work out as a C, but it's definitely worth the shot given his age. Moving him off position this early would indicate to other teams that he probably won't pan out. It takes a while for some catchers to mature (why the Mariners gave up on Varitek).
I just remember that 2-3 years ago, a lot of people (including Soxprospects) were saying that he may wind up at a different position (1b, 3b, corner OF). This isn't some new position that is being mentioned because of a dropped foul ball or two.
To me, the onus is on the pitching staff. I can't blame Swihart for their failings.