I agree with your point about confidence. If any player is that weak mentally, then he probably won't make it in the majors regardless of when he's called up.
That said, a downside to calling someone up too soon is that you start the player's clock, which means if he's not quite ready, you're trading some of his prime value in the future for perhaps lesser value of the player today.
If it's a dire situation, then it would make sense. Is our situation dire right now? I don't think so. I say let him develop some more in the minors until/unless we really need him.
All that said, I trust the managers/coaches to make the right decision on him either way.