This is a reasonable suggestion, outside of moving Panda which I don't think will happen, but trying Swihart at first base is something worth invesigating. If he works out there, it's a very inexpensive option, leaving more money for other areas of need, namely pitching.
I really like that Vazquez/Hanigan catching tandem, and Swihart could always catch in a pinch.
I know you disagree, but I think Panda will show improvement defensively next year. And with a year of experience under his belt and some more work at it, Hanley could even show some improvement in LF, at least to the point where he's not a liability.