If Story bounces back and garners any interest that would take his full contract, I'd trade him no matter what Mayer is looking like.
If Yoshida bounces back, I'm holding onto him.
Refsnyder and McGuire really wouldn't bring back much of a return except MiLB depth. There is a chance they move Wong instead as his framing is just awful and he can't really hit.
Also, depending on how Yorke and Grissom play, there could be a move with one of the two of them. It's not out of the realm of possibility that the future of the team is Mayer at SS and Story back at 2B too.