It's certainly not the top of my worries, but Yoshida has very little value as a guy who is not playing everyday.
You want your bench guys to have some combination of E.G. ability to play multiple positions well and to run fast, a back up catcher, and maybe a platoon bat. What can Yoshida do? he has a decent hit tool but he's a lefty, he doesn't run, he can't field, he has little pop. He's very limited in the bench role.
Again, at this point I'd live with him there to keep someone like Duran on this team and keep the lineup intact. Might pay dividends if an outfielder goes down and he moves into the DH position full time but Yoshida as a bench player is certainly less than ideal.