I feel that mental and physical errors are on the players, especially mental errors. Setting up a player to fail by employing him at a position he isn't used to, well, that falls on either the manager or general manager.
Also, there are players(Devers, for example) who are not very good defensively, and probably never will be, but the manager has to keep them in the line-up because of their bat.