No.
Dum dum Willl Clark was upset that Raffy didn't work out with him one day, but they talked it over and Clark apologized for being a blowhard about it (as he is about everything). Raffy didn't refuse to work out. You are misremembering. The only thing that happened back then was that he said he wanted to slowly transition into the position because he didn't want to be embarrassed in an MLB game since he had never played there before.