It's not a huge deal. A .373 home OBP isn't killing us, and he still takes a ton of pitches. Like jax said it will probably even out anyway.
update:
...looking at Swisher's home splits since this thread start, it looks like the numbers are already starting to even out:
.224/.384/.370/.754