You're right, there aren't any players that the Yankees and the Red Sox have both made substantial offers to that the Yankees haven't ended up signing. However, there are three things to consider...
1) There are players that the Yankees have shown interest in that ended up signing with the Red Sox.
2) There are players that the Yankees have actually been in contract negotiations with that ended up signing with other teams, for reasons I've already mentioned in two separate posts in this thread.
3) The Yankees haven't gone to incredible financial lengths to sign any of the players that both the Yankees and the Red Sox are interested in, as you seem to be implying.
Sure, if the Red Sox topped the Yankees' offer for Teixeira and Damon, it is very possible that the Yankees would have countered again, eventually signing them. But given the facts I've mentioned, to say this is an absolute certainty just isn't true.
Additionally, consider this. If the Red Sox were so sure that it would be foolish to top the Yankees' bid, because the Yankees would just increase their offer until the player signed with him, then why wouldn't they just increase their bid to drive up the price? They wouldn't be losing anything, and they would only be forcing the Yankees to overpay, which will hurt in the future.
My guess why they choose not to do this is because the increased price exceeds how much they believe the player is worth, and they're scared of getting stuck with that player for the increased price.