I always say we, out of habit, and because it's the easiest way to communicate amongst fans. It doesn't make me feel like a part of the team.
Anyway, I don't think it's wrong to say we. The fans and the players work together in a way. It's like a symbiotic relationship. If there isn't players, then the sport basically doesn't exist, therefore no fans. If there is no fans, there is no point to the sport. Sure, players can play in an empty stadium in front of only themselves, but why would someone want to do that? You have a talent you worked hard at, you want people to see you use that talent. You want to be cheered and have atmosphere. Not have what you do show up as some number in a newspaper.