Seriously, it isn't that big of a deal. Yes, you should all have the same. If someone forgets their bean bag and has to use the wrong color, seriously, who gives a flying fart in fairlyland. It just isn't all that important.
To answer the original question, wear what the rest of your crew wears. To cover all the bases, buy two white and two black. What are they, $5? If you don't have one the same color, someone else probably does, but if not, play the game.
Ask TxSkyBolt. I left my white hat on my dresser when we went to Midland for a playoff game a few years back so we all wore black hats. We managed!