That may be the rule, but I think it should be changed. The whole point of 'unsportsmanlike' fouls are that they are, by nature, unsportsmanlike actions. We can throw a flag for taking a swing at the opponent, flipping off the crowd (their side or opponents) or opposing sidelines, but not for being a jackass towards his own teammates or sideline, with potentially player safety implications, if they got into a fistfight and started swinging helmets. (yes, I know the flag won't make hitting someone with a helmet less likely to injure, but as in taunting, it's supposed to be a deterrent to inappropriate behavior.)
Regardless of who he hit or took a swing at or flipped off, it should be penalized the same, because it at its heart, an unsportsmanlike act, regardless of the intended audience. Just my .02 and change