Under NFHS rules, there is no distinction as to who the profanity was directed at. If it's at you, the opposing players, the weather, or even Lady Luck, profanity gets flagged. Whether or not they're talking to you when they say it is irrelevant. In my case my hearing is not what it used to be, so if I can hear it, then it's guaranteed that it was loud enough for a flag. I was working a 7-man game (between Catholic private schools, natch) where there was a double-turnover on one play. As the second turnover is being advanced, a coach on my sideline yells out, "I don't f***ing BELIEVE that!!" to nobody in general. The Side Judge dropped a flag at his feet and the coach was all confused. He didn't even remember using the f-bomb. It's still a flag though because it was definitely loud enough to be heard all over the field.
If they're moaning about a play that happened a couple of plays ago, I usually tell them, "Let it go, Coach. That play's over." If they ever start complaining about having to play against TWO teams that game, or otherwise imply that your impartiality might be suspect, that's an immediate flag with NO warnings. My first year I had a JV coach doing that all game and his players started believing him. By the end of the game they had given up trying because he convinced them they couldn't win. When I brought it up afterwards, all the veterans asked me why I didn't flag it. I said "because he didn't use profanity". They set me straight.