Sometimes, too much thinking can be as equally big a problem, especially if you worry excessively about what someone who rarely fully understands what you're doing, thinks. I'm not at all criticizing carrying your flag in your belt. If you're comfortable doing that, FINE, but carrying your flag in your back pocket is just as "professional", as long as you carry it properly and know when and how to use it correctly.
If you are reaching for your flag, "over and over" but not using it, there very likely is a problem, but it's probably not directly caused by where you decide to carry your flag. As always it usually pays to, "When in Rome to act like a Roman", so if there is some standard, especially about something trivial, it makes sense to follow the standard. Thankfully, not all areas react, or pay much attention, to such observations, which may cast more shadow on the observer, than the target.
Often descriptions of "professionalism" emerge mystically from deep within the "great somewhere" and simply reflect some unknown personal preference related to a particular option. Of course, it's always wise to at least consider every corrective or helpful solution, and how it might be of value.