The question has never been about whether, or not, to "flag it", there's no logic for that. The question is, and as evidenced by the idiot at the tennis stadium, the potential danger of one of these drones crashing (either accidentally or BY DESIGN) considering you have no idea who may be controlling it, or what their skills or intentions are.
Presuming there is some imaginary line separating dangerous from safe, drawn from the ground to the stars, is childish. As suggested earlier, regarding rotary powered aircraft in trouble, gravity determines the ultimate direction, but there is no guarantee that decline will be straight down, so EVERYONE on both sides of any "restraining" line are at (EQUAL) serious risk.
YES, the responsibility for removing the danger belongs to game management, the question REMAINS what should be done with the game, while that corrective action is underway? Suspend it until the threat is removed, or ignore it. Presuming the threat is removed because of an IMAGINARY restraining line, is insane.