FWIW, It seems the basic logic behind most/every rule remains, Is someone gaining an unfair advantage over their opponent. PLAYING with an extra player is clearly such an "Unfair Advantage", which would include attempting to confuse an opponent, or (actually) blocking/ impeding an opponent.
Judging whether the action of a player leaving the field lays FIRMLY in "the eye of the beholder" (NOT the Coaching staff, opponents, or spectators/sideline media) if YOU believe the action YOU are observing creates an UNFAIR ADVANTAGE, YOU have the right & responsibility to act upon YOUR conclusion, or to DECLINE taking action. That decision is YOUR'S and your's ONLY, and unsolicited advice about your decision simply DOESN'T MATTER (unless YOU deem it excessive and worthy of punitive action).