I'd also like to point out that my use of "down by contact" wasn't even used in its proper sense. So, yes, I know that's not a thing in Fed rules (or NCAA for that matter), but it was just the most descriptive phrase that I could think of.
The holder, with a knee on the ground, should be "down" by rule -- but gets an exception because he's the holder. However, once he's contacted by a defender, does he lose that exception and therefore is "down"? The answer seems to be "no".
Edit to add a follow up scenario:
10 seconds left in the game, 4th and 13 after a couple kneel downs -- A/K is up by 1 and is trying to run out the clock, B just called their last timeout after 3rd down. A/K lines up in a field goal/scrimmage kick formation. The ball is snapped, and the holder firmly holds the ball on the ground while the kicker makes no approach to kick the ball, but he is lined up to do so.
What can B do legally to try to end the down before time expires?