yeah, illegal batting on Team A.
ncaa enforcement is 10 yards from the previous spot and replay the down, or tack on 10 yards to the spot where the dead ball would belong to B.
also agree that since the ball was over the OOB area and stayed there, most teams wouldnt be hollering for a flag....but i'm sure team B wouldnt sneeze at the 10 yard tack on.