It's really a judgement call. But generally a good rule of thumb I use is did the player get knocked to the ground or did he just "bump" him. That doesn't mean that the "bump" can't be severe enough that it's roughing. Also watch for a good acting job where a kicker 'fakes it till he makes it'.
Make sure you see all the contact. Did the B player get blocked into the kicker? Did he make an obvious and concerted effort to avoid 'roughing' the kicker? Did he touch/tip the ball after it had been kicked before unavoidable contact was made? Did he just plow into and shove the kicker after the ball had been clearly kicked?
It's a very difficult judgement to make, try and find a lot of film to find obvious distinctions between the two. A great place are the Aloha clinic videos on kicking, there are some plays that cover this.