Therein lies the ultimate quandary for sports officials. Officials for virtually all amateur sports, and even for many professional sports, have to make the decision for themselves as to where their priority will fall, or find ways to compensate for the time away from family/occupation. For those who are not self-employed, or who may not be in a position to set their own schedule/hours, they simply have to decide whether to find other primary employment that can tolerate their absences during the season, or simply realize that higher level officiating that requires travel during otherwise normal business hours is not something their lifestyle will allow.
For most of my FBS career, I was totally blessed to have a job that allowed me to use my vacation and personal leave hours to get away from home on those Fridays (some Thursdays and Fridays). OK, sometimes I kinda ran out of those hours, but the 'blessed' part was that my 'superiors' knew what I doing, and simply looked the other way in those few instances, knowing that I made every effort to not abuse their trust and privilege, and worked hard to keep up with my work. Later, I became a partner in a business, and could come and go as needed, although I still tried hard to use vacation/personal leave, and not abuse the system that we expected our employees to follow. The other side of that coin is that I had a family that fully supported me, and knew that I would make up the 'family' time in other ways (especially by taking them with me to games, from time to time - and all bowl games). Without both of those blessings, working collegiate football would have been impossible.
I do know a few - very few - NFL officials that use the NFL as their only occupation. But, the vast majority still have other primary occupations. Obviously, they have been able to get past the time/distance obstacles that higher level officiating puts in the way.
There have been many folks I know/knew that were capable of working higher level football, in particular, but simply could not make the time/distance obligations work, and have either been happy to work high school sports, or gave up them up to focus on family and career. They are to be respected for making such a hard decision.
Major league baseball umpiring is pretty much a full-time occupation, during the season, although some umpires have regular off-season employment of one kind or another. I 'think' the same can be said for NBA, NHL, and other sports that play games almost every day of the week.
You cannot expect that your primary employer will simply let you take off at will, and retain your employment. And you certainly can not expect the sport you might want to work will change their scheduling or protocols to accommodate your primary employment and/or family. You may just have to accept that higher level officiating is just not something you will be able to do.
But, if you can make it work, what a joy it is to have the memories, the friends, and the stories you will have gained when it is all over.
Good luck!