Indiana uses a system that is 50% coach/AD vote and 50% other factors. The other factors are things like years licensed, years of playoff experience, playoff advancement, association attendance, and games worked (self reported scheduled since it's done as part of your application at the beginning of the year). Test score used to be a part of it, but they removed it last year. You still have to pass the test to remain eligible. Everything is on a scale of 1-5 and weighted. For example, you get full credit (5 points) for years licensed if the top 3 guys on your crew average 10 years of experience. It then scales down. I think max for playoff experience is 6 years for the top 3. Meeting attendance is 8 for football. Most associations will have 10-12 meetings and the state offers 2-4 online meetings that count. The summer clinic counts as well. You need to have 8 varsity games to get full credit, and we have 9 regular season weeks. To get full credit for playoff advancement your top 3 guys need to have worked a state final. A semi-final is 4, regional final is 3, sectional final is 2.
Most crews get full credit except for the advancement portion. That means the real differentiator is still the coach/AD vote. The coaches are the ones that are supposed to vote, but the application is accessed by the AD account so it could be one or the other or both that complete the vote. All schools are required to vote and are asked to only vote for the crews they had in the past 3 years. The IHSAA has no way to track who those schools are though so they let every school vote for every crew. Some crews get 80-100 votes. Most get 25-35. The general thought is the crews with 80-100 votes are getting a lot of extra votes from people who know them for various reasons (i.e. they have guys who work other sports especially basketball and baseball, they have guys are/were coaches/administrators, they have been doing this so long they know a LOT of people) and all those votes would be positive. This greatly increases their average score. The 80-100 vote crews will get a 4.7/4.8 average while the 25-35 vote crews will usually top out around 4.3/4.4 with many lower. They may have the same number of 1/2/3 votes but the extra 5s bring up the average much higher.
Once all the votes are in the crews are rated 1-x depending on how many there are. There are 125-130 playoff games week 1 and last year we had 142 crews apply. Week 2 has 96 games, week 3 has 48 games and so forth to the finals with 6 games. Starting with the sectional final round (48 games) at least half the crews have to have never worked a state final. That is determined by the referee. It's possible there is a crew with 3-4 guys who have worked a state final but now have a different referee because their state referee retired. I call these recycled crews. They still get full credit on the points because they have 3 state final officials, but they are on the "new" side because their current referee wasn't the white hat for their state final game. Of the 24 "new" sectional final crews it seems 6-8 of them are recycled crews. As they advance to the regional and semi-state rounds the percentage of recycled crews gets higher. It's not impossible for a truly "new" crew to advance, but it can be very difficult. They have very little margin for error in the vote.
A brand new crew can only work 2 rounds but usually only get 1 because they don't know enough coaches/ADs to get enough votes. A recycled crew is still considered new for advancement, but the can work up to 3 rounds. Any crew on their second year could advance to the third round (sectional final). From there you can't work a later round unless you worked the prior round in a previous year. Once you have worked a state final you can only advance to the sectional final the following year and then one additional round each year. That means a state final crew can only work a final every 4 years. Of the 24 crews that work a state final over a 4-year period, I would estimate 16-18 of them return in the next 4 years. It's hard to break into that group, but if you keep most of the guys together you'll likely return as long as you stay together.
It's a really bad system because quality of crew doesn't really play into it. It's not really an old boys network because it's not based on an individual or group getting together to decide ranking. It's based largely on the collective vote and the more people you know that vote the more likely you are to advance. So the ranking isn't a measure of quality. It's just a number assigned to each crew so the state can assign the crews.
I agree with FLAHL. No matter what system is used the same number of crews will be upset. There really isn't a way to create a completely subjective method of evaluation. I've heard the same issues with NFL ratings. They are graded on every play so theoretically the highest graded officials are working the playoffs. But I was told once they know by mid-season who they want to work the Super Bowl and all the sudden all their 50/50 calls are graded as correct.
Control what you can control. Work the games you are assigned to the best of your ability. And have fun doing this. Bad assignment processes aren't personal, and they aren't intended to be a reflection of your performance. Everybody wants a process that puts them in a state final, and there are only so many spots there.