Evaluations and improvement

[CaseyP]

This is the worst time of the year for me.  My crew studies hard, attends the only clinic our state provides, works hard to secure the best schedule of regular season games we can get and gets great evaluations once per year each year.

Maybe we should be satisfied with that.  The playoff game we get is almost always a good matchup.  That says something about the job we are doing.  But we want more.  We want to know what we can do to be better and get more playoff games.

What do other states do for evaluations?  Nebraska tries to evaluate crews once per year.  I hear rumors that they have crews rated on a 1-5 scale, but they do not share that with us.  The guys that evaluate us often call or otherwise contact us later and say we are doing well and compare well to the competition.  Do any states do more that give more specific feedback?  And what would you like to see as feedbacK?

[NorCalMike]

We do it totally different in California. Maybe because of the size of the state. Playoffs are done by section. After sections there are regional games and then state bowl games. Every game below the regional is assigned at the section level by the regular season assignors. Regional and above crews are selected by the State. Eligible officials apply for the game and they are selected by a committee. To be eligible you must have 10 years experience and have officiated a section final. This year was my first year eligible but I was not selected.

Are all playoff crews selected as a crew or do they select positions and make up crews? Do they publish which officials are selected? If so, you might want to give one of those officials a call and see if they are doing anything your not.

Does Nebraska have any statewide committees that you can work on? Might help your chances.

[HLinNC]

NC- while we are still "associations" in name, voting and boards of directors were effectively eliminated several years ago.  Regional supervisors select and assign the first two rounds of the playoffs, which are assigned to the home teams region.  After that, the RSO's submit candidates to the state supervisor who makes the selections.

Each RSO sets up how his officials are reviewed and evaluated.  Some have observers.  We have had in the past but not this season.   Our referees conference call with the RSO on Sunday evening and go over the G,B,U's from that weekend's games.

[wvoref]

Again, I am a member of a two state Board of Officials.  One relatively small state where the state office can be pretty familiar with most or all officials working in that state.  In this state playoff assignments basically come from a committee of one.  Officials are observed by state observers during the regular season and that information is sent to the committee of one.  Games are not usually assigned to a complete regular season crew (even though recently it appears that much more effort has gone into keeping as many members of a crew together as possible) but as much as possible all members of a playoff crew come from the same board.  This allows the state to avoid assigning someone a game involving a team from their home territory.  Four week playoffs, a crew can work a maximum of two games, but many crews receive only one.

The other state is a large state with many playoff crews assigned.  There is a complicated voting system (I'm sure it was designed by a former tax law writer) including local boards, ADs, coaches ratings, assigners, rules interpreters, board secretaries and I'm not sure who all else.  Crews are formed based on the individuals point totals with the highest ranked official at a position being placed on crew 1 and then working down.  The top 7 crews are supposed to work all 5 weeks including the final.  8 through 14 would work weeks 1 to 4 and so on.  The crews working multiple weeks can be comprised of officials from all over the state and can be assigned any game in the state.  Years ago we had one official from our board work a state final involving one of our local teams.  The crews only working the first week they attempt to put them together on a more regional basis and assign relatively close, even though three members of our board had a three hour drive to their game Friday.  This state was attempting to switch to a system of assigning playoff games to intact regular season crews but eventually saw the problems this could create and went back to their old system but with some heavy duty tweaking of the voting system.  Personally I think the ideal solution is the one used in the smaller state of assigning crews from the same board. Every playoff game has an observer evaluating the crew, but I cannot remember ever hearing of anyone being evaluated by the state during the regular season.  Crews have been removed from the playoff rotation due to a poor evaluation and another crew assigned the next game.  I believe there has also been cases of an individual being removed but the crew moving on when just the individual receives a poor evaluation.

[SouthGARef]

In Georgia, your local association somehow determines which playoff crew you are on. Determining your spot in your local association is not affected at all by the state. For the first three rounds, the state office gives a certain amount of games to each association totally at their discretion. I've never really heard a method of how that is done - it's mostly done by the prestige of the local association.

For the semifinals and state title games, each association can send in film of a game worked by each of their top two crews. These films are then evaluated by those appointed by the state office to do such things. There's a complicated system of evaluating these films. These grades factor into which crews are selected for the semifinals and championships, but at the end of the day the state still has the ability to assign who it wants. In theory, an association's #2 crew could get a title game while its #1 crew stays home. I'm not sure if that's ever happened, but it could.

I actually think for the most part the GHSA does the best it can with the limitations they have. There are hundreds of crews working and they can't be expected to watch all of them. The film system is a good way to ensure that assignments are based on skill and not just word of mouth. I'm sure there's some things that could be done better, but this is pretty solid IMO.

I've heard they're considering mixing associations together into playoff crews in the future. I think that's a bad idea, but they don't ask me about these things.

[Atlanta Blue]

I've heard they're considering mixing associations together into playoff crews in the future. I think that's a bad idea, but they don't ask me about these things.

That idea is being floated by someone who has never officiated a football game in his life (fact, not hyperbole).  Other learned minds are working hard to make sure it doesn't happen.

[SCHSref]

In South Cakalacky, we are given a grade (max 100) based on years of service, attendance of meetings, test score, and evaluations of varsity games only.  Therefore, many factors can negatively impact your rating, but only if you don't have many years in.

[Ralph Damren]

In Maine, we don't work as crews, so all of our assignments are as individuals. Each chapter (we have 4) has an internal rating system. Our chapter's is a rating of 100 that includes the following factors :
                            40% fellow officials' evaluation
                            30% result of NFHS test
                            20% attendance at meetings
                              8% seniority
                              2% submitting an evaluation of fellow officials
 These are used for regular season assignments.

Playoffs (3 regional rounds for 4 classes) :
                          Official must have attended rules interp clinic & worked 5 varsity games during that
                          season.
                         Each playoff-bound  school will submit a list of 20 (we have 45) "preferred " officials. 
                         Assigner will try to assign officials that are on both team's list to their game.
State championships :
                          Each football school can submit a list of 10 perferred officials.
                          Each chapter submits a list of their top 25% rated officials.
                          28 game officials are assigned (7 man crews - 4 games)
                         the assignments are prorate to the chapter membership (largest chapter gets 9
                         officials ,smallest chapter gets 4).
                          each chapter supplies timer, assist. timer & 4 man chain crew for a game.   
             

[WestGaHL]

In Georgia, your local association somehow determines which playoff crew you are on. Determining your spot in your local association is not affected at all by the state.

Oh, how I'd love for the state to evaluate crews instead of leaving it to associations to determine. However, like you said, with the number of crews that work on Friday nights in Georgia, it would be impossible to evaluate each one.

[Suudy]

In Maine, we don't work as crews, so all of our assignments are as individuals. Each chapter (we have 4) has an internal rating system. Our chapter's is a rating of 100 that includes the following factors :
                            40% fellow officials' evaluation
                            30% result of NFHS test
                            20% attendance at meetings
                              8% seniority
                              2% submitting an evaluation of fellow officials

I worked in an association where we did the bolded part above.  But I don't get it.  How are we supposed to provide any meaningful feedback on a fellow official, unless it is about things like communication, poise, or other basic stuff?  If I'm doing my job on the field (I'm usually HL), how can I possible comment on the LJ's or BJ's pre-snap or during the play mechanics/judgement?

And what ended up happening is that guys would cheat the system and give their buddies great ratings and everyone else average ratings.  And guess which 5 guys had the top ratings each season?  I'd rather have an observer (or observers) from the association sent to games to evaluate individuals.  We have been working toward that this season in our association.  Also, the state (WA) has been sending their own observers to assist with the selection of playoff officials.

The unfortunate thing with our state (so far), is that the feedback from the state observers is very limited.  In the couple of games we had an observer, we didn't get the report until 2 weeks after the game, and we never had a chance to have a post-game discussion.  The delay in the feedback made it very difficult.  Our association is starting to film games and do some internal evaluations, but so far the feedback has been very limited.  I suspect our association is no different than many others (I've worked in 3 in our state), but the training is usually _very_ limited.  It's usually just "get dressed, get on the field, and DO NOT blow your whistle or throw a flag."  In fact, my very first game ever officiating nearly 20 years ago was a varsity 8-man game where I was given very similar advice.

[HLinNC]

How are we supposed to provide any meaningful feedback on a fellow official, unless it is about things like communication, poise, or other basic stuff?

   

Agreed- if you are doing your job, how have you got time to evaluate someone else's?  If an official does something really outstanding or severly fouls something up, I'll e-mail the RSO and let him know general info.  Picking out specific details is best left to observers.

Communication, hustle, and appearance are all easy things to pick up and comment on but there are a whole lot of details I am not going to know.

[SD_Casey]

"get dressed, get on the field, and DO NOT blow your whistle or throw a flag."  In fact, my very first game ever officiating nearly 20 years ago was a varsity 8-man game where I was given very similar advice.

Hahaha! Hasn't changed much!  The R at my 1st varsity game (I was subbing as U) took my whistle and flag and threw them in the trunk of the car and said "you don't need these."

[Ralph Damren]

I worked in an association where we did the bolded part above.  But I don't get it.  How are we supposed to provide any meaningful feedback on a fellow official, unless it is about things like communication, poise, or other basic stuff?  If I'm doing my job on the field (I'm usually HL), how can I possible comment on the LJ's or BJ's pre-snap or during the play mechanics/judgement?

And what ended up happening is that guys would cheat the system and give their buddies great ratings and everyone else average ratings.  And guess which 5 guys had the top ratings each season?  I'd rather have an observer (or observers) from the association sent to games to evaluate individuals.  We have been working toward that this season in our association.  Also, the state (WA) has been sending their own observers to assist with the selection of playoff officials.

The unfortunate thing with our state (so far), is that the feedback from the state observers is very limited.  In the couple of games we had an observer, we didn't get the report until 2 weeks after the game, and we never had a chance to have a post-game discussion.  The delay in the feedback made it very difficult.  Our association is starting to film games and do some internal evaluations, but so far the feedback has been very limited.  I suspect our association is no different than many others (I've worked in 3 in our state), but the training is usually _very_ limited.  It's usually just "get dressed, get on the field, and DO NOT blow your whistle or throw a flag."  In fact, my very first game ever officiating nearly 20 years ago was a varsity 8-man game where I was given very similar advice.

          Our "fellow officials ratings" has several components, with 1-10 rating :
                          * Appearance
                          * Game control
                          * Rules knowledge
                          * Communication
                          * Judgement
                          * Hustle
...and a place for comments.

            These reports are passed out to the chapter members in pre-season. I'm convinced that there isn't a perfect ratings system; but most feel that ours is quite fair and little gripes.   

[Suudy]

These reports are passed out to the chapter members in pre-season. I'm convinced that there isn't a perfect ratings system; but most feel that ours is quite fair and little gripes.

I think all ratings systems only work as well as they are perceived by those being rated.  If people think they are being rated fairly, then the system works.  Getting back to the OP, there really is only one reason for evaluations in our association:  training.  Postseason assignments are based purely upon seniority.  Granted, you have to a a varsity official with at least 5 years experience (at at least 3 in our association) before being eligible for a postseason assignment, and getting onto a varsity crew requires some evaluation.  And the board decides the varsity crews, and as far as I can tell nobody grumbles about the varsity assignments.  However, in my previous association, postseason assignments were rating based.  And ratings had nothing to do with training.

The OP seems more concerned with training type feedback.  And this is an area I think is sorely lacking at the high school level (at least here in WA).  There's very poor training for young officials and nearly zero feedback for experienced guys.  There seems to be no effort in our state to improve the state of officiating.  If the state wants quality officials doing postseason games, they should invest in training and feedback.

[SCHSref]

Here is a more comprehensive insight as to how we evaluate

http://scfoa5.com/faq.html#3

[AlUpstateNY]

An unfortunate, age old, problem with post season assignments, in most environments, is simply that there are more qualified, quality and deserving officials than there are post season assignments. 

[SouthGARef]

Oh, how I'd love for the state to evaluate crews instead of leaving it to associations to determine. However, like you said, with the number of crews that work on Friday nights in Georgia, it would be impossible to evaluate each one.

Yea. I'd know we'd all love to criticize the GHSA in this area. But when you think about it - there are something like 300 games played every Friday night. Trying to evaluate every official over an 11(2) week season is just an impossible task.

So they leave it up to the local associations to make decisions to narrow that pool down to two crews each. That still leaves them 44 crews to evaluate. I think it's a pretty solid system. From what I understand, only 23 of the 44 eligible crews turned in film last year anyway.

[ECILLJ]

The Illinois High School Association has a complex rating formula (see below). For the most part I believe it is a fair system.

Ratings Explanation
How the Ratings Percentile is Determined: All ratings in a particular sport that are received during the current school term, plus the two previous school terms, are averaged and then divided by the number of ratings. The result is that an official who works often (and is rated often) will generally receive a better ranking than an official who obtains a high average on just a handful of ratings. These numbers are sorted from top to bottom and divided into 100 groups of equal size. The officials in the highest group are assigned to the 99th percentile, the next group to the 98th, and so on down to 0.
Why Your Rating Percentile Suddenly Went Down (Probably): At the start of the new school term, the ratings from three years ago are no longer included when computing your percentile. If you had a lot of good ratings three years ago, or if you simply had a lot of ratings three years ago, your percentile is likely to go down when the new rating is computed.
How the Top 15 Percentile is Determined: All Top 15 Lists in a particular sport that have been updated during the last two years (730 days) are used. Officials receive points based on their position in the lists: 15 points for each 1st-place mention, 14 points for each 2nd-place mention, and so on down to 1 point for each 15th-place mention. The total points are sorted from top to bottom and divided into 100 groups of equal size. The officials in the highest group are assigned to the 99th percentile, the next group to the 98th, and so on down to 0.
Why Your Top 15 Percentile Suddenly Went Down (Probably): Top 15 Lists expire exactly two years (730 days) after they were submitted. If you are at or near the top of a list that recently expired, your percentile is likely to go down when the new rating is computed.
Alternatively, you may recently have been moved down in the rankings (or completely removed from the rankings) on a Top 15 list or lists. Such a change can often result is a fairly dramatic drop in your percentile.

Power Rating Explanation
About the Power Rating (listed on main page): The Power Rating is calculated on or about the "snapshot date" of September 7 for fall sports. Any information that changes after the snapshot date -- including clinic attendance, change in promotion level, or ratings by schools or officials -- will not be included in the Power Rating. The IHSA administrator uses the Power Rating calculated on the snapshot date to make state series assignments. The Power Rating is viewable for two months after the snapshot date.

How the Power Rating is Determined: The power rating is a number from 0 to 40, derived from 8 different criteria, as follows:
Promotion level: Certified (C) -- 5 pts., Recognized (R) -- 3 pts., Registered (X) -- 1 pt.
Part 1 exam score: 96 -- 5 pts., 92 -- 4 pts., 88 -- 3 pts., 84 -- 2 pts, 80 -- 1 pt.
Previous tournament experience for contests by this gender: state final -- 5 pts., super-sectional or football semifinal -- 4 pts., sectional or football quarterfinal -- 3 pts., football second-round game -- 2 pts., regional or other football playoff game -- 1 pt.
Contest ratings (percentile rank of all officials): 90% -- 5 pts., 80% -- 4 pts., 70% -- 3 pts., 60% -- 2 pts., 50% -- 1 pt.
Top 15 lists (percentile rank of all officials): 90% -- 5 pts., 80% -- 4 pts., 70% -- 3 pts. 60% -- 2 pts., 50% -- 1 pt.
Clinic attendance (highest clinic in last three years): Level 2 -- 5 pts., Level 1 -- 2 pts., Level 0 -- 0 pts.
Clinic attendance (most recent clinic): this year -- 5 pts., last year -- 3 pts., two years ago -- 1 pt.
Varsity games worked (percentage of games a school is allowed to play in the regular season in that sport): 70% -- 5 pts., 60% -- 4 pts., 50% -- 3 pts., 40% -- 2 pts., 30% -- 1 pt. See table below.
    5 pts.   4 pts.   3 pts.   2 pts.   1 pt.
SPORT   MAX   70%   60%   50%   40%   30%
Football   9   7   6   5   4   3
MAX = number of contests (outside of tournaments) allowed per school by IHSA by-laws

[Atlanta Blue]

The result is that an official who works often (and is rated often) will generally receive a better ranking than an official who obtains a high average on just a handful of ratings.

Someone needs to explain math to the IHSSA.

[backjudge79]

The other state is a large state with many playoff crews assigned.  There is a complicated voting system (I'm sure it was designed by a former tax law writer) including local boards, ADs, coaches ratings, assigners, rules interpreters, board secretaries and I'm not sure who all else.  Crews are formed based on the individuals point totals with the highest ranked official at a position being placed on crew 1 and then working down.  The top 7 crews are supposed to work all 5 weeks including the final.  8 through 14 would work weeks 1 to 4 and so on.  The crews working multiple weeks can be comprised of officials from all over the state and can be assigned any game in the state.  Years ago we had one official from our board work a state final involving one of our local teams.  The crews only working the first week they attempt to put them together on a more regional basis and assign relatively close, even though three members of our board had a three hour drive to their game Friday.  This state was attempting to switch to a system of assigning playoff games to intact regular season crews but eventually saw the problems this could create and went back to their old system but with some heavy duty tweaking of the voting system.  Personally I think the ideal solution is the one used in the smaller state of assigning crews from the same board. Every playoff game has an observer evaluating the crew, but I cannot remember ever hearing of anyone being evaluated by the state during the regular season.  Crews have been removed from the playoff rotation due to a poor evaluation and another crew assigned the next game.  I believe there has also been cases of an individual being removed but the crew moving on when just the individual receives a poor evaluation.

Sounds like Ohio. Everyone knows in northwest Ohio the ratings and points mean nothing. One highly placed individual with a lot of say so picks who he wants even if they don't meet the minimum qualifications.

[CalhounLJ]

Oh, how I'd love for the state to evaluate crews instead of leaving it to associations to determine. However, like you said, with the number of crews that work on Friday nights in Georgia, it would be impossible to evaluate each one.

I'm not so sure you would like that. My state is attempting to rate and pick playoff officials and are doing a horrible job of it.  IMO, nobody knows the talent level of officials better than the coaches and the assigning secretary. But that's just me.


Sent from my iPhone using Tapatalk

[Magician]

I'm not so sure you would like that. My state is attempting to rate and pick playoff officials and are doing a horrible job of it.  IMO, nobody knows the talent level of officials better than the coaches and the assigning secretary. But that's just me.


Sent from my iPhone using Tapatalk

Coaches are not objective evaluators of the talent level of officials. I'm OK if they have some input, but making a correct but unpopular call can have a significantly negative impact on your ranking.

[Atlanta Blue]

MOST Coaches are not objective evaluators of the talent level of officials. I'm OK if they have some input, but making a correct but unpopular call can have a significantly negative impact on your ranking.

Let's not lump all of us together!

[Ralph Damren]

HUMAN NATURE : If you ask a coach shortly after a game, he'll remember the lone bad call(s) made against his team and not all the good ones made. They are more objective after the season is over or in preseason. We experimented way back last century by supplying post cards to the coaches for each game.....cards returned were 80%+ from losing coaches ......most were belittling a single call .....experiment failed

Since then we supply each school (both coach & AD need to sign off) a list of our officials (now 45) with 3 categories : Acceptable....Unknown.....Unacceptable with reason.

Several schools return the list with just a note saying "Send us whoever you feel is qualified"

Some send with unacceptable reasons of : "He works for the school", "His son is on the team", "He dates the coach's ex-wife" and the like.

Less will send with unacceptable reasons of : "Doesn't hustle", "Unapproachable", "Poor judgement", " Poor rule knowledge" and the like.. .......they are usually right

We require they have at least 25 on their acceptable list. That has never been a problem.

[FLAHL]

Wasn't it last year that the NFL officials were griping and moaning about who got the Super Bowl assignment?  If they can't eliminate the politics, griping, and moaning at their level, I don't expect us to have any better luck at the HS level.