KCBS Judging methods

Thanks MaxQ,

I want to be clear, the scoring system is as fair for one team as it is for another, but it does come down to more of the luck of getting a good table.

I've judged a couple dozen contests the last 18 months and was encouraged to do so by a good friend and a fine cook. I want to cook competitions and he told me to make sure I knew what I was getting in for. I do, and will cook.

What I thought seemed a little ironic, was judging the Royal Inv. this year. I got a chance to set with 3 judges, which were friends. I could tell by their judging book they only judge the AR and had started about 7 yrs ago, so yes they were trained to start at 6 and go up or down. Well we had a set of ribs that were way over cooked, but I felt had a decent profile and judged the ribs as presented.

Needless to say, one of the judges got talked to by the rep for being out of the 2 average range, but in all honesty, he judged it appropriately for how he was taught.

Reps seem to be good about balancing a table by the judges experience, but sometimes I wonder if they wouldn't be better served/balanced by how the judges were trained.

My wife and I are signed up to attend KCBS judging class April 13th. After attending the class I will reply to this thread on what/how we were trained to judge. Jer

Perhaps one distinction that should be mentioned is CBJ vs non CBJ judges. I recall a segment on BBQ Pitmasters a few years ago dealing with the The Royal. They had Myron on camera who was lamenting the fact that most of the 400+ judges were simply there to eat after the 3 minutes swearing in ceremony. He says in typical Myronese, "Hell, I wanna be judged by trained judges, not a bunch damned eaters." Who could argue his point?

The comp teams I know around New England always prefer to cook contests with a high ratio of CBJ's. So perhaps KCBS needs more CBJ's? I took the CBJ class a few years ago so I would have 1st hand knowledge what the judges were looking for. It was a 4 hr class + $60 (cheap). How do you level the playing field at a comp with CBJ vs non CBJ judges?

Good question, how do you make a comp use CBJ's when it is up to the organizer to use whoever he wants?

Lots of CBJ's around here if a comp WANTED to be 100%.

I'm a member of NEBS and a CBJ and I was taught to start at 7 and go from there.
BTW I have never been questioned on any of my scoring and haven't seen anyone else questioned either, this is the first I'm hearing about that.

Ken Dakai teach the class porkpie?

Judging seems to be an issue no matter what sanctioning body is being discussed. We do best when we cook our meat consistently and hit on all cylinders with our boxing routing. When we don't or if one thing is off, it affects our scores.

I've always been a believer that if we are getting sixes, sevens or even eights from more than one judge, something is wrong with that category.

quote:

Ken Dakai teach the class porkpie?

I took the class about and I'm guessing here, but about 7-8 years ago (My CBJ number is in the 1690's!) and don't really recall names as I was by myself and didn't know anybody up there at all MaxQ. It was a weekend deal with a cooking class one day and judging the next.

The only guy that I remember was the "Happy Holla'" guy with the beard and I don't know his name either. Sorry no help.

The reasoning we were gently given for starting at 7 was that everyone competing was spending money and time and effort to be there and so to have a even starting point to go from.
I agree with that, also an effort was made to explain how to keep the scores from swinging wildly up and down, to get the judges to narrow their focus somewhat so nobody would get blown away by errant 9's or 2's that weren't warranted.

All in all I thought the process was very fair and upright. But having said that I have judged contests where there weren't enough CBJ's and untrained people that just happened to be there, whether friends of friends or whatever have sat down and judged not knowing the first thing about what to look for or expect.

That is not right or fair for the competitor or the sport.

I've seen efforts by reps to try to level the playing field and school(ask) why a judge is out of the realm of normal. I applaud the effort that is made.

This may sound funny and a little strange, but I have been to 3 judging classes in the last 18 months and heck I've enjoyed it and have actually taken more thought away from it. It does seem that some type of follow up training can't be all bad, or at least those willing to do it share a common love of BBQ, isn't that why we all do it?

I took my KCBS judging class about 7 years ago. And help cook for one three years ago.

When I took mine,the class went more like a real comp, with the boxes having 6 diffenent types of cooking styles. I forced you to learn to judge the meat on it's flavor or merrits, and not looks alone. then after you gave it a score, you had to stand up and defend you score if it was out of line. Several "I like it hotter" or "I like it sweeter", were made. And the group discussed the rightness or wrongness of the comments.

But the session I helped cook for, there was no effort made to give the class a variety of flavors. But with the class getting the same meat in six different spots on the plate, the scores varied wildly.

I feel the starting point of 6 is a better place to begin. If the high score is a 9 then there is not much distance between the 7 and the 9.

But I do feel the table captains need to talk to anyone who gives score of 4 or lower, and they should be forced to give a comment card to address the judges issues with the entry.

At one event a few years several teams around us including ours, got scored on brisket that varied from 5's-9's on the same score sheet.

RandyE

Thanks Max,

I'm normally one of those "be part of the solution, not part of the problem" type of guys. But in this case, been there, done that, and got tired of being shot down. This isn't from me as moderator, this is from me as KCBS contestant.

I'll be shocked if the board responds. Over the last 10 years when I WAS a member, many efforts were started and taken to the board to address various issues, comments, concerns with the scoring system.

To date, it hasn't been fixed.

1. Judges and tracking scores. Why aren't judges scores tracked, so that those that obviously tended to one end or the other are addressed. Answer in the past was it wasn't worth it, too expensive to change the program, why do it? The competition committee makes a lot of the decisions, why not have a board of judges to it (and no, not master judges)

1a. Policing of judges. Reps to a poor job at best monitoring the judges and scores. In recent years it's gotten a little better, but even that meets with resistence. Just listen to some of the inside stories about scores at contests. AFTER the score is turned it, it's discussed, so that next time it won't happen. Uh, is someone tracking this judge for "next time".

2. Standards. There IS no standard for scoring or defining what is BBQ. As BBQ is subjective and KCBS wants to keep it that way (it's easier), the scoring system will always be up to individuals. And there in lies the problem that will always be the scoring problem. Individuals. My case in point. Chicken skin. For YEARS, the standard never addressed the tenderness of the skin. KCBS didn't. But judges in various forums started trending that way, discussing it in forum and it because a defacto standard. Is Bite through Skin a standard? Since when is THAT BBQ? I've seen judges give higher points for bite through skin than tender chicken meat. Really, I have to use Meat glue, cut the skin off, scrape the fat off, glue it back on, shape it in a muffin tin without a bone and boil it in butter and THAT'S BBQ? Sorry, not for me.

3. Judging class. This is THE big joke. I've cooked for many a class and the #1 requirement is to make the boxes have appearance issues. The classes are about taste and teaching what to look for in tasted. Tenderness has a little more time. The MAJORITY of the class is spent on illegal boxes. In the 200+ (estimate) boxes I've either judged of TC for, I can think of my 5 that had something illegal. Teach the TC's to look for illegal. Really, red kale? Shoot we had a hard time finding it to put in the box.

4. 100% CBJ's. To me, I don't like contests with this. Why? Because these judges are more likely to have pre-conceived ideas about BBQ. Not all, most likely not even the majority of them. Many of them would be like me, that you judge "as presented". But some will start at 6, some at 7, some at 9. Some will judge the Smoke Ring (yes they do and if they say so in public, they'll tell you in private they do) or they'll compare (many a time I've heard a CBJ say, I can cook better than that, so it just didn't taste like BBQ) or "I don't like chicken... I don't like sauce.... I don't like sweet" I wish I kept a book of the comments I've heard at the table after scoring BY CBJ's.

I would love to see change, but when it's a Very, Very, VERY subjective system, it's full and ready to have the judges do their own thing.

I'm voting with my personal $$$ and not competing nor joining KCBS for now. I like the mission, but if I'm not competing, there's nothing in it for me (no restaurant depot in OKC). My wife and I have judged a LOT and when we turn stuff in, knowing a general area it should score and to see the opposite, well we've seen it often enough to know it's TOO subjective. Same brisket getting 3 9's and also getting 3's (watch out Max, this happened TWICE in Great Bend)

I've suggested (as have others) in the past various ideas for improving, but the board has chosen over the years to not really tackle this. Example, put to a vote of the members what they would like to see happen with judging. Wouldn't matter. As the majority of the KCBS members are not really active members, then nothing changes.

I was on the NBBQA for over 5 years, another BBQ board and the same kind of challenges (different people with different agendas) always surfaced. The board means well, they just have a difficult job to do.

Well Said Russ.

I say the contests are much like a Hold Em' Poker Tournament.

You can know all of the odds and the psychology of the game and opponents but you are still at the mercy of the last two cards and the opponents correct or errant method of play that will actually kill your work.

Judges are much like a bad opponent who draws out on you with a poor hand to even start with.

Much like poker, If you cook consistent and often enough in contests, you will prevail. However, to the ametuer and infrenquent contest cooker, it is easy to say "Screw It" and save your money fighting the system.

quote:

Originally posted by David Qualls:
... However, to the ametuer and infrenquent contest cooker, it is easy to say "Screw It" and save your money fighting the system.

I know what you mean, but that wasn't the reason WE quit

The problem should be to grow the sport, bring in new teams, not scare them off.

The fact the % of teams that win multiple contests is because they cook 30 contests a year says something for them being consistent.

But not sure that makes it a fun thing.

I don't miss the judging, I miss the people. If I had unlimited funds I might compete again if I had a team, but not 100% my own $$$$. Just gets expensive doing 15+ contests a year with inconsistent, arbitrary judging.

We got a comment card last year that read something like; "the piece I got was not that good but in talking to the other judges afterwards, their's was better" WHAT? I think that meant, "I'm sorry, I didn't know what I was doing and messed up your score"
Anyway, I agree with Myron's theory that, CBJ or not, many judges are just there for the free food. I've only helped with one judging class, but like Russ said, it was mostly an appearance class. I would like to see a portion of the judging class dedication to the cooking process to give the new judges an idea of what was done to produce the samples in front of them.

I got a Comment card said my chicken was undercooked. I asked the rep if they tested the juices and did he verify.

"Nope, we were too busy".

Well I'm OCD about my chicken, I test EVERY piece I turn in and have never turned in undercooked. Now if there was red near the bones, maybe, but the juices were clear.

Standards, it's all about standards... or lack thereof

quote:

Just gets expensive doing 15+ contests a year with inconsistent, arbitrary judging.

That is an understatement if I ever heard one.

Just thinking out loud here but I bet it would be very interesting to see the results.

The KCBS board organizes a contest involving 20 teams, who are in on the plan. Each team is provided the same list of ingredients; meats, rubs, sauces and a small pantry of add-ons. The teams cook a KCBS timeline and turn-ins go to 10 CBJ's + 10 non CBJ's. The judges of course have no idea what's going on in this experiment.

The score cards are tallied 3 ways: combined CBJ, combined non-CBJ and total overall. I'd love to see those results.

I have a question. I have read the entire thread and find it all very interesting. I have never been to a competition nor have I judged anybody elses BBQ. I either like it or I don't.

How do you train 5, 10, 20 people to taste and score the exact same way? And then train them to apply the same standards to every piece of food they try?

I have been to a lot of beer tastings, where we describe what we taste. Some of the things people taste in the beer cracks me up. I have heard Urinal cake, horse blanket, and sweaty socks describe flavors in beer. I mention this, only because others would find qualities they loved in the beer. People taste differently. Right? So how do you train consistence?

Personally I think it's rediculous that presentation has anything to do with judging. It's pretty and all, but isn't it about the meat, and not who can arrange ruffage the best?

Just a couple of comments.I have been in these discussions,much like Smokin',so many times that I realize it is just killing time.

Some experienced cooks think maybe the approach in TX could be better,run like a chili cookoff.
Untrained civilians,given a quickie course and then senior cooks/finals table judges.

People in BBQ country know what bbq should taste ,smell,and eat like.
A TC can control the rules.

We mention Myron ,because he is a tv personality now.He often makes broad and colorful statements.He also ends them with "this is my business and I do it for the money".

Remember,Myron is arguably one of the best PORK cooks of all time.He spent many years doing that as a Memphis in May team.

Usually/often ,they have untrained judges and hopefully some trained judges to do the ONSITE judging of the team area,table settings,and 20 min presentation by the head cook.

We have judged Myron at onsite several times and he is the best at mesmerizing the judges.
Three years ago,at our "busman's holiday"where we take off from cooking and go judge a major Memphis contest,the wife/co-cook had the privilege of judging Finals and Myron is the best presenter-by far.

We have had the opportunity to cook beside Myron many times over the years.
We would question what he was doing to a product he would enter and his standard answer is"I don't eat this *****,it is for the judges.

Rick Salmon,another character from Olathe,always says"I'm just a whore to the judges".He means he does his best to produce his best and still not turn them off by anything in his entry.

Over a decade ago,Myron showed us how to French chicken legs ,or thighs so the judges would be amazed and not worry about the 7/8 of the score that was taste/tenderness.

Lots of areas of the country they have trouble getting CBJ s,in FL and much of the SE-maybe because of retirees-there is a waiting list,up to a year ahead of CBJs wanting to get on the judging list.
Many travel and judge every weekend.Maybe,they control too much of the scores?

No answers here,but CBJs have been good to us,been burned by rule changes,cold tables,etc.

Quick comment to the questions above the presentation is only 1/7 of your score,but folks do eat first with their eyes.Cooks often are going after 1/100 of a point out of 180 possible-so you strive to get these "given points"
You can teach standards for tender/texture as to cooked correctly.
Taste will always be subjective,so often it is to try and do nothing offensive to the judge.

Some of the things done for appearance,hopefully getting your boxes placed where there is even a small chance of helping you ,etc, may be for naught.

Some folks feel that if they are going to take the time to learn to cook their best,they try to "do no harm" to their entry.

When you look at the final scores ,at large/quality/competitive contests, where there could be 50 or more top cooks, the top of the category could all be grouped in a single point.

I won't talk about high or low scoring tables.

I've remained quiet because what do I know. But I pm'ed MaxQue and had some discussion much along the lines of Padrefan. Another novice to barbecue judging. But I come out where he does.

If someone ain't tasting all the entries, how can he/she know what's the best? There are rules, but you have to get inconsistency.

How many times have I heard, "Hope you can a good table!" or like Tom says, "Burned by cold tables?"

Everyone is doing their best. But tastes vary. What knocks the socks off someone may be just pretty decent to another.

Anyway, I've got a few buddies to agree to head to the Rib Cookoff in Reno, NV this September. Should be fun.

The problems Pags, lie in the lack of some judges to follow a simple logic, that seems hard for some judges to understand...you must leave personal tastes at the door and judge a product as it is"presented"...ie, you'll find NO or very few ribs that don't have some type of sauce on them. WHY?, because you can't find 5-6 judges that don't want to give you a fair judgement of what a unsauced rib should taste like...it's not what THEIR tastes are???

Like Tom said, why fight the system when others are winning by trying to give the judges what THEY like.

Too many times you see a piece of meat that is not cooked properly, also be hit for poor taste. Shouldn't that be judged entirely separate?

Honestly, I've noticed that most, not all, comp cooks make poor judges, why?...they want to compare the best they have ever ate to each entry...and theirs is always better. Give me 6 judges that have 4-6 contests judged under their belt and that is the table I'll prefer!

Pags,the thread is basically about KCBS,where the judges are not comparing product against the other entries.
In theory the traveling cook should receive the same score in AZ,as MO,as FL.

It could be possible that all six judges could give each entry identical scores.

Reference the Reno rib burn.

That is several vendors that pay a large fee to be there.I believe it is Curly's ribs supplied to them all.
The vendors bring as large a promo display as they can and setup.You will see vendors from PA,Ohio,etc with a truckload.

The goal is to sell as much to the inebriated,sunburned crowd as possible.The barrels of sauce poured on the ribs will be the most noticable item.
There could be ties and the computer can go out to a ten thousandth to determine the winner.There are also other possibilities to settle ties.

Other organizations you may compare against the Table,or of three teams you visit.
There are also other possibilities to settle ties.



Reference the Reno rib burn.

That is several vendors that pay a large fee to be there.I believe it is Curly's ribs supplied to them all.
The vendors bring as large a promo display as they can and setup.You will see vendors from PA,Ohio,etc with a truckload.

The goal is to sell as much to the inebriated,sunburned crowd as possible.The barrels of sauce poured on the ribs will be the most noticable item.

They do have a quasi-judging,where each vendor prepares some ribs apart from what the crowd eats.They are taken inside to the celebrity judges,Mayor,Fire Chief,,women's club,
etc.
Somehow the group designates what they felt was winner for the day.

I've never heard of a cookteam being there,altho some have very large vending companies.

We're talking humans here so there will always be inconsistencies. Judges are told not to compare entries to one another, but again that's just human nature to do so. There will never be a perfect system.
I sometimes wonder, and it's nothing more than a hunch, if some judges are more critical of the entries than they should be. Maybe that's how they were taught.
Last year our entry fees to winnings ratio was 5:1 so we're not in it for the money. We sometimes got scores back and thought the judges didn't know good bbq when they got it (of course I'm prejudiced )but in the end competition is something my wife and I do together and we enjoy it and the nice people we have met. So we keep plugging along and are happy when we do well.

While judges are supposed to judge each entry on its own and not as a comparison, My idea is to take their left side of the brain out of the picture and put a taste on their tongue that totally ceases their perception of what is a 9 in all three areas of judging.

I want to reprogram their thoughts on what is a 9 and use my entry as the baseline for what their creative mind tells them is the best.

I want to be the first entry on the sheet. I know that's a 1 in 6 chance but then again If I get my formula right, I will dominate the table anyway.

I'm in search of the "Holy Grail" of Umami! When I find it I will rule the BBQ World as long as I wish....

Thanks Cal and Tom, et al.

quote:

Originally posted by David Qualls:
...

I want to be the first entry on the sheet.

Well, in the old KCBS way, you could be the first if you entered a contest early. They would put the trays together to take to the table and put the entries in # order.

I for one wanted to be the last entry so I entered late. The idea being go ahead, eat the other five entries THEN taste mine.

Now KCBS just put them on the tray as they show up, no specific order allowed.

Funny ya say that Smokin'...TRUE STORY, When Sonja had just completed her judging class, she would come along to contests that I had already been signed up to judge and help out wherever she was wanted/needed.

About the third contest she did the entry window. She told me afterwards, ya know the big guy that helps Dave. Yeah, I know Dave's brother Martin Bouska(Butchers BBQ), why?

Well there were folks that wouldn't turn their brisket in on his tray...LOL!...go figure that one out.

Had something similar happen to me (not that I'm EVEN in a class with David).

When I was on a run, doing top 3 in ribs and pork for about two months straight I was walking up to turn in an entry and Donnie (Buffalo) and Rod (Pellet Envy) said... nope, we're gonna wait for the next pork tray.

Little secret a contests. I was always looking to see who's try I got on I learned it from Rod years earlier. I was more interesting in who's try I got on than fighting for position.

Yes I watched a couple of teams in Tecumseh stand with their warmer box for their entry and wait on a bunch of locals turn in and then literally run up to be the last in the tray. I thought it was a Chicken Sh*T move. And I really don't have a problem telling who they were. they are a country music group knockoff team name.

I'd rather lose with my head up rather than win and have to duck my head at the pay window.

Sorry you feel that way David, but if you spent a few years competing, seeing what everyone does to win. Spend a few years judging and see what judges do and how they judge.

When you're at a contest, and 10th's of a point matter, it's not Chicken Sh*T to be observant. Should it matter? No. Can it matter. maybe yes, maybe no.

Some people think using an FE is cheating.

I wish I could really say the things I've seen and others I trust have told me they've seen. It's just sad.

I turned in the best I could each week. Won plenty, lost plenty too.

I kept doing it while it was fun, but it's not fun anymore for us.

So I'll come out to Sam's next month, see everyone have a few beers and wait to hear the winners.

Smokin,
I agree and understand you. Its just the hyprocity of the issue. If I could pick, I want to be in the table that has what is considered the best of the contest so when I get hammered I know where the bar is set. And if I should win, then I know I'm on to something with some substance instead of just best of the draw.

If judging was done CORRECTLY, it shouldn't matter which table you hit or with who your product is turn in with, but as the veterans will tell you. A cook must control all the variables that he can, because it only takes 2 judges to ruin a good day/cook and I can tell you they do exist.

Oh, and don't EVEN ask me about why I didn't win a Grand Championship because the 4 (yes 4) KCBS reps at the contest mis-marked my pork entry (it was during that pork run) and I didn't get a pork score...

Still finished top 5, but that was with no pork score... and people were there that saw me hand it in, OH and they threw away the trays that had been turned in.

Now there is a rule call the Russ Rule that says "don't throw the trays away until after the contest".

I did get a check because of KCBS mistake, but I'd rather have the GC.

I may be a little sick but if I can put my box on the same tray and turn around and see a big time team right behind me going on the same tray I'm excited since internally I'm thinking, "Game on.". I'm in this to compete against the best and if I look at the score sheet and see I beat them at the same table I'm psyched. I agree with David that waiting for the new "local" teams to turn in before putting yours down is a strategy to some but to others it means they don't have faith in their product and are worried about it going up against another good box.

I think judges get a bad rap because we as cooks can be so inconsistent, often times not realizing it. One week I can score 169 in brisket and the next have it score 157 all because I changed one "minor" step in my boxing procedure. I've had chicken in the past get 1st only to find it sucking wind at the next five comps only to finally try all 10 pieces of chicken and realize half of them weren't consistently tender. I thought the damn judges were crazy the whole time but it was just the fact my chicken wasn't as good as I thought. As cooks, we have tasted incredibly less BBQ than judges, in most cases, and although we think it's awesome, it's only as good as what we think it is compared to what we have had.

One of my biggest lessons came when my Dad picked up a peoples choice rib from Rhythm n Que at a comp about a year ago. They absolutely kill it out here and at the time, the ribs I was making were by far the best I have ever had. One bite and I was like what in the world did they do to this rib?! It completely rocked my world as to what a BBQ rib could be. From that day on, I acknowledged the fact that I don't know jack about BBQ and someone elses BBQ may always be much better than what I even know possible. Prior to that day, I thought the judges were wacked since some days I would have some all 9's while another would be sixes and sevens. I couldn't explain it but once that realization hit me I could.

They mis-marked your entry?!?!!? I'm sure you were furious. That's crazy.

Did someone mention, at times the turn in box will actually not be able to be given out with the tray it was turned in on?

Sonja's judging cards(whole table), for brisket,got lost two years ago at the AR open, don't even know how that was resolved,oh well.

quote:

Originally posted by cal:
Did someone mention, at times the turn in box will actually not be able to be given out with the tray it was turned in on?

I have never worked the window but logistically speaking, other than chicken, what are the chances of your box hitting the same table as the rest of the tray? You may avoid turning in on the same tray and still end up on the same table as the team you were trying to avoid.

Yep!,

On small contests, they may have to hold the brisket trays to make sure that a few others can go to the individual tables. Seen them set 3 deep before having to switch out boxes.

There can be valuable strategies thou!