Subject: Digest for the period 5/26/2006 - 5/27/2006 Date: Sat, 27 May 2006 01:04:18 -0400 Table of contents ---------------------------------------------------------------------- 1. Number of entries and other musings (Tom Karnowski) 2. 50 points - another take (from South Africa) (Llew J van Rensburg) 3. how to contact the BJCP (Gordon Strong) 4. Objective Beer Scoring... Yet More (David Houseman) 5. RE: Objective Beer Scoring... (aboyce`at`mn.rr.com) 6. RE: Yet more... (John C. Tull) ---------------------------------------------------------------------- From: Tom Karnowski Date: 26 May 2006 03:00:55 -0000 Subject: Number of entries and other musings The honorable Mike Dixon asks "In response to 15 beers in a flight being too much, perhaps, but what makes 12 more appealing, and 8 even better?" Answer: The fact that there are fewer. Maybe 7 would be better than 8, and 6 would be better than 7. But too few may be bad too...bear with me...part of me thinks that's related to fellow TVHer John Peed's issue. I don't know when his beer was judged (was it first? was it last? was it 2nd?) but I know that when the first beer I judge seems really good, it probably gets a lower score than it would if it was the last one. If I have two beers to judge and one is WAY BETTER than the other, and the bad one isn't too bad, I may end up giving the good one a higher score than it deserves. Things get kind of relative when you are judging and even thought its not supposed to be that way (I should be able to get a brew and point-blank say where it lies on the score range), I know I have issues doing that and I bet 90+% of you do too especially at 9.00 in the morning. It makes me think of the last Blue Ridge brew-off I attended in Asheville several years ago, judging cream ales & American wheat beers. WE had a gob of them (probably over 12) and most were marginal, to me, but the other two judges who were maybe less experienced than me were giving them great scores. I was a little peeved until we got to the last one. It was totally fantastic. I tasted it and I instantly knew it was by far the best one, and it was a cream ale. It won best of show (how often does THAT happen?). It was brewed by George Fix, since deceased to the detriment of the homebrewing community. To me, that beer was a 50, and consequently the other beers should not have gone above the mid 30s if even that high. Dr Fix's beer kind of set the calibration point. I guess what I'm trying to say is that if we had encountered only 11 entries, we may have scored all the brews too high; and if Dr Fix's had been our first beer, we probably would have scored all of them too low (including Dr. Fix's although I suspect it would still have won best of show). I understand that John want to know how to make his beer better (and that's part of what makes him an outstanding brewer) but I think often when you have really good beers it gets harder to quantify what makes one better than another. You can say "that's unacceptable" but sorry, that's the best we will probably be able to do until someone comes up with a better idea. Finally, I like Spencer Thomas' food for thought and I really appreciate his posting. Nevertheless, I think he's too stingy. Sorry, I can't think of a better way to say it than that. He's only give out 5 scores above 45 out of 900 beers?? It reminds me a little of a time when I was judging in Birmingham AL and one of the judges - a professional brewer, I forget his name - kept slamming the beers. HE said a good commercial example of a beer should get about a 32. He also said it was common knowledge. IT was the first time I ever heard that and its the only time I've ever heard that. But it may fit into Spencer's stats. Further musing: assume Spencer's distributions are OK... Outstanding (45 - 50): 1% Excellent (38 - 44): 5% Very Good (30 - 37): 22% Good (21 - 29): 45% Fair (14 - 20): 17% Problematic (0 - 13): 10% ...we would expect that 28% of the entries would score 30 or more. Would that suggest that the optimal number of entries in a judge grouping would be 12, especially if you don't give ribbons to beers that score below 30? (actually I always hated that practice, competitions I organize give ribbons & prizes to the top 3 beers no matter what they score). Statistically that would mean you'd expect about 3 beers to be 30+ although such a small sample number may make that rule hard to follow. As a final observation, I still think 12 is too many for a judge team to evaluate in practice (see the first topic I discussed). Great discussion and interesting reads. ---------------------------------------------------------------------- From: Llew J van Rensburg Date: Fri, 26 May 2006 10:00:43 +0200 Subject: 50 points - another take (from South Africa) While studying for the BJCP exam, I summarised the points system in another way. It helped me keep track of the main issues: BJCP 2004 Guidelines (Llew's interpretation) World class example of style ( 45 46 47 48 49 50 ) * Flawless on technical merit * Fits style 100% * Great character * Balanced * Complex * Magical - everybody is going to talk about this wonderful beer afterwards Excellent beer ( 38 39 40 41 42 43 44 ) * Flawless on technical merit * Fits style 100% * Balanced * Complex * Lacks the magical intangibles of a world class beer Very good beer ( 30 31 32 33 34 35 36 37 ) * Very satisfactory beer, but has some of the following minor flaws: - Minor technical flaw - Minor stylistic flaw - Lacks balance - Lacks complexity Good beer ( 21 22 23 24 25 26 27 28 29 ) * Satisfactory beer * Generally fits style parameters * Upper end of range: - Few minor technical flaws - Few minor stylistic flaws (slightly out of style) - Lacking balance - Lacking complexity * Lower end of range: - More minor technical flaws - Stylistic inconsistencies flaws (more out of style) - Lacking balance - Lacking complexity Fair beer ( 14 15 16 17 18 19 20 ) * Drinkable, but quite a few problems: - Missed the style parameters - Has some off flavours and aromas - Balance problems - Some contamination - Other major flaws * At the lower end of range, beers exhibit more major flaws Problem beer ( 13 ): * Major problem (usually contamination) * Off-flavours overwhelms all other flavours and aromas * Undrinkable * Be courteous - have a lower limit of 13 Llew J van Rensburg Johannesburg, South Africa ---------------------------------------------------------------------- From: Gordon Strong Date: Fri, 26 May 2006 07:19:57 -0400 Subject: how to contact the BJCP > By the way, where is the BJCP leadership? Dave Houseman is not very > enthusiastic about this approach, but I know you're listening, Dave, > and > I hope you're noting the positive response. But where is everybody > else? What is the proper way to approach the BJCP leadership with > proposals like this? I'm sure all the leadership in the BJCP reads this forum, as well as several others. Personally, I like to see independent opinions and let conversations happen freely without trying to jump in too early and control them. When people with a BJCP title contribute, some folks instinctively back off their arguments. I'd rather see a range of opinions without this interference. This is an open forum for judges, not an official communication path to the BJCP; it serves a different purpose. Your argument is sort of like saying you wrote a letter to the editor of your newspaper but wonder why the senate didn't start drafting a new law. But since there's a direct question, I can give you a straight answer. If you want to approach the BJCP with a proposal, contact your regional representative. See http://www.bjcp.org/officers.html. You can even get stalker-friendly information in case you want to hunt us down at a competition near you, or in a random cafe in Belgium. You think that last part is a joke, but the same day as that picture of me was taken someone bumped into me at the Hop Festival in Poperinge and said she stewarded for me at the Spirit of Belgium competition earlier that year. That was cool... You can also try bringing up a conversation on the BJCP Forums, by talking to your rep privately or at a competition, or by coming to the AHA NHC and attending the annual BJCP member's meeting. Gordon Strong BJCP Mid-Atlantic Representative ---------------------------------------------------------------------- From: David Houseman Date: Fri, 26 May 2006 07:36:59 -0400 Subject: Objective Beer Scoring... Yet More Jon, Yes, I'm listening. I don't necessarily disagree but neither do I agree. So what does that mean? Well, I'm an engineer by training and practice, not a scientist. The difference? The scientist is more involved with the theory while the engineer with the practice. There are a number of competing elements in judging in a competition. First it is a competition so a key objective is to pick the best beers, to some objective criteria. And we do try to provide feedback to the brewer/entrant. But we have limited time to do this. Ten to 12 minutes in most competitions. The ideal world might have only 8 to 10 beer flights and 2 hours to judge them. In reality, competitions often don't have enough judges, get more entries than expected or the restaurant wants you out for the dinner crowd. There are pragmatic compromises that must be made. We may like certain styles but be called on to judge whatever the organizer needs to be judged. Go back and read Bev's excellent post today. A fine example of someone who gets it. I've judged with hundreds of judges, from other GMs to novices. There are styles of judges but in the end each is different. Some may start with 50 and deduct points. Others add up points. Many experienced judges can sense quickly the appropriate score for the each characteristic of the beer, provide full, succinct and accurate descriptions with useful feedback and do this for any of the styles and quickly, say 10-12 minutes. With 2000+ BJCP judges, don't expect everyone to judge the same way. Do what works for you. But hopefully we'll agree on the score within a few points and both provide helpful feedback to the entrant. Dave Houseman ---------------------------------------------------------------------- From: aboyce`at`mn.rr.com Date: Fri, 26 May 2006 10:42:33 -0400 Subject: RE: Objective Beer Scoring... Jon Tobey writes: >>As far as scoring a beer in competition I think that our goals >>are very unclear. Are we trying to pick the best beer, OR are >>we trying to pick the best beer AND help people make better beer. There are two "official" documents we can access for the answer to this question: http://www.bjcp.org/Judge_Procedures_Manual.pdf http://www.bjcp.org/SCP_JudgeInstructions.pdf ---------------------- >From the Judge Procedures Manual: =2E.=2Ethree primary goals for homebrew competitions: 1. To give the entrants valuable feedback on the quality of their brew = as perceived by the judges in order to enhance the quality of homebrewing.= 2. To provide training for aspiring beer judges. 3. To maintain valid standards of judging. >From the Judge Instruction sheet: The style adherence part of the scoring is separated into four sections: Aroma, Appearance, Flavor, and Mouthfeel. These sections are customarily used for recording OBJECTIVE (capitals mine - AB) descriptions of the entry. Beers should be judged as to how well they conform to the characteristics of their indicated style. The final section, Overall Impression, largely ignores stylistic considerations and represents how much you enjoyed the beer. Another good use of this section is giving suggestions on how to improve the beer. ---------------------- Looking at these quotes, we see no mention of "picking the best beer" as our purpose. That is the purpose of the competition, and a side-effect of our analyzing the beers. If we correctly evaluate and score each beer, then the "best beer" will be the one with the highest score. I concur with those who say we have to account for all 50 points. In BJCP prep classes held by the Minnesota Home Brewers Association, we are taught that any beer scored less than 50 IS OWED at least one or two specific, practical suggestions on how that beer may be improved. And that the lower a beer scores, the MORE specific suggestions that beer is owed. As for not having enough time in 12 minutes, I say - PRACTICE! I have received scoresheets from people like Kris England and Steve Piatz and other great judges who not only fill out the entire front side of the scoresheet in the time allocated, but also write several paragraphs on the back side. How do they do it? PRACTICE. Have TIMED scoresheet practice sessions at your club meetings. When you are trying a brand of beer that is new to you for the first time, look at the clock, then give yourself 12 minutes to do a complete scoresheet for it, including specific practical suggestions on how to improve it. Your ability to write complete scoresheets in 12 minutes will improve. Finally, by it's very nature, judging of ANYTHING is a subjective process.= =20 We try to capture it in the style guidelines, and we may recommend a way of judging (point deductions, top-down, bottom-up, etc), but when it's just a judge, alone with their thoughts at the table, they are going to judge each beer the best way they've learned. What we CAN and SHOULD continue to stress, however they come up with their scores, is that they NEED TO tell the brewer, in writing on the scoresheet, how they can improve their beer to score better next time. - Al Boyce BJCP, North Region Rep John Peed Writes: By the way, where is the BJCP leadership? PS - I tried to send this on 5/23, but JudgeNet had just added some edit function that disallowed email sent from my work, so I had to sign with a different email address. This has been one of the most useful threads on JudgeNet in some time. -------------------------------------------------------------------- mail2web - Check your email from the web at http://mail2web.com/ . ---------------------------------------------------------------------- From: John C. Tull Date: Fri, 26 May 2006 14:26:41 -0700 Subject: RE: Yet more... I have posted comments in the sections below. I also took the liberty of trying to reformat the text so it was not so horrible to read. Hopefully the mail server software will not re-reformat it so it looks like garbage again. > My last post got cut off after about the first third, but I'll avoid > rehashing it. > > Mike Dixon: I don't want to go on in great detail where a point > went. I > just want to know why it went. If a beer scores 44, I'd like to see > those 6 points justified - I want those 6 points; I want to zero in on > that big five oh. I would argue that a good score sheet will lay out the issues that account for those points. This is done implicitly, without the need to manually corroborate the difference with values. Again, any good score sheet should make this clear in the comments. > > As for the 37 and someone's term "mediocre" - if I'm shooting for 50 > (which I am), then I want to know why I'm 13 points short. When > you're > shooting for 50, 37 could almost be considered mediocre. But > that's not > really the point. Just assigning me a score of 37 and telling me > it was > a real nice beer, it was pale yellow, nice fruit, real clean, to > style, > yadda yadda - that just doesn't help me. And it doesn't say why or > how > the judge arrived at that score. Same as last comment. > > When I took the BJCP exam in February, one of the beers I judged was > pure, unadulterated bottled SNPA - I was 90% certain when I smelled it > and 99% certain when I tasted it. I somewhat flippantly scored it > something like a 49 (and said it was SNPA on the score sheet), > because I > consider it a benchmark for the style. I'm sure that will cost me, > because most homebrewers consider it a moderate, if not downright > mediocre, example of the style in today's outrageously hoppy brewing > environment. Be that as it may, let me score SNPA here, more > critically > (from memory), accounting for point deductions. Well, when you take the exam, your score is compared against the proctor scores in terms of points. So, if the proctors scored it 49 your in good shape. If they scored it 36, not such good shape. The comments are scored based on the quality of your feedback. Obviously, a 49 beer is harder to provide feedback on improvement, but you sure better be capturing all the nuances of the style that justify the beer being a 49. I would say that making guesses at the beer brand is a bad idea in general. Obviously a homebrew competition is about homebrewed beer, not commercial labels. It can be quite useful to say that a beer tastes, in my opinion, like a spot-on clone of a SNPA. The graders are likely not to make any deductions for your guess as exam beers are often commercial beers. Personally, I would note the inappropriateness of such a statement on a scoresheet as this is intended to be filled out as in a competition. > > Appearance: 2.5 Right color; brilliant; low head (-0.5), moderate > retention Aroma: 10 Really nice, delicately sweet malt with a > nice balance of citrus hops; hops are pretty light for style, though > (-2) Flavor: 16 Nice sweet malt, light caramel touch; good, > firm > bitterness; nice spicy/citrus hop flavor; superb balance, but hop > presence is subdued by today's standards (-4) Mouthfeel: 4 > Medium-light body, very crisp, a bit sharp in the finish (-1) Overall: > 8.5 Great light malt presence; superb balance; a terrific example of > the style, extremely drinkable, but lacks the hop punch that has > become > the hallmark of this style (-1.5) Total 41 I didn't give any > suggestions for improvement because it's obvious that the brewer is > perfectly capable, and to do so would be presumptuous, to say the > least So you just moved your near-perfect 49 to a 41, much more in line with the general grade curve for an excellent beer. This sounds like an argument against the original argument to use the 50-point scale. I would argue that you incorrectly deducted points for a lack of "hop presence" because SNPA is clearly within the statistics and description in the style guideline. This makes 4 of your points being deducted as quite subjective (another argument against yourself). "Sharp in the finish" is also subjective. Is this from hop astringency or harshness? My guess is yes, and this would be expected given the hops typically used in this style. Again, the "lack of hop punch that has become the hallmark of this style" rejects the range of possibilities for the style described in the guidelines. This is subjective as well. My point is that you can argue that the backwards point system is equally open to attack as the gestalt approach. Either way, you better be reasoning your score. > > Here's the thing that most people won't be able to stand, at least at > first: You write your comments and point deductions for a subsection > first, THEN you total the deductions, subtract from the max and that > gives you the numeric score. I still want to smell the beer and say, > "10", then start writing to justify that arbitrary number. That > number > always pops into my head before I start writing, but I get better > results when I write and deduct first and score later. You can do the > number first and then justify it if you want - that works too, because > if you cannot justify that number, then you have to change it. Perhaps some people need to think this way in order to verbalize their evaluation of a beer. I think judges that are doing a good job, no matter their rank, are going to validate their score in their writing anyway. Then this approach just becomes a layer of complexity that no one wants to deal with. > > By the way, where is the BJCP leadership? Dave Houseman is not very > enthusiastic about this approach, but I know you're listening, > Dave, and > I hope you're noting the positive response. But where is everybody > else? What is the proper way to approach the BJCP leadership with > proposals like this? Judgenet is a forum for discussing ideas related to beer judging and the BJCP, but there is no rule that "the leadership" have to respond in the list serve. If you want to push this idea, I would suggest contacting your regional representative and having it sent up the chain of command through that channel. You can find your rep on this page: http://www.bjcp.org/officers.html John Tull ---------------------------------------------------------------------- ***** Important Subscriber Information ***** To post a message to JudgeNet, send it to judge`at`synchro.com. Send plain text only, no HTML, MIME, encoded text or attachments. Make sure you use a meaningful subject. Quote only as much material as is needed for context. To manage your subscription, go to http://synchro.com/judge/subscriptions.html or send an email to judge-request`at`synchro.com with the subject: help judge. JudgeNet is also available as an NNTP newsgroup, go to news://news.synchro.com/synchro.judge