Clearly it has its limitations, such as personal subjectivity and patchiness of coverage, which means that it can only ever be used as a guide, not the prime factor, but it’s a lot better than nothing. I remember many years back pubs being put in the GBG (not locally) on the say-so of one member who may not have been there more than a couple of times. Given this, it’s a little worrying that there are still some branches who say they make no use whatsoever of NBSS scores in their GBG selections.
Before the NBSS, several local branches had instigated their own scoring systems, and I believe my own – Stockport & South Manchester – was perhaps the first. When I first joined the branch in 1985 I was struck by the amount of effort put into this, and by the fact that separate listings of Bitter and Mild scores were maintained, something unthinkable nowadays.
In fact we thought our system was superior to the NBSS, as ours was a 0-4 scale, allowing quarter scores, whereas NBSS is a 0-5 scale only allowing halves. Both set a 3 as the yardstick for the standard of beer you would reasonable expect in a GBG, but the NBSS rather stretches out the top end of the range, thus potentially making the averages more susceptible to distortion from the odd over-generous score. However, having used it for a few years now this doesn’t really seem to create a problem in practice.
Despite being a mostly urban branch, and setting a very low bar for the number of scores required for entry into the GBG – a mere six scores from two separate scorers over the course of a year – it is not unknown for well-regarded pubs to fall by the wayside on the grounds of insufficient scores. And it is very noticeable how half the total scores are accounted for by four or five popular multi-beer pubs. The days of CAMRA members spending their leisure time checking out a variety of pubs seem to be long gone.
In the July issue of What’s Brewing there’s a letter about the NBSS – shown on the right – which raises the idea that a score of zero should be given to a pub with no real ale. This seems to me completely wrong-headed. For a start, if the scores are going to be used to calculate averages, surely they should only refer to when a beer is actually bought and tasted. To my mind, a score of zero should represent undrinkable beer where the pub refused to either change it or give a refund.
If intermittent availability of real ale is an issue, then that is better picked up from local knowledge than beer scores. How is the branch meant to know whether a score of zero represents no real ale or dreadful beer? And the principle should be that each beer score relates to an individual beer, not the pub as a whole. It is wrong to say that a zero score should not be given if the pub has other real ales available.
This is something that needs further thought and clarification. The scoring scheme cannot be used for two different and contradictory purposes. And if it is upheld that zero means no real ale, then any zero scores must be excluded from the calculation of averages.
You could roll this logic out across all manner of things. Give a score to the cup of tea the missus makes you. A sandwich from the shop near the office. Then work out averages and stuff. Do on, fella, what's stopping you?
ReplyDeleteIf I was compiling the Good Cuppa Guide or the Good Butty Guide, then some form of assessment system would be very useful.
DeleteQuite. The the NBSS isn't for rating beer is it? It's for beer in pubs. So a pub with intermittent great beer will (given enough visits) score more than a pub that never has RA on, but less than a pub which always has great RA on. Which seems reasonable. If the pub wan't scored on those occasions when the RA wasn't on, but only when you were lucky enough to catch it, then it might end up displacing a pub (from the GBG) at which you could be sure of getting a pint (even if only good).
ReplyDeleteOn the question of averages, it's difficult to see why this should work. Is a 4 twice as good as a 2? Who knows? This scheme has more in common (I'd say) with a "5 star" ranking system. And the maths of converting a bunch of scores like that to a ranking isn't trivial. (It's a ranking we need to decide which pubs make the cut for the GBG). Y'all need a statistician.
Of course people are going to use it to calculate averages. What else are they meant to do with it?
DeleteYou don't want an "average". You want a ranking. (so you can decide which pubs get a slot in the GBG) The "average" is a (crap) way to get the ranking. Given the lack of any strong mathematical justification, it's just an magical process to absolve the committee of responsibility. Yay!
DeleteSo, how do we do this "ranking" then, O fount of all wisdom?
DeletePub has:
2 scores of 2
3 scores of 2.5
6 scores of 3
4 scores of 3.5
1 score of 4
Reducing it to an average of 2.96 makes it a lot easier to take in.
... ps. about that maths: Like this, maybe. But you do need a fair few scores to have any confidence in the final result.
ReplyDeletePersonally I only bother to submit scores when I'm in country pubs that may otherwise get few if any branch members visiting. I don't think our branch has a limit on the number of scores (I confess that I have never been to a selection meeting so I don't know for sure) but it is certainly a factor that will be discussed.
ReplyDeleteWe have the problem that our branch is partly urban. Most of our members live in Southampton. We have quite a few members in Winchester and in any case that's an easy place to go on the train from Southampton for a pub crawl so it is will covered. But the branch also covers a huge area of rural Hampshire which doesn't get anywhere near as well visited by active branch members. So I try to always submit a score when I do go to a country pub, certainly if the beer is GBG quality.
In general I think we do OK at GBG selection: there are rural pubs of GBG quality that miss out, but they're not clearly better than any of the Southampton pubs that get in. We have in the past had problems with POTY selection favouring Southampton, and made various attempts to improve things, which seem to be working: the last two have been rural pubs.
Incidentally mudgie, a branch has no excuse for not knowing what a zero means. (no RA). And neither should individual members be confused - is in black and white on the card they're filling in.
ReplyDeleteUnder our own superior system, 0 meant undrinkable, and that's still how I see it.
DeleteI have never been a camra member,so no nothing about beer scores,if i really like a beer i will note it down and if i really do not like a drink i put crap beer in my book.
ReplyDeleteI do disagree about this sort of scoring system as it could make a none real ale pub seem really bad when in reality it is a really good pub,i did a pub called Tinto Lounge in Bristol in April,there were no real ales on the bar,but there was keg Bath Ales Darkside which was a fantastic drink and the pub was also top notch in my opinion,i could have stayed in it for a session.
Then there are real ale pubs with five or more real ales on the bar,i went in the Old Post Office in Newark in June this year,i had a drink of Magpie Four for a Boy which made me shudder when ever i had a drink of it,it looked ok but it was crap,this pub may get a score of 1 or 2 making it better than Tinto Lounge which it is definatly not.
I have nothing against Camra i have been to every Nottingham Beer festival since 1982 plus other local beer festivals.
I thought that pubs got into or barred from the GBG if they did not provide Camra members with free food and the like when doing a survey visit.
I'm sure I recall a conversation YEARS ago about introducing -1 as "no real ale" and leaving zero for totally undrinkable and not replaced.
ReplyDeletePS. That pre-dated WhatPub where NBSS was the only way the branch could get feedback on a pub.
ReplyDeleteSometimes in organisations people want numbers to justify decision making. More often than not it really doesn't matter what numbers are so long as they sound fair and people accept them. People then consider their decision to be based on a technical process and not simply opinion and feeling. There is no difference in the quality of outcome but a feeling that there is, that there has to be, because the process was somehow "scientific".
ReplyDeleteThis is a process whereby a group generate some numbers to support what they are doing which is a subjective opinion on beer and pub quality. It is a process of group consensus more than a process of picking the best outcome.
If you go along to a beardy piss up and look at how the numbers are generated then the flaws are self evident. The group dynamics makes every group scoring process worthless. From dominant characters manipulating the result to weaker people following the lead of others, no meaningful information is accumulated. But you have some numbers you can pretend have meaning.
Then you get into what happens about the individual scoring. When at the beardy piss up identify the guy that does the accumulation, averages etc and ask him about it. Give him various examples like what happens if someone scores outside pre-existing expectations for an establishment etc. You see a guy not simply calculating averages but trying to come up with the right score.
Then go chuckle at how this lot go through this rigmarole for no other reason than when you question why the pubs in the book are a bit naff and not as good as other pubs you can think of, the justification is there. The process has been followed and somehow that is an accumulation of what a whole group thinks when actually it's what a hand full of people that really care about it think.
It's a fascinating process.
My Branch doesn't use NBSS. What Cooke says.
ReplyDeleteSo it's a matter of where you get the free pints, then? ;-)
DeleteThat kind of thing. No. We have no basic objection but membership shows no enthusiasm for it. But as Cookie says:
DeleteIf you go along to a beardy piss up and look at how the numbers are generated then the flaws are self evident. The group dynamics makes every group scoring process worthless. From dominant characters manipulating the result to weaker people following the lead of others, no meaningful information is accumulated. But you have some numbers you can pretend have meaning.
At every single one I have attended I have seen people want to hear others score before offering one themselves because they want to fit in and not disagree with the group. On the occasion that one person scores outside the range others consider acceptable you can witness another alter their own score up or down to compensate for another being "wrong" and trying to determine the end average. If you are a piss taker like me you might offer a dodgy score just to witness the compensatory behaviour of the wider group.
DeleteBut without *any* data then the problems of hearsay and favouritism are simply multiplied. Also it can provide the confidence to include more out-of-the-way pubs that most may not have visited.
DeleteIt's also amusing Cookie being so cynical about the machinations of CAMRA when I've heard his meat avatar really giving it the spiel to get young folks to join!
I never did the NBSS when it required filling in bits of paper. I did use the on-line version, then I heard some branches don't use it. To avoid wasting my time recording scores that might be ignored, I marked only pubs in our own area. I became fed up with this as I don't buy or use the GBG, and gave up altogether.
ReplyDeleteMy view, for what it's worth, is that scores from 0 to 5 should be where real ale is, or should be, available. Pubs that never serve real ale should perhaps be awarded an X, thus taking them out of the equation completely.
I agree with Nev on an X for no real ale.
DeleteStockport seems to embrace NBSS more than most and having observed beer scoring on a couple of recent staggers I'd say it works very well in helping the branch to choose the best pubs for the Guide. Scores for the same beer tend to cluster around the same score, which is what you'd hope rather than group-think.
This is a response to the letter from Peter Butler, our branch Pub Survey Officer, who takes the subject seriously and does a very good and conscientious job:
ReplyDeleteI wholeheartedly agree with Jim Scott's letter, and I'm glad this
topic has at last appeared in print. How can you score beer when there
was nothing to score?
The problem I and others have with the use of zero to mean "no real
ale available" is that because most branches simply put the scores
they receive into a spreadsheet or a database, they are counting a
zero meaning NRA as a score worse than any possible score for real
ale. This creates a perverse incentive for pubs aware of the scoring
system to keep selling deteriorating cask beer rather than taking it
off sale even if there's no cask ready to replace it. Going on serving
customers poor or undrinkable cask beer is clearly harmful to the
cause of real ale and to the reputations of the pub and the brewery -
and by implication to CAMRA for creating this stupidity. Brett Laniosh
still hasn't responded to this point even though I and others raised
it on the NBSS Yahoo group years ago - I'll try to locate what I
posted on this subject to quote it here.
I now don't count a "score" of zero as a score, I have to interpret it
as a presumed report that there was no real ale available on that
occasion, but I count this as a "no real ale day" - any pub can have
one or two of these in a year if there's a good reason, and in my
system it doesn't affect the pub's average score for real ale actually
sold. If an otherwise good pub has no real ale on occasions without
good justification, I will include a comment on our GBG selection
forms to indicate that the pub isn't a safe choice for the GBG. If a
pub lets its beer quality slip without taking it off sale, its minimum
score and its average will reflect this anyway.
CAMRA's online beer scoring used to include a tick box for "no real
ale", which showed up as ScoreType="N" in the downloads - so I could
treat the associated "score" of zero as a placeholder and not a real
score. At that time a zero score could still be used to indicate
totally undrinkable real ale with no redress offered (an extremely
rare experience but a far worse crime than only offering keg beer -
any undrinkable beer should certainly debar a pub from the GBG). The
current scale effectively forces scorers to give a positive score
(0.5) to undrinkable beer which has no merit whatever! (If I know this
has happened, I'll count it as the intended 0.)
Incidentally, one otherwise reliable and experienced scorer was
evidently confused by the use of 0 to mean "no real ale" - he put in a
couple of scores of 0 at the Magnet, apparently because he chose to
drink keg beers on those visits - he had neither encountered
undrinkable cask beer nor an absence of cask beer. I disregarded those
scores. I eventually put him straight on this (a bit rudely).
We don't score keg beers, but the present system can give non-members
the impression that we arrogantly regard keg beers as worthless.
They're not, they're just not what we're about, and keg versions of
cask beers are generally just an inferior version of the real thing,
not undrinkable.
Stockport & South Manchester branch pioneered the use of numerical
beer scoring, thanks to Phil Levison, and I think he and we got it
right - zero should mean undrinkable, not unavailable. Please be
assured that as far as possible I am continuing this policy.
Terrific, and as you say conscientious, response from Peter.
DeletePub visiting is all about enjoyment, but enhanced by the best beer, and getting the best beer in pubs into the Guide is fundamental to CAMRA aims.
Peter Butler has pretty much nailed it. However, since CAMRA changed the meaning of a zero score a few years back, it's easy to omit them from the average calculation and use the zeros as flags for unreliable availability.
DeleteThe other thing nobody has mentioned yet is the spread of scores. Consistency is important, so you want to see scores clustered at higher values rather than the two extremes of the scale. A mean value on its own isn't sufficient information, IMO.