This is an article I felt compelled to write after reading the comments on the Killzone 2 reviews – in particular the reviews by CVG and Edge. In it, I try to keep an impartial stance while talking about what exactly a review should do, and why exactly we should remove the ‘scoring system’ from them entirely.
Game reviews are subjective – that’s a fact of life. Not the meaning of life (unless your life is a particularly sad and games-orientated one), but a fact of it. The very nature of a review, and presumably why you read them in the first place, is that it is written by a person who is experienced in the field. This is why you buy magazines on the subject, it’s why you read video games websites. You feel like you should read the opinion of someone more experienced than yourself before spending your money on what could be a disappointing purchase.
In most cases, this is a good idea. On those games you’ve not heard much about, or those which might be on the edge of your interests within the medium. For example, Mirror’s Edge may be a first-person game, but it is nothing like a first-person shooter. Instead, it is more akin to a racing game and treated as such is much more enjoyable. However, people might buy the game expecting something more action packed and exciting, and therefore be disappointed. In this case, a review before a purchase would be highly beneficial. The game itself is often in two minds concerning exactly what it is, so the prospective buyers can hardly know what to expect before they buy it.
On the other hand, however, there are times when a review can do nothing but harm. This is when it concerns a big release, almost always a console exclusive, which has had a lot of coverage in the gaming press, and sometimes even the regular mainstream press. These games, such as Halo, Killzone, Metal Gear Solid, Bioshock and their subsequent sequels, represent much more than the games themselves. They also represent the console they run on, and by extension either validate or bring into question the reader’s choice of console. If Halo 3 was given a good review by a trusted publication, those people who bought a 360 feel like they made the right decision based solely on that. On the other hand, where Bioshock received less-than-perfect scores, many felt a misplaced sense of anger at the review, and by extension the writer, inevitably ending in semi-literate rants in the comments section of a website.
In these rants, the subject of console bias is always brought into question. On any website which takes it upon itself to review games on more than one platform you can find somewhere a string of comments claiming it scores games from ‘X’ console higher than games from console ‘Y’. Making such claims is almost always completely pointless in the first place, as it serves no purpose other than to stoke the fire of ill-informed Internet rage, but it’s also very hard to give any kind of proof to back up your statements. Yes, looking purely at the scores of games you might find that one console has games scoring higher than another console, but there’s always the possibility that those games are simply better in the view of the reviewer. Saying that, it could be claimed that the reviewer might be subconsciously marking up games for a certain console, but it is highly unlikely for an entire publication to be doing so. Take Edge for example – it has, I’m sure, a team larger than two reviewers. As a result, it is highly unlikely that the entire team will be pro-Xbox 360 or Anti-PS3. It is also highly unlikely that one member of the team would be trusted to write all the PS3/Xbox 360/Wii reviews, because I’m sure they are just as aware of possible bias as any of the readers. On top of this, Edge is one of the most trusted magazines in the business – it’s name has clout like no real competitor.
I feel that now I should come to the crux of the matter – the review scoring system that is inherent in pretty much every publication in existence. Let me start by saying that originally, I don’t believe that the idea of scoring was a bad one. It allowed people to get a quick overview of the game’s worth overall without studying the review in detail. However, this was when these ratings were generally 5/5, or at most 10/10. When it was at this stage, it could often be the case that a game could score 5/5 and still have flaws that would have meant that it wouldn’t score 10/10 or 100%, even though mathematically that is what it is equal to. When it was initially introduced, it was also never meant to be taken on it’s own – the idea was that you read the words that went with the score, and therefore saw why the reviewer had given the score they did.
It was only when people started giving scores out of 10 or, worse, 100 that it started to mean something to people other than a general feeling that the game is worth that much of some mystical currency. People began to look at a game rated 97% and comparing it to, say, a game with a rating of 95% and claiming that due to the 2% difference, it is undeniable that the first game is the better. People also started seeing scores like 70/100 as suddenly poor, whereas before it would equal the same as halfway between 3 and 4 stars, where the general consensus used to be that 3 stars is a worthy score.
Another problem is that each publication views each score slightly differently. While one magazine might call 70% a decent score, others might view at as average or middling, regardless of the mathematical value. This is only exacerbated when sites like Metacritic pool all the scores together, in an attempt to fairly give an overview of the gaming press’ view on a particular game. While a publication might view 70% as a decent score, it could easily be the lowest of a number of reviews and so despite saying basically the same thing as the other reviews, it comes under fire for the arbitrary scoring system it uses.
This is not the only place Metacritic fails – and this failing has perhaps an even larger impact due to the attitude towards scoring nowadays. On the metacritic website, it states:
All critic scores are converted to a 100-point scale. If a critic does not indicate a score, we assign a score based on the general impression given by the text of the review.
This means that a review that did receive a 5/5 rating in the old style, including some criticisms of the game that would be apparent on reading the review, would be converted into a 100% rating on Metacritic – something that looks much more like saying the game is ‘perfect’. Even more troubling is the phrase “we assign a score based on the general impression given by the text of the review”. Once you get to the stage that someone is giving an opinion of the opinion of someone else, you are bound to get pretty substantial inaccuracies. Only one person really knows the exact kind of score the review would award a game, and that is the reviewer. Who, seeing as they did not include a score in the first place, are unlikely to provide one. Often, the reason one doesn’t include a score at the end of a review is to avoid nonsense like this.
Metacritic goes further than this in weighting people’s opinions by introducing a colour scheme on the back of their overall scores – green for ‘universal acclaim’, yellow for ‘mixed reviews’ and red for ‘unfavourable reviews’. While this does not say anything about the games directly – mixed reviews can still contain reviews scoring the game very highly – it does give the impression that green coloured games are going to be better than yellow ones, which may not be the case.
Another worrying factor is the way Metacritic is weighted – this quote from their website explains:
The METASCORE is considered a weighted average because we assign more significance, or weight, to some critics and publications than we do to others, based on the overall stature and quality of those critics and publications. In addition, for music and movies, we also normalize the resulting scores (akin to “grading on a curve” in college), which prevents scores from clumping together.
Again, this is Metacritic having an opinion on which publications should be the most influential in the scoring process. This wouldn’t be a bid deal if it were between friends, but Metacritic is a widely trusted website which gets a substantial amount of traffic every day. Should it be that a weighted system is in placed to say who gets the biggest voice? Isn’t this why we got rid of weighted voting in politics – because we wanted everyone to have an equal say? Of course, Metacritic would doubtless say that they are protecting against the little publications making unsupported claims in order to deliberately bring a game’s score down, but they should still be allowed to. It should be obvious, through reading the reviews, which of them is telling the truth but it is the reader’s responsibility to do this, not Metacritic. Again, all too much stock is being placed in the scores by themselves – something not helped by Metacritic’s decision to emphasise the score, followed by a short excerpt of the review. It is highly unlikely that the casual visitor will stop to read even the exerpt of each review, let alone follow the small read link to the full one.
I appreciate that it might be time consuming for people to read the entire review for a game that they are only vaguely interested in in the first place, so I would suggest this – look at the score first, if it scores lowly, then you might not need to bother reading the entire review. However, if it scores relatively high, look back at the review and read it, to find out why it was scored so highly. Of course, I would recommend this for the low scored ones as well, in order to find out why exactly the reviewerr thought it was less that satisfactory, but I realise that this might be a bit too much effort for those just casually flicking through a magazine. For those who are desperately interested in a game, however, there is really no excuse to look at a score on it’s own and even comment on it without reading the review beforehand.
The label of ‘fanboy’ is thrown around a lot, particularly in what is known as the ‘console wars’. The majority of the hate, however, seems to converge on those high-budget console-exclusive games such as Halo 3 (Xbox 360) and, recently, Killzone 2 (PS3). What generally happens is one of three variations:
- A website gives a glowing review to the blockbuster game. It receives a perfect, or near-perfect score, despite some flaws which are mentioned in the main body of the review. As a result, a large number of fans from the other major console descend upon the site to express their outrage at the score, and ask why it scored higher that games X, Y or Z on their chosen platform. Questions of console bias, mental retardation and poor quality reviewing skills are brought up. Inevitably, the home console’s fan team show up and argue the case for the game/reviewer/console, despite the knowledge that had it been the other way round they would have acted in the exact same way.
- A website gives a generally positive review for the game. It receives a less-than-perfect score, generally detailing a number of flaws in the main body as justification for the score. As a result, a large number of fans from the home console descend upon the website, angrily questioning why it did not score as highly as games X, Y or Z on the opposing platform. Questions of console bias, mental retardation, poor quality reviewing skills or an inability to play the game well are brought up. Inevitably, the opposing console’s fans show up to defend the review, claiming that it is a fair and just review quietly gloating that it did not score as highly as games X, Y or Z.
- A website gives an average or poor review to the game. The Internet explodes with rage.
- In all of these possibilities, it is important to note that barely any of the angry commenters will have played the game, other than a demo, and nevertheless will act like they are better informed, more intelligent and have a larger penis than the reviewer. They will scream and shout about the review, but in the end will close their statement with something to the effect of “Well, it doesn’t matter, because I will preorder/have preordered/have bought it anyway”.
- It is also important to note that in the vast majority of cases, it would appear that the angry commenters have not read the review in it’s entirety, and instead have just looked at the score it ends with. This is evidenced in the way that almost exclusively start with something like “How can they give this a X/10?”. On occasion, they will pick at random a quote from the review which they believe to evidence their point.
I believe that by removing the scores from these reviews, not only would we force some reading upon these apparently near-illiterate Angry Internet Men, we would also allow them to make their own minds up about a game, rather than relying on a little number in their favourite publication. Instead, they might look through a review and think Hmm, this reviewer didn’t like this aspect of the game, but I think it sounds rather fun, or Hmm, this review enjoyed this feature, but it doesn’t sound like my cup of tea. That way, they would also stop becoming angry when they bought a game on the sole basis of the score a publication gave it and then discovering that it is, in fact, more of a racing game than a first-person shooter.
This review is worth every minute of your time. You should recommend it to others.