March 4/06: Making the Grade

Grading is not an exact science, whether you're reviewing contemporary fiction or the latest site to feature cum-crazed co-eds swallowing big mouthfuls of nut. It has to be subjective. At the end of the day, different people are going to see things differently. That's why I believe in keeping things simple. Maybe not so simple as most film critics, but simple. I mean, could anything be simpler than thumbs-up or thumbs-down? Or X stars out of five? Or out of four? I wanted a rating system that was a little more complex than that, so when I started grading sites I had a scale that ran from 1 to 9. This still fucks a lot of people up. They think the scale here runs from 0 to 10. It doesn't. There is nothing lower than a 1 and nothing higher than a 9. 

Why? Here's the story (and this is all a digression anyway so you can ignore it if you want): When we first launched we had a colour-coded scheme where scores of 1-3 were red, 4-6 were yellow, and 7-9 were green. Like a traffic light. I thought it made sense. But when we did a makeover a year or so ago we lost the colours so now they all look the same.

Anyway, that's not what I want to talk about. What I wanted to talk about here was how these grades break down. Because they don't. I don't have a scorecard or scoring system where I give a site so many points for this and so many points for that and then add them all up to get a magic number. The scores here reflect my total impression of a site.

Does the fact that I'm using a scale that only goes from 1-9 make my reviews any less comprehensive or exhaustive than those other sites that (typically) scale from 0-100? Does a score of 73.4 out of 100 mean more than a score of 7 out of 9?

Well, no. Do you trust Roger Ebert less because he just says "thumbs up" or "thumbs down" instead of giving Brokeback Mountain a score of 83.7? Hardly. Furthermore, and to be honest, much as I respect the ratings of most other review sites (though I still wish more of them would make more use of the full range of scores available instead of giving practically everything 70% or better), I think a lot of their report cards are a bunch of smoke and mirrors.

They can also be quite hard to figure out. Take something like "updates" (which is a criteria most of them use). Yes, it's good for a site to update frequently. But what a "good" update schedule is can depend a lot on the context. And what does it mean when the review plainly states that the site does not update or hasn't updated in six months, but they still get marks for it? I've seen this many times. It's maybe not something you notice if you're just looking for the bottom line, but it amazes me. And confirms my sense that I was right to just stick with a single score.

Another occasionally puzzling category (again widely employed) is "exclusivity" or "unique content." Again, this can mean different things in different contexts (contexts that are never made clear in the review). Then there is the problem of how you weigh it. I just recently reviewed an amateur site that I strongly suspected had a lot of third-party content. I mentioned that in my review, and took it under consideration when calculating my own score. Then I saw another review that had given it a very low score in this category, but which never mentioned anything about how exclusive the content was in the actual review. Did they know something I didn't? If so, I wish they'd tell me what it is. I always give credit to other reviews when they point out something I wouldn't have got or missed. 

But most of the other reviews of the site that I read simply gave a score of 8 out of 10 or 13 out of 15 in this category without any explanation. I suspect a lot of the numbers I saw were just being made up to fill out the score without any actual consideration for what they meant.

Cynical? Maybe a bit. But then I looked to see how these sites were grading the the video archives like VideoBox and Videosz for exclusivity. And you know what? Those sites scored pretty well. Which is hard to figure, since neither of them are producing any of their own content. What gives? My own instinct would be to say you just can't score archive sites for unique content because that's not what they're selling. But that would fuck up the report cards. And so, since these are both terrific sites, they both get solid scores.

I'm not trying to slag the other review sites here. Like I said, I think most of them are very good resources. I guess what I'm doing is just suggesting you take a closer look at what has become the standard reviewing practice of scoring a site out of 100 by offering what appear to be remarkably scientific, comprehensive, and detailed analyses and grade breakdowns. I think you'll find that most of it is just filler, a lazy substitute for real reviewing meant to give an impression of diligence and thoroughness. And I'm not sure it's even all that useful. I mean, how many sites have separate categories for (what I consider to be) real deal-breakers like only having streaming video?  Whenever I see that I'm always mentally knocking a site down at least one full point (over 10%). But it's hard to see where it even gets calculated on most of these report-card style reviews, and if it does it likely only results in a loss of a couple of percentage points. That just isn't realistic.

The most any reviewer can offer is their impression along with the relevant technical information. Grading is not an exact science. Just because it looks objective and empirical and has a bunch of fancy graphics and other stuff doesn't mean it's any more accurate or honest than thumbs-up and thumbs-down, rotten or ripe.

Current Rant