Rating system woes

bad ratings

As you may or may not have noticed in the last day or so, we’ve been beset by a bit of a fracas concerning the rating system on Protagonize. I feel it’s best to clarify where I stand on the matter, and I thought I’d solicit some feedback from our author community in the process. You may not be aware that I’ve also been reviewing the existing rating system on the site for some time, for other reasons than what I’m discussing today.

To recap the issues, we’ve had quite a bit of flux in the site’s top author rankings list over the course of the last 24 hours. This is due mainly to some (mildly) underhanded manipulation of the ratings system by authors trying to bump themselves up the charts, and knock others down in the process. It’s mean-spirited, and it just leads to a negative vibe throughout the whole site. The best I can do is to ask people to be fair to others if they want to be treated well themselves. I can provide you the tools to rate others and try and narrow things down to avoid overly negative ratings, but at the end of the day, the community really has to police itself in this kind of situation.

I’d like to be very clear about this right off the bat: I will not remove or reject overly negative or positive ratings on anything. While the site is effectively a dictatorship (well, from a management and editorial perspective, at least), people have a right to their opinions, even if they’re extreme. So, there’s definitely no point in asking me to remove ratings. No one has as of yet (though I have received a couple of complaints about what’s been happening, as well as a ream of comments on my profile), but I want to make sure I’m heard loud and clear on that front. I don’t think that’s a real solution to the problem, anyhow.

While I’m pleasantly surprised to see people watching the author rankings this closely, it worries me that people are getting so hung up on their ratings. As I’ve reiterated to various people in the last day, the more ratings we have in the system, the less these negative or positive ratings spikes will have an effect. We’re seeing a major impact from it on author rankings right now because several of our top-ranked authors don’t have a huge amount of ratings, so the influence of 5 overly negative or positive votes on their branches or chapters will be much more noticeable. What I want people to keep in mind is that these abnormalities usually iron themselves out over time; as frustrating as they may be in the short term, they will slowly be less and less important the more ratings people add to the system.

Now, as many people may already be aware of, 5-star ratings systems such as the one we have on Protagonize are inherently flawed in a variety of ways. At the same time, they are a simple, clean rating mechanic, and they comprise what is likely the most common rating system on the web in the last decade. The flaws with 5-star systems have been discussed to death, but if you’d like an overview, Life With Alacrity is an excellent blog that discusses ratings systems in depth. There are several blog posts there that are worth a read on the subject: Using 5-Star Rating Systems, Collective Choice: Rating Systems and Collective Choice: Experimenting with Ratings. What it comes down to is that 5-star rating systems are in many cases the least of all evils; if you can define the rating values clearly, make it easy to rate content, and accrue a large sample of votes, they become more valuable. However, as the sample size decreases, and the clarity of the different rating values becomes less obvious, votes tend to bunch up at the higher end of the scale and aggregate ratings become less valuable.

I’ve also seen several people request that authors no longer be able to rate their own branches and chapters. I tend to disagree with this in principle, but I think that the two solutions I offer below may soften the impact of someone going through and rating all of their own posts 5/5 stars, without removing the ability to rate your own content altogether. Note that many large, well-known existing systems allow you to rate your own material, from Digg, to Reddit, to Amazon.

I have a variety of different ways at my disposal to attack this problem, the first being implementing a Bayesian weighted rating system that would work hand-in-hand with the existing ratings. What this essentially means is that items and authors with less ratings would be weighted back towards the average, and statistical anomalies like a batch of 1/5 votes would affect users less. Of course, at the same time, a batch of 5/5 votes would also affect the overall rating less. You see the point. When I make this change, you won’t see anything different on the front-end, but you may notice that average ratings become a little smoother overall.

The other option available to me is to move away from a 5-star rating system entirely, and switch over to more of an item popularity system. There are several ways I could do this, but I’m leaning towards more of a Reddit-style system of +/neutral/- votes. There would likely no longer be an average rating on stories, but instead an aggregate count of votes. This would remove a bit of the ambiguity from the current system, and ratings would be a bit more meaningful overall. However, every system has its pitfalls. If we go this route, it may be necessary to institute some kind of karma system to force users to be a little more conscientious when voting. What this means is that you’d be given a certain number of karma points based on the length of your membership, updated regularly, and you’d have to spend those points wisely to rate content.

I’d like to hear what our members have to say about this matter; feel free to make yourself heard here and I’ll use this as an informal poll of the community on the subject.

About this entry

You’re currently reading “Rating system woes,” an entry on the protagonize blog: thoughts about collaborative writing

Published:
04.09.08 / 10am
Category:
Collaborative writing, Features, General, Site mechanics
Tags:
, , , , , ,
Bookmark and Share