Rating system woes

bad ratings

As you may or may not have noticed in the last day or so, we’ve been beset by a bit of a fracas concerning the rating system on Protagonize. I feel it’s best to clarify where I stand on the matter, and I thought I’d solicit some feedback from our author community in the process. You may not be aware that I’ve also been reviewing the existing rating system on the site for some time, for other reasons than what I’m discussing today.

To recap the issues, we’ve had quite a bit of flux in the site’s top author rankings list over the course of the last 24 hours. This is due mainly to some (mildly) underhanded manipulation of the ratings system by authors trying to bump themselves up the charts, and knock others down in the process. It’s mean-spirited, and it just leads to a negative vibe throughout the whole site. The best I can do is to ask people to be fair to others if they want to be treated well themselves. I can provide you the tools to rate others and try and narrow things down to avoid overly negative ratings, but at the end of the day, the community really has to police itself in this kind of situation.

I’d like to be very clear about this right off the bat: I will not remove or reject overly negative or positive ratings on anything. While the site is effectively a dictatorship (well, from a management and editorial perspective, at least), people have a right to their opinions, even if they’re extreme. So, there’s definitely no point in asking me to remove ratings. No one has as of yet (though I have received a couple of complaints about what’s been happening, as well as a ream of comments on my profile), but I want to make sure I’m heard loud and clear on that front. I don’t think that’s a real solution to the problem, anyhow.

While I’m pleasantly surprised to see people watching the author rankings this closely, it worries me that people are getting so hung up on their ratings. As I’ve reiterated to various people in the last day, the more ratings we have in the system, the less these negative or positive ratings spikes will have an effect. We’re seeing a major impact from it on author rankings right now because several of our top-ranked authors don’t have a huge amount of ratings, so the influence of 5 overly negative or positive votes on their branches or chapters will be much more noticeable. What I want people to keep in mind is that these abnormalities usually iron themselves out over time; as frustrating as they may be in the short term, they will slowly be less and less important the more ratings people add to the system.

Now, as many people may already be aware of, 5-star ratings systems such as the one we have on Protagonize are inherently flawed in a variety of ways. At the same time, they are a simple, clean rating mechanic, and they comprise what is likely the most common rating system on the web in the last decade. The flaws with 5-star systems have been discussed to death, but if you’d like an overview, Life With Alacrity is an excellent blog that discusses ratings systems in depth. There are several blog posts there that are worth a read on the subject: Using 5-Star Rating Systems, Collective Choice: Rating Systems and Collective Choice: Experimenting with Ratings. What it comes down to is that 5-star rating systems are in many cases the least of all evils; if you can define the rating values clearly, make it easy to rate content, and accrue a large sample of votes, they become more valuable. However, as the sample size decreases, and the clarity of the different rating values becomes less obvious, votes tend to bunch up at the higher end of the scale and aggregate ratings become less valuable.

I’ve also seen several people request that authors no longer be able to rate their own branches and chapters. I tend to disagree with this in principle, but I think that the two solutions I offer below may soften the impact of someone going through and rating all of their own posts 5/5 stars, without removing the ability to rate your own content altogether. Note that many large, well-known existing systems allow you to rate your own material, from Digg, to Reddit, to Amazon.

I have a variety of different ways at my disposal to attack this problem, the first being implementing a Bayesian weighted rating system that would work hand-in-hand with the existing ratings. What this essentially means is that items and authors with less ratings would be weighted back towards the average, and statistical anomalies like a batch of 1/5 votes would affect users less. Of course, at the same time, a batch of 5/5 votes would also affect the overall rating less. You see the point. When I make this change, you won’t see anything different on the front-end, but you may notice that average ratings become a little smoother overall.

The other option available to me is to move away from a 5-star rating system entirely, and switch over to more of an item popularity system. There are several ways I could do this, but I’m leaning towards more of a Reddit-style system of +/neutral/- votes. There would likely no longer be an average rating on stories, but instead an aggregate count of votes. This would remove a bit of the ambiguity from the current system, and ratings would be a bit more meaningful overall. However, every system has its pitfalls. If we go this route, it may be necessary to institute some kind of karma system to force users to be a little more conscientious when voting. What this means is that you’d be given a certain number of karma points based on the length of your membership, updated regularly, and you’d have to spend those points wisely to rate content.

I’d like to hear what our members have to say about this matter; feel free to make yourself heard here and I’ll use this as an informal poll of the community on the subject.

This entry was posted in Collaborative writing, Features, General, Site mechanics and tagged , , , , , , . Bookmark the permalink.

15 Responses to Rating system woes

  1. redhat says:

    I see your problem. I definitely like getting feedback from other users, but I’ll defer to whatever system you think is best. All I can say is, if anyone has a problem with something I’ve written, just let me know. I’d be happy to discuss your concerns. Going through and pinging all my posts with a bad score won’t really solve anything. This should be a friendly supportive place. It would be very cool if we could keep it that way.

  2. Crymzon says:

    One way of stopping people going on low or equally on high voting sprees is to make them leave a comment. This sounds rather draconian as you’re saying no comment no vote but I know I wouldn’t want to leave a comment on a ton of stories I was down voting and possibly hadn’t even read this also makes writers thing about what they’ve just read more which in turn might help their own work.

    Just an idea.

  3. Jimmy James says:

    I agree with redhat if theres an issue with my writing it doesn’t resolve issues just to stay silent and knock me down. I’ve definitely been affected by this as well as my ratings have gone WAY down in just a day. I suppose its not practical to be concerned about ratings but I’d like to make sure that I’m at least doing a good job, and just taking me down like that makes me feel like I’m doing a bad job. I think the evening out towards the average is the best line of fire.

  4. Jillian says:

    I agree with redhat; whatever system you think is best, Nick, is the system we should go with. I’m not particularly fussy either way. Although I am interested in ratings and I make a point to see who may or may not be on the top list from day to day, it’s really not worth bringing down the morale of the users nor tarnishing the great atmosphere that Protagonize has acquired.

    I’m not exactly sure why someone would give some really great authors such bad ratings except out of malolevence but as a helpful hint to other users/readers, perhaps its best to keep in mind, when rating others, to rate on a number of criteria such as spelling/grammar, subject, fluidity of writing and so on. I always try to keep these things in mind when I rate. That way, I can always give a fair rating; even if I don’t particularly like the topic of the story, I can still appreciate the skill of the writing.

  5. Rac7hel says:

    looks like a lot of stuff has been happening today while I was at the dentist getting a root canal… X(

    So… apparently the ordeal with redhat’s ratings has been settled. Still I would like to point out that I agree with Jillian… I tend to rate according to a few different criteria, including the ones she mentioned, as well as how the branch fits with the rest of the story, etc. (not simply how I feel about it, and certainly not to boost my own rating)

    I think the +/- karma thing would work out nicely if that’s what you decide to do, but I honestly don’t think there’s anything wrong with the 5 point system. As you’ve said… the more we rate, the less it will matter.

    I do wonder, as a side note, whether it’s intimidating to new members to see all these top rated and most popular authors displayed so prominently. I love the authors I know, but I wish there were a few more unfamiliar names writing on the same stories I’m reading. I see new members posting new stories, but I don’t see them frequently adding to other stories. Maybe that’s a different discussion altogether. I’m tired. Goodnight.

  6. nick says:

    I sat down with a friend last night over beers and we worked out a pretty decent solution. I think we’re going to be going to a Reddit-style model of +/neutral/- in the next little while. Existing votes will be ported over to the new system, likely 4-5 stars becoming +1, 2-3 stars becoming neutral, and 1 star becoming -1.

    We’ll have to work out the “karma” system (need a better name for that, too), but rating points will likely be accrued based on a combination of how many stories/branches/chapters you’ve posted, and how long you’ve been a member.

    - New members will gain 1 rating point on registration
    - Every week you’ve been a member, you’ll gain 1 rating point
    - Every post you make (story/branch/chapter) will add 1 rating point

    Every +1 or -1 vote will cost you a point, so you’ll have to make them count. Neutral votes won’t cost anything, but they’ll be stored more for future use (i.e. when we start providing suggested / recommended reading based on your ratings.)

  7. nick says:

    Regarding Rac7hel’s last comment, I’ll probably add a new “hot authors” block on the authors page to give a little variety. It’ll show the hottest authors in the last week, so we don’t necessarily see the same people at the top of that list all the time.

    I’m thinking of various ways to highlight posts from new authors a little more, too.

  8. Ganga says:

    I like the idea, Nick. Couldn’t have thought of a better way. It is a terrific site, and I’d like us to let go of this ratings/rankings obsession and continue to read and write stories, and have fun. Also fully agree with Jillian’s and Rachel’s comments about rating a story. I was getting the feeling recently that we’re some times rating the author rather than the content.

    Having said that, being new to fiction writing, I do like to see feedbacks/ratings on my stories.

    Crymzon’s suggestion about forcing people to leave a comment reminds me of some feedback surveys. For instance, if you voted negatively (like “disagree” or “strongly disagree”), you would be asked to explain why! Anyway, let’s not complicate it too much! ;)

    Also, like you said, you can provide the tools, but we, as a community, need to be responsible. So, the “karma” system (nothing wrong with the name either!) should help rein in everyone. Okay, back to protagonizing!

  9. nick says:

    Yeah, anything that requires a gut check before posting a rating is probably worthwhile, be it a forced comment (which I think may be a bit extreme in this case) to usage of some kind of karma points.

    Anything that costs you something to act on will probably filter out most of the crap, generally speaking.

    I’m going to see what I can do about tweaking the ratings system this weekend, after I finish up notifications (which are just about ready!)…

  10. Rac7hel says:

    What about not forcing us to leave a comment, but just allowing the author to see who has given you what ratings? I think that would filter the crap too.

  11. Jim says:

    I would like to see it enforced that people should have to comment when leaving a rating. I’ve had a few downplayed ratings, which some authors have now stood up to the plate for, but theres still a lot of anonymous ratings there.

    I’d like to know what about my work you don’t like and why you’re giving it a bad rating. It can really bring down the morale of a writer to get shitty ratings and no reason as to why.

    Either this or at least us being able to view who’s rating us, because right now its like a ‘rate and run’ system, where someone can rate me and never have to explain themselves because I’ll never know it was them.

  12. Dysphemism says:

    I actually like the idea of a forced comment – but not in the manner in which it has been expressed for the most part. I reckon it would make more sense if the forced comments were rejected by flagged moderation, and entirely anonymous.

    The karma system and activity-requirements seem crude. Some of us can get writer’s block for weeks on end and do nothing but read and rate – and do so quite thoughtfully. That can’t be done with such a system.

    So, if the ratings required an anonymous mini-critique readable to the public (and thus more likely for spam to be flagged among it and spam-ratings to be deleted).

    However, that doesn’t stop someone from coming across an amazing post with flawless grammar and format, intriguing characters and delicious symbolism and posting a dishonest comment of a twisted view. But we don’t live in paradise, yet, do we?

  13. nick says:

    You make a good point about users who post erratically but provide good feedback.

    Maybe a hybrid system where if you don’t have the necessary karma to rate something, you have to post a comment.

    The problem you run into with the required comment approach is suddently having to deal with spam comments, which may seem trivial to people used to using plugins on their blog to deal with it, but it’s actually a major undertaking when running a custom application like Protagonize.

    I’d have to tie into a third party web service like Akismet or something in that case, because I don’t have the time to sit there filtering every comment by hand, and I can’t expect users to do it for me via comment ratings.

    I may leave the existing system alone for the time being, because again, the more votes in the system, the less an individual can affect ratings. And with more users in the system, it’s becoming less and less of an issue.

  14. redhat says:

    Another option might be to have a breakdown of categories like ‘originality’, ‘execution’, ‘technical skill’ and the story’s score would be a total of all categories. Just a thought.

  15. Pingback: the protagonize blog: thoughts about collaborative writing » Blog Archive » An update from the front lines

Comments are closed.