Ratings, hate-raters, and you

I noticed a discussion topic in the Collaborative Corner this evening that I thought might be worth discussing on the blog.

Without going into too much detail, a topic was originally created with a parting mention of “no hate raters”, as well as a request for ratings at or above a certain level. Our moderation team works with a rather specific description of what hate rating is, which I had to clarify with the poster:

[A hate rater] is an individual who rates many pages with very low ratings, with the express purpose of bringing people down in the author ratings or story popularity rankings. What this means is that hate raters are trying to game the system or abuse a certain person (or people’s) rankings. Disliking a certain post on its own does not constitute “hate rating.”

This distinction is important — I don’t want people to feel like they have to give excellent ratings for everything written on Protagonize. That just waters down our ratings system and makes it worthless to everyone. A system where everyone gives each other 5.0s across the board doesn’t actually provide any useful feedback. In that case, removing the ratings altogether would almost be more worthwhile.

To paraphrase what I posted in the topic, ratings are only helpful to our authors if they’re honest and not self-serving. Either way, a negative rating on your writing does not mean you were the unfortunate target of a “hate rater.”

We do have a lot of folks on the site who take issue with getting a low rating… many of whom take the time to report this negative rating to the moderators as the hallmark of a hate rater targeting their work. Of course, their definition of a low rating and that of, say, your school teacher, university professor, book editor, or someone else grading their work seriously, may differ from that standard entirely. It does make me concerned about the ratings system we’re using here on Protagonize, and that maybe it requires a serious overhaul to make it more useful and less of a distraction to our authors. I get the sense that there are a number of members out there who are strictly interested in boosting up their ratings as much as possible, and not at all inclined to improve their writing.

This unfortunate trend leads to a lot of reciprocal “back patting” where no one actually ends up with realistic ratings because no one’s willing to actually provide an honest critique. Although I feel somewhat better about this when I see the contents of the Critiques Wanted discussion group, which has been an excellent area for feedback of late.

Back to the moderation side of things — most of you are likely unaware that our moderators have a variety of tools that allow them to discern a number of questionable patterns that arise on the site. We monitor things like spammers, users’ creation of fake or “mule” accounts, large quantities of overly positive ratings (yes, this happens more than you might think), and the same for overly negative ratings (what we refer to as hate raters.) And we don’t get involved in removing ratings, for instance, unless absolutely necessary.

Our moderators are very vigilant about this kind of behaviour. We do run into a number of thick-skulled individuals who think that creating fake accounts to rate up their own work, or to rate down the works of others, is something that we won’t notice. Repeatedly. Fortunately, our moderators are on the ball and tend to catch these miscreants pretty quickly, and they get banned, at first temporarily, and then permanently if they continue to abuse the site or its members.

But once again, let me re-iterate that folks should really not get riled up over single negative ratings. Especially not to the point of only requesting certain levels of ratings, or calling out users who dislike their work as “hate raters”. We look for trends; if there is no trend, it’s most likely an isolated incident. If it’s a middling rating and not even extremely low, it may be that someone just didn’t like your work — and yes, this actually does happen! In any case, it’s something we all have to deal with as writers. I’d just like for us not to get too hung up on numbers, and focus on quality. Maybe that’s just wishful thinking, but I’d like to think we’re all capable of critiquing honestly.

Getting higher ratings isn’t going to improve the quality of your writing — getting meaningful feedback will. If anything, artificially high ratings may give authors an inflated opinion of themselves, which isn’t all that helpful when they’re trying to grow as writers.

I’ll keep working on how to improve that feedback mechanism. What I’d like from the community is to focus on trying to provide better quality feedback.

This entry was posted in General, Site mechanics, writing and tagged , , , , . Bookmark the permalink.

16 Responses to Ratings, hate-raters, and you

  1. Pingback: Tweets that mention the protagonize blog » Ratings, hate-raters, and you -- Topsy.com

  2. Pingback: Dark Liquid

  3. Ana Cristina says:

    Great blog post. I’m glad you are drawing attention not just to the disruption that such “hater raters” can cause, but also to the fact that writers can sometimes become so consumed with bumping up their ratings that they lose focus on the very reason why they’re here in the first place — to hone their writing skills.

  4. SpookOfNight says:

    I think I know a pretty easy way of distinguishing “hater raters” from honest ratings.

    If you don’t like someone’s work, leave a constructive comment for them to improve – that way they can try and make their writing better without feeling they’ve been targetted.

    Simples, non? ;)

  5. Rac7hel says:

    The problem with @Spook’s suggestion is that readers shouldn’t feel obliged to explain why they dislike something. It’s certainly nice if they take the time to do it, but sometimes the work is just bad and they’d rather spend their time moving on and reading something else. Protagonize is here for readers as well as writers, and they have every right to think that someone’s work is crap.

    In other words, people are kidding themselves when they receive a 3.0 with no comment and then chalk it up to mindless hate rating instead of taking it to mean that someone actually thought their work was mediocre. This happens all the time, and more often than not, the thing with the medium rating is actually quite un-incredible. And would probably be a pain to critique.

  6. Tricia says:

    I think the point Nick is making is that *true* hate-raters don’t actually cause much disruption, or if they do, it’s limited, because the moderation team spot them quickly and take action to put a stop to it, and the same goes for those who create fake accounts to self-rate. The real issue is that many members are quick to protest against any rating below 4.00 with a ‘why are you hate-rating me?’ I’ve seen ‘No hate-rating’ in a lot of author guidance boxes too. Wanting everyone to give you a rating of 5.00 on everything is kind of missing the point of having a ratings system at all.

  7. SeeThomasHowl says:

    Personally, I know I’ve contracted if-you-don’t-have-something-nice-to-say-don’t-say-anything-at-all-itis to degree. If I read something I think is subpar, I just won’t rate it, and I’ll more avidly rate works I think lean toward the excellent end of the spectrum. And my intuition is that this is more or less a community wide sentiment that tends to drive overall ratings up to >4 (I would guess), and contributes to the false perception of anything <4 being a hate-rate.

    It's definitely a flawed system, but I'm not sure how it could be improved upon. Are even that it needs to be changed, necessarily. In a way it could be looked upon as an overflow of politeness, even empathy, as we're all touchy, fragile-egoed wannabe writers faced with the problem of attaching numeric values to one another's work.

    I think if there are more things like this blog entry that can set people straight about what does and doesn't constitute hate-rating, and more people are encouraged to rate according to what they really think without being so terrified that a fellow author might spontaneously combust at the sight of a 3.5, overall sentiments toward the subject may shift in a more healthy direction.

  8. Nick says:

    FYI — you guys may not have noticed Darkliquid’s trackback above, but his related blog post is very much worth a read…

    http://darkliquid.co.uk/2010/09/12/out-of-1-10-rating-systems-suck/

    Cheers,
    -nick

  9. Mark says:

    I am the original poster in question. I did ask for no hate raters, but I did not request that a low rating not be given. Although I didn’t state it correctly, I was trying to get the point across not to judge on opinion alone.

    I had noticed in the past that opinion based pieces receive low ratings in the range of 2.5 or lower. Even if the rating system was used as it was designed (which it isn’t), I would only give something that low if most of the piece was in shambles – grammar, concept, organization, etc …

    Which is what I plan on starting. But, don’t expect 2.5s or lower very often … because few pieces truly deserve that! I’ll find some, though. And, anything below a four will have a reason attached to it. Actually, any rating should have a comment, I think.

  10. Ixarux says:

    I haven’t been a great fan of the ratings out here. There have always been too many ‘perfect’ works out here.

    You get this feeling that a certain social norm has evolved out here that has been pushing the rating towards the higher scales; and anything low, no matter how objective, is treated with a sense of insult.

    As an economist, I am very fascinated in this sort of voting behavior in a community . Nick, I would really love to get in touch with you to know if it is possible for some sort of quantitative analysis to probably optimize the voting process or just simple analyze it from an academic perspective.

  11. Nick says:

    @Ixarux I completely agree — we’re doing some investigation into alternate ratings methods that would do the job better.

    If you’d like to get in touch, I’m always available — nick _at_ protagonize _dot_ com, as usual. I’d be happy to discuss the process with you in more detail.

    Cheers,
    -nick

  12. Juliia says:

    What about the people who go through the entirety of the Universal Truth and give 1s or 0.5s the whole way (typically because they consider it insulting)? Does this constitute hater-rating?

    • Nick says:

      @Juliia Yes, it does if it’s consistent across the whole thing. If you can send us a link to investigate (or easier still, just Flag the work and a moderator will look into it), we’ll see what’s going on.

  13. Anthony says:

    Well, what if you added a system where if someone wants to rate something, they also have to leave a comment. Or maybe they only have to leave a comment if they post a comment lower than 3.5, etc. etc.

    Something like this could fix everything.

    • Nick says:

      We’ve actually discussed the “forced comment” approach at length… The site already already (very clearly) suggests leaving a comment for 3.0 ratings and lower and that’s good enough with respect to “hater raters”.

      The problem here is two-prong and those “hater raters” are actually pretty rare. Self-rating is much more common and “low quality” ratings (i.e. ratings that are artificially inflated) are the worst culprit. Enforcing comments with <3.5 ratings would may actually make things worse in the long run. People are inherently lazy and if posting a below-3.5 rating (which is already very rare as everyone rates each other highly for the most part) actually REQUIRED a comment, even less people would do it. I very much doubt it would help the situation at all.

      It may prevent 1 hater rater, but it will equally discourage 10 normal raters from leaving a more appropriate rating.

      We have a discussion going on in the beta forums about how to re-invigorate the ratings system with some major-to-drastic changes. I’m going to close comments on this post from now on as I think the point has been made, in any case.

  14. Pingback: Commenting, fluffy kittens, and you | The Protagonize blog

Comments are closed.