Calling all Math People: Ratings

Calling all statistics wizards! Dare I open this topic again?

Anime people are kinda geeky in general, so I figured I'd ask and see if anyone with a background in statistics out there has a reasonably hack-proof way to make ratings representative of the majority of opinions. These ratings seem to be deathly important to many people and as I've said before... if I don't care for a feature but everyone else does, I'll work on it.

Please propose a new formula for rating galleries. Requirements:

* 3 ratings to be computed from 1-10: collection, presentation, overall.
* I'd like to preserve our non-display of ratings for galleries less than 5 votes.
* Reasonable resistance to common attacks: bombing ratings one direction or the other and creating new users to bomb ratings one direction or the other.
* Throws out ratings that do not fall within a certain statistical ... variance? Am I using that word correctly? I think I am not.

You have access to all ratings and the date they were rated for every user.

There you go. Base it on trends, base it on clever stat math, base it on whatever. Feel free to include things like corrections based on max ratings, numbers of ratings, or a "median" gallery (but tell me how to select a gallery for calibration). If you can put it into numbers, I can code it and declare that infernal ratings system done forever. Feel free to get as technical as you want in any statistics discussion and I will make my best attempt to Google my way through your proposal to understand it.

Create a "rating board of geek dreads" to "give out ratings" to those that like ratings instead..Volunteer-in some frustrated or fired anime artists that have an axe to grind because CG has put them out of a job =)

I think everyone has figured out between donuts and coffee that giving and receiving ratings out from "anyone" with a grudge, jealousy, drunk, hung-over, or anime ignorant will not change anything. And also remember, those clever weasles that give out ratings just to get more meaningless karma points (heh,heh) could be included in that..scenario.

Just my opinion.. ^-^ not to rain on anyone's ratings parade.

Hey, you already know I think it's all make-believe. Ratings were intended to be about as significant as a Magic 8-Ball (TM) prediction... but yet, many out there want it to be reasonably accurate. So, in the name of public good and against my own personal preference, I would like to devise a new system that attempts to be more significant than the one we have now.

A few people have mentioned wishing that in addition to (perhaps an option for instead of?) ratings, individual comments could be left pertaining to their galleries or cels.

Maybe we could use the feedback system to help each gallery owner create a library of comments that could be used in a new module for each individual gallery's home page? Something of a "gallery comment of the day" where the comment itself could be posted, along with not only the name of the person who sent the feedback comment, but a link to that person's RS gallery (if applicable).

It might be a way to encourage people to use the feedback system to give more detailed comments about galleries or cels than can be gleaned from the rating system.

Mathematically, there's a couple of ways I would consider. The first one, without getting into formulas (though I can give those if you want) is to use a weighted average with the weights' values coming from the normal distribution function centered around their current score. What that would mean is when a vote is cast, the farther away it is from the gallery's current average, the less of an effect it will have on the score. Jason, I wrote a little Visual Basic application to do this calculation, and can send you the executable (or even the source if you have VB6) if you care.

The second formula would involve calculating the mean and standard deviation for all the current votes. Then when a new vote comes in, if it's more than a certain number of standard deviations from the mean (we'd have to play with that factor to get the right number), it could be thrown out. I suppose you could combine both techniques and you'd get a system where the farther away from the average you are, it gets counted for less until a point where it doesn't get counted at all.

That just deals with the mathetical issues. There's other things to consider, like do the votes only get kept for a certain amount of time? I suppose you could also weigh the votes by how old they are, but now you're talking about weighing the votes on two different variables, which makes things super complicated. I'd be for just keeping them for a certain amount of time, and maybe have the time be something each user can specify in their account settings. Maybe there would also be the ability to purge all the current votes for their gallery and start over.

Now, after saying all that, let me just say that I think that's taking things too far. To me, a straight average system would have been fine, and less programming work. Does it matter if people know how the formula is calculated? Since the ratings don't count for anything, trying to vote myself higher doesn't really benefit me at all. And I don't think I have any enemies that would vote me really low just to spite me, though I can't speak for everybody. I think the issue is just that people are seeing (or at least think they're seeing) strange things like voting a gallery higher than its average and seeing the numbers go down. A simpler system might prevent that.

Wow, I can't believe I wrote that much. That's so unlike me usually. I guess I saw the call for math geeks and couldn't resist. If anybody has read all the way to here, kudos to you!