There are a lot of rating systems, so it can be hard to pick one. After some considering, I’ve been using this system:

⭐ Regret
⭐⭐ Meh
⭐⭐⭐ Satisfied
⭐⭐⭐⭐ Impressed
⭐⭐⭐⭐⭐ Transcendent

What makes for a good rating system? Let’s look at a few examples to draw some conclusions.

Thumbs up or down

The simple binary system is perhaps the perfect distillation of the critic’s task. Is the thing good or bad? No ambiguity.

Five stars then? Nope. I rate it ⭐⭐. There’s just too little nuance. It works for Siskel and Ebert because their rating is actually 4 distinct states:

Review Siskel Ebert
Two thumbs up 👍 👍
Siskel likes 👍 👎
Ebert likes 👎 👍
Two thumbs down 👎 👎

Even when they agree, there is nuance in their review that isn’t reflected in the rating. For instance, neither liked Brazil, but Gene liked it better on the second viewing and thought it could be saved with further editing. Roger just hated it.

Michelin Guide stars

The official description:

  • ⭐ “High-quality cooking, worth a stop” (Cuisine de qualité, mérite une halte)
  • ⭐⭐ “Excellent cooking, worth a detour” (Cuisine excellente, mérite un détour)
  • ⭐⭐⭐ “Exceptional cuisine, worth a special journey” (Une des meilleures cuisines, vaut le voyage).

Note that a Michelin star is a huge deal for a restaurant and it’s not a great scale for the whole range of things. It’s secretly a 4-point scale with almost every restaurant getting zero stars.

I rate this scale ⭐⭐ for anything besides high-end experiences. (For those, it gets ⭐⭐⭐, but you have to decide which system I used.)

Hundred point scale

So this is apparently how wines are rated. Metacritic uses 100 points too. It’s also effectively the Board Game Arena rating system because many people use decimals (6.3, 7.6, 8.1, etc.). This scale has plenty of nuance because there’s space in the rating to put slightly better things over slightly worse things.

The downside is a 100 point scale lacks even the semblance of conviction. An 82 point thing is clearly better than a 79 point thing. But where do you draw the line between worth the investment and not? Because of this, scores tend toward inflation. If 80 is the culturally accepted cutoff, a 79 feels harsher than it otherwise would be.1

Unless you review hundreds of things, this system is ridiculous. ⭐

Ten point scale

This includes IGN, IMDb and Famitsu2. This scale shares many of the same problems as the 100 point scale. People tend to let 1-5 go entirely unused and the real system works out to be:

  • 6 Bad
  • 7 Ok
  • 8 Good
  • 9 Great

In other words, a 4 point rating. I give it ⭐.

5 ⭐s

The 5 star system thrives in product and service reviews. If you ever look at Amazon ratings, you’ll notice that people tend to use 1 ⭐ and complain that there isn’t a way to give zero. 5 ⭐ ratings seem way more common than they have an right to be. J-shaped distributions occur because people tend to either like a product (it affirms their decision to buy it) or hate a product (because it disappointed somehow). My favorite reviews are 1 ⭐ reviews in the form “I ordered this for [some important event] and it was delivered late!” That must suck for you, but it tells me nothing about the product itself.

Eagle-eyed observers will notice this is essentially my system. If it’s to work, I need to be vigilant about giving more stars in the middle of the range than towards the end. So ⭐⭐⭐ out of 5.

Solomon scale

Named after Paul Salomon who explained the scale in Episode 184 of the Decision Space podcast. (Go to around 15:00.) The scale is simple:

  1. Needs more work
  2. Fine
  3. Good

He pitched it for game designers looking for ratings from play testers. It has the sharp decision of the thumb scale with a tiny bit of room for greatness at the top. It’s designed for regular things in exactly the way Michelin stars are not. For that purpose, it’s a ⭐⭐⭐⭐⭐ on my scale.

So why use the 5 ⭐ system?

Ultimately I want to sharply divide the stuff I really like from the stuff I don’t with a tiny bit of room for nuance. For instance I don’t recommend Cibele or Wasted Pizza. But one (Cibele) is substantially better than the other. A binary recommendation works better when aggregating many opinions. Something with a tiny bit more give suits me better.

Ratings are subjective even though they look objective

I suspect the 100 point scale feels good for many people because it feels objective. 99 > 98, so The Legend of Zelda: Ocarina of Time is better than SoulCalibur. I’m not so sure. When my friend pulled out his Dreamcast, we had a great time. I started Ocarina of Time on 3DS and it was, you know, fine. The best console game of all time, of course, is Metroid Prime Trilogy, but that only gets a 91 on Metecritic. 91 < 99, so either I’m wrong or the scoring system is broken.

Ultimately people who assign numbers to things like games or wines or movies are translating their subjective opinion into a form that can be compared to other people’s opinions. There are about a million different things that might be factors in the final score. My evaluation of Metroid Prime is heavily influenced by the circumstances that led up to playing it.

Here’s a (misguided) review from someone who hated it:

Garbage. Same games with no graphics upgrade, but you get to pay for them again! But wait, that’s not all, now we have MANDATORY MOTION CONTROLS. No controller support of any kind. Steer clear.

That earned the game a 0, which seems a little harsh, but:

Opinions are fossilized feelings

What’s left when I experience a book, movie or game? Well, memories, of course. Memories include a mix of facts and feelings. We like to think our opinions arise from facts, but more often they’re derived from our feelings. I felt a sense of wonder as I played the Metroid. The fellow who gave it a 0 felt aggrieved.

Star rankings don’t normally have a solid definition. We just sorta know what ⭐⭐ means compared to ⭐⭐⭐. But I decided to attach them to feelings. To repeat the scale from the top of this post:

⭐ Regret
⭐⭐ Meh
⭐⭐⭐ Satisfied
⭐⭐⭐⭐ Impressed
⭐⭐⭐⭐⭐ Transcendent

A lot of stuff sits in the 3-star category and there’s nothing wrong with those things. A 3-star toaster reliably makes toast every time you use it and there’s hardly anything to notice. Entertainment products (books, movies, TV shows, games, etc.) are different because the only thing they produce is memories.

Back in 2009, I posted on Facebook:

Last night we stayed up way too late watching “Escape from Alcatraz” starring Clint Eastwood. About a third of the way through the movie I started thinking it was a bit formulaic but a good thriller. It wasn’t until the final scene that I realized I’d seen the movie before. Now I can’t decide if it was good because it still held my interest or bad because I forgotten it.

Well now I know. It’s ⭐⭐. I still don’t remember much about Escape from Alcatraz and that’s a problem because other than a pleasant way to spend a few hours, what good is a movie that leaves almost no impression?

On the other end of the spectrum stand movies like 2001: A Space Odyssey that left an impression on me and frequently returns to my mind. It holds up to rewatching not because I’ve forgotten it, but because each time I experience it adds to my stock of memories. A clear ⭐⭐⭐⭐⭐ on my scale.

  1. Additionally, the US grading system probably influences people to assume 60 as a failing grade.↩︎

  2. Ok, technically Famitsu gives games scores between 0 and 40 because their scale starts at zero and there are 4 critics.↩︎