7 ms·

Analysis of whether IMDB ratings suffer recency bias

72 points by eoinmurray92 7y ago

minimaxir 7y ago
Recency bias came up a few days ago in a previous HN submission by the OP, and the OP also commented in: https://news.ycombinator.com/item?id=20099023 https://news.ycombinator.com/item?id=20099023 In that subthread, I posted a heat map data visualization I made of IMDB rating vs. release year, which is suspiciously similar to the one OP made: https://news.ycombinator.com/item?id=20099344 https://news.ycombinator.com/item?id=20099344 It's not plagiarism, but if the OP got the idea for checking for recency bias from the HN thread, they should explicitly credit that.
- mykowebhn 7y ago
  I agree with you. Does the opening sentence count? "I've read a lot recently about the recency bias of movie ratings, IMDB ratings in particular."
- KyleOS 7y ago
  Hey - not OP but I wrote the article. There was a lot of discussion about recency bias in that thread, including the graph you shared and I decided to expand on the analysis. Note there are a few other graphs in my post. I did just draw this up quickly yesterday. But you're right, sorry - I should have cited your comment in my post. Adding it now.
  - minimaxir 7y ago
    Part of the reason I make data vizzes is so people can expand on the ideas, so there's no issue there! (as long as proper credit is given to the original work)
    - KyleOS 7y ago
      Completely understand & I agree. Citation added.
  - bscphil 7y ago
    Hi, there are what seem to me (not a statistician) to be a number of serious errors in the article. The most egregious is the first conclusion, which threatens to undermine the whole thing. > There does not seem to be any strong connection between number of votes and a movie's IMDB rating. Well, this just isn't true. I happen to know this because I've looked at this data before myself. The issue is that you're cutting off the graph at a maximum of 60k votes with no explanation or even pointing out that you're doing it! I'm sure this is just an honest mistake, but it cuts out basically every movie that is actually popular! This slowly ate at me until it was enough to get me out of bed to redo this graph myself. I've uploaded my quick and dirty result in R here: https://i.imgur.com/TTuCFEL.png https://i.imgur.com/TTuCFEL.png As you can see from the graph, there's a direct and obvious link between the number of votes and the average rating. The reason I say this threatens to undermine the whole thing is that I also know from previous experience that more recent films tend to have a lot more ratings on average. This (naively) suggests to me that we should see some recency bias if only because more recent blockbusters get a much larger number of votes than others and the sort of people who vote on blockbusters have different standards for what makes a great movie. (Avengers: Endgame was briefly the #1 movie of all time on the top 250 list.) I don't have time to plot all the different graphs you did, but perhaps you can recheck your results after fixing at least this issue, and get back to us?
    - bscphil 7y ago
      Follow up: I redid one of the graphs to satisfy my own interest in the question. Instead of averaging over all the movies ever released like the article did, I averaged over all the votes. This was an attempt to answer the question of whether the average viewing experience of a film from year x is better or worse than that of a film from year y. I found that there has been a noticeable decline in the average rating of over half a point (out of 10) since ~1930 or so. https://i.imgur.com/bY9vPvk.png https://i.imgur.com/bY9vPvk.png I think this should be explained in a combination of two ways: * History acts as a filter. If I choose to watch a movie from 1947 it's probably because a lot of people over the years have said it's good. * To some extent, older movies may really be better than more recent ones. I still suspect that the blockbuster effect means that the movies people are mostly to watch are going to receive higher ratings than the average film overall. And this is mostly because people rating blockbusters are less critical than the film-viewing community as a whole. So while there might be no "recency bias" of the kind this article was looking for, there might be blockbuster bias where in the last several decades studios have figured out how to capitalize on a less critical / cynical audience. (That hockey stick graph of 8-10 ratings in the article is certainly suggestive.)
djtriptych 7y ago
I was recently thinking about this as Chernobyl (which was excellent to be fair) quickly became the highest rated TV drama ever.
- KyleOS 7y ago
  A recent post on the same topic: https://news.ycombinator.com/item?id=20098900 https://news.ycombinator.com/item?id=20098900
- watwut 7y ago
  I mean, if you can suggest other similar shows I will be only glad.
iforgotmypass 7y ago
I feel like author is missing the point. Shouldn't the recency bias show up in recent votes (while the movie is "hot" and trendy) vs. later votes?
- eoinmurray92 7y ago
  I think it's been interpreted as recent movies not recent votes.
PaulHoule 7y ago
It's well known that movie rankings do. This was key to the resolution of https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize The time series of reviews of movies like https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film) https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film) and https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Justice https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Jus... tend to have many early bad reviews by mainstream people who can't get past things that they find squicky and the lack of the usual "Hollywood Happy Ending" but that get better reviews from "true fans" later on who enjoy the fact that the directors tried to make something different even if it was flawed.
- deanCommie 7y ago
  Interestingly, the opposite of what happened with Star Wars: The Last Jedi. Casual fans and critics loved the subversion of genre tropes, story roles, character archetypes, and plot points. "true fans" reviled that the direction align with their vision for the franchise, and it sounds like Episode 9 will be a return to the comfortable and familiar.
- slyall 7y ago
  A quick Google shows a few sites that keep history of IMDB ratings. Unfortunately they seem to concentrate on movies actually in the IMDB 250 Some interesting ones: Avatar - http://top250.info/movie/?0499549 http://top250.info/movie/?0499549 Se7en - http://top250.info/movie/?0114369 http://top250.info/movie/?0114369 Heat - http://top250.info/movie/?0113277 http://top250.info/movie/?0113277 Star Wars: Episode VII - The Force Awakens http://top250.info/movie/?2488496 http://top250.info/movie/?2488496 Terminator 2 - http://top250.info/movie/?0103064 http://top250.info/movie/?0103064 Note: There seems to be a big shifts in movie ranks in May 2005 and July 2012, probably due to a change in the ranking formula.
fileyfood500 7y ago
I see from one of the charts that the average movie rating from 2019 is over .5 higher than the previous 5 years, in what appears to be a big jump. Is it possible that recency bias is only relevant on a shorter timescale? It looks like ratings are higher for movies that are a few months old, based on the data. I wonder if this is generally true, or if early 2019 happens to have higher rated movies.
- ac29 7y ago
  Sure - for example, fans of a particular actor, director, or series are more likely to go see a movie soon after release in theaters as opposed to later via rental or streaming. Fans are likely to rate higher than the general public, so over the first, lets say, 3-12 months, ratings often trend down.
- justadog 7y ago
  Fake reviews and ratings must be a factor. For big movie releases it is hard to find IMDB reviews that seem genuine. I assume good portion of the ratings are fake as well. It must have an effect given how common it is.
olalonde 7y ago
Doesn't this analysis rest on the assumption that movies are equally good each year? Maybe I suffer from nostalgia (or survivorship bias) but I'm not sure this is a good assumption.
- drollcrops 7y ago
  Good is relative. And you could easily rabbit hole on this. Do people have the same sense of quality? Have people changed their rating scale based on production/effects quality? Etc etc. Probably best to just take the subjective rating.
- jdsully 7y ago
  The problem with past movies is you only see the good ones. I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies - because while both suffer from horrible plots usually modern movies at least have interesting visuals. However for certain categories the past was most definitely better. Dramas being the main one. Without a lot of fancy effects the plot was the only thing you had to carry the movie. They also don't beat things over your head and can embrace subtlety in a way modern movies do not. I also really liked the 70s trend of movies that didn't really have a satisfying ending, you just get to experience the lives of people who don't really improve. 5 easy pieces or The Last Detail fit that category very well.
  - EugeneOZ 7y ago
    So you want to say that Shutter Island is a bad example of drama? It's all just old-farts rants, nothing more. Everything is evolving. In old movies actors play often too dumb, just few of them have good examples of how to play a role. Some "legendary" movies from the past would fail in theaters nowadays without their legendary reputation. We have more movies right now, more choices, so our demands are higher - otherwise we will not have enough time to watch them all. Also, we have much more choices of how to watch them - age of digital distribution. Higher demands give higher quality, movies evolve both in visuals and in acting art.
    - 7y ago
mynewstartup 7y ago
In terms of shows, Sopranos has long been dethroned, possibly due to this effect. Then again, production values will only get better over time, so maybe the latest shows are of the best quality overall. Best line: Fucking Queeahs! (Paulie throws chair at his murder victim spirits)
in3d 7y ago
I wonder if recording ratings now and then every month or so would show a decline in ratings. IMDb must know already.
dahart 7y ago
> Number of Movies By Ratings I’m dying to see all of these as percentages rather than absolute numbers! I was even thinking that while scrolling down through, and then there the totals are right at the bottom. @KyleOS are you up for normalizing them?
bscphil 7y ago
Incidentally, in case the creator sees this, this site is borderline unusable for me on mobile Firefox. Text gets cut off in strange places, I can't zoom out properly to see all of the animations. "Request desktop site" seems to fix some things but break others - at least I can zoom all the way out but the text becomes hard to read. Situation might be the same on Chrome (didn't check), but regardless it could use some work since it's pretty broken as is.
- eoinmurray92 7y ago
  Hi - founder of Kyso here, thanks for your feedback we will get to work fixing those issues now
  - bscphil 7y ago
    Thanks! After playing with it a bit it's clear that part of the problem is that the charts are getting generated on the fly and that's very performance intensive. My brand new Android phone is just choking on the page, to the point I can actually feel the phone overheating. Perhaps static images would be an improvement for mobile devices, with the option to load the full data on request.
8innovate 7y ago
This is definitely not the 'recency bias' I expected. I've always noticed that when a show is released it gets WAY higher ratings in the first week or so of the release date. I've learned not to trust the ratings until at least a 2 weeks after a release. It looks like this analysis focuses on the year a film was released. This seems like it would be a lot harder to determine because film quality changes (subjective) and the audience writing reviews is probably changing over time.
ajhurliman 7y ago
An alternative way of reading the same data: "Analysis of whether movies are getting better over time"
- richardhod 7y ago
  The problem with your formulation is that 'better' is perhaps the most subjective word there is. Define it in measurable terms, and perhaps you have a reading there. There are many factors to consider which may lead you to find things less clear-cut, for example here's one or two, with some sub-factors - Modern audience Preference for familiar patterns in: (eg) -- acting styles -- music used -- visual tricks and norms -- special effects styles -- acting styles -- soundtrack loudness -- colour and high resolution -- High Concept movies -- Fantasy and Comic-related films -- modern movie themes and topics - Internationalisation of movies, so that fans from abroad prefer modern films which are more universalised and less American-specific (see literature)
  - ajhurliman 7y ago
    Right, I'm not asserting that my alternative title is the correct one, I'm lampooning the assertion that IMDB is "suffering from recency bias" as opposed to a more neutral claim like "ratings for more recent movies are higher." Answering the question of _why_ they went up is a lot harder to prove.
ddffre 7y ago
Of course, there are plenty of examples of that. Like the assassination of the NK leader movie, which at first received 10 star rating. Now look at it. Sane goes with all the new blockbuster movies, you should usually wait awhile for the real ratings to show.
z3t4 7y ago
What I find most amusing is ratings (and also awards/nominations) before a movie has been released.
IAmEveryone 7y ago
Series tend to be rated higher than movies, probably because people that don't enjoy them stop watching and rating after one or a few episodes. That effect may well be spilling over into movies, considering far more movies are now sequels.
ravenstine 7y ago
Great job! I'd love to see a similar analysis for Rotten Tomatoes since there's separate metrics for critics and audience, and I find RT ratings to be more extreme than IMDb. I'm often surprised to see how many bad movies get a rating of ~6 on IMDb.

72 points by eoinmurray92 7y ago

Recency bias came up a few days ago in a previous HN submission by the OP, and the OP also commented in: https://news.ycombinator.com/item?id=20099023 https://news.ycombinator.com/item?id=20099023 In that subthread, I posted a heat map data visualization I made of IMDB rating vs. release year, which is suspiciously similar to the one OP made: https://news.ycombinator.com/item?id=20099344 https://news.ycombinator.com/item?id=20099344 It's not plagiarism, but if the OP got the idea for checking for recency bias from the HN thread, they should explicitly credit that.

mykowebhn 7y ago
I agree with you. Does the opening sentence count? "I've read a lot recently about the recency bias of movie ratings, IMDB ratings in particular."
KyleOS 7y ago
Hey - not OP but I wrote the article. There was a lot of discussion about recency bias in that thread, including the graph you shared and I decided to expand on the analysis. Note there are a few other graphs in my post. I did just draw this up quickly yesterday. But you're right, sorry - I should have cited your comment in my post. Adding it now.
- minimaxir 7y ago
  Part of the reason I make data vizzes is so people can expand on the ideas, so there's no issue there! (as long as proper credit is given to the original work)
  - KyleOS 7y ago
    Completely understand & I agree. Citation added.
- bscphil 7y ago
  Hi, there are what seem to me (not a statistician) to be a number of serious errors in the article. The most egregious is the first conclusion, which threatens to undermine the whole thing. > There does not seem to be any strong connection between number of votes and a movie's IMDB rating. Well, this just isn't true. I happen to know this because I've looked at this data before myself. The issue is that you're cutting off the graph at a maximum of 60k votes with no explanation or even pointing out that you're doing it! I'm sure this is just an honest mistake, but it cuts out basically every movie that is actually popular! This slowly ate at me until it was enough to get me out of bed to redo this graph myself. I've uploaded my quick and dirty result in R here: https://i.imgur.com/TTuCFEL.png https://i.imgur.com/TTuCFEL.png As you can see from the graph, there's a direct and obvious link between the number of votes and the average rating. The reason I say this threatens to undermine the whole thing is that I also know from previous experience that more recent films tend to have a lot more ratings on average. This (naively) suggests to me that we should see some recency bias if only because more recent blockbusters get a much larger number of votes than others and the sort of people who vote on blockbusters have different standards for what makes a great movie. (Avengers: Endgame was briefly the #1 movie of all time on the top 250 list.) I don't have time to plot all the different graphs you did, but perhaps you can recheck your results after fixing at least this issue, and get back to us?
  - bscphil 7y ago
    Follow up: I redid one of the graphs to satisfy my own interest in the question. Instead of averaging over all the movies ever released like the article did, I averaged over all the votes. This was an attempt to answer the question of whether the average viewing experience of a film from year x is better or worse than that of a film from year y. I found that there has been a noticeable decline in the average rating of over half a point (out of 10) since ~1930 or so. https://i.imgur.com/bY9vPvk.png https://i.imgur.com/bY9vPvk.png I think this should be explained in a combination of two ways: * History acts as a filter. If I choose to watch a movie from 1947 it's probably because a lot of people over the years have said it's good. * To some extent, older movies may really be better than more recent ones. I still suspect that the blockbuster effect means that the movies people are mostly to watch are going to receive higher ratings than the average film overall. And this is mostly because people rating blockbusters are less critical than the film-viewing community as a whole. So while there might be no "recency bias" of the kind this article was looking for, there might be blockbuster bias where in the last several decades studios have figured out how to capitalize on a less critical / cynical audience. (That hockey stick graph of 8-10 ratings in the article is certainly suggestive.)

djtriptych 7y ago

I was recently thinking about this as Chernobyl (which was excellent to be fair) quickly became the highest rated TV drama ever.

KyleOS 7y ago
A recent post on the same topic: https://news.ycombinator.com/item?id=20098900 https://news.ycombinator.com/item?id=20098900
watwut 7y ago
I mean, if you can suggest other similar shows I will be only glad.

iforgotmypass 7y ago

I feel like author is missing the point. Shouldn't the recency bias show up in recent votes (while the movie is "hot" and trendy) vs. later votes?

eoinmurray92 7y ago
I think it's been interpreted as recent movies not recent votes.

PaulHoule 7y ago

It's well known that movie rankings do. This was key to the resolution of https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize The time series of reviews of movies like https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film) https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film) and https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Justice https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Jus... tend to have many early bad reviews by mainstream people who can't get past things that they find squicky and the lack of the usual "Hollywood Happy Ending" but that get better reviews from "true fans" later on who enjoy the fact that the directors tried to make something different even if it was flawed.

deanCommie 7y ago
Interestingly, the opposite of what happened with Star Wars: The Last Jedi. Casual fans and critics loved the subversion of genre tropes, story roles, character archetypes, and plot points. "true fans" reviled that the direction align with their vision for the franchise, and it sounds like Episode 9 will be a return to the comfortable and familiar.
slyall 7y ago
A quick Google shows a few sites that keep history of IMDB ratings. Unfortunately they seem to concentrate on movies actually in the IMDB 250 Some interesting ones: Avatar - http://top250.info/movie/?0499549 http://top250.info/movie/?0499549 Se7en - http://top250.info/movie/?0114369 http://top250.info/movie/?0114369 Heat - http://top250.info/movie/?0113277 http://top250.info/movie/?0113277 Star Wars: Episode VII - The Force Awakens http://top250.info/movie/?2488496 http://top250.info/movie/?2488496 Terminator 2 - http://top250.info/movie/?0103064 http://top250.info/movie/?0103064 Note: There seems to be a big shifts in movie ranks in May 2005 and July 2012, probably due to a change in the ranking formula.

fileyfood500 7y ago

I see from one of the charts that the average movie rating from 2019 is over .5 higher than the previous 5 years, in what appears to be a big jump. Is it possible that recency bias is only relevant on a shorter timescale? It looks like ratings are higher for movies that are a few months old, based on the data. I wonder if this is generally true, or if early 2019 happens to have higher rated movies.

ac29 7y ago
Sure - for example, fans of a particular actor, director, or series are more likely to go see a movie soon after release in theaters as opposed to later via rental or streaming. Fans are likely to rate higher than the general public, so over the first, lets say, 3-12 months, ratings often trend down.
justadog 7y ago
Fake reviews and ratings must be a factor. For big movie releases it is hard to find IMDB reviews that seem genuine. I assume good portion of the ratings are fake as well. It must have an effect given how common it is.

olalonde 7y ago

Doesn't this analysis rest on the assumption that movies are equally good each year? Maybe I suffer from nostalgia (or survivorship bias) but I'm not sure this is a good assumption.

drollcrops 7y ago
Good is relative. And you could easily rabbit hole on this. Do people have the same sense of quality? Have people changed their rating scale based on production/effects quality? Etc etc. Probably best to just take the subjective rating.
jdsully 7y ago
The problem with past movies is you only see the good ones. I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies - because while both suffer from horrible plots usually modern movies at least have interesting visuals. However for certain categories the past was most definitely better. Dramas being the main one. Without a lot of fancy effects the plot was the only thing you had to carry the movie. They also don't beat things over your head and can embrace subtlety in a way modern movies do not. I also really liked the 70s trend of movies that didn't really have a satisfying ending, you just get to experience the lives of people who don't really improve. 5 easy pieces or The Last Detail fit that category very well.
- EugeneOZ 7y ago
  So you want to say that Shutter Island is a bad example of drama? It's all just old-farts rants, nothing more. Everything is evolving. In old movies actors play often too dumb, just few of them have good examples of how to play a role. Some "legendary" movies from the past would fail in theaters nowadays without their legendary reputation. We have more movies right now, more choices, so our demands are higher - otherwise we will not have enough time to watch them all. Also, we have much more choices of how to watch them - age of digital distribution. Higher demands give higher quality, movies evolve both in visuals and in acting art.
  - 7y ago

mynewstartup 7y ago

In terms of shows, Sopranos has long been dethroned, possibly due to this effect. Then again, production values will only get better over time, so maybe the latest shows are of the best quality overall. Best line: Fucking Queeahs! (Paulie throws chair at his murder victim spirits)

in3d 7y ago

I wonder if recording ratings now and then every month or so would show a decline in ratings. IMDb must know already.

dahart 7y ago

> Number of Movies By Ratings I’m dying to see all of these as percentages rather than absolute numbers! I was even thinking that while scrolling down through, and then there the totals are right at the bottom. @KyleOS are you up for normalizing them?

bscphil 7y ago

Incidentally, in case the creator sees this, this site is borderline unusable for me on mobile Firefox. Text gets cut off in strange places, I can't zoom out properly to see all of the animations. "Request desktop site" seems to fix some things but break others - at least I can zoom all the way out but the text becomes hard to read. Situation might be the same on Chrome (didn't check), but regardless it could use some work since it's pretty broken as is.

eoinmurray92 7y ago
Hi - founder of Kyso here, thanks for your feedback we will get to work fixing those issues now
- bscphil 7y ago
  Thanks! After playing with it a bit it's clear that part of the problem is that the charts are getting generated on the fly and that's very performance intensive. My brand new Android phone is just choking on the page, to the point I can actually feel the phone overheating. Perhaps static images would be an improvement for mobile devices, with the option to load the full data on request.

8innovate 7y ago

This is definitely not the 'recency bias' I expected. I've always noticed that when a show is released it gets WAY higher ratings in the first week or so of the release date. I've learned not to trust the ratings until at least a 2 weeks after a release. It looks like this analysis focuses on the year a film was released. This seems like it would be a lot harder to determine because film quality changes (subjective) and the audience writing reviews is probably changing over time.

ajhurliman 7y ago

An alternative way of reading the same data: "Analysis of whether movies are getting better over time"

richardhod 7y ago
The problem with your formulation is that 'better' is perhaps the most subjective word there is. Define it in measurable terms, and perhaps you have a reading there. There are many factors to consider which may lead you to find things less clear-cut, for example here's one or two, with some sub-factors - Modern audience Preference for familiar patterns in: (eg) -- acting styles -- music used -- visual tricks and norms -- special effects styles -- acting styles -- soundtrack loudness -- colour and high resolution -- High Concept movies -- Fantasy and Comic-related films -- modern movie themes and topics - Internationalisation of movies, so that fans from abroad prefer modern films which are more universalised and less American-specific (see literature)
- ajhurliman 7y ago
  Right, I'm not asserting that my alternative title is the correct one, I'm lampooning the assertion that IMDB is "suffering from recency bias" as opposed to a more neutral claim like "ratings for more recent movies are higher." Answering the question of _why_ they went up is a lot harder to prove.

ddffre 7y ago

Of course, there are plenty of examples of that. Like the assassination of the NK leader movie, which at first received 10 star rating. Now look at it. Sane goes with all the new blockbuster movies, you should usually wait awhile for the real ratings to show.

z3t4 7y ago

What I find most amusing is ratings (and also awards/nominations) before a movie has been released.

IAmEveryone 7y ago

Series tend to be rated higher than movies, probably because people that don't enjoy them stop watching and rating after one or a few episodes. That effect may well be spilling over into movies, considering far more movies are now sequels.

ravenstine 7y ago

Great job! I'd love to see a similar analysis for Rotten Tomatoes since there's separate metrics for critics and audience, and I find RT ratings to be more extreme than IMDb. I'm often surprised to see how many bad movies get a rating of ~6 on IMDb.