to join the CVG community. Not a member yet? Join now!
CVG
News

Reported Metacritic weighting system is 'wholly inaccurate'

Metacritic disputes ranking scheme

A report that claims to have determined how Metacritic ranks media outlets in terms of their influence is "wholly inaccurate," according to the site.

Zoom

The original report aimed to model the weightings of a list of game media in terms of influence. The result was a six-tier model where outlets were ranked from 'highest' to 'lowest' influence.

The research was carried out by Adams Greenwood-Ericksen of Full Sail University, with the results announced during a presentation at GDC. Shortly after Gamasutra picked up the story, Metacritic has released an official statement on its Facebook page describing the findings as "wildly, wholly inaccurate".

The full Metacritic statement:


Today, the website Gamasutra "revealed" the weights that we assign to each gaming publication (for the purpose of calculating our Metascores), based on a presentation given at the Game Developers Conference this morning. There's just one major problem with that: neither that site, nor the person giving the presentation, got those weights from us; rather, they are simply their best guesses based on research (the Gamasutra headline is misleading in this respect).

And here's the most important thing: their guesses are wildly, wholly inaccurate. Among other things:

  • We use far fewer tiers than listed in the article.
  • The disparity between tiers listed in the article is far more extreme than what we actually use on Metacritic. For example, they suggest that the highest-weighted publications have their scores counted six times as much as the lowest-weighted publications in our Metascore formula. That isn't anywhere close to reality; our publication weights are much closer together and have much less of an impact on the score calculation.
  • Last but definitely not least: Our placement of publications in each tier differs from what is displayed in the article. The article overvalues some publications and undervalues others (while ignoring others altogether), sometimes comically so. (In addition, our weights are periodically adjusted as needed if, over time, a publication demonstrates an increase or decrease in overall quality.)

Comments