Troels Nielsen · @syndereNDota
8th Jun 2018 from TwitLonger
My thoughts on the new calibration and game quality
The following is based on the expectation that calibration is done using the previous season's end MMR. If it is, as some people claim, somehow based on MMR values of 7 months ago, the entire calibration process makes absolutely zero sense to me.
As far as I understand, the calibration system will put you at a high uncertainty from the beginning, making the games push you far in either direction depending on whether you win or lose. If you win a lot, it will put you in harder and harder games, and if you lose a lot, it will put you in easier games. The idea of this makes perfect sense, but the values seem pretty ridiculous to me.
10 games is an extremely little sample size. Let's say players A and B start on the same MMR (6000). Player A goes 2-8 and player B goes 8-2. Is B a much better than A based on those results? B could have had a few games where an opponent abandoned or where the enemy team picked worse heroes against the lineup B was playing in. Over the course of 10 games, it doesn't seem that improbable that this could happen a few times, _even if_ B's games get increasingly more difficult with each win compared to A's, all it takes is for a guy in the enemy team to abandon (from connection or... other issues) or an excellent last pick that can win the game alone (mind you, this hero may not even be played by B but by one of his team mates, so he may not even have contributed much to the victory but still win regardless), or a significant outdraft, which happens frequently in all brackets. Also, remember that loads of players are calibrating at the same time, so the uncertainty among the team mates is also really high. B may, even on his win streak, end up getting strong team mates who just had a bad start in calibration (lost a few of their first games, maybe they are 1-4 or something). Dota 2 is a game with way too many variables to even remotely accurately calibrate people based on 10 games. Such a low amount of games might produce accurate results in single-player games, though, where there are way less variables.
Based on the information gathered from various people's posts, A's end MMR could be roughly 5200, while B's may end up on 6800 (these numbers can be less or more extreme based on which games were won or lost - the games you were supposed to be favored in or the ones in which you weren't, from what I understood). The 3 extra games B won and A lost therefore amount to a total of 1600 MMR, or 64 straight wins.
"Who cares, people will go back to where they belong eventually". Yes, they will. The problem is just that in the very high bracket (and any bracket, I would imagine), the game quality is going to be drastically lower for EVERYONE, A or B, over the coming weeks at least if nothing is changed about this, because some players will have calibrated way too low or way too high, and while they will eventually "get back where they belong", it will take a really long time with the massive amounts gained and lost. The games will flat out be worse and way less balanced and therefore the game will be less enjoyable to play. So what to do?
Perhaps one of the following two ideas could be good if tweaked correctly?
1) Revert the whole calibration. Do it again but with a bigger sample size to calibrate by to limit the variance. Perhaps players should play 25 games to calibrate? Perhaps it's a dynamic number of games needed based on how reliable the results of the games they played are? (I like this idea less because people don't know when to expect calibration, and it's something exciting to look forward to for many). 15-10 (60% win rate) is way more reliable than 6-4 (60% win rate) and could still give a sizable reward (for example, let's say it gave triple the amount of normal MMR on average: 5*25*3 = 375).
2) Change the amount that the current calibration contributes. There's no way that A and B end up 1600 MMR apart after each playing 10 games like this. Halving the amounts (or even more, maybe putting it to 3/8s of what it is currently -- in our example A would then be 5700 and B 6300, corresponding to 24 straight wins) still makes the calibration push MMRs a decent amount in either direction (and way more than last season where we seemingly just had a flat deflation and then every calibration game seemed to just count +/- 25)