🔥👩🏻‍💻New Contest Rating Algorithm👨🏽‍💻🔥 - It's HERE!
LeetCode
107322
Official

Update: More information about "Contest Season and Absence in Participation" here: 👉 Contest Season and Absence in Participation

Happy New Year everyone! 🎉

In the beginning of 2020, LeetCode contests are ready to begin afresh with a new contest rating algorithm! We've put in lots of thoughts behind this change, so in the sections below, we will highlight what drive this change of contest algorithm and how this change will affect our participants.

Please note that this new rating algorithm will NOT be implemented immediately. We are looking to hear some feedback from you before making any changes. To keep you on top of this change, there will be a seperate annoucement when the new rating algorithm is indeed implemented, so you will not be left in the dark.

Now, what's this new rating algorithm all about?


What Do We Hope To Achieve With This New Rating Algorithm?

By replacing the current rating algorithm with the new rating algorithm, we hope to overcome the following 2 issues caused by our current contest rating algorithm and its setups:

  1. With the old/current rating algorithm, a user’s rating varies little compared to the relatively large number of contest this user has participated in.

    At the same time, we want to avoid inducing abnormally large rating fluctuation that does not truly reflect a participant's actual contest rankings, which has been a reported problem from our user. In any case, The new rating algorithm should produce rating variation that will best represent the changes in participants actual rankings over time during each contest, in more consistent and small variations.

  2. With the old/current contest algorithm setup, if a user had signed up for a contest but was absent in participation for any reason, this user’s rating would be heavily and negatively affected, and the rating would never return to the same height again.

    Here we define 'the absence of participation' as the situation when a user registered before contest, but did not make any submissions (including wrong ones). In the old rating algorithm, a user with an absence of participation is considered to be losing to all other participations in the same contest.

    But we understand that sometimes life gets in the way of contests, so with the new rating algorithm set up, we want to treat this absence of participation differently than the other participations so its penalty will not be as severe and long-lasting as the one we have now.



Where Does Our New Rating Algorithm Come From - Introduction To Elo Rating Algorithm:

Elo rating algorithm is an algorithm used in calculating the relative skill level of players participating in 1 vs 1 competitions. Before a game, set the rating of user A to be R_A, and the rating of user B, R_B. The mean-winning percentage would be:

After the game, the new rating of user A would become:

Among which, S_A is user A’s actual rating, (win=1, draw=0.5, lost=0), K is a constant.



Introduction to LeetCode’s new rating algorithm

LeetCode’s new rating algorithm takes reference from the Open Codeforces rating system (here: https://codeforces.com/blog/entry/20762).

Since Elo rating algorithm is mostly used for 1 vs. 1 competitions, we have to make some adaptations to this rating algorithm in order for it to work for our contests, which involve multiple participants. In LeetCode’s new rating algorithm, each contest participant will be playing against each of the other participants.

Participants with the relatively better rankings wins and the relatively worse rankings lose. For instance, if there were 4 participants attending a contest, a total of 6 matches would be played (P1 v.s. P2, P1 v.s. P3, P1 v.s. P4, P2 v.s. P3, P2 v.s. P4, P3 v.s. P4). There will be no draws for contests.

Here are more details. To determine the ratings of a participant, we first calculate every participant’s Expected Ranking (ERank) after a contest, which is equivalent to the sum of all other participants’ winning rate against this participant, or:

Secondly, we calculate the geometric mean (m) of the expecting-ranking and actual-ranking:

We can then come up with the Expected Rating (ER) of this participant like this:

And by using binary search algorithm to come up with the ER, the variation of the ratings would be like this:

From which,

With k as the number of times this participant has participated in our contest, the function f has its initial as 0.5, and this initial will be reduced as the number of participated contest increases to 0.2222…, or 2/9.

Thus, the final rating from this new rating algorithm would be:

Additionally, the rating calculation here only applies to participants who have made submissions during the contest. For users who have absence of participation, contest ratings will be deduced by ways discussed in the Conclusions section of this post.



LeetCode Global Rating Data

👉 Global Top 5000 Ratings List

🟢Global Top 10

The graphs below are taken from Contest 83 to 167, capturing the changes from ratings 1,500 and above. For each participant, shown on the graph on top is the rating from every contest, with the blue line of rating is generated from the new rating algorithm, and the orange line of rating is generated from the old rating algorithm. And on the graph below the ranking of that participant in each contest.



Conclusions

From the graphs above, we have observed:

  • The new rating algorithm effectively reduces rapid rating convergence while steadly produce visible rating variations that truthfully represent changes in the contest rankings of participants over time.
  • The average contest ranking among these global top 10 participants generated by the new rating algorithm is noticeably higher than that from the current rating algorithm, which indicates that the new rating algorithm is better at reflecting participants contest ranking than the old one.
  • The new rating algorithm is time sensitive and puts more weights on the most recent contests, so participants who have better rankings from more recent contests have higher ratings on average.


FAQ

Q1: Will contest seasons share the same or different rating algorithms?
A1: All contest seasons will be sharing the same rating algorithm.

Q2: How do we determine everyone's rating in the beginning of the contest season?
A2: In the beginning of each contest season, k will be set to 0, which means everyone will start with a clean slate for contest history.

Q3: Why are we changing the rating algorithm?
A3: We are changing the rating algorithm due to 2 reasons:

  1. With the old rating algorithm, users' rating growth slows down as they partake in more and more contest；

  2. in the current rating rules, absence in participation is equivalent to losing to all other partcipants, causing the participant's rating to dip sharply. If this user happens to have participated in a large number of contest already (which we've mentioned in the beginning of this article), it's extremely difficult and time consume for this user's rating to climb back to the normal state.

Q4: How does the new rating algorithm and its setup solve the current contest concerns mentioned above?
A4: Without a sigma parameter and with a new variance, the new rating algorithm will overcome the issue of stagnant growth in ratings after an increasing number of contest participation. And as for those who have absence in participation, the solution/penalization setup is done seperately from the new rating algorithm.

Q5: If recent participations have bigger impact on one's ratings than participations from a more distant time, does it mean participations in the beginning of a contest season is less important than the ones later on?
A5: Yes, this is true if you participate in our contests often, because your contest ranking might increase throughout this period of time.



What Do You Think About The Following?

  • Do you want to have contest seasons? By contest season, each contest season will use the same rating algorithm, with k=0, so participants start anew.

    • Yes
    • No

  • What do you think is the best "fresh-start" rating for each seaon?

    • Participation ratings will be put back to 0 in the beginning of each season, and ratings of each season will be kept seperately.
    • Your suggestion.

  • If you want to have contest seasons, which of the following options do you prefer?

    • Bi-anually/ twice a year
    • Annually

  • How will users were absent in participation be penalized? We offer 2 options here:

    • For participants who were absent in participation, their rating will drop by a set percentage. For example, let’s say the set percentage is 3%. If a participant had a contest rating of 2000 before a contest, his/her contest rating would be reduced to 2000*(1-0.03)=1940 at the end of this contest.
    • Every participant will be given 1 and 1 chance only of absence forgiveness per year/ per contest season, which enables them to maintain their ratings if they forget to participate or submit due to any reasons. The reduction of ratings from such absence will be strictly enforced in the following contests.

Share your experience from the previous contests and what you think about the changes in the rating algorithms in the comments below! If you have any questions or needs more clarifications on some parts of this post, tag us below and we'll clarify your doubts asap ❤️
Comments (50)