Runtime results are misleading

In order to provide more meaningful runtime results, number of test cases for each question should be increased and each test case should be measured multiple times.

Sometimes a solution that takes 6ms is only faster than 50% of submissions and a solution that takes 5ms is faster than 100% of submissions. If these solutions are benchmarked many times each (e.g. 1000) and their average performance is measured instead, the difference in performance might become much less than 20% (5ms vs 6ms).

And in general, I suggest using JMH for measuring the runtime performance of submissions.

Comments (1)