Is the ISIC 2018 online score for Task1 the same it was in 2017? Can we compare the online score of 2018 with results published for last year’s challenge?
Hi @fredguth,
The images used for the Final Test set in this year’s 2018 Challenge are entirely new, and come from a larger (1000 images in 2018 vs 600 in 2017) and more diverse population of images. Additionally, the “threshold Jaccard index” metric, used as the primary ranking metric in 2018, is a new metric (developed in part from lessons learned during the analysis of the 2017 Challenge); the 2017 Challenge used a non-thresholded Jaccard index.
However, this is an interesting topic. We are interested tracking the performance of state-of-the-art algorithms over time: has over one year of technology advancement lead to improved algorithms? We may attempt to explore this topic in research following the conclusion of the 2018 Challenge, but I’d be very interested to hear thoughts on the best methodologies for analyzing this.