Hi,
I wonder to know if you provide the validation leader board like the previous year challenge. I think it was very useful to compare the performance before the final test submission.
Thanks,
Amirreza
Hi @amirreza.mahbod,
We are not currently planning on making the validation leader board public again this year. However, you will still be able to immediately see the score of your own validation submissions.
Due to the very small data size (100 images) of the validation set, the immediate scoring of submissions, and the ability to make unlimited submissions (which is susceptible to “wacky boosting”), we do not feel that the validation phase is a truly robust evaluation of an algorithm’s performance, so it could be misleading to post validation scores on a public leaderboard.
The primary purpose of the validation set is to be an optional tool for participants to check for data format issues in submissions and to help check for data issues that may lower scores substantially (e.g. a bug in your script causes every segmentation to be empty).
Of course, your feedback is still extremely helpful to our design of the ISIC Challenge. If you (or other participants) have other thoughts on how we could improve the Challenge, please feel free to continue posting here.
Thanks again!
Hello,
Do I need to formally register somewhere in order to be able to post validation predictions and receive the evaluation scores?
Nothing in the menu here demonstrate how to do that:
https://challenge2018.isic-archive.com/participate/
https://challenge2018.isic-archive.com/task3/
Many thanks,