Could you please tell me how the ISIC score the prediction for task 3 if it has two maximal positions such as [0, 0, 0, 0.5, 0, 0.5]? Thanks!
The short answer is that such a row is always scored as false / incorrect, so you’d be much better to break any ties (add or subtract a very small value from one of the tied probabilities) before submitting (even if you had to do so randomly).
We have already considered that this situation may happen. In such a case of more than one maximal value, we consider that the detector is “undecided” (which we consider to not be a diagnostically useful category). Internally, our scoring confusion matrix actually counts this image as having a diagnosis of “undecided” too; of course, since the ground truth contains no “undecided” samples, such a diagnosis is always wrong.
I hope this helps.
Thank you very much!