Task 3:How to score if the prediction have two maximal postitions

ISIC 2018 Challenge
1

Hello,
Could you please tell me how the ISIC score the prediction for task 3 if it has two maximal positions such as [0, 0, 0, 0.5, 0, 0.5]? Thanks!

2

Hi @tat-dat,

The short answer is that such a row is always scored as false / incorrect, so you’d be much better to break any ties (add or subtract a very small value from one of the tied probabilities) before submitting (even if you had to do so randomly).

We have already considered that this situation may happen. In such a case of more than one maximal value, we consider that the detector is “undecided” (which we consider to not be a diagnostically useful category). Internally, our scoring confusion matrix actually counts this image as having a diagnosis of “undecided” too; of course, since the ground truth contains no “undecided” samples, such a diagnosis is always wrong.

I hope this helps.

1 Like
3

Thank you very much!