Task3 Metric Confusion

ISIC 2018 Challenge
1

Hi there,

I am doing the task 3 and I have tried many times to submit the validation results. the format is like the following: It is really weird all my models got the similar results around 0.15. But there is a big difference between my training accuracy results. Is there anyone encountering this situation? Appreicate any suggestions. Many thanks!

ISIC_0034328,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034446,1.0,0.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034395,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034372,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034430,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034489,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034336,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034427,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034412,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034456,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034398,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034355,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034471,0.0,0.0,0.0,0.0,0.0,1.0,0.0
ISIC_0034482,1.0,0.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034352,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034420,0.0,1.0,0.0,0.0,0.0,0.0,0.0
ISIC_0034476,0.0,1.0,0.0,0.0,0.0,0.0,0.0

2

Try probabilities for each class instead of 1’s and 0’s

3

Thank you for your suggestions, Devansh! Actually,I tried but failed with same scores. Originally I use confidence but I always got around 0.15, then I reset 1 and 0.

4

I had a similar problem. It turned out that the image ids and the predictions were getting mixed up in my submission file. Changing shuffle from True to False in gen.flow_from_directory in Keras fixed it for me:

test = gen.flow_from_directory(directory,
target_size=(224,224),
class_mode=‘categorical’,
shuffle=False,
batch_size=batch_size)

1 Like
6

Thank you for your sharing,Richard! It you set shuffle as False that means the results you got would keep the order ?So you mean that the image order matters for the submission? Do you mind to show a image order you submitted for a reference if that works for you? BTW, do you keep the probabilities or only 1’s and 0’s? Appreciate really!

7

If you encode the classes like so:

'MEL': 0, 'NV': 1, 'BCC': 2, 'AKIEC': 3, 'BKL': 4, 'DF': 5, 'VASC': 6

Then a sample submission may look like so:
image

1 Like
8

Sorry, no. Image order does not matter for the submission (I just checked).

My problem with Keras was that I was retrieving the file / image names from the directory and then running shuffled data through the model which was resulting in a mismatch between the image names and predictions.

Can you verify that your image names and predictions are being matched up correctly? Perhaps you can run one image through your model and comparing that prediction with your master submission.

I am submitting fractional probabilities as well.

1 Like
10

Thank you for the information, Richard! I found the problems because messing up features. Good luck on your ranking though!

11

Good luck to you too!