Rules for external models / training data

ISIC 2018 Challenge
1

Hi,

Is there a rules section? I have two questions:

  1. Are pre-trained models allowed?, e.g., on imagenet dataset

  2. What about external datasets? From a ISIC 2017 challenge submitted abstracts, i saw one that states that they used external data from the ISIC archive, source: https://arxiv.org/pdf/1703.04197.pdf
    So can we use data from that archive to get more training data?

Best wishes,
Christian

2

Hi @ck1.kromm,

This is a great question. I’ll get back to you soon with a full clarification on the rules around this.

1 Like
3

Hi @ck1.kromm,

Here’s some information that may be helpful. If you have any further questions, please feel free to let us know.

  1. You may use pre-trained networks.
  2. You may use any external datasets for pre-training as well.
  3. You may NOT use any human annotations of the test sets.
  4. You may NOT use aggregate statistics of the test dataset for analysis. All decisions should be on an individual image basis.

We may award an “honorable mention” category for the team to achieve highest performance without the use of any external training data, beyond that which is freely and publicly available and not in-domain. Though the details are currently being worked out.

Thank you.

4

Thank you!

5

The submission asks if we are using external data. If I use a pretrained network, but no external lesion image, should I say I am using external data or not?

6

I would regard ImageNet-pretrained weights of architectures a baseline in current deep learning and not as “additional” data in the sense of this challenge.

In turn, using a pretrained network that has previously seen medical or dermatologic data, even if the data was not used during final training runs, should count as additional data. This counts equally for e.g. hyperparameter tuning or ensemble calibration.