About only the latest submission being valid: I think it would be more interesting if each team could submit a handful of submissions (three or five). That fosters more daring explorations, since at least one of the submissions can be really risky. It also allows making mandatory, for example, that one of the submissions uses only the provided data.
In any case, I’d like you to clarify what constitutes a “team”, because last year we had different user accounts with essentially (or exactly) the same team composition with different submissions. Will this be allowed for everyone? To which extent?
We allow and encourage a single team (whether that’s a single person or a whole lab) to try multiple approaches to the task and submit them all, if that team is willing to do the extra work to fully explore and writeup the arXiv abstract for each of these approaches.
However, the rule
You may make unlimited and independent submissions, but only the most recent submission will be used for official judging
just reflects a technical limitation with our submission system where each registered user will only have their most recent submission to a given task scored on the leaderboard.
We are currently investigating improvements to make the system more friendly to submitting multiple approaches. In event that we are not able to provide this technical capability by test time (July 9), we’d suggest that teams create multiple registered users at that time, and have each user submit results for a different approach; indeed, in the 2017 ISIC Challenge several teams did exactly this.
As a further clarification, we will allow each team to submit a maximum of 3 separate approaches / algorithms to be evaluated in parallel in the final scoring.
Within a given approach, submitters may still upload unlimited times (in case of data format failures or last-minute tuning), but each new upload with overwrite the previous ones. Of course, submitters are blinded to the scores of their uploads (as described in the rules), to prevent “wacky boosting”.
We’ll have further details about the exact method for teams to indicate that multiple uploads should be scored as separate approaches (as mentioned in my last post) well in advance of the time that the submission system opens.
For multiple submission related to different approaches, are we required to create separate user accounts or is there a provision for that using a single account.
The submission system now allows individual users to submit separate approaches.
On the submission form, use the “brief description of your algorithm’s approach” field to distinguish the approaches. Previously submitted approach names are available in the dropdown menu. The most recent submission for each approach will be used for official judging.
The submission instructions on the challenge website have been updated with this information.