ISIC Challenge task 3 submission

[daisyz (Tiantian Zhang)]

Hi, I have two questions about Task 3.
(1) How to submit the results of Task 3. Is the submission like the ground truth(the .csv file)?
(2) I want to attend the ISIC 2019. Does ISIC 2019 have similar validation or test submissions to get the result score?

[sam.univ2013 (sam ben)]

hi,
About your first question task3 isic-test currently does not work !!
for your second question, yes it has similar (validation, test) submissions to get the result score
https://challenge2019.isic-archive.com/

[daisyz (Tiantian Zhang)]

Thank you for information. I found that the submission system for ISIC 2019 isn’t open at present.

[sam.univ2013 (sam ben)]

your welcome , does isic2018 test task 3 works with you !!?

[daisyz (Tiantian Zhang)]

No, only validation

[philipp.tschandl]

Hi @sami and @daisyz

Please make sure you submit to the “Live-Leaderboard” for the ISIC 2018 challenge: https://submission.challenge.isic-archive.com/#challenge/5bedea14c5eaea2c335effca (the “original” challenge leaderboard is fixed, therefore you cannot submit to that one anymore).

The CSV-format for the test-set should be the same as for the validation-set.

[chenzj (zijian chen)]

Hi, @ tschandl
I have an error in submission with task 1, too. I submit to the “Live-Leaderboard” for the ISIC 2018 challenge: https://submission.challenge.isic-archive.com/#phase/5bee430fc5eaea4f24b5ebee , and i’m sure my submission format is correct, but an error occur:

covalic.error: Internal error: unknown ground truth phase number: 1.
DockerException: Non-zero exit code from docker container (1).
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/celery/app/trace.py”, line 375, in trace_task
R = retval = fun(args, **kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/task.py”, line 153, in call
results = super(Task, self).call(_t_args, **_t_kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/celery/app/trace.py”, line 632, in protected_call
return self.run(*args, **kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/tasks.py”, line 19, in run
return core.run(*pargs, **kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/core/utils.py”, line 123, in wrapped
return fn(*args, **kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/core/init.py”, line 183, in run
task_outputs=task_outputs, **kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/plugins/docker/executor.py”, line 239, in run
stream_connectors=stream_connectors, **extra_run_kwargs)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/docker/tasks/init.py”, line 333, in _docker_run
_run_select_loop(task, container, read_streams, write_streams)
File “/home/ubuntu/.venvs/girder/local/lib/python2.7/site-packages/girder_worker/docker/tasks/init.py”, line 184, in _run_select_loop
raise DockerException(‘Non-zero exit code from docker container (%d).’ % exit_code)

could you fix it ,thank you very much!

[philipp.tschandl]

Hi,
Unfortunately I don’t have access to any technical backend so I don’t know what is going on here at the moment. If you need urgent results (e.g. for a submission/revision of a manuscript) regarding Task 3 I may be able to help out producing specified metrics, but I won’t be back to my office until July…

[brian.helba]

Hi @chenzj,

We’ve just resolved this issue. You should now be able to submit normally.

Apologies, we’ve been making some improvements to the scoring system which were released prematurely. Please let us know if you have any other issues.

[chenzj (zijian chen)]

@brianhelba Thank you for your efforts, i have successfully made a submission just now.