Since these low suggest almost random guessing, getting them is almost always a problem with having your predicts aligned correctly with the image name row and/or class column.
I guess you will have to search for problems in your pipeline, e.g. in this thread people found described shuffling of Keras’ flow_from_directory() caused such a problem.