Once an initial design is collected, SMAC should use the initial design evaluations to conduct some early diagnostics.
- If all configurations crash --> run function/pipeline is probably broken
- If all perform identically --> metric or space is likely misconfigured
Maybe we can think of more but if the two aforementioned cases arise, we can safely abort the HPO run early since in the first case we have no data to build the surrogate on (except for crashes, yielding the same penalty cost) and if all the performances are the very same this might be either due to a huge plateau or actually a bug. In either case we have no real feedback signal for the surrogate model.
Once an initial design is collected, SMAC should use the initial design evaluations to conduct some early diagnostics.
Maybe we can think of more but if the two aforementioned cases arise, we can safely abort the HPO run early since in the first case we have no data to build the surrogate on (except for crashes, yielding the same penalty cost) and if all the performances are the very same this might be either due to a huge plateau or actually a bug. In either case we have no real feedback signal for the surrogate model.