This tool is especially useful in cases when evaluations are expensive (e.g. LM harness eval) and you want to track model performance during training.

github: github.com/amoudgl/assa...