In my limited experience testing it (it got released ~this week) the schedule_free_adamw optimizer that's now in axolotl has outperformed the various adamw variants for me. The new adopt optimizer on the other hand hasn't delivered for me.

Sonnet is still King 👑 for summarization:

Sonnet 3.6 vs 4o 11-20 (n=210):
Claude Sonnet 3.6: 54% (113 wins)
GPT-4o (11/20): 44% (92 wins)
Ties: 2% (5)

Sonnet 3.6 vs Gemini Exp 11-21 (n=202):
Claude Sonnet 3.6: 60% (122 wins)
Gemini-exp-1121: 38% (76 wins)
Ties: 2% (4)