"Can I just get an LLM to do my error analysis?"

We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.

Full podcast: youtu.be/BsWxPI9UM4c?...

I recently sat down with Lenny Rachitsky to discuss why AI Evals are becoming the most sought after skill for product builders.

As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.

Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.

Links to more resources in the reply