Lightnews — Scholar-powered news

Reposted by Hamel Husain

Jeremy Lewi @jeremy.lewi.us · 13d

If your looking to get started with evals check out this cookbook from @hamel.bsky.social
cookbook.openai.com/examples/eva...

Building resilient prompts using an evaluation flywheel | OpenAI Cookbook

This cookbook provides a practical guide on how to use the OpenAI Platform to easily build resilience into your prompts. A resilient prom...

cookbook.openai.com

1 8

Hamel Husain @hamel.bsky.social · 16d

"Can I just get an LLM to do my error analysis?"

We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.

Full podcast: youtu.be/BsWxPI9UM4c?...

6

Hamel Husain @hamel.bsky.social · 16d

Full podcast. youtu.be/BsWxPI9UM4c?...

We are teaching our final course of the year on evals starting this Monday. You can enroll with this link to get 35% off: maven.com/parlance-lab...

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation,…

youtu.be

1

Hamel Husain @hamel.bsky.social · 16d

I recently sat down with Lenny Rachitsky to discuss why AI Evals are becoming the most sought after skill for product builders.

As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.

1 6

Hamel Husain @hamel.bsky.social · 24d

I think it’s more like 99% 🤣 the 1% worked super hard on data modeling

2

Hamel Husain @hamel.bsky.social · 25d

This one is going to be spicy. 80% of the time I've seen a graph DB in production, it's been an overcomplicated mess (especially in AI applications).

In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense.

Sign up here:

You Don't Need a Graph DB

Many teams adopt graph databases believing they need specialized tools for relationship data, adding unnecessary complexity to their stack. This session reveals that for most use cases, the…

maven.com

1 15

Reposted by Hamel Husain

Kyle Baxter @kbaxter.bsky.social · Aug 26

For technical domains especially, getting non-DSes involved in analyzing outputs is vital. It’s hard to build anything good without it bc v1s almost always have major fail modes. Finding the appropriate system design—let alone optimizing—requires a tight coupling of output analysis and system design

Hamel Husain @hamel.bsky.social · May 17

Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.

Links to more resources in the reply

1 2 4

Hamel Husain @hamel.bsky.social · Aug 23

Can you screenshot it and tell me what it is so I can troubleshoot it

1

Hamel Husain @hamel.bsky.social · Aug 23

If you're wanted to learn applied AI evals but not sure if its for you, @sh-reya.bsky.social and I put together something that might help.

This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books.

Here's the link: ai.hamel.dev/eval-course

AI Evals Email Course

A free 17-part email series on the principles of application-centric LLM evals.

ai.hamel.dev

1 1 8

Reposted by Hamel Husain

Jeremy Lewi @jeremy.lewi.us · Aug 8

I've been eval pilled by @hamel.bsky.social . Everyone is all let's build some MCP servers and ship a minimal AISRE as quickly as possible. And I'm writing a design doc about how to build evals with @runme.dev so we can iterate rapidly on the AI

2 8

Hamel Husain @hamel.bsky.social · Jun 8

Last chance to signup for this free lesson with OpenAI on evals, Including a sneak peek of their new eval products!

Link: maven.com/p/d2dc30/how...

2

Hamel Husain @hamel.bsky.social · May 17

Link to full talk youtube.com/live/jJMYWfQ...

AI Evals For Engineers: Book Review

YouTube video by Hamel Husain

youtube.com

Hamel Husain @hamel.bsky.social · May 17

We'll be discussing this in our upcoming course on May 19th - AI Evals For Engineers & PMs (This link has a 35% discount code):

maven.com/parlance-lab...

AI Evals For Engineers & PMs by Hamel Husain and Shreya Shankar on Maven

Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.

maven.com

1

Hamel Husain @hamel.bsky.social · May 17

Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.

Links to more resources in the reply

1 7

Hamel Husain @hamel.bsky.social · May 12

If you are writing evals without error analysis, our course AI Evals for Engineers & PMs is for you. Begins monday next week. Full syllabus in this link: maven.com/parlance-lab...

4

Hamel Husain @hamel.bsky.social · May 8

It is very easy to make mistakes when creating evals for your AI product. @sh-reya.bsky.social and I run through the most common errors in this talk.

35% discount code to our upcoming course in the video notes

youtu.be/GL0XhAj5LPE?...

LLM Evals: Common Mistakes

YouTube video by Hamel Husain

youtu.be

1 5

Hamel Husain @hamel.bsky.social · May 1

I thought this was a meme 🤣 … but it’s real

1 9

Reposted by Hamel Husain

Hamel Husain @hamel.bsky.social · Apr 29

GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!

In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product

maven.com/p/da8264/how...

2 4 25

Reposted by Hamel Husain

Eugene Yan @eugeneyan.com · Apr 30

@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work.

Enrollment closes in 4 days.

Secret 35% discount code: maven.com/parlance-lab...

2 4

Hamel Husain @hamel.bsky.social · Apr 29

GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!

In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product

maven.com/p/da8264/how...

2 4 25

Reposted by Hamel Husain

Hamel Husain @hamel.bsky.social · Apr 27

I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan

maven.com/p/544677/wha...

1 1 6

Hamel Husain @hamel.bsky.social · Apr 27

I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan

maven.com/p/544677/wha...

1 1 6

Reposted by Hamel Husain

Kyle @kylestratis.com · Apr 26

As genAI projects mature, proper evals are becoming table stakes for production deployment. But how do we evaluate probabilistic machines?

Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!

AI Evals For Engineers & PMs by Hamel Husain and Shreya Shankar on Maven

Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.

maven.com

1 1

Hamel Husain @hamel.bsky.social · Apr 16

Last chance to sign up for this. Recording sent to everyone who signs up. maven.com/p/29a33a/hyb...

4

Reposted by Hamel Husain

Eugene Yan @eugeneyan.com · Apr 16

@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts:
• Don't skip error analysis
• Don't skip looking at your data
• Don't gatekeep who can write prompts
• Don't let zero users be a roadblock
• Don't be blindsided by criteria drift

1 3 15