This, of course, is not only great puzzle design: it's great practice for working with code and data IRL. It can definitely feel frustrating initially---but you can't beat the high of feeling the gears in your brain /click/ into place once you truly grok the logic. And it helps Santa, so double win!

Great choice of show for a first watch.

That's why they pay us the big bu---wait

Now whether they *do* pull those levers remains to be seen. More what I want to insist is that we are not necessarily doomed. There are futures yet possible where online sampling is still feasible. Ask your sample providers hard questions. Demand that they do what they can too. Here’s to hope!

Good sample will become more expensive. That will exacerbate structural (dis)advantages. No argument. It sucks. But I strongly doubt that it will explode in price to the point where only the big players will be able to afford it. Sample providers have levers to pull yet.

"I don’t believe you that they care about quality."

Cool. But the existence of platforms like Prolific disproves that point. They don’t _all_ care, but if you're basing your science off the metaphorical equivalent of a back-alley surgeon, I don't know what to tell you.

"Uh, Peter, you dashing dimwit, have you not heard of VPNs?"

Don't track it by IP address, track it by payment profile.

"Well how about bot farms?!"

We already have those---and pretty sophisticated methods for reputable panels to suss them out. This will make it harder but not impossible.

Remove anyone from your panel who answer superhuman numbers of surveys per day. Or at least give researchers the option to filter them out of your sample.

Westwood showed that there's very little that we can do on the buyer side, once people are in, to detect an AI. I think there are still a few tricks left untested in this paper but his larger point stands: Your normal tactics won't work.

But there's a pretty easy way from the supplier side:

Why? Because if everyone believes that the providers' sample is just bots, people stop buying from them. Unless the buyers believe that silicon samples are good enough. Which, lol. That's on them.

"Peter, you beautiful fool," I hear you say, "it's not about the number of users, it's the number of surveys they can 'complete'! It will flood the zone!"

That assumes that the sampling providers don’t have an interest in ensuring a minimal level of sampling quality. Which they do.

First, it's not at all easy to pull off making an agentic workflow that operates as well as Westwood's. Agentic AI is fiddly. The barrier to entry is pretty high. Your average survey respondent won't be able to do this.

I'm wondering if we could lean on the "hidden" html tag to do the same thing rather than text color.

www.w3schools.com/tags/att_hid...

This is not me saying that the scan found something and that they're lying about it (though if I read verified report confirming that they lied I wouldn't be shocked). More just that even banal things like "the scan shows nothing to be concerned about" is given the "Dear Leader" treatment.

I chuckled a bit at its use of "our brain".

Between this and when you posted the Matt Parker survivor bias video a few weeks ago, I'm convinced that we have the same algorithm.