#PerplexityBot
LightNews LightNews
weightlessbooks.bsky.social
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist

robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
October 28, 2025 at 1:41 PM
chrismlong.bsky.social
Their study found that most major AI crawlers can fetch JavaScript files (between 10%-25%) but do not execute it. GPTBot, ClaudeBot, PerplexityBot and more do not currently fully render JavaScript content.

This means that if you're using JS to load content, many AI crawlers will be missing them.
January 7, 2025 at 1:43 PM
sequr.be
Pretty extensive robots.txt so far

`DisallowAITraining` is also a newer option, again only for cooperative robots
www.ietf.org/archive/id/d...
Contents of a robots.txt file: User-Agent: AI2Bot User-Agent: AI2Bot-Dolma User-Agent: Amazonbot User-Agent: anthropic-ai User-Agent: Applebot User-Agent: Applebot-Extended User-Agent: AwarioBot User-Agent: AwarioRssBot User-Agent: AwarioSmartBot User-Agent: Bytespider User-Agent: CCBot User-Agent: ChatGPT-User User-Agent: Claude-Web User-Agent: ClaudeBot User-Agent: cohere-ai User-Agent: cohere-training-data-crawler User-Agent: DataForSeoBot User-Agent: diffbot # User-Agent: DuckAssistBot User-Agent: FacebookBot User-Agent: Google-CloudVertexBot User-Agent: Google-Extended User-Agent: Googlebot-Extended User-Agent: GPTBot User-Agent: ICC-Crawler User-Agent: ImagesiftBot User-Agent: img2dataset User-Agent: Kangaroo Bot User-Agent: Meltwater User-Agent: Meta-ExternalAgent User-Agent: Meta-ExternalFetcher User-Agent: OAI-SearchBot User-Agent: Omgili User-Agent: Omgilibot User-Agent: PanguBot User-Agent: peer39_crawler User-Agent: Perplexity User-Agent: PerplexityBot User-Agent: Petalbot User-Agent: Scrapy User-Agent: Seekr User-Agent: SemrushBot-OCOB User-Agent: Sentibot User-Agent: Timpibot User-Agent: TurnitinBot User-Agent: VelenPublicWebCrawler User-Agent: webzio-extended User-Agent: YouBot Disallow: / DisallowAITraining: /
June 11, 2025 at 11:28 AM
carlhendy.com
Just updated robots.txt.

𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗣𝗧𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗼𝗼𝗴𝗹𝗲-𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /

LLMs:
May 1, 2025 at 11:41 AM
johnspurlock.com
‘I went into Perplexity and asked "What's on this page rknight․me/PerplexityBot?". Immediately I could see the log and just like Lewis, the user agent didn't include their custom user agent’

rknight.me/blog/perplex...
Perplexity AI Is Lying about Their User Agent
Perplexity AI claims it sends a user agent and respects robots.txt but it absolutely does not
rknight.me
June 15, 2024 at 10:51 PM
hex.ooo
just checked and the ones I'm blocking right now are:
AISearchBot anthropic-ai Applebot Bytespider ChatGPT-User Claude-Web cohere-ai Diffbot FacebookBot Google-Extended GPTBot omgili PerplexityBot YouBot
July 21, 2025 at 8:37 AM
se-davidobrien.mastodon.scot.ap.brid.gy
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: Scrapy
User-agent: Timpibot
User-agent […]
Original post on mastodon.scot
mastodon.scot
June 1, 2025 at 8:06 PM
weightlessbooks.bsky.social
Ok actually, because robots.txt is optional and ignorable, here is our current, full AI bot blocking solution using the non-optional .htaccess mod_rewrite (see alt text)
<IfModule mod_rewrite.c> RewriteEngine on RewriteBase / # block AI bots RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Ahrefsbot|AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|AspiegelBot|aspiegel|AwarioRssBot|AwarioSmartBot|Brightbot\s1.0|Bytespider|CCBot|chatgpt|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|DataForSeoBot|Datanyze|Diffbot|DotBot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo\sBot|Kinza|OPPO\sA33|magpie-crawler|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|Omgilibot|PanguBot|peer39_crawler|PerplexityBot|PetalBot|serpstatbot|spaziodati|Scrapy|SemrushBot|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade\sindexer\sbot|thesis-research-bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YandexBot|YouBot|zh-CN|zh_CN) [NC] RewriteRule ^ – [F] </IfModule>
October 28, 2025 at 5:09 PM
jornalexpresso.bsky.social
"O motor de busca da Google ainda é todo-poderoso, mas pela primeira vez em mais de uma década caiu abaixo dos 90% de quota global. Sobretudo no desktop, há cada vez mais utilizadores que optam por fazer perguntas ao ChatGPT, Claude, PerplexityBot ou outro sistema de inteligência artificial"
Uma entrada a pés juntos na guerra dos motores de busca
Uma entrada a pés juntos na guerra dos motores de busca
expresso.pt
August 18, 2025 at 4:49 PM
planesailinggames.com
And (details in alt-text again)
User-agent: DuckAssistBot Disallow: / User-agent: AI2Bot Disallow: / User-agent: Bytespider Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User -agent: YouBot Disallow: /
December 19, 2024 at 6:45 PM
xkeeper.net
it is frankly amazing how much internet traffic is just literal garbage
150+ requests from three IPs all next to one another surprise, it was "Perplexitybot" crawling everything at max speed
March 9, 2025 at 10:50 PM
vercel.com
Search engine crawlers still outpace AI crawlers, but it's growing significantly.

Last month GPTBot, Claude, AppleBot, and PerplexityBot combined account for nearly 1.3 billion fetches—a little over 28% of Googlebot's volume.
December 17, 2024 at 6:16 PM
aimeecozza.com
Wake up babes, new version of Simple NoAI WordPress plugin just dropped. Adds more crawlers to deny including:
AppleBot-Extended, Bytespider, Cohere-ai, Diffbot, ImagesiftBot, PerplexityBot, FacebookBot, Omigili

Get 1.6.3 here: wordpress.org/plugins/simp...
Simple NoAI and NoImageAI
This plugin very simply adds a line of code to your header that tells AIs not to use anything on your website for indexing.
wordpress.org
June 11, 2024 at 3:13 PM
ehsankhosraviseo.bsky.social
یادآوری: robots.txt می‌تواند خواندن محتوای شما توسط #چت_بات‌ ها را روشن/خاموش کند. نمونه مینیمال: User-agent: GPTBot → Allow: / (برای مسدودسازی، Disallow: / بگذارید.) همین قانون را برای PerplexityBot، Claude، Google-Extended و… تکرار کنید.
مدیریت دسترسی چت‌بات‌های هوش ‌مصنوعی از طریق robots.txt
August 20, 2025 at 9:30 AM
aleyda.bsky.social
.... ByteDance (Bytespider), Perplexity (PerplexityBot).

* Google's Gemini leverages Googlebot's infrastructure, enabling full JavaScript rendering.

* AppleBot renders JavaScript via browser crawler, like Googlebot. Processing JS, CSS, Ajax requests, and other resources for full-page rendering...
December 18, 2024 at 3:38 PM
danreif.bsky.social
⚠️Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites. 🤨
blog.cloudflare.com/perplexity-i... #Perplexity #AICrawler #PerplexityBot
Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.
blog.cloudflare.com
August 5, 2025 at 4:41 PM
parthshahseo.bsky.social
I’ve read that ChatGPT's search results often match Bing’s, which makes sense given the underlying integration. Gemini taps into Google Search, while Perplexity uses its own crawler, PerplexityBot. Since Perplexity essentially built its own search engine, its results feel a bit more limited.
April 2, 2025 at 9:01 PM
mitstek.bsky.social
robots.txt

User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
July 21, 2025 at 3:49 AM
quinnypig.com
You can keep up to date on which AI search companies haven’t bent the knee to Amazon pretty easily via their robots.txt:
Amazon’s robots.txt shows a diff with PerplexityBot being barred from everything now.
November 22, 2024 at 8:34 PM
ppival.bsky.social
An interesting read. I've been really happy with Perplexity, but I was just able to verify that it pulled content from a site that explicitly excluded PerplexityBot via robots.txt That doesn't seem cool.

Perplexity Is a Bullshit Machine www.wired.com/story/...

#ai #manners
Perplexity Is a Bullshit Machine
A WIRED investigation shows that the AI-powered search startup Forbes has accused of stealing its content is surreptitiously scraping—and making things up out of thin air.
www.wired.com
June 20, 2024 at 7:36 PM
danabramov.bsky.social
this is kind of weird but i guess it's interesting? one thing i still don't get after reading the post — do any of these integrations actually work?

like, i see in my logs: Meta-ExternalAgent, ChatGPT-User, PerplexityBot, PerplexityBot-user. would these actually pay? and what price should i set?
July 1, 2025 at 11:44 PM
chrismlong.bsky.social
In this example, I used WP Engine to look at the Go Fish Digital site and search for crawl instances from Perplexitybot (Perplexity) and GPTBot (OpenAI).

Reviewing this data, I can confidently say that both are discovering the site.
January 9, 2025 at 1:34 PM
anilinanilina.bsky.social
The Vercel article;

“The results consistently show that none of the major AI crawlers currently render JavaScript.

This includes: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot) Anthropic (ClaudeBot) Meta (Meta-ExternalAgent) ByteDance (Bytespider) Perplexity(PerplexityBot)”

7/10
The rise of the AI crawler - Vercel
New research reveals how ChatGPT, Claude, and other AI crawlers process web content, including JavaScript rendering, assets, and other behavior and patterns—with recommendations for site owners, devs,...
vercel.com
January 29, 2025 at 2:00 PM
cyrez.dev
Are there any plans to block AI crawler bots within Bluesky's robots.txt file?
robots.txt file example to block AI crawler bots: # Hello Friends! # If you are considering bulk or automated crawling, you may want to look in # to our protocol (API), including a firehose of updates. See: https://atproto.com/ # By default, may crawl anything on this domain. HTTP 429 ("backoff") status # codes are used for rate-limiting. Up to a handful concurrent requests should # be ok. User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Omgilibot Disallow: / User-Agent: FacebookBot Disallow: / User-Agent: Applebot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: Claude-Web Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: YouBot Disallow: /
November 15, 2024 at 6:03 PM
ki-news.bsky.social
Are AI Bots Reading llms.txt Files? – Live Tracking Dashboard – Live experiment tracks whether major AI bots like GPTBot, ClaudeBot, and PerplexityBot are reading llms.txt files across the web. Each time an AI crawler accesses an llm.txt file, we log an anonymo... https://tinyurl.com/yutpad7u #AIBot
Are AI Bots Reading llms.txt Files? – Live Tracking Dashboard
This live experiment tracks whether AI bots like GPTBot, ClaudeBot, and PerplexityBot are reading llms.txt files across real websites. View crawler activity in real time.
llmstxt.ryanhoward.dev
July 5, 2025 at 6:39 PM