Mojeek is also an option for a search engine with it’s own webcrawler. Can’t speak to it’s use though, I gave up on it pretty quick. They made a visualization some years ago: www.searchenginemap.com
here's how you remove the AI from the latest Firefox update
October 21, 2025 at 6:25 PM
Everybody can reply
1 likes
Makes me wonder if it could be found with a different, older, less money-focused browser, like Webcrawler or something...
October 13, 2025 at 3:09 PM
Everybody can reply
2 likes
The first web search I remember doing (on my Lynx text browser, presumably using Webcrawler) was "Thomas Aquinas". I think the article came from Michigan university.
Fascinating @theguardian.com piece here from Tim Berners-Lee @timbl.bsky.social about where the Internet goes from here...
www.theguardian.com/technology/2...
www.theguardian.com/technology/2...
Why I gave the world wide web away for free | Tim Berners-Lee
My vision was based on sharing, not exploitation – and here’s why it’s still worth fighting for
www.theguardian.com
September 28, 2025 at 4:21 PM
Everybody can reply
6 likes
They think current AI is bordering on Skynet, when it’s really just a webcrawler and database scraper that’s great at plagiarism and getting math wrong.
September 18, 2025 at 11:44 PM
Everybody can reply
1 reposts
11 likes
September 3, 2025 at 5:22 AM
Everybody can reply
3 reposts
1 quotes
24 likes
This is an ancient artifact. This might as well be a tablet engraved with Linear A
August 24, 2025 at 12:10 AM
Everybody can reply
5 likes
@trans.bsky.social @juno.bsky.social @fumblebee.bsky.social
WebCrawler run for today:
Results for Joey and Juno, there does not seem to come up much lately for you Sage
Results have been mailed
WebCrawler run for today:
Results for Joey and Juno, there does not seem to come up much lately for you Sage
Results have been mailed
September 12, 2024 at 8:54 PM
Everybody can reply
2 likes
🎶I wish I was a little webcrawler
A stick to the waller
I wanna meet my girl on the bridge
She’s a faller🎶
A stick to the waller
I wanna meet my girl on the bridge
She’s a faller🎶
December 4, 2024 at 10:23 PM
Everybody can reply
1 reposts
1 likes
@trans.bsky.social
@juno.bsky.social
@fumblebee.bsky.social
WebCrawler run for today:
Links for Sage & Joey
Have been mailed :)
@juno.bsky.social
@fumblebee.bsky.social
WebCrawler run for today:
Links for Sage & Joey
Have been mailed :)
September 6, 2024 at 3:11 PM
Everybody can reply
3 likes
Back in the primordial days of the information superhighway (I'm talking the age of 28.8k fax modems) I searched "giant women" on Webcrawler (by way of Netscape Navigator) and stumbled upon Giantesszone.
The was before Gary even started making films. It was just an archive of pics, stories & links.
The was before Gary even started making films. It was just an archive of pics, stories & links.
Alright so how did you discover the size fetish scene? Mine is kinda hilarious. I misspelled “gigantes Zelda” while looking for info in Spanish about the giants from Majora’s Mask when I was like 12 and that lead me to VGGTS.
November 16, 2024 at 4:28 AM
Everybody can reply
4 likes
Google search is now about as useful as shopping on ebay.
I'll do it if I have to, but sifting through the garbage to find what I'm looking for is making me miss webcrawler and altavista.
I'll do it if I have to, but sifting through the garbage to find what I'm looking for is making me miss webcrawler and altavista.
August 24, 2023 at 9:01 PM
Everybody can reply
2 likes
Tucows?
I found this story via the Webcrawler search engine.
I'm going to make a page on my Lycos website about this. Sign my guest book to let me know you liked it.
I found this story via the Webcrawler search engine.
I'm going to make a page on my Lycos website about this. Sign my guest book to let me know you liked it.
Fake election news ads are luring people into investment schemes. We got some taken down #Canada
Meta, Tucows shut down pages flagged by CBC News
www.cbc.ca/news/canada/...
Meta, Tucows shut down pages flagged by CBC News
www.cbc.ca/news/canada/...
Fake election news ads are luring people into investment schemes. We got some taken down | CBC News
Fake ads that purport to show CBC News articles are once more circulating online — with the end goal of convincing Canadians to sign up for sketchy investment schemes. The CBC’s Visual Investigations ...
www.cbc.ca
March 29, 2025 at 6:12 AM
Everybody can reply
3 likes
April 22, 2025 at 4:04 AM
Everybody can reply
Crawl4AI: Die Zukunft der KI-gestützten Webcrawler und Datenextraktion
https://dasgeld.co/posts/526ADD47-FA5E-45D4-9633-6EF256FCC6F2
https://dasgeld.co/posts/526ADD47-FA5E-45D4-9633-6EF256FCC6F2
May 18, 2025 at 8:48 PM
Everybody can reply
Nepenthes: eine Teergrube für KI-Webcrawler
Webcrawler für KI-Modelle machen auch vor dem Urheberrechtsschutz oft nicht halt – das Tool Nepenthes stellt ihnen eine Falle […]
Webcrawler für KI-Modelle machen auch vor dem Urheberrechtsschutz oft nicht halt – das Tool Nepenthes stellt ihnen eine Falle […]
Original post on social.heise.de
social.heise.de
January 24, 2025 at 4:00 PM
Everybody can reply
1 reposts
Bluesky has a #Python API.
atproto.blue/en/latest/
It can't be hard to write a webcrawler for my twitter feed to automate crossposting here.
atproto.blue/en/latest/
It can't be hard to write a webcrawler for my twitter feed to automate crossposting here.
Documentation
The AT Protocol SDK documentation for Python
atproto.blue
November 20, 2024 at 7:22 AM
Everybody can reply
3 likes
This is Webcrawler erasure.
June 27, 2024 at 4:40 PM
Everybody can reply
1 likes
So, we’re all going to start calling Google…
Alta Vista
Lycos
Excite
Webcrawler
Ask Jeeves
Or we could just go with complicit cowards.
Alta Vista
Lycos
Excite
Webcrawler
Ask Jeeves
Or we could just go with complicit cowards.
February 12, 2025 at 12:10 AM
Everybody can reply
1 reposts
1 quotes
4 likes
The first time I got to try out the internet at a friend’s house, the very first thing I typed in WebCrawler was “The Crow.” We were obsessed with that film. About wore out my stereo playing “It Can’t Rain All the Time” over and over. (I realize how much this comment dates me. I care not. Lol)
July 24, 2025 at 6:58 PM
Everybody can reply
2 likes
"You sure about that, webcrawler?"
Yeah. I think I get it.
I'm smart.
I'm smart.
June 21, 2024 at 12:59 PM
Everybody can reply
1 quotes
„DuckDuckGo verfügt über einen eigenen Suchindex, der von seinem Webcrawler namens DuckDuckBot aufgebaut wird. […] Zusätzlich greift DuckDuckGo jedoch auch auf externe Quellen wie Bing, Yahoo und andere Suchdienste zurück, um die Suchergebnisse zu ergänzen und möglichst umfassend zu gestalten.“
April 5, 2025 at 4:47 PM
Everybody can reply
Ok, i'll remove the string from the WebCrawler again, my apologies if i overstept boundaries
September 27, 2023 at 7:07 PM
Everybody can reply
2 likes
Webcrawler . Boolean logic to search the net
June 25, 2024 at 11:50 PM
Everybody can reply
1 likes
Webcrawler is still okay.
February 11, 2025 at 1:51 AM
Everybody can reply