Skip to main content

Slashdot: Websites are Blocking the Wrong AI Scrapers

Websites are Blocking the Wrong AI Scrapers
Published on July 29, 2024 at 11:53PM
An anonymous reader shares a report: Hundreds of websites trying to block the AI company Anthropic from scraping their content are blocking the wrong bots, seemingly because they are copy/pasting outdated instructions to their robots.txt files, and because companies are constantly launching new AI crawler bots with different names that will only be blocked if website owners update their robots.txt. In particular, these sites are blocking two bots no longer used by the company, while unknowingly leaving Anthropic's real (and new) scraper bot unblocked. This is an example of "how much of a mess the robots.txt landscape is right now," the anonymous operator of Dark Visitors told 404 Media. Dark Visitors is a website that tracks the constantly-shifting landscape of web crawlers and scrapers -- many of them operated by AI companies -- and which helps website owners regularly update their robots.txt files to prevent specific types of scraping. The site has seen a huge increase in popularity as more people try to block AI from scraping their work. "The ecosystem of agents is changing quickly, so it's basically impossible for website owners to manually keep up. For example, Apple (Applebot-Extended) and Meta (Meta-ExternalAgent) just added new ones last month and last week, respectively," they added.

Read more of this story at Slashdot.

Labels:

Comments

Post a Comment

Popular posts from this blog

Slashdot: Spain-Backed Fund Joins FOSSA's Sovereign Satellite Communications Push

Spain-Backed Fund Joins FOSSA's Sovereign Satellite Communications Push Published on 2026-06-28T22:05:00Z Spanish startup FOSSA Systems "has raised about $10.5 million to expand its connectivity constellation," reports Space News, noting some funding is backed by Spain's government: The support from the Spanish Society for Technological Transformation (SETT) comes a year after the fund injected 14 million euros into Spain's Sateliot , which is also developing a satellite connectivity network with security and defense applications. Spanish private investment firm Kibo Ventures led FOSSA's funding round, the six-year-old venture announced June 24, bringing its total raised to date to nearly 20 million euros. The proceeds will help fuel FOSSA's push beyond the tiny picosatellites it once used to connect low-power monitoring devices toward larger cubesats in low Earth orbit, enabling additional sovereign communications and space-based intelligence capab...
Post a Comment
Read more

Slashdot: AT&T Outlines $250 Billion US Investment Plan To Boost Infrastructure In AI Age

AT&T Outlines $250 Billion US Investment Plan To Boost Infrastructure In AI Age Published on 2026-03-10T20:00:00Z AT&T plans to invest more than $250 billion over the next five years to expand U.S. telecom infrastructure for the AI age. The company says it will also hire thousands of technicians while partnering with AST SpaceMobile to extend coverage to remote areas. Reuters reports: Rapid adoption of artificial intelligence, cloud computing and connected devices has prompted telecom operators to invest heavily in fiber and 5G networks as they also seek to fend off intensifying competition from cable broadband providers. AT&T, which has about 110,000 employees in the U.S., said the new hires will help build and maintain its infrastructure. The outlay includes capital expenditure and other spending, the company said. The spending will focus on expanding its fiber and wireless networks, including accelerating deployment of fiber broadband, 5G home internet and satellite co...
Post a Comment
Read more