Slashdot: Websites are Blocking the Wrong AI Scrapers

Websites are Blocking the Wrong AI Scrapers
Published on July 29, 2024 at 11:53PM
An anonymous reader shares a report: Hundreds of websites trying to block the AI company Anthropic from scraping their content are blocking the wrong bots, seemingly because they are copy/pasting outdated instructions to their robots.txt files, and because companies are constantly launching new AI crawler bots with different names that will only be blocked if website owners update their robots.txt. In particular, these sites are blocking two bots no longer used by the company, while unknowingly leaving Anthropic's real (and new) scraper bot unblocked. This is an example of "how much of a mess the robots.txt landscape is right now," the anonymous operator of Dark Visitors told 404 Media. Dark Visitors is a website that tracks the constantly-shifting landscape of web crawlers and scrapers -- many of them operated by AI companies -- and which helps website owners regularly update their robots.txt files to prevent specific types of scraping. The site has seen a huge increase in popularity as more people try to block AI from scraping their work. "The ecosystem of agents is changing quickly, so it's basically impossible for website owners to manually keep up. For example, Apple (Applebot-Extended) and Meta (Meta-ExternalAgent) just added new ones last month and last week, respectively," they added.

Comments

Slashdot: US Army Soldier Arrested In AT&T, Verizon Extortions

US Army Soldier Arrested In AT&T, Verizon Extortions Published on January 01, 2025 at 02:35AM An anonymous reader quotes a report from KrebsOnSecurity: Federal authorities have arrested and indicted a 20-year-old U.S. Army soldier on suspicion of being Kiberphant0m, a cybercriminal who has been selling and leaking sensitive customer call records stolen earlier this year from AT&T and Verizon. As first reported by KrebsOnSecurity last month, the accused is a communications specialist who was recently stationed in South Korea. Cameron John Wagenius was arrested near the Army base in Fort Hood, Texas on Dec. 20, after being indicted on two criminal counts of unlawful transfer of confidential phone records. The sparse, two-page indictment (PDF) doesn't reference specific victims or hacking activity, nor does it include any personal details about the accused. But a conversation with Wagenius' mother -- Minnesota native Alicia Roen -- filled in the gaps. Roen said that prio...

Slashdot: US Army Soldier Pleads Guilty To AT&T and Verizon Hacks

US Army Soldier Pleads Guilty To AT&T and Verizon Hacks Published on February 20, 2025 at 01:31AM Cameron John Wagenius pleaded guilty to hacking AT&T and Verizon and stealing a massive trove of phone records from the companies, according to court records filed on Wednesday. From a report: Wagenius, who was a U.S. Army soldier, pleaded guilty to two counts of "unlawful transfer of confidential phone records information" on an online forum and via an online communications platform. According to a document filed by Wagenius' lawyer, he faces a maximum fine of $250,000 and prison time of up to 10 years for each of the two counts. Wagenius was arrested and indicted last year. In January, U.S. prosecutors confirmed that the charges brought against Wagenius were linked to the indictment of Connor Moucka and John Binns, two alleged hackers whom the U.S. government accused of several data breaches against cloud computing services company Snowflake, which were among the ...

Slashdot: AT&T Now Lets Customers Lock Down Account To Prevent SIM Swapping Attacks

AT&T Now Lets Customers Lock Down Account To Prevent SIM Swapping Attacks Published on July 02, 2025 at 01:30AM AT&T has launched a new Account Lock feature designed to protect customers from SIM swapping attacks. The security tool, available through the myAT&T app, prevents unauthorized changes to customer accounts including phone number transfers, SIM card changes, billing information updates, device upgrades, and modifications to authorized users. SIM swapping attacks occur when criminals obtain a victim's phone number through social engineering techniques, then intercept messages and calls to access two-factor authentication codes for sensitive accounts. The attacks have become increasingly common in recent years. AT&T began gradually rolling out Account Lock earlier this year, joining T-Mobile, Verizon, and Google Fi, which already offer similar fraud prevention features. Read more of this story at Slashdot.

Shameless Plug

Search This Blog