Slashdot: Websites are Blocking the Wrong AI Scrapers

Websites are Blocking the Wrong AI Scrapers
Published on July 29, 2024 at 11:53PM
An anonymous reader shares a report: Hundreds of websites trying to block the AI company Anthropic from scraping their content are blocking the wrong bots, seemingly because they are copy/pasting outdated instructions to their robots.txt files, and because companies are constantly launching new AI crawler bots with different names that will only be blocked if website owners update their robots.txt. In particular, these sites are blocking two bots no longer used by the company, while unknowingly leaving Anthropic's real (and new) scraper bot unblocked. This is an example of "how much of a mess the robots.txt landscape is right now," the anonymous operator of Dark Visitors told 404 Media. Dark Visitors is a website that tracks the constantly-shifting landscape of web crawlers and scrapers -- many of them operated by AI companies -- and which helps website owners regularly update their robots.txt files to prevent specific types of scraping. The site has seen a huge increase in popularity as more people try to block AI from scraping their work. "The ecosystem of agents is changing quickly, so it's basically impossible for website owners to manually keep up. For example, Apple (Applebot-Extended) and Meta (Meta-ExternalAgent) just added new ones last month and last week, respectively," they added.

Comments

Slashdot: US Army Soldier Pleads Guilty To AT&T and Verizon Hacks

US Army Soldier Pleads Guilty To AT&T and Verizon Hacks Published on February 20, 2025 at 01:31AM Cameron John Wagenius pleaded guilty to hacking AT&T and Verizon and stealing a massive trove of phone records from the companies, according to court records filed on Wednesday. From a report: Wagenius, who was a U.S. Army soldier, pleaded guilty to two counts of "unlawful transfer of confidential phone records information" on an online forum and via an online communications platform. According to a document filed by Wagenius' lawyer, he faces a maximum fine of $250,000 and prison time of up to 10 years for each of the two counts. Wagenius was arrested and indicted last year. In January, U.S. prosecutors confirmed that the charges brought against Wagenius were linked to the indictment of Connor Moucka and John Binns, two alleged hackers whom the U.S. government accused of several data breaches against cloud computing services company Snowflake, which were among the ...

Slashdot: AT&T Now Lets Customers Lock Down Account To Prevent SIM Swapping Attacks

AT&T Now Lets Customers Lock Down Account To Prevent SIM Swapping Attacks Published on July 02, 2025 at 01:30AM AT&T has launched a new Account Lock feature designed to protect customers from SIM swapping attacks. The security tool, available through the myAT&T app, prevents unauthorized changes to customer accounts including phone number transfers, SIM card changes, billing information updates, device upgrades, and modifications to authorized users. SIM swapping attacks occur when criminals obtain a victim's phone number through social engineering techniques, then intercept messages and calls to access two-factor authentication codes for sensitive accounts. The attacks have become increasingly common in recent years. AT&T began gradually rolling out Account Lock earlier this year, joining T-Mobile, Verizon, and Google Fi, which already offer similar fraud prevention features. Read more of this story at Slashdot.

Slashdot: Protecting 'Funko' Brand, AI-Powered 'BrandShield' Knocks Itch.io Offline After Questionable Registrar Communications

Protecting 'Funko' Brand, AI-Powered 'BrandShield' Knocks Itch.io Offline After Questionable Registrar Communications Published on December 16, 2024 at 01:04AM Launched in 2013, itch.io lets users host and sell indie video games online — now offering more than 200,000 — as well as other digital content like music and comics. But then someone uploaded a page based on a major videogame title, according to Game Rant. And somehow this provoked a series of overreactions and missteps that eventually knocked all of itch.io offline for several hours... The page was about the first release from game developer 10:10 — their game Funko Fusion, which features characters in the style of Funko's long-running pop-culture bobbleheads. As a major brand, Funko monitors the web with a "brand protection" partner (named BrandShield). Interestingly, BrandShield's SaaS product "leverages AI-driven online brand protection," according to their site, to "detect...

Shameless Plug

Search This Blog