Slashdot: Websites are Blocking the Wrong AI Scrapers

Websites are Blocking the Wrong AI Scrapers
Published on July 29, 2024 at 11:53PM
An anonymous reader shares a report: Hundreds of websites trying to block the AI company Anthropic from scraping their content are blocking the wrong bots, seemingly because they are copy/pasting outdated instructions to their robots.txt files, and because companies are constantly launching new AI crawler bots with different names that will only be blocked if website owners update their robots.txt. In particular, these sites are blocking two bots no longer used by the company, while unknowingly leaving Anthropic's real (and new) scraper bot unblocked. This is an example of "how much of a mess the robots.txt landscape is right now," the anonymous operator of Dark Visitors told 404 Media. Dark Visitors is a website that tracks the constantly-shifting landscape of web crawlers and scrapers -- many of them operated by AI companies -- and which helps website owners regularly update their robots.txt files to prevent specific types of scraping. The site has seen a huge increase in popularity as more people try to block AI from scraping their work. "The ecosystem of agents is changing quickly, so it's basically impossible for website owners to manually keep up. For example, Apple (Applebot-Extended) and Meta (Meta-ExternalAgent) just added new ones last month and last week, respectively," they added.

Comments

Slashdot: AT&T Now Lets Customers Lock Down Account To Prevent SIM Swapping Attacks

AT&T Now Lets Customers Lock Down Account To Prevent SIM Swapping Attacks Published on July 02, 2025 at 01:30AM AT&T has launched a new Account Lock feature designed to protect customers from SIM swapping attacks. The security tool, available through the myAT&T app, prevents unauthorized changes to customer accounts including phone number transfers, SIM card changes, billing information updates, device upgrades, and modifications to authorized users. SIM swapping attacks occur when criminals obtain a victim's phone number through social engineering techniques, then intercept messages and calls to access two-factor authentication codes for sensitive accounts. The attacks have become increasingly common in recent years. AT&T began gradually rolling out Account Lock earlier this year, joining T-Mobile, Verizon, and Google Fi, which already offer similar fraud prevention features. Read more of this story at Slashdot.

Slashdot: Protecting 'Funko' Brand, AI-Powered 'BrandShield' Knocks Itch.io Offline After Questionable Registrar Communications

Protecting 'Funko' Brand, AI-Powered 'BrandShield' Knocks Itch.io Offline After Questionable Registrar Communications Published on December 16, 2024 at 01:04AM Launched in 2013, itch.io lets users host and sell indie video games online — now offering more than 200,000 — as well as other digital content like music and comics. But then someone uploaded a page based on a major videogame title, according to Game Rant. And somehow this provoked a series of overreactions and missteps that eventually knocked all of itch.io offline for several hours... The page was about the first release from game developer 10:10 — their game Funko Fusion, which features characters in the style of Funko's long-running pop-culture bobbleheads. As a major brand, Funko monitors the web with a "brand protection" partner (named BrandShield). Interestingly, BrandShield's SaaS product "leverages AI-driven online brand protection," according to their site, to "detect...

Slashdot: 'Investors in Limbo'. Will the TikTok Deal's Deadline Be Extended Again?

'Investors in Limbo'. Will the TikTok Deal's Deadline Be Extended Again? Published on December 15, 2025 at 04:29AM An anonymous reader shared this report from the BBC: A billionaire investor keen on buying TikTok's US operations has told the BBC he has been left in limbo as the latest deadline for the app's sale looms. The US has repeatedly delayed the date by which the platform's Chinese owner, Bytedance, must sell or be blocked for American users. US President Donald Trump appears poised to extend the deadline for a fifth time on Tuesday. "We're just standing by and waiting to see what happens," investor Frank McCourt told BBC News... The president...said "sophisticated" US investors would acquire the app, including two of his allies: Oracle chairman Larry Ellison and Dell Technologies' Michael Dell. Members of the Trump administration had indicated the deal would be formalised in a meeting between Trump and Xi in October — howeve...

Shameless Plug

Search This Blog